[raft] Internal storage choice

[the-glu]

In the context of the raft store implementation described in #1525, we need a way to store in-memory objects (such as subscriptions, operational intents, etc.).

A quick analysis revealed various solutions:

• Direct maps/objects
• Go-memdb
• buntdb
• sqlite

These solutions offer different tradeoffs: using direct maps means we need to manually maintain our own indexes/structure, whereas a 'real' database allows us to use standard queries.

Performance is our primary concern, as we want to apply changes coming from raft as quickly as possible, and the objects involved are relatively simple.

To evaluate this, a small prototype simulating subscriptions was developed:

type Subscription struct {
    ID                string
    Manager           string
    Version           int64
    NotificationIndex int
    StartTime         time.Time
    EndTime           time.Time
    USSBaseURL        string
    Cells             []uint64
}

with the following operations:

Add(s *Subscription) error
GetByID(id string) (*Subscription, error)
Update(s *Subscription) error
Delete(id string) error
SearchByCells(cells []uint64) ([]*Subscription, error)
IncrementByCells(cells []uint64) (int, error)
SearchExpired(now time.Time) ([]*Subscription, error)

Performance was tested with 100, 1,000, 10,000, 100,000 and 1,000,000 entries.

Results are:

The results show that:

• Simple memory maps are almost always faster and use less memory
• The only slower case is searching for expired entries, as end_date is not indexed - at least in this test

Based on this analysis, I propose using simple maps for raft internal storage instead of the listed frameworks/libraries.

Note that this underlying storage can be changed at any time, and different implementations can even coexist within a DSS pool - this only concerns the way in-memory information is stored.