cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
18,317 Commits 24 Branches 342 Tags 314 MiB
7aa6663127
Commit Graph

435 Commits

Author SHA1 Message Date
Peter Rabbitson
7bced1532b Convert the chainstore lock to RW 
High read-API concurrency facilitates multiple calls to GetHeaviestTipSet
which in turn could slow down chainsync.
 2021-04-06 13:01:48 +02:00
Raúl Kripalani
b1b452bc0f remove dependency from blockstore/splitstore => chain/store. 2021-03-05 14:46:18 +02:00
Raúl Kripalani
1a804fbdec move splitstore into blockstore package. 2021-03-05 14:46:18 +02:00
Raúl Kripalani
1b51c10d78 split off lmdb support to a different branch. 2021-03-05 14:46:18 +02:00
vyzo
0fc2f3a26f fix post-rebase compilation errors 2021-03-05 14:46:18 +02:00
vyzo
3282f856ec fix tests 2021-03-05 14:46:18 +02:00
vyzo
99c6e4f48f adjust min bloom filter size 2021-03-05 14:46:18 +02:00
vyzo
cae5ddce88 dynamically size bloom filters 2021-03-05 14:46:18 +02:00
vyzo
5639261e44 make compaction parameters variable 2021-03-05 14:46:18 +02:00
vyzo
7587ab6234 quiet the stupid linter 2021-03-05 14:46:18 +02:00
vyzo
05fee27840 remove stale references to lmdb from splitstore implementation 2021-03-05 14:46:18 +02:00
vyzo
f62999d2b8 use named constants for bloom filter parameters 2021-03-05 14:46:18 +02:00
vyzo
44aadb931a rehash salted keys in bloom filter 2021-03-05 14:46:17 +02:00
vyzo
88849201ff fix tests 2021-03-05 14:46:17 +02:00
vyzo
f5ce7957f3 size bloom filter for 50M objects 2021-03-05 14:46:17 +02:00
vyzo
f4c6bc6a86 comment nomenclature 2021-03-05 14:46:17 +02:00
vyzo
4cc672d0c5 batch move objects from coldstore to hotstore 2021-03-05 14:46:17 +02:00
vyzo
97abbe1eca add (salted) bloom filter liveset 2021-03-05 14:46:17 +02:00
vyzo
aba6530411 batch deletion for purging the tracking store 2021-03-05 14:46:17 +02:00
vyzo
09cd1175a1 structured log for beginning of compaction 2021-03-05 14:46:17 +02:00
vyzo
e52c709d8a more accurate setting of skip params 2021-03-05 14:46:17 +02:00
vyzo
2426ffb277 better logging plus moving some code around 2021-03-05 14:46:17 +02:00
vyzo
2f26026991 compactSimple should walk the cold epoch at depth 1 
So that it finds reachable objects that should stay in the hotstore
 2021-03-05 14:46:17 +02:00
vyzo
783dcda19c add Sync to the tracking store 2021-03-05 14:46:17 +02:00
vyzo
364076c1ea set NoSync option for bolt livesets 2021-03-05 14:46:17 +02:00
vyzo
73259aa350 add configuration for splitstore and default to a simple compaction algorithm 2021-03-05 14:46:17 +02:00
vyzo
2e4d45ef07 test for bolt backed tracking store 2021-03-05 14:46:17 +02:00
vyzo
f1c61c4753 implement bolt backed tracking store 2021-03-05 14:46:17 +02:00
vyzo
2c1a9781cf add test for bolt liveset 2021-03-05 14:46:17 +02:00
vyzo
27a9b974db implement bolt-backed liveset 2021-03-05 14:46:17 +02:00
vyzo
68b6f913c7 propagate useLMDB option to splitstore through DI 2021-03-05 14:46:17 +02:00
vyzo
923a3db4b0 abstract tracking store and live set construction 2021-03-05 14:46:17 +02:00
vyzo
8f0ddac41a add comment 2021-03-05 14:46:17 +02:00
vyzo
e79445123f handle MDB_KEY_EXIST in tracking store Puts 2021-03-05 14:46:17 +02:00
vyzo
9977f5c3ec rewrite sweep logic to avoid doing writes/deletes nested in a read txn 2021-03-05 14:46:17 +02:00
vyzo
ee751f88cd refactor lmdb specific snoop/liveset code into their own files 
paves the way for different back ends
 2021-03-05 14:46:17 +02:00
vyzo
8e12377e69 handle consistency edge case 2021-03-05 14:46:17 +02:00
vyzo
31268ba685 walk snapshot the same way snapshot exporting does; skip old msgs and receipts by default. 
so that we don't panic with missing blocks in non-archival nodes
 2021-03-05 14:46:17 +02:00
vyzo
5068d51ac3 use CompactionCold epochs for delinating the cold epoch cliff 
this allows us to change the thresholds for testing.
 2021-03-05 14:46:17 +02:00
vyzo
d44719dbd3 amend confusing comment 2021-03-05 14:46:17 +02:00
vyzo
842ec43c2f get rid of goroutine iteration in tracking store; long live ForEach 2021-03-05 14:46:17 +02:00
vyzo
a586d42c3b make hot store DI injectable in the split store, default to badger. 2021-03-05 14:46:17 +02:00
vyzo
7044e623f9 flag to enable GC during compaction, disabled for now 2021-03-05 14:46:17 +02:00
vyzo
f6c930d0aa crank up blockstore max readers to 16K, reduce retry delays to 10us 2021-03-05 14:46:17 +02:00
vyzo
95befa1e41 set lmdb max readers retry delay to 1ms 2021-03-05 14:46:17 +02:00
vyzo
874ecd3573 adjust hot store options, redux. 2021-03-05 14:46:17 +02:00
vyzo
ca8a673b5f adjust hot store options 2021-03-05 14:46:17 +02:00
vyzo
69a88d41b6 fix snoop test 2021-03-05 14:46:17 +02:00
vyzo
cdf5bd0500 return annotated xerrors where appropriate 2021-03-05 14:46:17 +02:00
vyzo
ea05fd9d86 use xerrors instead of fmt.Errorf 2021-03-05 14:46:17 +02:00
vyzo
d91b60df91 fix potential panic with max readers retry and cursor channel 2021-03-05 14:46:17 +02:00
vyzo
b9f8a3d587 log MDB_READERS_FULL retries 2021-03-05 14:46:17 +02:00
vyzo
c89ab1a990 retry on MDB_READERS_FULL errors 2021-03-05 14:46:17 +02:00
vyzo
2080e467ba don't set MaxReaders for tracking store 2021-03-05 14:46:17 +02:00
Raúl Kripalani
877ecab960 update go-bs-lmdb and migrate to ledgerwatch/lmdb-go. 2021-03-05 14:46:17 +02:00
vyzo
5b4e6b7b26 don't set max readers for livesets 2021-03-05 14:46:17 +02:00
vyzo
58a8434825 temporary log level for splitstore to DEBUG 2021-03-05 14:46:17 +02:00
vyzo
8b0087524f adjust walk boundaries for marking 2021-03-05 14:46:17 +02:00
vyzo
76d6edbb52 fix max readers for tracking store 2021-03-05 14:46:17 +02:00
vyzo
1a23b1f6af make CompactionThreshold a var to fix lotus-soup build 
finality is not a constant there!
 2021-03-05 14:46:17 +02:00
vyzo
6e51e6db9a better handling of MDB_KEYEXIST in Put 2021-03-05 14:46:17 +02:00
vyzo
ce41e394dc handle MDB_KEYEXIST in liveset marking 2021-03-05 14:46:17 +02:00
vyzo
843fd09a62 deal with MDB_KEY_EXIST errors 2021-03-05 14:46:17 +02:00
vyzo
f44cf0f2c4 appease linter 2021-03-05 14:46:17 +02:00
vyzo
facdc555b1 add nil check for curTs -- some tests don't have chain state 2021-03-05 14:46:17 +02:00
vyzo
e07c6c71c0 splitstore constructor 2021-03-05 14:46:17 +02:00
vyzo
b0f48b500f use CAS for compacting state 2021-03-05 14:46:17 +02:00
vyzo
0af7b16ad5 simplify Has 2021-03-05 14:46:17 +02:00
vyzo
37e391f133 add TODO note about map size 2021-03-05 14:46:17 +02:00
vyzo
5db314f422 fallback to coldstore if snooping fails. 2021-03-05 14:46:17 +02:00
vyzo
d20cbc0c28 protect against potential data races 
overkill, but let's not have race detectors scream at us.
 2021-03-05 14:46:17 +02:00
vyzo
da478832cb quiet linter 2021-03-05 14:46:17 +02:00
vyzo
4763397281 add tracking store test 2021-03-05 14:46:17 +02:00
vyzo
0d7476c5b2 implement LMDB-backed tracking store 2021-03-05 14:46:17 +02:00
vyzo
83f8a0ab12 quiet linter 2021-03-05 14:46:17 +02:00
vyzo
5043f31adf liveset unit test 2021-03-05 14:46:17 +02:00
vyzo
3f92a000c7 implement lmdb-backed LiveSet 2021-03-05 14:46:17 +02:00
vyzo
0bf1a78b39 stubs for tracking store and live set 2021-03-05 14:46:17 +02:00
vyzo
17bc5fcd85 move splitstore implementation to its own directory 2021-03-05 14:46:17 +02:00
vyzo
2c9b58aaec add some logging 2021-03-05 14:46:17 +02:00
vyzo
c1b1a9ce2a avoid race with compacting state variable 2021-03-05 14:46:17 +02:00
vyzo
3083d80f5e no need to import go-ipfs-blockstore, lib/blockstore will do 2021-03-05 14:46:17 +02:00
vyzo
101e5c6540 close keys channel when dome emitting keys 2021-03-05 14:46:17 +02:00
vyzo
b945747eb2 satisfy linter 2021-03-05 14:46:17 +02:00
vyzo
2bed6c94cd use dual live set marking algorithm to keep all hotly reachable objects in the hotstore 2021-03-05 14:46:17 +02:00
vyzo
c2cc198316 fix off by 1 in marking 2021-03-05 14:46:17 +02:00
vyzo
fd08786048 track base epoch in metadata ds 2021-03-05 14:46:17 +02:00
vyzo
b192adfd2e trigger compaction from head changes 2021-03-05 14:46:17 +02:00
vyzo
c8f1139e0d compaction algorithm 2021-03-05 14:46:17 +02:00
vyzo
6577cc8ea6 splitstore struct and Blockstore interface implementation 2021-03-05 14:46:17 +02:00
Raúl Kripalani
1ac0c9a926 address review comments. 2021-03-02 21:29:24 +00:00
Raúl Kripalani
2047a74958 implement blockstore.Union, a union blockstore. 
The union blockstore takes a list of blockstores. It returns the first
satisfying read, and broadcasts writes to all stores.

It can be used for operations that require reading from any two blockstores,
for example WalkSnapshot.
 2021-03-02 17:03:11 +00:00
Raúl Kripalani
b1c348b4a7 address review comments. 2021-03-02 16:31:01 +00:00
Raúl Kripalani
3795cc2bd2 segregate chain and state blockstores. 
This paves the way for better object lifetime management.

Concretely, it makes it possible to:
- have different stores backing chain and state data.
- having the same datastore library, but using different parameters.
- attach different caching layers/policies to each class of data, e.g.
  sizing caches differently.
- specifying different retention policies for chain and state data.

This separation is important because:
- access patterns/frequency of chain and state data are different.
- state is derivable from chain, so one could never expunge the chain
  store, and only retain state objects reachable from the last finality
  in the state store.
 2021-02-28 22:49:44 +00:00
Raúl Kripalani
7f0f7d0b36 Merge branch 'master' into refactor/lib/blockstore 2021-02-28 19:55:23 +00:00
vyzo
5c6988a7c1 refactor coalescing logic into its own function, take both cancellation sets into account 2021-02-02 21:26:50 +02:00
Raúl Kripalani
d1104fec4c rename blockstores for consistency. 2021-01-29 23:17:25 +00:00
Raúl Kripalani
b0cbc932bd consolidate all blockstores in blockstore package. 2021-01-29 20:01:00 +00:00
Raúl Kripalani
a1da1dab85 add context to LockedRepo#Datastore(). 2021-01-26 11:01:43 +00:00
Raúl Kripalani
ce3af308ed Merge branch 'master' into chore/snake_context_through_blockstore_init 2021-01-25 19:31:41 +00:00