vyzo
4cc672d0c5
batch move objects from coldstore to hotstore
2021-03-05 14:46:17 +02:00
vyzo
97abbe1eca
add (salted) bloom filter liveset
2021-03-05 14:46:17 +02:00
vyzo
aba6530411
batch deletion for purging the tracking store
2021-03-05 14:46:17 +02:00
vyzo
09cd1175a1
structured log for beginning of compaction
2021-03-05 14:46:17 +02:00
vyzo
e52c709d8a
more accurate setting of skip params
2021-03-05 14:46:17 +02:00
vyzo
2426ffb277
better logging plus moving some code around
2021-03-05 14:46:17 +02:00
vyzo
2f26026991
compactSimple should walk the cold epoch at depth 1
...
So that it finds reachable objects that should stay in the hotstore
2021-03-05 14:46:17 +02:00
vyzo
783dcda19c
add Sync to the tracking store
2021-03-05 14:46:17 +02:00
vyzo
364076c1ea
set NoSync option for bolt livesets
2021-03-05 14:46:17 +02:00
vyzo
73259aa350
add configuration for splitstore and default to a simple compaction algorithm
2021-03-05 14:46:17 +02:00
vyzo
2e4d45ef07
test for bolt backed tracking store
2021-03-05 14:46:17 +02:00
vyzo
f1c61c4753
implement bolt backed tracking store
2021-03-05 14:46:17 +02:00
vyzo
2c1a9781cf
add test for bolt liveset
2021-03-05 14:46:17 +02:00
vyzo
27a9b974db
implement bolt-backed liveset
2021-03-05 14:46:17 +02:00
vyzo
68b6f913c7
propagate useLMDB option to splitstore through DI
2021-03-05 14:46:17 +02:00
vyzo
923a3db4b0
abstract tracking store and live set construction
2021-03-05 14:46:17 +02:00
vyzo
8f0ddac41a
add comment
2021-03-05 14:46:17 +02:00
vyzo
e79445123f
handle MDB_KEY_EXIST in tracking store Puts
2021-03-05 14:46:17 +02:00
vyzo
9977f5c3ec
rewrite sweep logic to avoid doing writes/deletes nested in a read txn
2021-03-05 14:46:17 +02:00
vyzo
ee751f88cd
refactor lmdb specific snoop/liveset code into their own files
...
paves the way for different back ends
2021-03-05 14:46:17 +02:00
vyzo
8e12377e69
handle consistency edge case
2021-03-05 14:46:17 +02:00
vyzo
31268ba685
walk snapshot the same way snapshot exporting does; skip old msgs and receipts by default.
...
so that we don't panic with missing blocks in non-archival nodes
2021-03-05 14:46:17 +02:00
vyzo
5068d51ac3
use CompactionCold epochs for delinating the cold epoch cliff
...
this allows us to change the thresholds for testing.
2021-03-05 14:46:17 +02:00
vyzo
d44719dbd3
amend confusing comment
2021-03-05 14:46:17 +02:00
vyzo
842ec43c2f
get rid of goroutine iteration in tracking store; long live ForEach
2021-03-05 14:46:17 +02:00
vyzo
a586d42c3b
make hot store DI injectable in the split store, default to badger.
2021-03-05 14:46:17 +02:00
vyzo
7044e623f9
flag to enable GC during compaction, disabled for now
2021-03-05 14:46:17 +02:00
vyzo
f6c930d0aa
crank up blockstore max readers to 16K, reduce retry delays to 10us
2021-03-05 14:46:17 +02:00
vyzo
95befa1e41
set lmdb max readers retry delay to 1ms
2021-03-05 14:46:17 +02:00
vyzo
874ecd3573
adjust hot store options, redux.
2021-03-05 14:46:17 +02:00
vyzo
ca8a673b5f
adjust hot store options
2021-03-05 14:46:17 +02:00
vyzo
69a88d41b6
fix snoop test
2021-03-05 14:46:17 +02:00
vyzo
cdf5bd0500
return annotated xerrors where appropriate
2021-03-05 14:46:17 +02:00
vyzo
ea05fd9d86
use xerrors instead of fmt.Errorf
2021-03-05 14:46:17 +02:00
vyzo
d91b60df91
fix potential panic with max readers retry and cursor channel
2021-03-05 14:46:17 +02:00
vyzo
b9f8a3d587
log MDB_READERS_FULL retries
2021-03-05 14:46:17 +02:00
vyzo
c89ab1a990
retry on MDB_READERS_FULL errors
2021-03-05 14:46:17 +02:00
vyzo
2080e467ba
don't set MaxReaders for tracking store
2021-03-05 14:46:17 +02:00
Raúl Kripalani
877ecab960
update go-bs-lmdb and migrate to ledgerwatch/lmdb-go.
2021-03-05 14:46:17 +02:00
vyzo
5b4e6b7b26
don't set max readers for livesets
2021-03-05 14:46:17 +02:00
vyzo
58a8434825
temporary log level for splitstore to DEBUG
2021-03-05 14:46:17 +02:00
vyzo
8b0087524f
adjust walk boundaries for marking
2021-03-05 14:46:17 +02:00
vyzo
76d6edbb52
fix max readers for tracking store
2021-03-05 14:46:17 +02:00
vyzo
1a23b1f6af
make CompactionThreshold a var to fix lotus-soup build
...
finality is not a constant there!
2021-03-05 14:46:17 +02:00
vyzo
6e51e6db9a
better handling of MDB_KEYEXIST in Put
2021-03-05 14:46:17 +02:00
vyzo
ce41e394dc
handle MDB_KEYEXIST in liveset marking
2021-03-05 14:46:17 +02:00
vyzo
843fd09a62
deal with MDB_KEY_EXIST errors
2021-03-05 14:46:17 +02:00
vyzo
f44cf0f2c4
appease linter
2021-03-05 14:46:17 +02:00
vyzo
facdc555b1
add nil check for curTs -- some tests don't have chain state
2021-03-05 14:46:17 +02:00
vyzo
e07c6c71c0
splitstore constructor
2021-03-05 14:46:17 +02:00
vyzo
b0f48b500f
use CAS for compacting state
2021-03-05 14:46:17 +02:00
vyzo
0af7b16ad5
simplify Has
2021-03-05 14:46:17 +02:00
vyzo
37e391f133
add TODO note about map size
2021-03-05 14:46:17 +02:00
vyzo
5db314f422
fallback to coldstore if snooping fails.
2021-03-05 14:46:17 +02:00
vyzo
d20cbc0c28
protect against potential data races
...
overkill, but let's not have race detectors scream at us.
2021-03-05 14:46:17 +02:00
vyzo
da478832cb
quiet linter
2021-03-05 14:46:17 +02:00
vyzo
4763397281
add tracking store test
2021-03-05 14:46:17 +02:00
vyzo
0d7476c5b2
implement LMDB-backed tracking store
2021-03-05 14:46:17 +02:00
vyzo
83f8a0ab12
quiet linter
2021-03-05 14:46:17 +02:00
vyzo
5043f31adf
liveset unit test
2021-03-05 14:46:17 +02:00
vyzo
3f92a000c7
implement lmdb-backed LiveSet
2021-03-05 14:46:17 +02:00
vyzo
0bf1a78b39
stubs for tracking store and live set
2021-03-05 14:46:17 +02:00
vyzo
17bc5fcd85
move splitstore implementation to its own directory
2021-03-05 14:46:17 +02:00
vyzo
2c9b58aaec
add some logging
2021-03-05 14:46:17 +02:00
vyzo
c1b1a9ce2a
avoid race with compacting state variable
2021-03-05 14:46:17 +02:00
vyzo
3083d80f5e
no need to import go-ipfs-blockstore, lib/blockstore will do
2021-03-05 14:46:17 +02:00
vyzo
101e5c6540
close keys channel when dome emitting keys
2021-03-05 14:46:17 +02:00
vyzo
b945747eb2
satisfy linter
2021-03-05 14:46:17 +02:00
vyzo
2bed6c94cd
use dual live set marking algorithm to keep all hotly reachable objects in the hotstore
2021-03-05 14:46:17 +02:00
vyzo
c2cc198316
fix off by 1 in marking
2021-03-05 14:46:17 +02:00
vyzo
fd08786048
track base epoch in metadata ds
2021-03-05 14:46:17 +02:00
vyzo
b192adfd2e
trigger compaction from head changes
2021-03-05 14:46:17 +02:00
vyzo
c8f1139e0d
compaction algorithm
2021-03-05 14:46:17 +02:00
vyzo
6577cc8ea6
splitstore struct and Blockstore interface implementation
2021-03-05 14:46:17 +02:00
Raúl Kripalani
1ac0c9a926
address review comments.
2021-03-02 21:29:24 +00:00
Raúl Kripalani
2047a74958
implement blockstore.Union, a union blockstore.
...
The union blockstore takes a list of blockstores. It returns the first
satisfying read, and broadcasts writes to all stores.
It can be used for operations that require reading from any two blockstores,
for example WalkSnapshot.
2021-03-02 17:03:11 +00:00
Raúl Kripalani
b1c348b4a7
address review comments.
2021-03-02 16:31:01 +00:00
Raúl Kripalani
3795cc2bd2
segregate chain and state blockstores.
...
This paves the way for better object lifetime management.
Concretely, it makes it possible to:
- have different stores backing chain and state data.
- having the same datastore library, but using different parameters.
- attach different caching layers/policies to each class of data, e.g.
sizing caches differently.
- specifying different retention policies for chain and state data.
This separation is important because:
- access patterns/frequency of chain and state data are different.
- state is derivable from chain, so one could never expunge the chain
store, and only retain state objects reachable from the last finality
in the state store.
2021-02-28 22:49:44 +00:00
Raúl Kripalani
7f0f7d0b36
Merge branch 'master' into refactor/lib/blockstore
2021-02-28 19:55:23 +00:00
vyzo
5c6988a7c1
refactor coalescing logic into its own function, take both cancellation sets into account
2021-02-02 21:26:50 +02:00
Raúl Kripalani
d1104fec4c
rename blockstores for consistency.
2021-01-29 23:17:25 +00:00
Raúl Kripalani
b0cbc932bd
consolidate all blockstores in blockstore package.
2021-01-29 20:01:00 +00:00
Raúl Kripalani
a1da1dab85
add context to LockedRepo#Datastore().
2021-01-26 11:01:43 +00:00
Raúl Kripalani
ce3af308ed
Merge branch 'master' into chore/snake_context_through_blockstore_init
2021-01-25 19:31:41 +00:00
Łukasz Magiera
2b3d66da3b
Merge pull request #5362 from filecoin-project/feat/fullnode-restore
...
Implement full-node restore option
2021-01-18 23:42:20 +01:00
Lucas Molas
69eb2163d2
fix(sync): enforce fork len when changing head ( #5244 )
2021-01-18 12:34:25 -03:00
Łukasz Magiera
38f9559080
Implement full-node restore option
2021-01-15 20:57:32 +01:00
Peter Rabbitson
446a24ef3f
Fix forgotten test-call
2021-01-04 15:46:09 +01:00
Jakub Sztandera
13ff98416f
Fix cs.tipsets being unbounded
...
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-11-26 23:37:59 +01:00
Łukasz Magiera
54637d5093
Merge pull request #4767 from zgfzgf/feat-print-multiple-blocks
...
print multiple blocks from miner cid
2020-11-20 16:14:37 +01:00
Raúl Kripalani
0c6072a1a0
chainstore lifecycle: close via Close() instead of context.
2020-11-16 22:28:59 +00:00
Raúl Kripalani
bac51dd4e3
daemon --import-snapshot: force head silently, without triggering reorg.
2020-11-16 16:03:47 +00:00
Raúl Kripalani
d7b4f92f1f
add context to ChainStore.
2020-11-16 15:52:19 +00:00
Aayush Rajasekaran
1af1ceb1e0
Merge pull request #3646 from filecoin-project/feat/delayed-pubsub-subscribe
...
delayed pubsub subscribe for messages topic
2020-11-13 17:44:32 -05:00
Łukasz Magiera
99031bae17
Merge pull request #4709 from filecoin-project/feat/mute-logs
...
Make some logs quieter
2020-11-12 20:40:10 +01:00
Aayush Rajasekaran
735c04f999
Merge pull request #4681 from filecoin-project/badger-viewable
...
move to native badger blockstore; leverage zero-copy View() to deserialize in-place
2020-11-10 18:08:12 -05:00
Raúl Kripalani
38c404e04e
vm: construct CBOR store properly.
2020-11-10 15:49:33 +00:00
Raúl Kripalani
af7236f7e7
migrate direct usages blockstore.{Get=>View}.
2020-11-10 13:35:36 +00:00
vyzo
185653f85a
improve coalescing
2020-11-09 13:35:42 +02:00
zgfzgf
8c8396af59
print multiple blocks from miner
2020-11-09 10:57:36 +08:00
vyzo
378d7a1ad9
stupid linter
2020-11-06 22:26:47 +02:00
vyzo
9a76c648f1
unit test for head change coalescer
2020-11-06 22:26:47 +02:00
vyzo
08aab1c48a
make Close satisfy io.Closer
2020-11-06 22:26:24 +02:00
vyzo
9149ae2164
godoc and final dispatch when closing the coalescer
2020-11-06 22:26:24 +02:00
vyzo
ec13c5f80d
coalesce both reverts and applys in the same manner
2020-11-06 22:26:24 +02:00
vyzo
4058f5642f
head change coalescer
2020-11-06 22:26:24 +02:00
Jakub Sztandera
5a34e5b2bf
Merge pull request #4599 from filecoin-project/feat/sync-manager-redux
...
rewrite sync manager
2020-11-06 21:01:26 +01:00
Raúl Kripalani
3577300aee
Merge branch 'master' into badger-viewable
2020-11-06 19:34:25 +00:00
vyzo
32b7797394
only subscribe to messages topic if we are synced within 10 epochs
2020-11-05 18:08:13 +02:00
Raúl Kripalani
370ef93db6
chainstore: new ForceHeadSilent to support benchmarks.
2020-11-04 14:39:01 +00:00
Łukasz Magiera
a1e1b03ca4
Optionally allow bitswap for chainstore
2020-11-03 23:44:44 +01:00
Łukasz Magiera
9270ac6358
Make some logs quieter
2020-11-03 13:28:41 +01:00
Raúl Kripalani
099c4b5e1d
migrate repo.Datastore(/chain) to repo.Blockstore().
2020-11-01 13:03:21 +00:00
Jakub Sztandera
e575b5fe8a
Add error if weights are the same
...
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-10-28 16:53:13 +01:00
Jakub Sztandera
c7e6c8eacc
Fix flaky TestChainExportImportFull
...
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-10-23 21:45:08 +02:00
Łukasz Magiera
6b9a4c4a34
Merge pull request #4536 from filecoin-project/zgfzgf-fix-chain-export
...
Small chain export optimization
2020-10-22 17:35:58 +02:00
Łukasz Magiera
ae04f3029b
chain: Add more robust test for ChainExport
2020-10-22 16:01:11 +02:00
Steven Allen
4e730b5ec8
port to v2 imports
2020-10-21 12:16:23 -07:00
zgfzgf
b7726c4ca8
optimize walkSnapshot recurseLinks
2020-10-12 16:44:39 +08:00
Peter Rabbitson
23f5a99117
Add comment clarifying current codepath
2020-10-10 15:36:32 +02:00
Peter Rabbitson
85abca4b16
Delint
2020-10-10 04:00:12 +02:00
Peter Rabbitson
6610a247cf
Dump the block validation cache whenever we perform an import
...
This solves a problem with folks resurrecting long-out-of-sync nodes via
snapshot imports.
The interface switch to Batching is necessary: startup is too long otherwise
( 8 minutes just to clear everything on a relatively old node )
2020-10-10 03:41:04 +02:00
Steven Allen
748d2e82a7
unshare the journal
...
Motivation:
* Run lotus with the race detector enabled (primary motivation).
* Allow multiple lotus nodes in a process (not a high priority).
Previously, the journal was shared between all lotus instances, but it was
initialized for every new node. This caused safety problems in tests (at a
minimum).
This patch explicitly passes the journal to all services that need it.
2020-10-09 13:23:07 -07:00
Steven Allen
b1818a1055
fix base fee tests for different smoke heights
2020-10-06 13:12:16 -07:00
Steven Allen
256be285ba
only enable gas tampering if the breeze upgrade is enabled
2020-10-06 13:00:06 -07:00
Steven Allen
8292d60196
Merge branch 'master' into asr/spec-v1
2020-10-02 18:03:03 -07:00
Łukasz Magiera
f147072284
Merge remote-tracking branch 'origin/master' into feat/pruning-command
2020-10-02 23:37:25 +02:00
Steven Allen
4989b1cc90
Merge branch 'master' into asr/spec-v1
2020-09-30 09:19:12 -07:00
Aayush Rajasekaran
a4e7117429
Add lotus shed util to validate a tipset
2020-09-30 01:29:04 -04:00
Steven Allen
233d8a9b72
update even more imports
2020-09-28 14:25:58 -07:00
Dirk McCormick
cfe6f595b0
fix: unsafe append in LoadTipSet
2020-09-28 16:54:18 +02:00
Dirk McCormick
6b16d48bad
refactor: fetch tipset blocks in parallel
2020-09-28 15:56:44 +02:00
Travis Person
a13d10e42d
add logging to chain export
2020-09-25 20:20:18 +00:00
Łukasz Magiera
17a84c9a2c
Merge pull request #3975 from filecoin-project/steb/abstract-actor-policy
...
Add some actors policy setters for testing
2020-09-24 15:51:41 +02:00
Łukasz Magiera
2867b31d2b
Merge pull request #3208 from austinabell/extratsload
...
Update beacon entry load error message
2020-09-24 14:19:00 +02:00
Steven Allen
32a699d6a3
Add some actors policy setters for testing
...
Addresses:
* a307e4593a (r491966115)a307e4593a (r491966634)
2020-09-23 15:00:52 -07:00
Aayush Rajasekaran
a876a0ba44
Use actor state addresses
2020-09-23 02:32:40 -04:00
Steven Allen
1bf3b4989d
rename imports to match actors code
...
`sed -i 's/\bv0\(\w\)\(\w*\)/\L\1\E\20/g' **/*.go`
2020-09-18 14:59:27 -07:00
Łukasz Magiera
70faa36b7f
Merge remote-tracking branch 'origin/master' into refactor/net-upgrade
2020-09-18 19:29:06 +02:00
Aayush Rajasekaran
053cfc1cc7
Migrate verifreg actor
2020-09-17 03:55:23 -04:00
Aayush Rajasekaran
b5ba7a0fad
Miner migration
2020-09-17 02:21:16 -04:00
Aayush Rajasekaran
b4ee519282
Partial progress towards switching to miner and power interfaces
2020-09-16 01:37:49 -04:00
whyrusleeping
6670d22fb5
add command to (slowly) prune lotus chain datastore
2020-09-15 20:20:48 -07:00
Aayush Rajasekaran
952f2c2f1e
Sever chainstore's dependence on the state tree
2020-09-15 20:41:34 -04:00
Steven Allen
cc4d5306eb
Progress
2020-09-14 15:43:12 -07:00
Steven Allen
02dcb5e182
Merge branch 'master' into refactor/net-upgrade
2020-09-14 14:53:57 -07:00
Łukasz Magiera
06ec571c20
Merge remote-tracking branch 'origin/master' into refactor/net-upgrade
2020-09-14 15:09:52 +02:00
Łukasz Magiera
683a58195e
More terraforming in chain/
2020-09-14 14:46:38 +02:00
Raúl Kripalani
6d29d75724
Merge branch 'master' into inmem-journal
2020-09-14 12:17:45 +01:00
whyrusleeping
0e3dd3cb3b
Add faster and slimmer option to chain export via lotus-shed
2020-09-10 17:28:25 -07:00
