Commit Graph
37 Commits
Author SHA1 Message Date
Jorropo 6c01310728 chore: migrate to boxo
This migrates everything except the `go-car` librairy: https://github.com/ipfs/boxo/issues/218#issuecomment-1529922103

I didn't migrated everything in the previous release because all the boxo code wasn't compatible with the go-ipld-prime one due to a an in flight (/ aftermath) revert of github.com/ipfs/go-block-format. go-block-format has been unmigrated since slight bellow absolutely everything depends on it that would have required everything to be moved on boxo or everything to optin into using boxo which were all deal breakers for different groups.

This worked fine because lotus's codebase could live hapely on the first multirepo setup however boost is now trying to use boxo's code with lotus's (still on multirepo) setup: https://filecoinproject.slack.com/archives/C03AQ3QAUG1/p1685022344779649

The alternative would be for boost to write shim types which just forward calls and return with the different interface definitions.

Btw why is that an issue in the first place is because unlike what go's duck typing model suggest interfaces are not transparent https://github.com/golang/go/issues/58112, interfaces are strongly typed but they have implicit narrowing. The issue is if you return an interface from an interface Go does not have a function definition to insert the implicit conversion thus instead the type checker complains you are not returning the right type.

Stubbing types were reverted https://github.com/ipfs/boxo/issues/218#issuecomment-1478650351

Last time I only migrated `go-bitswap` to `boxo/bitswap` because of the security issues and because we never had the interface return an interface problem (we had concrete wrappers where the implicit conversion took place).
2023-06-19 14:45:05 -07:00
zenground0 bd0c010be6 Configur buffer and threshold 2023-03-09 08:40:14 -07:00
zenground0 a994153e27 GC respects target for max hotstore space 2023-03-07 07:39:17 -07:00
ZenGround0andzenground0 4ffded6fef feat:splitstore:single compaction that can handle prune aka two marksets one compaction (#9571)
* begin

* rough draft -- this should probably actually work?

* WIP

* Start testing

* message mode

* Fix tests, make gen

* Better default

* docsgen-cli

* Review Response

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-11-07 16:31:12 -05:00
ZenGround0andzenground0 8b7be6d47e feat:chain:splitstore auto prune (#9123)
Auto Prune

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-08-08 16:06:32 -04:00
vyzo 6bd2d89e30 use memory blockstore for miner manifest loading 2022-04-14 19:34:26 +03:00
vyzo 9d25464703 use functional options for hotstore gc, rename MovingGC to FullGC 2021-07-27 09:53:22 +03:00
vyzo aa0bd51b2c thread GCFrequency option into the splitstore config 2021-07-26 15:05:57 +03:00
Łukasz Magiera da5aeda197 Merge branch 'master' into feat/splitstore-hot-messages 2021-07-22 12:58:06 +02:00
vyzo ebbaf23af8 support out-of-chain reference protection 2021-07-20 09:02:40 +03:00
vyzo 1b77361301 add option for hotstore message retention 2021-07-17 08:35:35 +03:00
vyzo e003203bea implement exposed splitstore 2021-07-15 13:12:10 +03:00
vyzo c0a1cfffa1 rename noopstore to discardstore 2021-07-09 19:19:37 +03:00
vyzo 190cb18ab0 housekeeping
- remove defunct tracking store implementations
- update splitstore node config
- use mark set type config option (defaulting to mapts); a memory constrained node
  may want to use an on-disk one
2021-07-04 18:38:28 +03:00
vyzo 5f7ae1f489 update splistore DI constructor 2021-07-04 18:38:28 +03:00
vyzo 68bc5d2291 skip moving cold blocks when running with a noop coldstore
it is a noop but it still takes (a lot of) time because it has to read all the cold blocks.
2021-07-04 18:38:28 +03:00
vyzo 649b7dd162 add config option for hot headers 2021-07-04 18:38:28 +03:00
vyzo 997f2c098b keep headers hot when running with a noop splitstore 2021-07-04 18:38:28 +03:00
vyzo 04f2e102a1 kill full splitstore compaction, simplify splitstore configuration 2021-07-04 18:38:28 +03:00
vyzo 5cca29d1db hook noop blockstore for splitstore in DI 2021-07-04 18:38:28 +03:00
Łukasz Magiera 7e4cb9da84 Fix fallback chainstore 2021-04-09 16:11:02 +02:00
vyzo 6b8c60a659 don't ID wrap the hotstore 2021-03-05 14:46:18 +02:00
vyzo dd0c308427 move Blockstore config to FullNode, rename to Chainstore and add default for HotStoreType 2021-03-05 14:46:18 +02:00
Raúl Kripalani 4b1e1f4b52 rename liveset => markset; rename snoop => tracking store; docs. 2021-03-05 14:46:18 +02:00
Raúl Kripalani 8cfba5b092 renames and polish. 2021-03-05 14:46:18 +02:00
Raúl Kripalani b1b452bc0f remove dependency from blockstore/splitstore => chain/store. 2021-03-05 14:46:18 +02:00
Raúl Kripalani 1a804fbdec move splitstore into blockstore package. 2021-03-05 14:46:18 +02:00
Raúl Kripalani 1b51c10d78 split off lmdb support to a different branch. 2021-03-05 14:46:18 +02:00
vyzo 0fc2f3a26f fix post-rebase compilation errors 2021-03-05 14:46:18 +02:00
vyzo 97abbe1eca add (salted) bloom filter liveset 2021-03-05 14:46:17 +02:00
vyzo 73259aa350 add configuration for splitstore and default to a simple compaction algorithm 2021-03-05 14:46:17 +02:00
vyzo b83994797a separate LMDB options for hotstore and tracking stores 2021-03-05 14:46:17 +02:00
vyzo 68b6f913c7 propagate useLMDB option to splitstore through DI 2021-03-05 14:46:17 +02:00
vyzo 99c7d8e3eb more informative names for the hotstore directories 2021-03-05 14:46:17 +02:00
vyzo a586d42c3b make hot store DI injectable in the split store, default to badger. 2021-03-05 14:46:17 +02:00
vyzo 622b4f7d9d hook splitstore into DI 2021-03-05 14:46:17 +02:00
Raúl Kripalani 3795cc2bd2 segregate chain and state blockstores.
This paves the way for better object lifetime management.

Concretely, it makes it possible to:
- have different stores backing chain and state data.
- having the same datastore library, but using different parameters.
- attach different caching layers/policies to each class of data, e.g.
  sizing caches differently.
- specifying different retention policies for chain and state data.

This separation is important because:
- access patterns/frequency of chain and state data are different.
- state is derivable from chain, so one could never expunge the chain
  store, and only retain state objects reachable from the last finality
  in the state store.
2021-02-28 22:49:44 +00:00