cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
20,336 Commits 24 Branches 342 Tags 314 MiB
ec43903ad6
Commit Graph

179 Commits

Author SHA1 Message Date
Raúl Kripalani
3795cc2bd2 segregate chain and state blockstores. 
This paves the way for better object lifetime management.

Concretely, it makes it possible to:
- have different stores backing chain and state data.
- having the same datastore library, but using different parameters.
- attach different caching layers/policies to each class of data, e.g.
  sizing caches differently.
- specifying different retention policies for chain and state data.

This separation is important because:
- access patterns/frequency of chain and state data are different.
- state is derivable from chain, so one could never expunge the chain
  store, and only retain state objects reachable from the last finality
  in the state store.
 2021-02-28 22:49:44 +00:00
Raúl Kripalani
7f0f7d0b36 Merge branch 'master' into refactor/lib/blockstore 2021-02-28 19:55:23 +00:00
Raúl Kripalani
d1104fec4c rename blockstores for consistency. 2021-01-29 23:17:25 +00:00
Raúl Kripalani
b0cbc932bd consolidate all blockstores in blockstore package. 2021-01-29 20:01:00 +00:00
Raúl Kripalani
a1da1dab85 add context to LockedRepo#Datastore(). 2021-01-26 11:01:43 +00:00
Peter Rabbitson
47141042b6
Add comment 
Co-authored-by: raulk <raul@protocol.ai>
 2021-01-12 18:11:18 +01:00
Peter Rabbitson
9334e73396 Snake a context through the Chain-blockstore creation 2021-01-04 15:27:52 +01:00
Raúl Kripalani
2ef8acdfc8 back out index cache usage in badger blockstore. 2020-12-02 21:35:13 +00:00
Raúl Kripalani
e8a5e0d2fe integrate memory watchdog; impose limits on badger caches. 2020-12-02 16:40:28 +00:00
Raúl Kripalani
44040f0f3f badger options: enable IndexCache and CompactL0OnClose. 2020-11-20 00:33:43 +00:00
Raúl Kripalani
55061ae6fb fix blockstore directory not created automatically. 2020-11-19 15:55:02 +00:00
Peter Rabbitson
7be7a9b5e3 Stop referring to github.com/ipfs/go-ipfs-blockstore outside of lib 2020-11-12 06:01:35 +01:00
Raúl Kripalani
3577300aee Merge branch 'master' into badger-viewable 2020-11-06 19:34:25 +00:00
Raúl Kripalani
0b2a02c8ea remove redundant import. 2020-11-06 18:57:04 +00:00
Raúl Kripalani
85e37e4b70 make the value log loading mode mmap, to leverage zero-copy access. 2020-11-06 18:39:08 +00:00
Raúl Kripalani
0b8a21ee16 badger: restore prev. max table size (64MiB; default). 2020-11-01 17:26:37 +00:00
Raúl Kripalani
72e573d98d fix lint. 2020-11-01 17:09:14 +00:00
Raúl Kripalani
5a98660d1b make repo.Blockstore() idempotent; wrap in IDStore. 2020-11-01 16:56:20 +00:00
Raúl Kripalani
d8d85373f5 fix lotus-shed datastore commands. 2020-11-01 13:50:41 +00:00
Raúl Kripalani
a16d7f221e Merge branch 'master' into badger-viewable 2020-11-01 13:10:56 +00:00
Raúl Kripalani
d2e2322fd2 make the lotus node use the new native badger blockstore. 2020-11-01 13:01:26 +00:00
Jakub Sztandera
0297be4b9a
Add lotus-shed datastore rewrite command 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-10-28 23:24:45 +01:00
Jakub Sztandera
ecc1d94b3a
Reduce badger ValueTreshold to 128 
It should significntly size of the LSM index, and thus increase the
performance with bigger datastores

Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-10-28 22:17:01 +01:00
Łukasz Magiera
8783c7434e Merge remote-tracking branch 'origin/master' into feat/signing-backends 2020-10-09 00:50:41 +02:00
Łukasz Magiera
9b5a0815fc backup: open datastores in readonly in offline mode 2020-10-01 17:55:47 +02:00
Aayush Rajasekaran
6abccc4d5e Add an option to set config 2020-09-30 03:19:04 -04:00
Frank
dcae513487 make nested dir 2020-09-14 16:49:35 +08:00
Łukasz Magiera
780f6dba34 Remote wallet backends 2020-09-05 21:39:09 +02:00
Jakub Sztandera
997d49b63f
Disable GC on chain badger datastore 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-09-01 20:39:58 +02:00
Raúl Kripalani
efdc428d5d keep storage-fsm (renamed to storage-sealing) and sector-storage in extern. 2020-08-17 14:26:18 +01:00
Raúl Kripalani
3c17cd655e integrate extern/sector-storage into lotus proper. 2020-08-16 11:09:58 +01:00
Łukasz Magiera
73b8868046 gofmt 2020-08-06 03:32:59 +02:00
Łukasz Magiera
f8b8ecc0c3 Consensus filter 2020-08-06 03:16:39 +02:00
hannahhoward
8e6862f7ca fix(lint): fix lint errors 2020-07-31 14:19:40 -07:00
hannahhoward
3cac1080cc feat(retrievalstoremgr): add retrievalstoremgr 
add manager for retrievals to handle different cases for IPFS/non-ipfs
 2020-07-31 14:16:18 -07:00
Łukasz Magiera
1fd23248ff gofmt 2020-07-28 16:38:26 +02:00
Łukasz Magiera
09b90773d8 Merge remote-tracking branch 'origin/next' into feat/cid-builder 2020-07-28 16:06:06 +02:00
hannahhoward
5a623cccb5 feat(markets): update markets mulitple deal stores 2020-07-27 23:13:28 -07:00
hannahhoward
dd885b7302 feat(multistore): extract multistore code to repo 
Extract multistore + multiread blockstore to a seperate repo
 2020-07-24 14:47:22 -07:00
Steven Allen
b7a4dbb07f Support inline CIDs 
And use the new CidBuilder from the spec actors.

This patch does not switch over to inline CIDs by default, but paves the way.
 2020-07-23 23:12:32 -07:00
Frank
c38b8b1265 close file 2020-07-23 18:21:13 +08:00
Jakub Sztandera
0f5e56289e
Update stores list when deleting 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-07-17 22:44:16 +02:00
Jakub Sztandera
f2ebeaa7e4
Fix lint 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-07-17 22:29:39 +02:00
Jakub Sztandera
cc1d23a94c
Use single multi ds 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-07-17 22:14:37 +02:00
Łukasz Magiera
ede57c0b12 Fix retrieval datastore-key-not-found error 2020-07-10 18:33:26 +02:00
Łukasz Magiera
5a117d8edf gofmt 2020-07-08 23:05:54 +02:00
Łukasz Magiera
66237415c5 Merge remote-tracking branch 'origin/next' into feat/client-multi-bstore 2020-07-08 22:48:47 +02:00
Łukasz Magiera
5ca1be567d
Merge pull request #2323 from filecoin-project/feat/fastretrieval 
Support fast-retrieval deals
 2020-07-08 21:54:46 +02:00
Jakub Sztandera
d3d4612575
Re-eanble compact on close 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-07-08 20:59:46 +02:00
Łukasz Magiera
99796220f9 Support fast-retrieval deals 2020-07-08 20:42:07 +02:00
Jakub Sztandera
40286e4192
Use mostly defaults for badger options 
This options are the defults in badger `master` now.
We should update to next badger version as soon as it is stable, it has
some nice improvments.

Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-07-08 17:54:33 +02:00
Łukasz Magiera
8e0d33a1fd Updates for sectors-storge next 2020-07-08 17:23:27 +02:00
Łukasz Magiera
55f089d997 client: Unbreak retrieval 2020-07-07 14:35:02 +02:00
Łukasz Magiera
d370e0ae2d Go is hard 2020-07-07 13:48:13 +02:00
Łukasz Magiera
18fc3337ff client: API/Command to drop imports 2020-07-07 13:45:02 +02:00
Łukasz Magiera
7175b1dd65 gofmt 2020-07-07 11:38:22 +02:00
Łukasz Magiera
24ed43d541 client: Fix import labeling 2020-07-07 11:38:09 +02:00
Łukasz Magiera
f59eb94d92 client: Set correct dag builder params 2020-07-07 11:14:13 +02:00
Łukasz Magiera
47f0898ce9 Wire up client import manager 2020-07-07 10:52:19 +02:00
Łukasz Magiera
8942967223 Client Import manager 2020-07-07 10:52:04 +02:00
Łukasz Magiera
92e4507cf7 fsrepo: multi-datastores 2020-07-06 23:03:04 +02:00
laser
fb1d5197fa pass SetConfig a mutator func, as per PR feedback 
- fsLockedRepo.config gets a mutex
- add missing checkToken call to lockedMemRepo#GetStorage and lockedMemRepo#SetStorage
- LockedRepo#SetConfig accepts a mutating function as per @magik
 2020-06-11 08:49:19 -07:00
laser
cf321f7667 add support for replacing config after node starts 
- TODO: does a "locked repo" need fine-grained (i.e. field-level) locking?
 2020-06-11 08:49:19 -07:00
laser
d4dbfdbe91 rename Config to GetConfig 2020-06-11 08:49:19 -07:00
Jakub Sztandera
5605aae269
Fix even more lint warnings 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-06-02 16:29:39 +02:00
Łukasz Magiera
c73326e96d fsrepo: Use LevelDB for metadata datastore 2020-06-01 20:13:07 +02:00
Łukasz Magiera
7003960fdd Merge remote-tracking branch 'origin/master' into next 2020-06-01 18:52:06 +02:00
laser
92b8d9ad17 tell user why repo is already locked 
Fixes #1829
 2020-06-01 09:37:10 -07:00
Łukasz Magiera
e5c24ff32b Update sector-storage with ReadPiece impl 2020-05-29 19:53:35 +02:00
Łukasz Magiera
f1ccbe496a cli: Trim spaces from token file 2020-05-28 22:11:49 +02:00
Ignacio Hagopian
6e63ea6cce revert file 
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
 2020-05-28 22:11:49 +02:00
Ignacio Hagopian
de3edc2ec7 remote ipfs support & automatic env 
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
 2020-05-28 22:11:49 +02:00
Łukasz Magiera
02afd7d71f cli: Trim spaces from token file 2020-05-27 10:13:06 +02:00
Łukasz Magiera
95ee1c1cde Extract sector-storage 2020-03-28 01:32:39 +01:00
Łukasz Magiera
c66d3dfad7 sectorstorage: Untangle from lotus deps 2020-03-27 22:35:33 +01:00
Łukasz Magiera
5e4a7e54df Rename agvmgr+sealmgr to sectorstorage 2020-03-23 12:47:10 +01:00
Łukasz Magiera
d87b7c264d stores: Use index in local store 2020-03-19 16:11:04 +01:00
Łukasz Magiera
2c7f579838 workers: Implement SectorIndex 2020-03-13 12:59:19 +01:00
Łukasz Magiera
bc7fede68b workers: Wire up most pieces 2020-03-13 02:37:38 +01:00
Łukasz Magiera
0c2af7a8ee repo: Fix fsrepo.SetStorage on init 2020-03-09 22:46:17 +01:00
Łukasz Magiera
01087e01d2 storage: Improve thread safety 2020-03-09 20:22:30 +01:00
Łukasz Magiera
1cf26fc652 repo: Implement memrepo.SetStorage 2020-03-09 07:21:42 +01:00
Łukasz Magiera
16cb27b5f9 Get node_test to pass 2020-03-05 23:27:30 +01:00
Łukasz Magiera
1461e475da storage: Support adding paths at runtime 2020-03-05 23:02:01 +01:00
Łukasz Magiera
a5e5918fc5 mod tidy; gofmt 2020-03-05 20:21:06 +01:00
Łukasz Magiera
a0dbb6bdd6 Storage Manager refactor 2020-03-03 23:19:22 +01:00
Jakub Sztandera
49167cc55f
Expose JWT permission error 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-02-12 21:25:29 +01:00
Jakub Sztandera
1ed62628a7
Update go-log to v2 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-01-08 20:22:31 +01:00
Jakub Sztandera
886898b137
Update to badger v2 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-01-08 19:56:45 +01:00
whyrusleeping
613a0a05a0 more of that sweet staticcheck goodness 2019-12-04 18:04:09 -08:00
whyrusleeping
59d9c681ad Rework miner test setups to fix tests 2019-11-30 17:21:11 -06:00
Jakub Sztandera
d9520a05dd
Make sure pond works, don't override config 
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2019-11-13 16:37:54 +01:00
Jakub Sztandera
70419c0f55
Use API ListenAddress from config 
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2019-11-13 16:37:40 +01:00
Łukasz Magiera
73ab6c0c66 WorkerCount on storageminer config 2019-11-12 18:59:38 +01:00
Jakub Sztandera
3ea0997c93
Add support for different configs 
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2019-10-30 17:38:39 +01:00
Łukasz Magiera
5432987030 repo: enable badger log truncation 2019-10-29 19:04:35 +01:00
whyrusleeping
63b81af23e handle changing default address 2019-10-18 20:42:00 +09:00
Jakub Sztandera
1bf713cb0a
Cleanup imports after rename 
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2019-10-18 13:47:41 +09:00
whyrusleeping
d818e20df5 Wallets record default address in keystore 2019-10-17 19:18:40 +09:00
Łukasz Magiera
1534536590 Store libp2p key in keystore 2019-10-03 02:11:31 +02:00