cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
17,706 Commits 24 Branches 342 Tags 314 MiB
485d161a5e
Commit Graph

1446 Commits

Author SHA1 Message Date
Geoff Stuart
e684248f48 Added api call to get actors cids 2022-06-23 14:07:23 -04:00
Łukasz Magiera
db676acdf8 storage: Cleanup interfaces a bit 2022-06-17 13:52:19 +02:00
Łukasz Magiera
58416d0881 storage: Don't depend on specs-storage, move to storiface 2022-06-17 13:36:33 +02:00
Łukasz Magiera
9c4d10ec73 api: handle no-precommit in StateSectorPreCommitInfo gracefully 2022-06-16 15:20:58 +02:00
Łukasz Magiera
2dc1df878d api: Move StateComputeDataCID to full-node 2022-06-16 13:29:35 +02:00
Łukasz Magiera
05cdeb80c3 chore: remove redundant import prefixes 2022-06-15 12:06:22 +02:00
Łukasz Magiera
28099a3905 storage: Move storage/sealer/stores to storage/paths 2022-06-14 20:25:52 +02:00
Łukasz Magiera
a9600b8a6f storage: Move extern/sector-storage to storage/sealer 2022-06-14 20:03:38 +02:00
Łukasz Magiera
98a48a47f8 storage: Move extern/storage-sealing to storage/pipeline 2022-06-14 19:41:59 +02:00
Łukasz Magiera
717bda63b1 storage: Move control address logic to storage/ctladdr 2022-06-14 19:32:29 +02:00
Łukasz Magiera
dff1bf2868 storage: Move window-post logic into a separate package 2022-06-14 19:27:09 +02:00
Łukasz Magiera
e65fae28de chore: fix imports 2022-06-14 17:00:51 +02:00
Geoff Stuart
b7010c9e60 Implement function to migrate actors with only code changes 2022-06-10 15:52:32 -04:00
Geoff Stuart
801c670edd Remove ChainPutMany 2022-06-10 14:09:05 -04:00
Geoff Stuart
5c0f2c8ae6 Add putObj and putMany to apiBlockstore 2022-06-09 15:13:42 -04:00
Aayush
b28c11a57d Merge branch 'feat/nv16' 2022-06-03 14:01:49 -04:00
Jennifer Wang
2e59d0129d Merge branch 'release/v1.15.3' into jen/mergev1153to16 2022-05-31 16:33:18 -04:00
Aayush
4fa04abb8c Add disable flag for Skyr upgrade 2022-05-26 18:20:49 -04:00
Aayush Rajasekaran
6924a3d5f2 Fix calculation of Drand round from Filecoin epochs 2022-05-25 12:43:52 -04:00
Aayush
8cca9b1970 Use new go-state-types accessors 2022-05-17 15:21:27 -04:00
LexLuthr
7e492f23b1 add dagstore register-shard command 2022-05-12 19:46:53 +05:30
Jakub Sztandera
5fb69281d8
Estimate gas across the upgrade with per message multipliers 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2022-05-09 17:08:13 +02:00
Aayush Rajasekaran
8282484f65 Gas estimation: Overestimate by 2 around the M1 upgrade 2022-05-06 11:50:43 -04:00
Aayush Rajasekaran
26a780a198 Gas estimation: Refactor the special PayCh collect case 2022-05-06 11:42:26 -04:00
simlecode
962e37e893 add StateGetNetworkParams api 2022-05-05 14:16:30 +08:00
Łukasz Magiera
d7c52f8adf sealing: ComputeDataCid miner API 2022-04-26 21:37:48 +02:00
Travis Person
ea4f599233 Make MarketListDeals and DealsList return points for MarketDeal 2022-04-22 23:26:29 +00:00
Geoff Stuart
49d6c07280 Implemented StateLoockupRobustAddress 2022-04-14 17:41:18 -04:00
Aayush
339b7db2b7 Integrate FIP 0027 2022-04-14 19:32:56 +03:00
Łukasz Magiera
6bef1aeb82 feat: cli: lotus client list-asks --protocols 2022-04-11 19:49:52 +02:00
Łukasz Magiera
2b3fad1261 fix: node: Fix market node startup 2022-04-01 17:32:55 -04:00
Łukasz Magiera
54cb55a7ae feat: miner: API/CLI to compute window-post 2022-03-28 17:06:51 -04:00
Łukasz Magiera
82343a06e7 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-03-25 16:49:46 -04:00
Łukasz Magiera
945b93590d ping: Match go-ipfs in cmd output 2022-03-21 15:11:19 +01:00
Łukasz Magiera
701d0a111e Merge branch 'feat/net-ping' of github.com:ychiaoli18/lotus into feat/net-ping 2022-03-21 13:06:11 +01:00
Łukasz Magiera
92e4caa151 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-03-21 12:09:47 +01:00
Anton Evangelatov
0ed8647b6f extract unixfs filestore into lib 2022-03-21 10:48:52 +01:00
Łukasz Magiera
5365ccfdb1 cli: Show separate worker types in miner info 2022-03-18 21:31:15 +01:00
Łukasz Magiera
6ddbe41376 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-03-18 10:54:44 +01:00
Darko Brdareski
79453663b3
feat: Add additional test annotations (#8272) 
* Annotate api,proxy_util,blockstore_badger, policy tests

* Annotate splitstore: bsbadger / markset

* Annotate splitstore feature

* Annotate union/timed blockstore tests

* Annotate openrpc, diff_adt tests

* Annotate error,drand,events tests

* Annotate predicates_test

* Fix annotations

* Annotate tscache, gen tests

* Annotate fundmanager test

* Annotate repub and selection tests

* Annotate statetree_test

* Annotate forks_test

* Annotate searchwait_test.go

* Fix duplicated @@ symbols

* Annotate chain stmgr/store tests

* Annotate more (types) tests

* More tests annotated

* Annotate conformance chaos actor tests

* Annotate more integration tests

* Annotate journal system tests

* Annotate more tests.

* Annotate gas,head buffer behaviors

* Fix markset annotations

* doc: test annotations for the markets dagstore wrapper

* Annotate miner_api test in dagstore

* Annotate more test files

* Remove bad annotations from fsrepo

* Annotate wdpost system

* Remove bad annotations

* Renamce "conformance" to "chaos_actor" tests

* doc: stm annotations for blockheader & election proof tests

* Annotate remaining "A" tests

* annotate: stm for error_test

* memrepo_test.go

* Annotate "b" file tests

* message_test.go

* doc: stm annotate for fsrepo_test

* Annotate "c" file tests

* Annotate "D" test files

* message_test.go

* doc: stm annotate for chain, node/config & client

* docs: stm annotate node_test

* Annotate u,v,wl tests

* doc: stm annotations for various test files

* Annotate "T" test files

* doc: stm annotate for proxy_util_test & policy_test

* doc: stm annotate for various tests

* doc: final few stm annotations

* Add mempool unit tests

* Add two more memPool Add tests

* Update submodules

* Add check function tests

* Add stm annotations, refactor test helper

* Annotate api,proxy_util,blockstore_badger, policy tests

* Annotate splitstore: bsbadger / markset

solving merge conflicts

* Annotate splitstore feature

* Annotate union/timed blockstore tests

* Annotate openrpc, diff_adt tests

* Annotate error,drand,events tests

* Annotate predicates_test

* Fix annotations

* Annotate tscache, gen tests

* Annotate fundmanager test

* Annotate statetree_test

* Annotate forks_test

* Annotate searchwait_test.go

* Fix duplicated @@ symbols

* Annotate chain stmgr/store tests

* Annotate more (types) tests

* More tests annotated

* Annotate conformance chaos actor tests

* Annotate more integration tests

* Annotate journal system tests

* Annotate more tests.

* Annotate gas,head buffer behaviors

solve merge conflict

* Fix markset annotations

* Annotate miner_api test in dagstore

* Annotate more test files

* doc: test annotations for the markets dagstore wrapper

* Annotate wdpost system

* Renamce "conformance" to "chaos_actor" tests

* Annotate remaining "A" tests

* doc: stm annotations for blockheader & election proof tests

* annotate: stm for error_test

* Annotate "b" file tests

* memrepo_test.go

* Annotate "c" file tests

* message_test.go

* Annotate "D" test files

* doc: stm annotate for fsrepo_test

* Annotate u,v,wl tests

* message_test.go

* doc: stm annotate for chain, node/config & client

* docs: stm annotate node_test

* Annotate "T" test files

* doc: stm annotations for various test files

* Add mempool unit tests

solve merge conflict

* doc: stm annotate for proxy_util_test & policy_test

* doc: stm annotate for various tests

* doc: final few stm annotations

* Add two more memPool Add tests

* Update submodules

* Add check function tests

solve conflict

* Add stm annotations, refactor test helper

solve merge conflict

* Change CLI test kinds to "unit"

* Fix double merged test

* Fix ccupgrade_test merge

* Fix lint issues

* Add stm annotation to types_Test

* Test vectors submodule

* Add file annotation to burn_test

Co-authored-by: Nikola Divic <divicnikola@gmail.com>
Co-authored-by: TheMenko <themenkoprojects@gmail.com>
 2022-03-16 18:37:34 +01:00
Łukasz Magiera
135aef78d7 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-03-11 17:04:58 +01:00
Łukasz Magiera
81f0cfdd1e
Merge pull request #8276 from filecoin-project/feat/faster-clientcalccommp 
feat: api: Make ClientCalcCommP multithreaded
 2022-03-10 23:22:48 +01:00
Łukasz Magiera
0d9e34d651 client impl: Fix typo in ClientCalcCommP CopyBuffer error 2022-03-10 22:41:45 +01:00
Łukasz Magiera
9c4f8254ee post workers: Fix build post-merge 2022-03-10 14:20:07 +01:00
Łukasz Magiera
a715fb3063 feat: Make ClientCalcCommP multithreaded 2022-03-09 23:25:29 +01:00
hannahhoward
49742f8fdc feat(deps): update to graphsync v0.13.0 with 2.0 protocol 2022-03-09 18:06:35 +00:00
Łukasz Magiera
046a9f8af0 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-03-09 16:27:03 +01:00
Łukasz Magiera
7dcfeee0ea use old go-path 2022-03-03 14:17:25 +00:00
Masih H. Derkani
1bb65189af
Upgrade to go-path v0.2.1 and remove replace directive 
The dependency was used in the chain API. Integrated the new fetcher
factory.
 2022-03-03 13:29:32 +00:00
Masih H. Derkani
aacc246ba3
Merge branch 'master' into feat/cid-to-piece-idx 2022-03-02 14:06:30 +00:00
Łukasz Magiera
e9a6f5f287 Merge remote-tracking branch 'origin/master' into feat/paych-avail-reuse 2022-03-02 12:56:56 +00:00
zenground0
77a954c7c3 Merge branch 'master' into feat/cid-to-piece-idx 2022-02-28 12:57:11 -07:00
ZenGround0
d711775cdf
Merge pull request #8177 from filecoin-project/fix/fault-tracker-snap-deals 
fix:proving:post check sector handles snap deals replica faults
 2022-02-24 11:30:10 -07:00
Aayush
fd8cb44e67 fix: client: calculate commps for pieces bigger than 32GB 2022-02-24 10:27:37 -05:00
zenground0
abe04c33c1 Fix fault tracker to handle snap deals 2022-02-23 09:56:47 -07:00
Aarsh Shah
e3f5b7c59f update dagstore top level index db 2022-02-18 14:00:01 +04:00
Kevin Li
ba72eff3e6 feat: cli/net: implement 'net ping' command 2022-02-17 17:22:52 +08:00
Łukasz Magiera
36a1934845 paychmgr: Fix tests after api changes 2022-02-14 20:16:30 +01:00
Łukasz Magiera
f61eb23f8f api: separate method for paych funding 2022-02-14 19:56:02 +01:00
Łukasz Magiera
10af768c60 Merge commit 'origin/release/v1.15.0~2' into feat/paych-avail-reuse 2022-02-14 19:27:12 +01:00
Jennifer Wang
6ed97f0d04 Merge branch 'release/v1.14.0' into jen/v15 2022-02-09 03:17:38 -05:00
Aarsh Shah
399a1afd94 Merge remote-tracking branch 'origin/master' into feat/cid-to-piece-idx 2022-02-08 12:37:57 +04:00
Will Scott
7247f8e41f
add net protect api methods 
fix #8015
 2022-02-03 17:24:49 +01:00
Łukasz Magiera
98f4d37f1a Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-01-31 19:50:52 +00:00
Darko Brdareski
e51ce5c508 Merge remote-tracking branch 'upstream/master' into bloxico/system-test-matrix 2022-01-27 10:57:56 +01:00
Darko Brdareski
8c41e17c93 Fix typo in client_test annotations 2022-01-27 10:56:33 +01:00
Aayush
1b18236f91 feat: sealer: allow users to abort in-flight snap upgrades 2022-01-25 13:01:51 -05:00
Łukasz Magiera
8b19b84140 paych: option to force off-chain get 2022-01-20 18:19:26 +01:00
Łukasz Magiera
8e46b9ea5d paych: API to pre-fund channels 2022-01-20 18:15:46 +01:00
Aarsh Shah
9ee940098d Merge remote-tracking branch 'origin/master' into feat/cid-to-piece-idx 2022-01-20 16:21:48 +04:00
Łukasz Magiera
e476cf7968 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-01-20 13:15:48 +01:00
vyzo
554848fe6b add resource manager Net(Set)Limit api 2022-01-20 11:36:11 +02:00
vyzo
d05d5bcb70 add resource manager NetStat api 2022-01-20 11:36:11 +02:00
Łukasz Magiera
b38141601c Untangle ffi from api 2022-01-18 11:57:04 +01:00
Łukasz Magiera
f2496d87c9 post workers: Snapdeals updates 2022-01-18 11:25:04 +01:00
zenground0
d6aa17e21f Snap Deals Integration 
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
 2022-01-14 17:14:32 -05:00
Aarsh Shah
8c8652e7bb add a new cli 2022-01-12 18:06:48 +04:00
Aarsh Shah
edfc2c215c Merge remote-tracking branch 'origin/main' into feat/cid-to-piece-idx 2022-01-12 12:48:54 +04:00
zenground0
33f2d24f54 Snap Deals Integration 
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
 2022-01-10 15:39:38 +05:30
hannahhoward
1078dff3a9 feat(storageminer): also list data transfers with no id 2021-12-22 13:41:56 -08:00
hannahhoward
8a923fb420 fix(storageminer): fix conditional 2021-12-22 13:41:56 -08:00
hannahhoward
177282d9f9 fix(storageminer): lint/mod-tidy/extra checks 2021-12-22 13:41:55 -08:00
hannahhoward
cddf63efe9 feat(storageminer): add api for transfer diagnostics 
Add API + CLI for inspecting in depth diagnostics on graphsync transfers with a given peer
 2021-12-22 13:41:29 -08:00
Aayush Rajasekaran
1bc987772c
Merge pull request #7818 from filecoin-project/asr/vm-nvfactor 
Refactor: VM: Remove the NetworkVersionGetter
 2021-12-21 12:19:53 -05:00
Darko Brdareski
86cf17c1f4 Remove bad annotation from gas_test.go 2021-12-20 16:51:09 +01:00
Darko Brdareski
dda1a42a2a Merge branch 'bloxico/system-test-matrix' of https://github.com/filecoin-project/lotus into merge_lotus 2021-12-20 15:48:16 +01:00
Aayush Rajasekaran
3f4eaa99d5 Refactor: State: Rename stmgr::GetNtwkVersion to GetNetworkVersion 2021-12-17 18:43:39 -05:00
Dirk McCormick
b1734f84b3 feat: retrieval ask CLI command 2021-12-17 15:55:12 +01:00
vyzo
dd327f0b22 plumb more contexts 2021-12-17 11:42:09 +02:00
Darko Brdareski
3e32aa896c Annotate client storage deals feature 2021-12-16 14:36:02 +01:00
vyzo
22ce395130 fix some unit tests 2021-12-14 17:05:10 +02:00
Darko Brdareski
0addca1070 Fix bad annotations 2021-12-13 13:41:04 +01:00
Aayush Rajasekaran
dfb65ed89f Plumb contexts through 2021-12-11 17:04:00 -05:00
Łukasz Magiera
d019853687
review: Cleanup some comments 
Co-authored-by: Peter Rabbitson <ribasushi@protocol.ai>
 2021-11-29 21:29:00 +01:00
Łukasz Magiera
5b5e6b9e44 retrieval: DagSpec.MatchPath -> ExportMerkleProof 2021-11-29 21:14:00 +01:00
Łukasz Magiera
61791b90ea retrieval: Only output matching nodes, MatchPath dagspec 2021-11-29 20:40:55 +01:00
Peter Rabbitson
d5b4d93575 Cleanup partial retrieval codepaths ( zero functional changes ) 
- Adjust the ExportRef comment to reflect change from #7673
- Simplify control flow in parseDagSpec() ( read diff under -w )
 2021-11-25 10:35:05 +01:00
Łukasz Magiera
0c884734a3
Merge pull request #7673 from filecoin-project/chore/DM-level-tests_plus_merkle-proof-cars 
Chore/dm level tests plus merkle proof cars
 2021-11-24 20:43:39 +01:00
Łukasz Magiera
bd4927b494 retrieval: Cleanup some comments 2021-11-24 20:13:49 +01:00
Peter Rabbitson
46937a1b9f Add tests cementing the DM-level behavior on simple paths 2021-11-24 15:31:15 +01:00
Łukasz Magiera
78949d6c05 retrieval: Fix deadlock in ClientGetRetrievalUpdates 2021-11-23 18:32:56 +01:00
Łukasz Magiera
407c2ed114 retrieval: Drop the RootSelector hack 2021-11-23 17:42:43 +01:00
Łukasz Magiera
25e89d3a7a retrieval: require unixfs exports to be aligned on block boundary 2021-11-22 13:29:09 +01:00
Łukasz Magiera
2c583b03ff retrieval: Support DM-paths in ls 2021-11-22 12:46:29 +01:00
Łukasz Magiera
74f645a098 retrieval: Fix parseDagSpec root check 2021-11-22 12:46:29 +01:00
Łukasz Magiera
597b72e286 retrieval: Fix lint, cli docsgen 2021-11-22 12:46:29 +01:00
Łukasz Magiera
9c119bfdad retrieval: Make the ls command work 2021-11-22 12:46:17 +01:00
Łukasz Magiera
450d0687da retrieval: REST export endpoint 2021-11-22 12:46:17 +01:00
Łukasz Magiera
b83a9b902a gofmt 2021-11-22 12:46:17 +01:00
Łukasz Magiera
b26906963b retrieval: Support multi-root export 2021-11-22 12:46:17 +01:00
Łukasz Magiera
b9bd061bdd fix unixfs selector root node selection 2021-11-22 12:46:04 +01:00
Łukasz Magiera
b868769ec8 more retrieval api work 2021-11-22 12:46:02 +01:00
Łukasz Magiera
89138bab4d Simplify retrieval APIs 2021-11-22 12:45:42 +01:00
Łukasz Magiera
60ea33b1c7 client: cleanup clientRetrieve 2021-11-22 12:45:42 +01:00
Łukasz Magiera
08e297a217 client: Support json selectors in retrieval 2021-11-22 12:45:42 +01:00
Aayush Rajasekaran
32fc03886d CLI: Add a lotus multisig cancel command 2021-11-18 19:21:17 -05:00
Anton Evangelatov
5454aebf13
CLI tools for the DAGStore inverted index (#7361) 
* cli commands for dagstore

* address comments from Dirk

* rename inverted index to piece index
 2021-11-17 15:39:45 +02:00
Dirk McCormick
501308239f feat: CLI command to announce deal to indexers 2021-11-17 12:16:22 +01:00
Aarsh Shah
bfee6699d2 update go mod and inject store the hash index in dagstore 2021-11-17 11:42:14 +04:00
TheMenko
fb911a4542 annotated repo_test 2021-11-01 01:24:26 +01:00
Jiaying Wang
afe54ab5f3
Merge pull request #7511 from filecoin-project/frrist/fix-lily 
fix: support node instantiation in external packages
 2021-10-13 20:33:17 -04:00
frrist
6cd6732225 fix: support node instantiation in external packages 
- implement ChainGetPath on ChainModuleAPI
 2021-10-13 15:24:28 -07:00
Anton Evangelatov
6e5ccc87cf cli: add retry for deals stuck in Publish with no funds 2021-10-11 14:03:55 +02:00
Peter Rabbitson
a40aa9f90b Normlize selector use within lotus 2021-10-07 11:01:29 +02:00
Peter Rabbitson
f8ac98bf46 Adjust error wording 2021-10-05 18:16:14 +02:00
Peter Rabbitson
5e6aceeb0e Merge remote-tracking branch 'origin/master' into feat/datamodel-selector-retrieval 2021-10-05 18:15:51 +02:00
Jiaying Wang
cbb147de23
Merge pull request #7405 from filecoin-project/feat/update-graphsync-0.10.0 
Update to graphsync to v0.10.0, enable seperate storage and retrieval transfer limits
 2021-10-05 11:50:59 -04:00
Rod Vagg
97c2cdb8d6 traversals: limit link load count for SelectiveCar use as well 2021-10-05 10:47:49 +02:00
Peter Rabbitson
af0d9b650d Error on corner cases with corresponding negative tests 2021-10-04 23:21:24 +02:00
Peter Rabbitson
2a1094fc17 Merge banch feat/go-fil-markets-v1.13.0 into feat/datamodel-selector-retrieval 2021-10-04 21:56:11 +02:00
Jennifer Wang
38e01cba83 Merge branch 'release/v1.12.0' into jen/12tomaster 2021-10-04 02:38:00 -04:00
Aayush Rajasekaran
c3c46e9097 Fix Drand fetching around null tipsets 2021-09-29 11:57:59 -04:00
Aayush Rajasekaran
b9bfcc4b69 Randomness: Move getters from ChainAPI to StateAPI 2021-09-21 14:20:15 -04:00
Aayush Rajasekaran
1d6db306ec Randomness: Move getters from ChainAPI to StateAPI 2021-09-15 00:03:13 +02:00
Peter Rabbitson
0444435589 Expose basic text-based datamodel selector on retrieval 
Syntaxt of selection is located at
https://pkg.go.dev/github.com/ipld/go-ipld-selector-text-lite#SelectorSpecFromPath

Example use, assuming that:
  - The root of the deal is a plain dag-pb unixfs directory
  - The directory is not sharded
  - The user wants to retrieve the first entry in that directory

  lotus client retrieve --miner f0XXXXX --datamodel-path-selector 'Links/0/Hash' bafyROOTCID ~/output

For a much more elaborate example see the top of ./itests/deals_partial_retrieval_test.go
 2021-09-10 09:44:11 +02:00
Peter Rabbitson
2c5f4386ba Prepare the retrieval codepath for selectors 
Slight import-name shuffle, no functional changes
 2021-09-09 12:42:58 +02:00
Łukasz Magiera
95b128b7bc chain: Cleanup consensus logic 2021-09-02 18:09:37 +02:00
Łukasz Magiera
5366821144 api for getting alert states 2021-08-26 15:44:45 +02:00
Aarsh Shah
928ac2191b fix throttling bug 2021-08-25 16:19:02 +05:30
Aarsh Shah
e1a056db46 ignore nil throttler 2021-08-24 10:42:00 +05:30
Łukasz Magiera
12875a9664 api/command for encoding actor params 2021-08-20 17:32:57 +02:00
Łukasz Magiera
0beb48f11d Fix error handling in SectorAddPieceToAny api impl 2021-08-19 12:56:40 +02:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671) 
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
 2021-08-16 23:34:32 +01:00
ZenGround0
6da26dcefc Remove network version panic 2021-08-10 13:07:30 -04:00
Steven Allen
57bf5c2143 feat: add ChainGetTipSetAfterHeight 
This is identical to ChainGetTipSetByHeight, but returns the tipset
following any null tipsets. This is what the user usually wants anyways.

(and I need it for another PR)
 2021-08-05 12:07:14 -07:00
vyzo
1d50dadc18 fix bug in message exclusion 2021-07-29 21:58:59 +03:00
Raúl Kripalani
07f40b9488 fix docs and nits. 2021-07-28 20:03:25 +01:00
Anton Evangelatov
de4a847078 add RuntimeSubsystems API method; use it in lotus-miner info 2021-07-28 16:02:05 +03:00
vyzo
a0d6fdba33 add ChainBlockstoreInfo APIv1 endpoint 2021-07-26 08:30:07 +03:00
vyzo
2dc72d5849 satisfy linter who wants to be a spell checker in comments 2021-07-25 11:47:21 +03:00
vyzo
3d2ae433ee add ChainCheckBlockstore API 2021-07-25 11:14:48 +03:00
Łukasz Magiera
660829703a Merge remote-tracking branch 'origin/master' into feat/split-net-api 2021-07-22 15:38:06 +02:00
Anton Evangelatov
fe31956713 fetch peer id from chain 2021-07-21 11:43:19 +02:00
Łukasz Magiera
812dc266cf builder: Don't require specific NetAPI impl in StorageMinerAPI 2021-07-15 12:01:13 +02:00
Łukasz Magiera
49e26cce7d api: Separate the Net interface from Common 2021-07-15 11:41:30 +02:00
hunjixin
bdaa73a413 add ChainGetMessagesInTipset api 2021-07-15 14:52:09 +08:00
Łukasz Magiera
837322ea59
Merge pull request #6356 from filecoin-project/nonsense/split-market-miner-processes 
Support standalone miner-market process
 2021-07-13 17:16:07 +02:00
hannahhoward
92ec239202 fix(node): ClientRetrieve stops on cancel 
When a deal is cancelled, any in progress calls to ClientRetrieve or ClientRetrieveWithEvents should
return
 2021-07-12 14:46:18 -07:00
Anton Evangelatov
a423c46f2a rename mock.go to net.go 2021-07-12 11:35:03 +02:00
Łukasz Magiera
ed0cc0b56d Handle bad peerIDs in MarketListRetrievalDeals 2021-07-08 19:27:03 +02:00
Anton Evangelatov
ff2772a58c resolved conflicts 2021-07-06 16:00:41 +02:00
Anton Evangelatov
9e85492b85 fix test 2021-07-02 12:24:07 +02:00
Anton Evangelatov
4194c7fa24 revert back optional for Miner; remove ActorAddress call in node.MinerHandler 2021-07-02 11:58:31 +02:00
Raúl Kripalani
1869de7662 unwire host from storage.Miner. 2021-07-01 13:53:20 +01:00
frrist
d02a2a2cfa polish(statetree): accept a context in statetree diff for timeouts 
- this operation can take a while, this changes gives users the option
to abort if it takes "too long".
 2021-06-30 16:40:43 -07:00
Anton Evangelatov
4f0a96c9c7 resolved conflicts 2021-06-30 13:16:52 +02:00
Anton Evangelatov
9ec7cd7239 enable libp2p options 2021-06-30 12:55:43 +02:00
Łukasz Magiera
3f3336340f Fix wallet error messages 2021-06-25 10:48:47 +02:00
Anton Evangelatov
4217ec5308 make auth on remote rpc server optional 2021-06-23 13:27:03 +02:00
Anton Evangelatov
d45bb14015 Merge branch 'master' into nonsense/split-market-miner-processes 2021-06-16 18:49:15 +02:00
Łukasz Magiera
0514f38dc4 Merge remote-tracking branch 'origin/master' into chore/merge-1.10 2021-06-14 13:27:53 +02:00
Anton Evangelatov
beb5bffaf6 fix log.Error; rename RemoteStore 2021-06-09 13:05:54 +02:00
Anton Evangelatov
9f3ec82cc1 initial resolution of conflicts 2021-06-08 13:01:22 +02:00
Jakub Sztandera
f9acd07987
Update libp2p to 0.14.1 
This does not resolve the CI issue but prepares us for it after https://github.com/libp2p/go-libp2p/pull/1116

Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2021-06-07 18:51:48 +02:00
Łukasz Magiera
8625da3479 Bump miner/worker api versions 2021-06-07 16:57:24 +02:00
Łukasz Magiera
a7746961fb Merge remote-tracking branch 'origin/master' into feat/list-retrievals 2021-06-04 20:49:01 +02:00
Anton Evangelatov
8bd09e39ca resolve merge conflicts 2021-06-04 16:17:00 +02:00
Łukasz Magiera
dc642d0b7b Merge remote-tracking branch 'origin/feat/nv13' into feat/nv13-1.11 2021-06-01 21:06:58 +02:00
Łukasz Magiera
d69032b2f0
Merge pull request #6367 from filecoin-project/fix/robustify-commit-batcher 
Robustify commit batcher
 2021-06-01 17:49:46 +02:00
Łukasz Magiera
482e1110c2 precommit batcher: Improve error propagation 2021-06-01 14:35:30 +02:00
Łukasz Magiera
9fcb564bef Make commit batcher more robust 2021-06-01 11:56:19 +02:00
Aayush Rajasekaran
9ceee6028b More tweaking of context handling in the messagepool 2021-05-31 18:13:23 -04:00
Aayush Rajasekaran
ed93d0725f Protect mp.localAddrs and mp.pending behind helper functions 2021-05-31 18:13:23 -04:00
Aayush Rajasekaran
1f03a618f9 Plumb contexts through 2021-05-31 18:13:23 -04:00
Łukasz Magiera
ffa47659a1 Merge remote-tracking branch 'origin/feat/nv13' into feat/nv13-1.11 2021-05-31 21:38:34 +02:00
Łukasz Magiera
c3e8eddb9b Merge remote-tracking branch 'origin/master' into feat/nv13-1.11 2021-05-31 21:24:56 +02:00
Anton Evangelatov
d195a12565 Merge branch 'master' into nonsense/split-market-miner-processes 2021-05-31 11:26:11 +02:00
Łukasz Magiera
c8d603557b storagefsm: Fix batch deal packing behavior 2021-05-30 17:46:22 +02:00
Aayush Rajasekaran
fba5c65ffd Extend the default deal start epoch delay 2021-05-28 12:35:48 -04:00
hannahhoward
19b6dc8d1e feat(cli): add a list retrievals command 
Currently, there is no way to inspect retrievals on a client. This adds said command, allow with
corresponding APIs
 2021-05-27 11:48:25 -07:00
Łukasz Magiera
1e4456138e Merge master into feat/nv13 2021-05-27 12:28:20 +02:00
Aayush Rajasekaran
d9e86afa24 Tweak client calcDealExpiration to consider genesis miners 2021-05-26 21:46:37 -04:00
Aayush Rajasekaran
21b4741e30 Fix randomness fetching around null blocks 2021-05-26 21:06:06 -04:00
Dirk McCormick
885564fe24 Revert "chore: update go-libp2p" 
This reverts commit f7fbaef361.
 2021-05-20 11:22:19 -06:00
Anton Evangelatov
76bb424de0 update StorageMinerAPI 2021-05-20 13:01:14 +02:00
Anton Evangelatov
a989f60e27 add SectorAddPieceToAny and SectorUnsealPiece to StorageMiner iface; model moved to api package - PieceDealInfo, DealSchedule 2021-05-19 13:05:07 +02:00
Łukasz Magiera
c7ba083fa4 Import precommit batcher 2021-05-18 16:51:06 +02:00
Aayush Rajasekaran
9a6e601754
Merge pull request #5961 from filecoin-project/feat/stateless-offline-dealflow 
Introduce stateless offline dealflow, bypassing the FSM/deallists
 2021-05-17 12:35:39 -04:00
Łukasz Magiera
506f39b294 WIP: Integrate FIP0013 2021-05-11 22:10:29 -04:00
Peter Rabbitson
d54ed1b0c6 Merge remote-tracking branch 'origin/master' into feat/stateless-offline-dealflow 2021-05-11 04:54:22 +02:00