cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
18,304 Commits 24 Branches 342 Tags 314 MiB
25babace8f
Commit Graph

1499 Commits

Author SHA1 Message Date
Łukasz Magiera
ba67431731 Merge branch 'update-go-libp2p-v0.21' of github.com:MarcoPolo/lotus into deps/go-libp2p-v0.21 2022-08-19 11:22:04 -04:00
Shrenuj Bansal
b7227d3a6b run make gen 2022-08-16 15:56:32 -04:00
Shrenuj Bansal
fa4a479b97 Add uuid to mpool message sent to chain node from miner 2022-08-16 15:39:06 -04:00
Yu
0ad96858ff chore: do make docsgen 2022-08-12 15:12:22 +08:00
Yu
bb05aeb860 chore: fix typo in comment 2022-08-12 11:53:06 +08:00
Marco Munizaga
766d809f0c Update to go-libp2p v0.21-RC 2022-08-11 11:15:20 -07:00
Shrenuj Bansal
f390c186e6 Close all storage paths on worker shutdown 2022-08-10 17:06:01 -04:00
ZenGround0
0c91b0dc10
feat:chain:splitstore chain prune (#9056) 
* Splitstore chain prune
* Protect on reification for simpler logic and sound cold compact protect
* Recovery from checkpoint during chain prune
* Splitstore (discard and universal mode) running in itests
* Add pause and restart functions to itest block miner
* Add config options to itest full nodes
* Add FsRepo support for itest full ndoes

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
 2022-08-05 16:34:16 -04:00
Łukasz Magiera
cb2b4fa4aa Merge remote-tracking branch 'origin/master' into feat/storage-redeclare 2022-08-05 21:26:15 +02:00
Łukasz Magiera
11e4914531 Merge branch 'feat/SchedRemoveRequest' of github.com:LexLuthr/lotus into LexLuthr-feat/SchedRemoveRequest 2022-08-05 10:40:45 +02:00
Łukasz Magiera
8a39b79ee8 Merge branch 'desc' of github.com:ysrotciv/lotus into docs/ysrotciv-desc 2022-08-05 09:33:26 +02:00
Łukasz Magiera
819314de46 Merge remote-tracking branch 'origin/master' into feat/storage-redeclare 2022-08-05 09:01:30 +02:00
LexLuthr
cdc08e566f fix everything pointed as per review 2022-08-04 02:00:24 +05:30
LexLuthr
c736dedfa6 move requestremove inside runSched 2022-08-04 02:00:24 +05:30
LexLuthr
cf78fa99ee Rebase on master 2022-08-04 01:59:48 +05:30
Łukasz Magiera
4b47881950 Merge branch 'master' into sbansal/lotus-worker-stop-cmd 2022-08-03 10:42:19 +02:00
LexLuthr
f5a7a650e7 rename to ActorWithdrawBalance 2022-08-02 13:04:54 +05:30
LexLuthr
e0e0b0b62f don't wait for msg to land on chain 2022-08-01 23:41:24 +05:30
Łukasz Magiera
3cd8d9eb06 itests: Worker tests for storage detach/redeclare 2022-08-01 15:58:06 +02:00
Łukasz Magiera
8cff52aef6 Storage detach/attach in lotus-miner, cli commands 2022-08-01 15:58:06 +02:00
Łukasz Magiera
943d2a72c6 storage: StorageRedeclareLocal on lotus-worker 2022-08-01 15:58:06 +02:00
Łukasz Magiera
e252e666bd storage: Storage detach on lotus-worker 2022-08-01 15:58:06 +02:00
Łukasz Magiera
f947e65f15 wip storage redeclare/detach 2022-08-01 15:58:06 +02:00
Shrenuj Bansal
dfea74cca8 Add stop cmd for lotus worker 2022-07-29 18:33:30 -04:00
LexLuthr
498334e860 fix spelling mistakes 2022-07-29 20:18:10 +05:30
LexLuthr
4f03486011 miner withdrawbalance API 2022-07-29 20:09:49 +05:30
Łukasz Magiera
6ac5c16d2b storage: Path type filters 2022-07-12 21:45:12 +02:00
Łukasz Magiera
413183e4fa fix: Delegate storage auth on market nodes 2022-07-06 22:22:41 +02:00
Jennifer Wang
c3f3eb0812 Merge branch 'releases' into jen/masterbp 2022-06-27 15:13:12 -04:00
Geoff Stuart
83d7db88cf review fixes 2022-06-23 14:32:14 -04:00
Geoff Stuart
e684248f48 Added api call to get actors cids 2022-06-23 14:07:23 -04:00
Łukasz Magiera
58416d0881 storage: Don't depend on specs-storage, move to storiface 2022-06-17 13:36:33 +02:00
Łukasz Magiera
9c4d10ec73 api: handle no-precommit in StateSectorPreCommitInfo gracefully 2022-06-16 15:20:58 +02:00
Łukasz Magiera
2dc1df878d api: Move StateComputeDataCID to full-node 2022-06-16 13:29:35 +02:00
Łukasz Magiera
6d770f6e10 make gen 2022-06-15 16:51:30 +02:00
Łukasz Magiera
05cdeb80c3 chore: remove redundant import prefixes 2022-06-15 12:06:22 +02:00
Łukasz Magiera
a9600b8a6f storage: Move extern/sector-storage to storage/sealer 2022-06-14 20:03:38 +02:00
Łukasz Magiera
98a48a47f8 storage: Move extern/storage-sealing to storage/pipeline 2022-06-14 19:41:59 +02:00
Łukasz Magiera
e65fae28de chore: fix imports 2022-06-14 17:00:51 +02:00
Geoff Stuart
b7010c9e60 Implement function to migrate actors with only code changes 2022-06-10 15:52:32 -04:00
Geoff Stuart
801c670edd Remove ChainPutMany 2022-06-10 14:09:05 -04:00
Geoff Stuart
5c0f2c8ae6 Add putObj and putMany to apiBlockstore 2022-06-09 15:13:42 -04:00
Aayush
b28c11a57d Merge branch 'feat/nv16' 2022-06-03 14:01:49 -04:00
Jennifer Wang
2e59d0129d Merge branch 'release/v1.15.3' into jen/mergev1153to16 2022-05-31 16:33:18 -04:00
Łukasz Magiera
032e598962 feat: gateway: OpenRPC support 2022-05-27 17:03:56 +02:00
Aayush Rajasekaran
6924a3d5f2 Fix calculation of Drand round from Filecoin epochs 2022-05-25 12:43:52 -04:00
Jennifer Wang
5cfedacf9f make jen 2022-05-23 15:11:46 -04:00
Aayush
3b23ba0520 Address review 2022-05-17 15:52:59 -04:00
Aayush
8cca9b1970 Use new go-state-types accessors 2022-05-17 15:21:27 -04:00
LexLuthr
7e492f23b1 add dagstore register-shard command 2022-05-12 19:46:53 +05:30
simlecode
962e37e893 add StateGetNetworkParams api 2022-05-05 14:16:30 +08:00
zenground0
33f190c9ce Unneeded todo 2022-05-03 11:48:22 -04:00
vyzo
b29a182da7 fix docgen 2022-04-27 17:57:04 +03:00
Łukasz Magiera
d7c52f8adf sealing: ComputeDataCid miner API 2022-04-26 21:37:48 +02:00
Łukasz Magiera
1aed631fe5 sealing: DataCid on workers 2022-04-26 20:36:23 +02:00
Travis Person
ea4f599233 Make MarketListDeals and DealsList return points for MarketDeal 2022-04-22 23:26:29 +00:00
Geoff Stuart
49d6c07280 Implemented StateLoockupRobustAddress 2022-04-14 17:41:18 -04:00
Aayush
339b7db2b7 Integrate FIP 0027 2022-04-14 19:32:56 +03:00
Łukasz Magiera
6bef1aeb82 feat: cli: lotus client list-asks --protocols 2022-04-11 19:49:52 +02:00
Łukasz Magiera
54cb55a7ae feat: miner: API/CLI to compute window-post 2022-03-28 17:06:51 -04:00
Łukasz Magiera
7401fa234e
Merge pull request #7971 from filecoin-project/feat/post-worker 
feat: PoSt workers
 2022-03-25 20:31:49 -04:00
Łukasz Magiera
82343a06e7 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-03-25 16:49:46 -04:00
Łukasz Magiera
5f9753933a
chore: Remove temp gomock reflect file 2022-03-25 16:46:50 -04:00
Łukasz Magiera
701d0a111e Merge branch 'feat/net-ping' of github.com:ychiaoli18/lotus into feat/net-ping 2022-03-21 13:06:11 +01:00
Łukasz Magiera
6ddbe41376 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-03-18 10:54:44 +01:00
Łukasz Magiera
f2c0ba1df5
Merge pull request #8329 from filecoin-project/fix/snap-worker-cleanup 
fix: sealing: Remove sector copies from workers after snapdeals
 2022-03-16 18:48:21 +01:00
Darko Brdareski
79453663b3
feat: Add additional test annotations (#8272) 
* Annotate api,proxy_util,blockstore_badger, policy tests

* Annotate splitstore: bsbadger / markset

* Annotate splitstore feature

* Annotate union/timed blockstore tests

* Annotate openrpc, diff_adt tests

* Annotate error,drand,events tests

* Annotate predicates_test

* Fix annotations

* Annotate tscache, gen tests

* Annotate fundmanager test

* Annotate repub and selection tests

* Annotate statetree_test

* Annotate forks_test

* Annotate searchwait_test.go

* Fix duplicated @@ symbols

* Annotate chain stmgr/store tests

* Annotate more (types) tests

* More tests annotated

* Annotate conformance chaos actor tests

* Annotate more integration tests

* Annotate journal system tests

* Annotate more tests.

* Annotate gas,head buffer behaviors

* Fix markset annotations

* doc: test annotations for the markets dagstore wrapper

* Annotate miner_api test in dagstore

* Annotate more test files

* Remove bad annotations from fsrepo

* Annotate wdpost system

* Remove bad annotations

* Renamce "conformance" to "chaos_actor" tests

* doc: stm annotations for blockheader & election proof tests

* Annotate remaining "A" tests

* annotate: stm for error_test

* memrepo_test.go

* Annotate "b" file tests

* message_test.go

* doc: stm annotate for fsrepo_test

* Annotate "c" file tests

* Annotate "D" test files

* message_test.go

* doc: stm annotate for chain, node/config & client

* docs: stm annotate node_test

* Annotate u,v,wl tests

* doc: stm annotations for various test files

* Annotate "T" test files

* doc: stm annotate for proxy_util_test & policy_test

* doc: stm annotate for various tests

* doc: final few stm annotations

* Add mempool unit tests

* Add two more memPool Add tests

* Update submodules

* Add check function tests

* Add stm annotations, refactor test helper

* Annotate api,proxy_util,blockstore_badger, policy tests

* Annotate splitstore: bsbadger / markset

solving merge conflicts

* Annotate splitstore feature

* Annotate union/timed blockstore tests

* Annotate openrpc, diff_adt tests

* Annotate error,drand,events tests

* Annotate predicates_test

* Fix annotations

* Annotate tscache, gen tests

* Annotate fundmanager test

* Annotate statetree_test

* Annotate forks_test

* Annotate searchwait_test.go

* Fix duplicated @@ symbols

* Annotate chain stmgr/store tests

* Annotate more (types) tests

* More tests annotated

* Annotate conformance chaos actor tests

* Annotate more integration tests

* Annotate journal system tests

* Annotate more tests.

* Annotate gas,head buffer behaviors

solve merge conflict

* Fix markset annotations

* Annotate miner_api test in dagstore

* Annotate more test files

* doc: test annotations for the markets dagstore wrapper

* Annotate wdpost system

* Renamce "conformance" to "chaos_actor" tests

* Annotate remaining "A" tests

* doc: stm annotations for blockheader & election proof tests

* annotate: stm for error_test

* Annotate "b" file tests

* memrepo_test.go

* Annotate "c" file tests

* message_test.go

* Annotate "D" test files

* doc: stm annotate for fsrepo_test

* Annotate u,v,wl tests

* message_test.go

* doc: stm annotate for chain, node/config & client

* docs: stm annotate node_test

* Annotate "T" test files

* doc: stm annotations for various test files

* Add mempool unit tests

solve merge conflict

* doc: stm annotate for proxy_util_test & policy_test

* doc: stm annotate for various tests

* doc: final few stm annotations

* Add two more memPool Add tests

* Update submodules

* Add check function tests

solve conflict

* Add stm annotations, refactor test helper

solve merge conflict

* Change CLI test kinds to "unit"

* Fix double merged test

* Fix ccupgrade_test merge

* Fix lint issues

* Add stm annotation to types_Test

* Test vectors submodule

* Add file annotation to burn_test

Co-authored-by: Nikola Divic <divicnikola@gmail.com>
Co-authored-by: TheMenko <themenkoprojects@gmail.com>
 2022-03-16 18:37:34 +01:00
Łukasz Magiera
9f6f94bd02
Merge pull request #8215 from filecoin-project/feat/remove-mark-for-upgrade 
chore:sealing:remove endpoint from cli
 2022-03-16 15:33:21 +01:00
Łukasz Magiera
45b07674e5 stores: http: Support multiple storage IDs in ?keep 2022-03-16 12:28:56 +01:00
Jennifer Wang
8a2a2d065d Merge branch 'releases' into jen/backport 2022-03-15 02:07:35 -04:00
Łukasz Magiera
135aef78d7 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-03-11 17:04:58 +01:00
Łukasz Magiera
9c4f8254ee post workers: Fix build post-merge 2022-03-10 14:20:07 +01:00
hannahhoward
49742f8fdc feat(deps): update to graphsync v0.13.0 with 2.0 protocol 2022-03-09 18:06:35 +00:00
Łukasz Magiera
046a9f8af0 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-03-09 16:27:03 +01:00
Aayush
e9e28f75e2 Revert "add resource manager NetStat api" 
This reverts commit d05d5bcb70.
 2022-03-02 15:39:22 -05:00
Aayush
b8747e3afc Revert "add resource manager Net(Set)Limit api" 
This reverts commit 554848fe6b.
 2022-03-02 15:39:21 -05:00
Aayush
bac17061d4 Revert "add examples to docgen" 
This reverts commit 39bf59d372.
 2022-03-02 15:39:21 -05:00
Aayush
7dbf0ced82 Revert "make gen and friends" 
This reverts commit b360c9403f.
 2022-03-02 15:39:14 -05:00
Masih H. Derkani
aacc246ba3
Merge branch 'master' into feat/cid-to-piece-idx 2022-03-02 14:06:30 +00:00
Łukasz Magiera
e9a6f5f287 Merge remote-tracking branch 'origin/master' into feat/paych-avail-reuse 2022-03-02 12:56:56 +00:00
zenground0
a463c90057 remove endpoint from cli 2022-03-01 11:27:03 -07:00
zenground0
77a954c7c3 Merge branch 'master' into feat/cid-to-piece-idx 2022-02-28 12:57:11 -07:00
jennijuju
c6361b391c Merge branch 'releases' into jen/v15 2022-02-25 21:01:31 -05:00
zenground0
e5de237990 Fix fault tracker to handle snap deals 2022-02-24 17:53:21 -05:00
zenground0
abe04c33c1 Fix fault tracker to handle snap deals 2022-02-23 09:56:47 -07:00
Łukasz Magiera
a0cb6094c6 Merge remote-tracking branch 'origin/master' into feat/paych-avail-reuse 2022-02-22 10:55:03 +01:00
Aarsh Shah
e3f5b7c59f update dagstore top level index db 2022-02-18 14:00:01 +04:00
Kevin Li
ba72eff3e6 feat: cli/net: implement 'net ping' command 2022-02-17 17:22:52 +08:00
Łukasz Magiera
384999556c paychmgr: AvailableAmt -> NonReservedAmt 2022-02-16 20:39:43 +01:00
Kevin Li
f677995603 feat: add MsigGetVestingSchedule to gateway api 
expose MsigGetVestingSchedule method to gateway api
 2022-02-16 16:11:46 +08:00
Łukasz Magiera
60b9acc09d gen 2022-02-14 21:16:41 +01:00
Łukasz Magiera
df5bd147fa
paychmgr: Review comments 
Co-authored-by: Hannah Howard <hannah@hannahhoward.net>
Co-authored-by: dirkmc <dirkmdev@gmail.com>
 2022-02-14 20:07:13 +01:00
Łukasz Magiera
f61eb23f8f api: separate method for paych funding 2022-02-14 19:56:02 +01:00
Łukasz Magiera
10af768c60 Merge commit 'origin/release/v1.15.0~2' into feat/paych-avail-reuse 2022-02-14 19:27:12 +01:00
Łukasz Magiera
53e1f75c2c Add FinalizeReplicaUpdate into some more places 2022-02-10 17:27:49 -05:00
Łukasz Magiera
212f5ddb4f wip FinalizeReplicaUpdate 2022-02-10 17:24:26 -05:00
Jennifer Wang
de1f4792cd Merge remote-tracking branch 'origin/master' into jen/v15 2022-02-09 19:27:34 -05:00
Jennifer Wang
6ed97f0d04 Merge branch 'release/v1.14.0' into jen/v15 2022-02-09 03:17:38 -05:00
Łukasz Magiera
09cfad9d71 Add FinalizeReplicaUpdate into some more places 2022-02-08 17:22:41 +01:00
Łukasz Magiera
142ba6660a wip FinalizeReplicaUpdate 2022-02-08 17:22:41 +01:00
Will Scott
7247f8e41f
add net protect api methods 
fix #8015
 2022-02-03 17:24:49 +01:00
Aayush
1b18236f91 feat: sealer: allow users to abort in-flight snap upgrades 2022-01-25 13:01:51 -05:00
Łukasz Magiera
8b19b84140 paych: option to force off-chain get 2022-01-20 18:19:26 +01:00
Łukasz Magiera
8f6f21c94c paych: Print available amounts in paych status 2022-01-20 18:16:00 +01:00
Łukasz Magiera
9715113898 paych: Test pre-funding 2022-01-20 18:15:59 +01:00
Łukasz Magiera
8e46b9ea5d paych: API to pre-fund channels 2022-01-20 18:15:46 +01:00
Aarsh Shah
9ee940098d Merge remote-tracking branch 'origin/master' into feat/cid-to-piece-idx 2022-01-20 16:21:48 +04:00
Łukasz Magiera
e476cf7968 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-01-20 13:15:48 +01:00
vyzo
b360c9403f make gen and friends 2022-01-20 11:44:01 +02:00
vyzo
39bf59d372 add examples to docgen 2022-01-20 11:36:11 +02:00
vyzo
554848fe6b add resource manager Net(Set)Limit api 2022-01-20 11:36:11 +02:00
vyzo
d05d5bcb70 add resource manager NetStat api 2022-01-20 11:36:11 +02:00
Łukasz Magiera
b38141601c Untangle ffi from api 2022-01-18 11:57:04 +01:00
Łukasz Magiera
f2496d87c9 post workers: Snapdeals updates 2022-01-18 11:25:04 +01:00
Łukasz Magiera
0c062f9c08 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-01-18 10:49:42 +01:00
Łukasz Magiera
7bd9af7856 address review 2022-01-14 17:14:32 -05:00
zenground0
d6aa17e21f Snap Deals Integration 
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
 2022-01-14 17:14:32 -05:00
zenground0
d1480c36c0 RemoveData and Decode 
- Unsealing replica update with sector key works and tested
- Sector key generation added and tested
 2022-01-14 17:14:32 -05:00
zenground0
93656e65f8 WIP sector storage and integration test 2022-01-14 17:14:32 -05:00
Łukasz Magiera
4a874eff70 post workers: Cleanup, tests 2022-01-14 14:17:52 +01:00
Aarsh Shah
8c8652e7bb add a new cli 2022-01-12 18:06:48 +04:00
Aarsh Shah
edfc2c215c Merge remote-tracking branch 'origin/main' into feat/cid-to-piece-idx 2022-01-12 12:48:54 +04:00
Łukasz Magiera
960759d22b address review 2022-01-11 17:31:27 +01:00
Łukasz Magiera
e216aefd23 fix make gen 2022-01-10 18:24:00 +01:00
Łukasz Magiera
cf7dd36829 Merge branch 'feat/decoupling-post-worker' of github.com:wusirdcenter/lotus into feat/decoupling-post-worker 2022-01-10 15:06:04 +01:00
zenground0
33f2d24f54 Snap Deals Integration 
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
 2022-01-10 15:39:38 +05:30
mz-sirius
793b5c7cc3 fix ci err 2022-01-05 21:41:21 +08:00
mz-sirius
3fd55fa56b decoupling winningpost and windowpost from lotus-miner 2022-01-05 01:50:49 +08:00
zl
4172a3c8b7 ExampleValue for a silce is nil 2022-01-04 14:27:10 +08:00
hannahhoward
cddf63efe9 feat(storageminer): add api for transfer diagnostics 
Add API + CLI for inspecting in depth diagnostics on graphsync transfers with a given peer
 2021-12-22 13:41:29 -08:00
Dirk McCormick
b1734f84b3 feat: retrieval ask CLI command 2021-12-17 15:55:12 +01:00
Aayush Rajasekaran
80d5e52923 Merge branch 'master' into next 2021-12-13 13:24:28 -05:00
zenground0
a5be80828a RemoveData and Decode 
- Unsealing replica update with sector key works and tested
- Sector key generation added and tested
 2021-12-03 15:21:06 -05:00
Łukasz Magiera
727765b248 Command to list active sector locks 2021-12-03 12:33:23 +01:00
Łukasz Magiera
6d52d8552b Fix docsgen 2021-11-30 02:06:58 +01:00
Łukasz Magiera
c9a2ff4007 cleanup worker resource overrides 2021-11-30 02:06:58 +01:00
Clint Armstrong
4ef8543128 Permit workers to override resource table 
In an environment with heterogenious worker nodes, a universal resource
table for all workers does not allow effective scheduling of tasks. Some
workers may have different proof cache settings, changing the required
memory for different tasks. Some workers may have a different count of
CPUs per core-complex, changing the max parallelism of PC1.

This change allows workers to customize these parameters with
environment variables. A worker could set the environment variable
PC1_MIN_MEMORY for example to customize the minimum memory requirement
for PC1 tasks. If no environment variables are specified, the resource
table on the miner is used, except for PC1 parallelism.

If PC1_MAX_PARALLELISM is not specified, and
FIL_PROOFS_USE_MULTICORE_SDR is set, PC1_MAX_PARALLELSIM will
automatically be set to FIL_PROOFS_MULTICORE_SDR_PRODUCERS + 1.
 2021-11-30 02:06:58 +01:00
Clint Armstrong
93e4656a27 Use a float to represent GPU utilization 
Before this change workers can only be allocated one GPU task,
regardless of how much of the GPU resources that task uses, or how many
GPUs are in the system.

This makes GPUUtilization a float which can represent that a task needs
a portion, or multiple GPUs. GPUs are accounted for like RAM and CPUs so
that workers with more GPUs can be allocated more tasks.

A known issue is that PC2 cannot use multiple GPUs. And even if the
worker has multiple GPUs and is allocated multiple PC2 tasks, those
tasks will only run on the first GPU.

This could result in unexpected behavior when a worker with multiple
GPUs is assigned multiple PC2 tasks. But this should not suprise any
existing users who upgrade, as any existing users who run workers with
multiple GPUs should already know this and be running a worker per GPU
for PC2. But now those users have the freedom to customize the GPU
utilization of PC2 to be less than one and effectively run multiple PC2
processes in a single worker.

C2 is capable of utilizing multiple GPUs, and now workers can be
customized for C2 accordingly.
 2021-11-30 02:06:58 +01:00
Clint Armstrong
c4f46171ae Report memory used and swap used in worker res 
Attempting to report "memory used by other processes" in the MemReserved
field fails to take into account the fact that the system's memory used
includes memory used by ongoing tasks.

To properly account for this, worker should report the memory and swap
used, then the scheduler that is aware of the memory requirements for a
task can determine if there is sufficient memory available for a task.
 2021-11-30 02:06:58 +01:00
Łukasz Magiera
320b36495d gofmt 2021-11-29 21:39:27 +01:00
Łukasz Magiera
d019853687
review: Cleanup some comments 
Co-authored-by: Peter Rabbitson <ribasushi@protocol.ai>
 2021-11-29 21:29:00 +01:00
Łukasz Magiera
5b5e6b9e44 retrieval: DagSpec.MatchPath -> ExportMerkleProof 2021-11-29 21:14:00 +01:00
Łukasz Magiera
61791b90ea retrieval: Only output matching nodes, MatchPath dagspec 2021-11-29 20:40:55 +01:00
zenground0
7d2b3f05db WIP sector storage and integration test 2021-11-29 10:24:00 -05:00
Peter Rabbitson
d5b4d93575 Cleanup partial retrieval codepaths ( zero functional changes ) 
- Adjust the ExportRef comment to reflect change from #7673
- Simplify control flow in parseDagSpec() ( read diff under -w )
 2021-11-25 10:35:05 +01:00
Łukasz Magiera
407c2ed114 retrieval: Drop the RootSelector hack 2021-11-23 17:42:43 +01:00
Łukasz Magiera
bdac11ade7 retrieval: Update some API comments 2021-11-22 14:19:03 +01:00
Łukasz Magiera
763659b8a3 retrieval: Update lotus-soup 2021-11-22 12:46:29 +01:00
Łukasz Magiera
ec2bfb99bb make gen 2021-11-22 12:46:25 +01:00
Łukasz Magiera
b83a9b902a gofmt 2021-11-22 12:46:17 +01:00
Łukasz Magiera
b26906963b retrieval: Support multi-root export 2021-11-22 12:46:17 +01:00
Łukasz Magiera
b0c043cc2f make gen 2021-11-22 12:46:15 +01:00
Łukasz Magiera
b868769ec8 more retrieval api work 2021-11-22 12:46:02 +01:00
Łukasz Magiera
89138bab4d Simplify retrieval APIs 2021-11-22 12:45:42 +01:00
Travis Person
17cb5117f4 add additional methods to lotus gateway 2021-11-22 12:19:27 +01:00
Aayush Rajasekaran
32fc03886d CLI: Add a lotus multisig cancel command 2021-11-18 19:21:17 -05:00
Anton Evangelatov
5454aebf13
CLI tools for the DAGStore inverted index (#7361) 
* cli commands for dagstore

* address comments from Dirk

* rename inverted index to piece index
 2021-11-17 15:39:45 +02:00
Dirk McCormick
501308239f feat: CLI command to announce deal to indexers 2021-11-17 12:16:22 +01:00
Jennifer Wang
724dfc6a4d tidy 2021-10-26 20:10:18 -04:00
Anton Evangelatov
6e5ccc87cf cli: add retry for deals stuck in Publish with no funds 2021-10-11 14:03:55 +02:00
Peter Rabbitson
c4a7de9d37 Remove dead example code + dep 2021-10-07 09:44:50 +02:00
Peter Rabbitson
2a1094fc17 Merge banch feat/go-fil-markets-v1.13.0 into feat/datamodel-selector-retrieval 2021-10-04 21:56:11 +02:00
Aayush Rajasekaran
1d6db306ec Randomness: Move getters from ChainAPI to StateAPI 2021-09-15 00:03:13 +02:00
Peter Rabbitson
0444435589 Expose basic text-based datamodel selector on retrieval 
Syntaxt of selection is located at
https://pkg.go.dev/github.com/ipld/go-ipld-selector-text-lite#SelectorSpecFromPath

Example use, assuming that:
  - The root of the deal is a plain dag-pb unixfs directory
  - The directory is not sharded
  - The user wants to retrieve the first entry in that directory

  lotus client retrieve --miner f0XXXXX --datamodel-path-selector 'Links/0/Hash' bafyROOTCID ~/output

For a much more elaborate example see the top of ./itests/deals_partial_retrieval_test.go
 2021-09-10 09:44:11 +02:00
Łukasz Magiera
e4044151f0 Show deal sizes is sealing sectors 2021-09-07 18:49:53 +02:00
hannahhoward
91804d5746 feat(deps): update go-graphsync v0.9.0 2021-09-03 16:22:51 +02:00
Steven Allen
1cf556c3a2 feat: expose ChainGetPath on the gateway 2021-08-30 16:43:21 -07:00
Łukasz Magiera
e2ba650a8c cli for checking alerts 2021-08-26 15:45:17 +02:00
Łukasz Magiera
5366821144 api for getting alert states 2021-08-26 15:44:45 +02:00
Łukasz Magiera
12875a9664 api/command for encoding actor params 2021-08-20 17:32:57 +02:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671) 
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
 2021-08-16 23:34:32 +01:00
ZenGround0
b914e95f34 revert pricelist by version to pricelist by epoch 2021-08-11 09:49:23 -04:00
Steven Allen
57bf5c2143 feat: add ChainGetTipSetAfterHeight 
This is identical to ChainGetTipSetByHeight, but returns the tipset
following any null tipsets. This is what the user usually wants anyways.

(and I need it for another PR)
 2021-08-05 12:07:14 -07:00
Raúl Kripalani
07f40b9488 fix docs and nits. 2021-07-28 20:03:25 +01:00
Anton Evangelatov
de4a847078 add RuntimeSubsystems API method; use it in lotus-miner info 2021-07-28 16:02:05 +03:00
Mike Greenberg
76a9f4241b feat: Graceful error when api impl is nil 2021-07-26 13:51:37 +02:00
vyzo
1f6935f8c4 make gen 2021-07-26 12:15:01 +03:00
vyzo
3c409d5023 require admin for running checks on the splitstore. 2021-07-26 12:13:41 +03:00
vyzo
77604db716 make gen 2021-07-26 08:33:25 +03:00
vyzo
a0d6fdba33 add ChainBlockstoreInfo APIv1 endpoint 2021-07-26 08:30:07 +03:00
vyzo
21bb2bda09 make gen 2021-07-25 11:25:29 +03:00
vyzo
5b2e4d8ad4 add permission tag to ChainCheckBlockstore API 2021-07-25 11:24:06 +03:00
vyzo
3d2ae433ee add ChainCheckBlockstore API 2021-07-25 11:14:48 +03:00
ZenGround0
fe5e1b23d5 Update api version to account for new network version 6.5 + new underlying repr 2021-07-22 09:49:47 -04:00
Łukasz Magiera
660829703a Merge remote-tracking branch 'origin/master' into feat/split-net-api 2021-07-22 15:38:06 +02:00
whyrusleeping
d89e2f065c add StateReadState to gateway api 2021-07-21 17:21:28 -07:00
Łukasz Magiera
d5c9cccf21 address net interface split review 2021-07-15 15:51:47 +02:00
Łukasz Magiera
b20581bac2 api: check that only the top-level internal struct gets picked up 2021-07-15 15:45:41 +02:00
Łukasz Magiera
49e26cce7d api: Separate the Net interface from Common 2021-07-15 11:41:30 +02:00
hunjixin
bdaa73a413 add ChainGetMessagesInTipset api 2021-07-15 14:52:09 +08:00
Anton Evangelatov
d89ddb9315 resolve conflicts 2021-07-12 11:34:37 +02:00
Łukasz Magiera
ee97aa0a8a dealpublisher: Use address selector 2021-07-07 19:12:03 +02:00
Łukasz Magiera
8f5c0c60f5 Config for deal publishing control addresses 2021-07-07 18:00:54 +02:00
Anton Evangelatov
4f0a96c9c7 resolved conflicts 2021-06-30 13:16:52 +02:00
Łukasz Magiera
c1303f1eac gateway: Add support for Version method 2021-06-28 19:05:27 +02:00
Łukasz Magiera
69e0dff56c api: Fix Wrap for nested proxy structs 2021-06-28 19:00:37 +02:00
Anton Evangelatov
d45bb14015 Merge branch 'master' into nonsense/split-market-miner-processes 2021-06-16 18:49:15 +02:00
Aayush Rajasekaran
bee548face Add utils to use multisigs as miner owners 2021-06-15 16:57:52 -04:00
Anton Evangelatov
090a16316c bump miner version apiv0 2021-06-15 13:59:51 +02:00
Anton Evangelatov
31e6fb154e resolve conflicts 2021-06-14 11:27:54 +02:00