cerc-io/lotus
15
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
15,459 Commits 24 Branches 342 Tags 314 MiB
0d7d02da35
Commit Graph

1107 Commits

Author SHA1 Message Date
Aayush Rajasekaran
b9bfcc4b69 Randomness: Move getters from ChainAPI to StateAPI 2021-09-21 14:20:15 -04:00
Łukasz Magiera
2437a4182d
Merge pull request #7322 from filecoin-project/asr/drand-state 
Randomness: Move getters from ChainAPI to StateAPI
 2021-09-21 09:20:12 +01:00
Anton Evangelatov
605953565b unit test where StateMarketStorageDeal return nil, err 2021-09-17 15:14:53 +02:00
Anton Evangelatov
34e3dc2b37 GetCurrentDealInfo err: handle correctly err case 2021-09-17 14:55:56 +02:00
Łukasz Magiera
438d5ce78e fix a panic in HandleRecoverDealIDs 2021-09-15 18:13:01 +02:00
Łukasz Magiera
ef03314c6d storagemgr: Cleanup workerLk around worker resources 2021-09-15 16:35:19 +02:00
Aayush Rajasekaran
1d6db306ec Randomness: Move getters from ChainAPI to StateAPI 2021-09-15 00:03:13 +02:00
swift-mx
b0b016b58c fix: check padSector Cid 2021-09-10 18:13:07 +08:00
Jennifer Wang
f751291f1a update to proof v0.9.2 2021-09-08 01:03:11 -04:00
frank
179458efe9 index out of range 2021-09-06 16:20:23 +08:00
Jennifer Wang
6379d5129f integrate the proof patch: tag proofs-v9-revert-deps-hotfix 2021-09-02 15:40:15 -04:00
Łukasz Magiera
7ef1b62b41 sealing: Fix sector state accounting with FinalizeEarly 2021-09-02 19:27:10 +02:00
Łukasz Magiera
41db98d49f sealing: Fix tests 2021-09-01 12:09:44 +02:00
Łukasz Magiera
b280e29409 sealing: Fix retry loop in SubmitCommitAggregate 2021-09-01 11:33:23 +02:00
Łukasz Magiera
3118bd1039 stores: Fix reserved disk usage log spam 2021-08-31 13:36:09 +02:00
Steven Allen
1cf556c3a2 feat: expose ChainGetPath on the gateway 2021-08-30 16:43:21 -07:00
Łukasz Magiera
2293ecd8e8 Reduce lotus-miner startup spam 2021-08-27 19:41:54 +02:00
Łukasz Magiera
b42171dc04 sealing: Check piece CIDs after AddPiece 2021-08-26 13:02:15 -07:00
Łukasz Magiera
62769e3b11 sealing: Fix RecoverDealIDs loop with changed PieceCID 2021-08-20 16:00:40 +02:00
Aayush Rajasekaran
113ea6dc4b Update to latest FFI 2021-08-17 18:15:12 -04:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671) 
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
 2021-08-16 23:34:32 +01:00
Steven Allen
498644acab
Merge pull request #7011 from filecoin-project/fix/no-t-aux 
fix: don't check for t_aux when proving
 2021-08-11 11:28:29 -07:00
Aayush Rajasekaran
a52352b13a PreCommitPolicy: Don't try to align expirations on proving period boundaries 2021-08-11 12:44:12 -04:00
ZenGround0
6da26dcefc Remove network version panic 2021-08-10 13:07:30 -04:00
Steven Allen
18f39be3ba fix: don't check for t_aux when proving 
We don't need it.
 2021-08-09 11:07:35 -07:00
mx
cfc10e9b52 fix: more logging in maybeStartBatch error 2021-08-06 16:50:37 +08:00
He Weidong
4273fb067c reduce diff 2021-07-29 15:01:52 +08:00
He Weidong
ef6620ad8e reasonable max value for initial sector expiration 2021-07-29 14:57:06 +08:00
He Weidong
ee20d30480 reasonable min and max value for initial sector expiration 2021-07-29 14:55:11 +08:00
Steven Allen
dbe4891a39
Merge branch 'master' into fix/check_ticket_msg 2021-07-28 15:25:40 -07:00
Aayush Rajasekaran
21244704fc
Merge branch 'master' into mg/feat/configurable-sector-expirations 2021-07-28 16:45:27 -04:00
Jennifer Wang
2790c9439f Update to proof v8.0.3 2021-07-27 13:29:07 -04:00
hunjixin
ac8937245c fix ticket check 2021-07-27 16:21:47 +08:00
Mike Greenberg
41bce7925a fix: Remove unnecessary Truncate 2021-07-23 20:12:30 -04:00
Mike Greenberg
3829d6bd9a fix: Remove actor method from autogen source; Move inline to pkg 2021-07-22 15:10:49 -04:00
Mike Greenberg
123a976f62 chore: Add unit test for custom CC lifetime value expiration 2021-07-22 15:07:03 -04:00
Mike Greenberg
7ee46ad4e0 fix: PreCommitPolicy unit tests 2021-07-22 15:07:03 -04:00
Mike Greenberg
adb62a3fff chore: Move cfg getter into PCPolicy; Clamp values on get 2021-07-22 15:07:03 -04:00
Mike Greenberg
9e34cce1f2 feat(config): support configurable CC sector expiration 2021-07-22 15:07:03 -04:00
Łukasz Magiera
8029a9ffe3 sealing: Address riview on preCommitParams send fix 2021-07-20 17:46:19 -07:00
johnli-helloworld
aea338ecf0 fix handleSubmitPreCommitBatch() preCommitParams error 2021-07-20 17:46:19 -07:00
Steven Allen
865e7ca80a test(sealing): test message not found 2021-07-20 12:09:50 -07:00
Steven Allen
ad1dd36193 fix: better error message on missing message 2021-07-20 11:20:45 -07:00
xloem
e0208e3595 fixes #6786 segfault 2021-07-20 11:20:45 -07:00
Łukasz Magiera
583a8a13d9
Merge pull request #6629 from filecoin-project/feat/pledge-from-miner-balance 
Config for collateral from miner available balance
 2021-07-13 17:22:00 +02:00
Łukasz Magiera
0678b3fa05 sealing: Cleanup DisableCollateralFallback handling 2021-07-13 17:06:41 +02:00
Łukasz Magiera
7526a074d9 sealing: collateral buffer / falback config 2021-07-13 17:04:58 +02:00
Łukasz Magiera
83f2368507 Add CollateralFromMinerBalance config 2021-07-12 14:11:58 +02:00
Anton Evangelatov
16784aa2cc remove pieceProvider from DI; small refactors 2021-07-12 11:30:26 +02:00
Anton Evangelatov
566a9d5541 fix signature 2021-07-06 18:12:30 +02:00
Anton Evangelatov
7a78527fde remove not used sa; cleanup 2021-07-06 17:46:21 +02:00
Anton Evangelatov
604be5fc82
Update extern/sector-storage/stores/remote.go 
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2021-07-06 17:24:56 +02:00
Anton Evangelatov
ff2772a58c resolved conflicts 2021-07-06 16:00:41 +02:00
Anton Evangelatov
4be0a7a215 resolve conflicts with master 2021-07-05 13:13:32 +02:00
Peter Rabbitson
a544fa0f97 Fix more CID doucle-encoding as hex 2021-07-05 12:51:15 +02:00
Łukasz Magiera
e2f48b21ab
Merge pull request #6653 from filecoin-project/fix/finalize-in-storage 
storage: Fix FinalizeSector with sectors in stoage paths
 2021-07-02 20:38:25 +02:00
Łukasz Magiera
8a94ab676e storage: Fix FinalizeSector with sectors in stoage paths 2021-07-02 19:54:45 +02:00
Łukasz Magiera
35c0e3cf8c
Merge pull request #6658 from sunday527/master 
Fix: precommit_batch method used the wrong cfg.CommitBatchWait
 2021-07-02 15:18:48 +02:00
Łukasz Magiera
60ecbdee8f
Merge pull request #6635 from llifezou/fix_checkTicketExpired 
fix ticket expiration check
 2021-07-02 15:18:04 +02:00
zhoutian527
71e1577a62 Fix: precommit_batch method used the wrong cfg.PreCommitBatchWait 2021-07-02 15:53:21 +08:00
llifezou
df53b97fe4
Update extern/storage-sealing/states_sealing.go 
fix sector precommitted but expired judgment

Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2021-07-02 11:44:46 +08:00
llifezou
fe3ad4fd60
Update extern/storage-sealing/states_sealing.go 
fix log

Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2021-07-02 11:38:04 +08:00
Łukasz Magiera
7c2c8b2a95 commit batch: Regression test nil FailedSectors map 2021-07-01 13:51:11 +02:00
Łukasz Magiera
c094aa82ec commit batch: AggregateAboveBaseFee config 2021-07-01 13:33:54 +02:00
Łukasz Magiera
88bb9f422e commit batch: Initialize the FailedSectors map 2021-07-01 12:15:58 +02:00
llifezou
73e58f7af1 fix getTicket: sector precommitted but expired case 2021-07-01 10:53:42 +08:00
johnli-helloworld
229d5e5c80 handleSubmitCommitAggregate() exception handling 2021-06-30 15:29:21 +02:00
Łukasz Magiera
2dad06d59f
Merge pull request #6634 from GFZRZK/GFZRZK/correct_check_commit 
remove precommit check in handleCommitFailed
 2021-06-30 15:23:33 +02:00
Łukasz Magiera
f45340461d
gofmt 2021-06-30 15:19:29 +02:00
Anton Evangelatov
4f0a96c9c7 resolved conflicts 2021-06-30 13:16:52 +02:00
johnli-helloworld
b0b070ffe7 to optimize the batchwait 2021-06-30 16:56:40 +08:00
llifezou
3dd3476bfd fix ticket expiration check, otherwise it may cause a large number of loops to retry GetTicket when retrying PreCommit1 2021-06-30 16:32:44 +08:00
wangchao
a4342f3997 remove precommit check in handleCommitFailed 2021-06-30 11:09:07 +08:00
Łukasz Magiera
7b1b082381 Merge branch 'master' into fix/prove-commit-aggregate 2021-06-29 11:48:52 +02:00
Jerry
73704c92ba ensure agg fee is adequate 2021-06-29 14:36:26 +08:00
Jerry
5c3d67a811 fix: miner balance is not enough, so that ProveCommitAggregate msg exec failed 2021-06-29 14:06:41 +08:00
Łukasz Magiera
84dbcb63d2
Merge pull request #6566 from GFZRZK/GFZRZK/correct_sector_expire_time 
scale up sector expiration to avoid sector expire in batch-pre-commit waitting
 2021-06-24 10:59:49 +02:00
Łukasz Magiera
c3480dc0e8 Merge remote-tracking branch 'origin/releases' into chore/merge-release 2021-06-24 09:39:11 +02:00
Aayush Rajasekaran
616e5688fc Remove MinPreCommitBatch 2021-06-23 12:30:32 -04:00
Anton Evangelatov
21b51328f9 adding TestDealWithMarketAndMinerNode 2021-06-23 12:44:52 +02:00
wangchao
03261771e2 scale up sector expiration to avoid sector expire in batch-pre-commit waitting 2021-06-23 14:10:28 +08:00
Aayush Rajasekaran
93f7cbe587 Add a helpful comment 2021-06-22 19:31:36 -04:00
Aayush Rajasekaran
6ca2a14819 Always flush when timer goes off 2021-06-22 19:16:36 -04:00
Anton Evangelatov
6720463799 resolve merge conflicts 2021-06-22 11:28:23 +02:00
Raúl Kripalani
684cce198f add a unit test. 2021-06-21 20:49:24 +01:00
Raúl Kripalani
59eab2df25 move scheduling filtering logic down. 2021-06-21 20:49:16 +01:00
Raúl Kripalani
83839362c5 fix boolean condition. 2021-06-21 20:35:51 +01:00
Raúl Kripalani
b6147fb27f extern/storage: retype resource filtering strategy to enum. 2021-06-21 20:28:15 +01:00
Raúl Kripalani
f3b6f8de1a add ability to ignore worker resources when scheduling. 2021-06-21 20:08:18 +01:00
Łukasz Magiera
b37a66c7c9 Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10 2021-06-21 17:03:03 +02:00
Łukasz Magiera
738ac926d1 Update ffi 2021-06-18 16:27:37 +02:00
Łukasz Magiera
5516f3492e ffiwrapper: Assert aggregate length 2021-06-18 12:02:42 +02:00
Łukasz Magiera
7f00a6e59c Proofs v8.0.2 with fixed aggregate lengths 2021-06-18 11:17:18 +02:00
yaohcn
7c6736b3ef fix commit finalize failed 2021-06-18 12:09:02 +08:00
Łukasz Magiera
0a5a7cf45d storagefsm: Fix too-long log handling 2021-06-17 20:52:46 -04:00
Łukasz Magiera
b0128bd99e storagefsm: Fix race spawning more than one new sector at once 2021-06-17 20:52:42 -04:00
Łukasz Magiera
097baeb9b0 Make batch deal input test less flaky 2021-06-17 20:52:09 -04:00
Łukasz Magiera
5c8498b603 storagefsm: Fix batch deal packing behavior 2021-06-17 20:50:14 -04:00
Łukasz Magiera
44de67cf7d
Merge pull request #6175 from filecoin-project/feat/dynamic-retreival-pricing 
Dynamic Retrieval pricing
 2021-06-17 10:25:48 +02:00
aarshkshah1992
9002fcbe47 local unsealed file wont have the unsealed piece 2021-06-17 09:46:29 +05:30
Anton Evangelatov
d45bb14015 Merge branch 'master' into nonsense/split-market-miner-processes 2021-06-16 18:49:15 +02:00
Łukasz Magiera
1f914053b4 sealing: Wire up context to batchers 2021-06-16 11:43:06 +02:00
Łukasz Magiera
4bd2587e47
Merge pull request #6479 from filecoin-project/test/multicore-sdr 
Test multicore SDR support
 2021-06-16 11:02:09 +02:00
Łukasz Magiera
81b412399e sealing: Fix restartSectors race 2021-06-15 17:55:58 -04:00
Łukasz Magiera
74db586fdf sealing: Fix restartSectors race 2021-06-15 17:43:32 -04:00
Aayush Rajasekaran
520a0091f5 Merge branch 'releases' into release/v1.10.0 2021-06-15 17:40:36 -04:00
Łukasz Magiera
c7c593c74e TestMulticoreSDR: Setup rust logger after envvar check 2021-06-15 17:59:46 +02:00
Łukasz Magiera
6b0aed9317 Setup rust logger in the test 2021-06-15 17:53:26 +02:00
Łukasz Magiera
265afd696e Run TestMulticoreSDR on Circle 2021-06-15 17:46:06 +02:00
Łukasz Magiera
9ae780902a Test multicore SDR support 2021-06-15 17:36:10 +02:00
Łukasz Magiera
20c4fe9bba Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10 2021-06-15 16:50:23 +02:00
Aayush Rajasekaran
18b13f6f58 Update to fixed Bellperson 2021-06-14 20:19:26 -04:00
Łukasz Magiera
f9596dd730
Merge pull request #6432 from filecoin-project/tests/batching-unit 
Unit tests for sector batchers
 2021-06-14 21:05:34 +02:00
wangchao
4da30931c9 failed sectors should be added into res correctly 2021-06-14 11:33:38 -04:00
Łukasz Magiera
40efafb7cd Update ffi with fixed multicore sdr support 2021-06-14 17:29:21 +02:00
Łukasz Magiera
ff43d29bbd Update ffi with fixed multicore sdr support 2021-06-14 16:59:55 +02:00
Łukasz Magiera
0514f38dc4 Merge remote-tracking branch 'origin/master' into chore/merge-1.10 2021-06-14 13:27:53 +02:00
Anton Evangelatov
31e6fb154e resolve conflicts 2021-06-14 11:27:54 +02:00
wangchao
f392c1295c failed sectors should be added into res correctly 2021-06-12 23:22:32 +08:00
aarshkshah1992
ea9bed2746 undo ffi change 2021-06-11 16:22:09 +05:30
Łukasz Magiera
733240a2bb sealing: Test early finalization fsm planners 2021-06-11 11:52:00 +02:00
Łukasz Magiera
05d9b5ce0f sealing: Add missing planner for CommitFinalizeFailed 2021-06-11 11:44:37 +02:00
Łukasz Magiera
ec06f086ef sealing: Early finalization option 2021-06-11 11:41:28 +02:00
aarshkshah1992
fed5afa704 merge master 2021-06-11 09:35:20 +05:30
Łukasz Magiera
06195bc8e1 Unit tests for sector batchers 2021-06-10 13:46:36 +02:00
Aayush Rajasekaran
3766980331 Merge branch 'release/v1.10.0' into asr/merge-release 2021-06-09 18:41:10 -04:00
Łukasz Magiera
52199c9af3 Proofs v8.0.1 2021-06-10 00:17:13 +02:00
Łukasz Magiera
f0a2e97cb5 fee config for sector batching 2021-06-09 14:23:06 -04:00
Aayush Rajasekaran
dede1b4758 Merge branch 'release/v1.10.0' into asr/merge-release 2021-06-09 14:05:41 -04:00
Aayush Rajasekaran
b0c9dd49f0 Fund miners with the aggregate fee when ProveCommitting 2021-06-09 13:08:54 -04:00
Aayush Rajasekaran
dbb4e9fcc5 Drop soms logs 2021-06-09 12:26:20 -04:00
Anton Evangelatov
beb5bffaf6 fix log.Error; rename RemoteStore 2021-06-09 13:05:54 +02:00
Łukasz Magiera
92bb874327
Use correct batch fee config in commit batcher 
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2021-06-09 12:17:18 +02:00
Aayush Rajasekaran
ba27d45214 Split the getSectorCutoff methods between precommit and commit batchers 2021-06-08 16:46:35 -04:00
Łukasz Magiera
78171055e7 fee config for sector batching 2021-06-08 15:45:32 +02:00
Anton Evangelatov
30ba9a2751 remove double http handler decl 2021-06-08 15:36:54 +02:00
Anton Evangelatov
4dd093b160 fix piece provider test 2021-06-08 14:57:45 +02:00
Anton Evangelatov
9f3ec82cc1 initial resolution of conflicts 2021-06-08 13:01:22 +02:00
aarshkshah1992
4e9bb16532 changes as per review 2021-06-08 10:09:27 +05:30
aarshkshah1992
212d0fc264 fix remote store diff 2021-06-08 09:54:45 +05:30
aarshkshah1992
e4e60d7af4 fix mock 2021-06-08 09:37:12 +05:30
aarshkshah1992
16166504f4 merged master 2021-06-08 09:17:40 +05:30
Aayush Rajasekaran
b13169f071 Rename deadlines to cutoffs in the batchers 2021-06-07 20:20:06 -04:00
aarshkshah1992
ddd9bf610e fix CI 2021-06-07 16:45:52 +05:30
aarshkshah1992
1a90d3bbb5 update ffi 2021-06-07 16:12:39 +05:30
Anton Evangelatov
b0cb0c1a4a do not depend on filecoin-ffi in api package 2021-06-07 11:51:25 +02:00
aarshkshah1992
8d9cef17af changes as per review 2021-06-07 15:03:09 +05:30
aarshkshah1992
2a134887c3 logs to debug read & unseal 2021-06-07 15:03:09 +05:30
aarshkshah1992
ec6a49693f logs to debug read and unseal 2021-06-07 15:03:09 +05:30
aarshkshah1992
207f0d901a integration test should remove unsealed files 2021-06-07 15:03:09 +05:30
aarshkshah1992
73f7825fbb clean up logging 2021-06-07 15:03:09 +05:30
aarshkshah1992
3b792a32c3 better logging 2021-06-07 15:03:09 +05:30
aarshkshah1992
22f36483cb more logging 2021-06-07 15:03:09 +05:30
aarshkshah1992
acfa3d7370 finish integration tests 2021-06-07 15:03:09 +05:30
aarshkshah1992
ad4b182bfe remove read task type and run gen and docsgen 2021-06-07 15:03:06 +05:30
aarshkshah1992
35a0dbfa8c fix go mod 2021-06-07 15:02:52 +05:30
aarshkshah1992
4efaa8d688 unit tests for the remote store Reader 2021-06-07 15:02:52 +05:30
aarshkshah1992
0d88800eb9 use mockgen 2021-06-07 15:02:52 +05:30
aarshkshah1992
77b5e8d045 use an actual worker in the integration tests 2021-06-07 15:02:52 +05:30
Dirk McCormick
6879ae9e6a feat: TestPieceProviderReadPiece 2021-06-07 15:02:52 +05:30
aarshkshah1992
74372d3e81 fix linting problems 2021-06-07 15:02:52 +05:30
aarshkshah1992
759d8f090b test http handler 2021-06-07 15:02:52 +05:30
aarshkshah1992
db5c88196d address review comments 2021-06-07 15:02:52 +05:30
Aarsh Shah
c853350bdf Apply suggestions from code review 
Co-authored-by: dirkmc <dirkmdev@gmail.com>
 2021-06-07 15:02:52 +05:30
aarshkshah1992
65eb610ec3 docs, logs and green ci 2021-06-07 15:02:49 +05:30
aarshkshah1992
670835fca0 bypass task scheduler for reading unsealed pieces 2021-06-07 15:02:04 +05:30
Anton Evangelatov
d9a7348ae1 use masters filecoin-ffi 2021-06-04 16:20:40 +02:00
Anton Evangelatov
8bd09e39ca resolve merge conflicts 2021-06-04 16:17:00 +02:00
Anton Evangelatov
88756f3ebf fix TestAPIDeal tests 2021-06-04 15:41:38 +02:00
aarshkshah1992
05ba3de5cc changes as per review 2021-06-04 10:27:43 +05:30
Łukasz Magiera
565bb4f589 mock: Log debug info on bad aggregates 2021-06-03 17:35:10 +02:00
aarshkshah1992
43453bb633 fix broken test 2021-06-03 10:15:23 +05:30
Łukasz Magiera
40cc29d723 Skip FD check in TestDownloadParams 2021-06-02 19:50:17 +02:00
Łukasz Magiera
d77d3df698 Merge remote-tracking branch 'origin/feat/nv13' into feat/nv13-1.11 2021-06-02 15:06:50 +02:00
Łukasz Magiera
f6963523f8 Use filecoin-ffi master 2021-06-02 11:13:24 +02:00
Łukasz Magiera
dc642d0b7b Merge remote-tracking branch 'origin/feat/nv13' into feat/nv13-1.11 2021-06-01 21:06:58 +02:00
Łukasz Magiera
8b84e499c5 storagefsm: Handle preCommitParams errors better 2021-06-01 20:36:20 +02:00
Łukasz Magiera
482e1110c2 precommit batcher: Improve error propagation 2021-06-01 14:35:30 +02:00
Łukasz Magiera
cb4eb487f4 commit batcher: Fix min aggregate size check 2021-06-01 12:27:22 +02:00
Łukasz Magiera
084b0e7f60 Handle collateral when submitting aggregated commits 2021-06-01 12:09:15 +02:00
Łukasz Magiera
9fcb564bef Make commit batcher more robust 2021-06-01 11:56:19 +02:00
Anton Evangelatov
0a79f49dc5 fix tests 2021-06-01 11:45:34 +02:00
Łukasz Magiera
ffa47659a1 Merge remote-tracking branch 'origin/feat/nv13' into feat/nv13-1.11 2021-05-31 21:38:34 +02:00
Łukasz Magiera
c3e8eddb9b Merge remote-tracking branch 'origin/master' into feat/nv13-1.11 2021-05-31 21:24:56 +02:00
Aayush Rajasekaran
61554cf3e0 Update to latest actors 2021-05-31 14:47:41 -04:00
Anton Evangelatov
d195a12565 Merge branch 'master' into nonsense/split-market-miner-processes 2021-05-31 11:26:11 +02:00
Łukasz Magiera
f3bf773152 storagefsm: Fix too-long log handling 2021-05-30 19:24:42 +02:00
Łukasz Magiera
6e1919c67f storagefsm: Fix race spawning more than one new sector at once 2021-05-30 18:30:38 +02:00
Łukasz Magiera
9475079b97 Make batch deal input test less flaky 2021-05-30 17:46:22 +02:00
Łukasz Magiera
c8d603557b storagefsm: Fix batch deal packing behavior 2021-05-30 17:46:22 +02:00
Łukasz Magiera
1e4456138e Merge master into feat/nv13 2021-05-27 12:28:20 +02:00
yaohcn
d04e7d98ce Get current seal proof when necessary 2021-05-27 11:53:33 +08:00
Łukasz Magiera
fc76a09b19 mock: Use real aggregate lengths 2021-05-26 10:06:41 +02:00
Łukasz Magiera
7bd0fcbb24 sealing: Don't start batch timers with empty batches 2021-05-25 16:47:42 +02:00
Łukasz Magiera
04658e1cae fix lint 2021-05-25 16:26:59 +02:00
Łukasz Magiera
f5409845b5 Some review addressing 2021-05-25 16:07:45 +02:00
aarshkshah1992
0f3ff9e06a fix typo 2021-05-24 15:22:11 +05:30
aarshkshah1992
dc6dbc9a11 dpr changes and test based on new unsealing PR 2021-05-22 22:40:21 +05:30
Łukasz Magiera
58203559e7
Merge pull request #6304 from yaohcn/fix-ticket 
fix ticket expired
 2021-05-21 16:07:28 +02:00
aarshkshah1992
50e023edd3 changes as per review 2021-05-21 19:15:05 +05:30
aarshkshah1992
2c9f5922b5 logs to debug read & unseal 2021-05-21 19:02:37 +05:30
aarshkshah1992
8ff5bce7a3 logs to debug read and unseal 2021-05-21 19:01:17 +05:30
aarshkshah1992
fb29f782df integration test should remove unsealed files 2021-05-21 16:15:08 +05:30
aarshkshah1992
536d7c4275 clean up logging 2021-05-21 15:50:25 +05:30
aarshkshah1992
40642b2cad better logging 2021-05-21 15:16:20 +05:30
aarshkshah1992
85f2ac8789 more logging 2021-05-21 14:56:37 +05:30
aarshkshah1992
78a0458ada finish integration tests 2021-05-21 11:00:17 +05:30
yaohcn
00a1e2c05b test ticket expired 2021-05-21 11:45:57 +08:00
aarshkshah1992
c17300dc1f remove read task type and run gen and docsgen 2021-05-20 15:25:46 -06:00
aarshkshah1992
50d7acfa0c fix go mod 2021-05-20 15:25:46 -06:00
aarshkshah1992
bd9959070e unit tests for the remote store Reader 2021-05-20 15:25:46 -06:00
aarshkshah1992
9b34494501 use mockgen 2021-05-20 15:25:46 -06:00
aarshkshah1992
31a5f68dfa use an actual worker in the integration tests 2021-05-20 15:25:46 -06:00
Dirk McCormick
ec85a973b1 feat: TestPieceProviderReadPiece 2021-05-20 15:25:46 -06:00
aarshkshah1992
8c4c26ca4c fix linting problems 2021-05-20 15:25:46 -06:00
aarshkshah1992
dd9c9fd4fd test http handler 2021-05-20 15:25:46 -06:00
aarshkshah1992
c58048d16a address review comments 2021-05-20 15:25:46 -06:00
Aarsh Shah
d33d426692 Apply suggestions from code review 
Co-authored-by: dirkmc <dirkmdev@gmail.com>
 2021-05-20 15:25:46 -06:00
aarshkshah1992
73613ee883 docs, logs and green ci 2021-05-20 15:25:46 -06:00
aarshkshah1992
2a40c802ea bypass task scheduler for reading unsealed pieces 2021-05-20 15:25:46 -06:00
Łukasz Magiera
2d4eaf08c4
Merge pull request #6233 from nonsense/fix-finalizesector-storage-req 
consider storiface.PathStorage when calculating storage requirements
 2021-05-20 21:43:55 +02:00
Łukasz Magiera
5f2e163bbb
Merge pull request #6259 from filecoin-project/raulk/docs-storage 
`storage` module: add go docs and minor code quality refactors
 2021-05-20 21:42:30 +02:00
Łukasz Magiera
e3255a06ea sealing: Add missing states to SubmitPreCommitBatch planner 2021-05-20 18:39:59 +02:00
Łukasz Magiera
d71334cc24 Address self-review 2021-05-20 18:04:07 +02:00
yaohcn
e21d4d7a7e fix ticket expired 2021-05-20 20:08:24 +08:00
Anton Evangelatov
9a7b0b657e comment 2021-05-20 13:18:56 +02:00
Anton Evangelatov
19bd5beb96 update remoteGetSector 2021-05-20 13:15:57 +02:00
Anton Evangelatov
1a9b5760a7 add remoteGetAllocated http handler 2021-05-20 13:14:25 +02:00
Anton Evangelatov
90928991b3 remove DealInfo and DealSchedule from storage-sealing gen/main.go 2021-05-20 13:11:10 +02:00
Anton Evangelatov
76bb424de0 update StorageMinerAPI 2021-05-20 13:01:14 +02:00
Anton Evangelatov
cb603c62d9 update retrievaladapter ; add piece_provider 2021-05-20 12:49:53 +02:00
Anton Evangelatov
c12d802811 update reader to use ReadAtLeast 2021-05-20 12:38:38 +02:00
Anton Evangelatov
2aad7b6979 update sectorstorage.New (Manager) interface 2021-05-20 12:32:29 +02:00
Łukasz Magiera
bb889a5976 sealing: Fix max commit batch size check 2021-05-20 12:11:11 +02:00
Łukasz Magiera
7edffcd37b Update ffi 2021-05-20 11:06:40 +02:00
Łukasz Magiera
51139361c7 sealing: Handle full batches correctly 2021-05-19 20:34:50 +02:00
Łukasz Magiera
e088c71b9a marketadapter: Handle batch sealing messages 2021-05-19 20:07:20 +02:00
Łukasz Magiera
a5677d1b7a ffiwrapper: Separate Prover interface 2021-05-19 15:20:23 +02:00
Anton Evangelatov
83e55dc09e move handleProvingSector to correct file - states_proving.go 2021-05-19 13:17:59 +02:00
Anton Evangelatov
55401116bb add UnpaddedByteIndex.Valid() 2021-05-19 13:16:29 +02:00
Anton Evangelatov
e275b54f55 export PartialFile and OpenPartialFile 2021-05-19 13:14:16 +02:00
Anton Evangelatov
a989f60e27 add SectorAddPieceToAny and SectorUnsealPiece to StorageMiner iface; model moved to api package - PieceDealInfo, DealSchedule 2021-05-19 13:05:07 +02:00
Łukasz Magiera
dd393b470f Fix aggregation inputs 2021-05-18 20:36:47 +02:00
Łukasz Magiera
0419c64a06 CLI for precommit batching 2021-05-18 20:35:35 +02:00
Łukasz Magiera
e400bdf87a Order proofs before aggregation 2021-05-18 18:58:41 +02:00
Łukasz Magiera
9690bc882c Test to trigger batching logic 2021-05-18 18:41:42 +02:00
Łukasz Magiera
1946d2ffd4 Wire up Precommit Batching 2021-05-18 17:37:52 +02:00
Łukasz Magiera
f66b9c5663 Maybe working precommit batching 2021-05-18 17:21:16 +02:00
Łukasz Magiera
d92c5e1001 Missing precommit batcher 2021-05-18 16:54:55 +02:00
Łukasz Magiera
c7ba083fa4 Import precommit batcher 2021-05-18 16:51:06 +02:00
Łukasz Magiera
56145201db
Merge pull request #6282 from filecoin-project/feat/fip-0013-addenda 
FIP 13 addenda: correctly handle commit batch timer
 2021-05-18 16:23:23 +02:00
vyzo
c544f4ce44 avoid extraneous rpc call for storage start epoch 2021-05-18 16:59:11 +03:00
vyzo
6b3e04b9b1 cache sector deadlines. 2021-05-18 15:28:51 +03:00
vyzo
74bad49068 correctly handle commit batch timer 2021-05-18 14:30:47 +03:00
Łukasz Magiera
fe9311e435 update ffi 2021-05-18 13:24:10 +02:00
vyzo
357c0868b7 proper config for termination batching and commit wait 2021-05-18 12:20:19 +03:00
Łukasz Magiera
ba2032c642 Fix some aggregation bugs 2021-05-17 22:51:29 +02:00
Jennifer Wang
5af3af5a22 Merge branch 'release/v1.9.0' into releases 2021-05-17 16:16:34 -04:00
Łukasz Magiera
6278bdc69a Make things build 2021-05-17 20:47:41 +02:00
Łukasz Magiera
8f42f375cf Update ffi 2021-05-17 19:55:44 +02:00
Łukasz Magiera
71bd005594 wip 2021-05-17 13:31:43 -04:00
Aayush Rajasekaran
5f8c80533a Update to latest actors and FFI 2021-05-17 13:15:07 -04:00
Raúl Kripalani
2d7f4b1c61 docs: add godocs to storage module. 2021-05-14 19:45:47 +01:00
Łukasz Magiera
506f39b294 WIP: Integrate FIP0013 2021-05-11 22:10:29 -04:00
Anton Evangelatov
eb13c74dce panic on unknown pathType 2021-05-11 18:14:01 +02:00
Anton Evangelatov
e07438417c consider storiface.PathStorage when calculating storage requirements 2021-05-11 13:19:26 +02:00
Jennifer Wang
b1d046ce16 Update to proof v7.0.1 2021-05-06 17:31:15 -04:00
Travis Person
323bb02ca5 Update to new proofs with gpu support enabled in release binaries 2021-05-04 20:31:26 +00:00
Łukasz Magiera
7c37618b50 Update ffi to proofs v7 2021-04-30 14:30:12 +02:00
Łukasz Magiera
19ced50d81 Update ffi to proofs v7 2021-04-30 14:28:36 +02:00
whyrusleeping
ac77c51d5e address nit 2021-04-29 10:24:16 -07:00
whyrusleeping
48feb52cbf add a test for adding padded pieces 2021-04-27 16:28:25 -07:00
whyrusleeping
6daaf6ac2e attempt to do better padding on pieces being written into sectors 2021-04-27 11:39:06 -07:00
Łukasz Magiera
a24a73bb52 Merge tag 'v1.8.0' into release/v1.9.0 2021-04-27 10:30:33 +02:00
Łukasz Magiera
c4137a438d Merge commit 'f1ded63d608e1b191d79c780b8065f8cff367c98' into feat/nv12-1.11 2021-04-27 08:22:55 +02:00
Aayush Rajasekaran
f1ded63d60 v4 specs-actors integration, nv12 migration 2021-04-27 02:02:44 -04:00
Dirk McCormick
61344644a4 feat: add more debug logging for unsealing 2021-04-22 11:33:28 +02:00
Łukasz Magiera
e76935147f Merge remote-tracking branch 'origin/master' into feat/v1-api 2021-04-13 14:24:22 +02:00
Łukasz Magiera
335a2df062
Merge pull request #6006 from filecoin-project/asr/proofs-update 
Update to latest proofs
 2021-04-13 11:27:02 +02:00
Łukasz Magiera
b54b1303f9
Merge pull request #5962 from filecoin-project/fix/storagefsm-allow-broken-global 
storagefsm: Apply global events even in broken states
 2021-04-11 22:30:56 +02:00
yaohcn
844d1d364d fix health report 2021-04-11 14:47:04 +08:00
Aayush Rajasekaran
738cc2555c Update FFI 2021-04-10 01:40:24 -04:00
Łukasz Magiera
9d4935a736
Merge pull request #5976 from filecoin-project/fix/storage-demote-stat-err 
localstorage: Demote reservation stat error to debug
 2021-04-09 21:08:08 +02:00
Łukasz Magiera
bed5ec81bd localstorage: Demote reservation stat error to debug 2021-04-06 16:06:47 +02:00
Łukasz Magiera
9fb345af67 storagefsm: Test applying global events in broken state 2021-04-06 15:38:13 +02:00
Peter Rabbitson
25a77d905d Centralize everything on ipfs/go-log/v2 
I am not entirely sure this is right, but everything seems to build...
 2021-04-06 15:04:32 +02:00
Łukasz Magiera
81bd27911f Propagate StateMsg api changes 2021-04-05 19:56:53 +02:00
Łukasz Magiera
9a2ec07837 storagefsm: Apply global events even in broken states 2021-04-05 18:25:29 +02:00
Łukasz Magiera
f372964a60 fix lint 2021-04-01 16:40:11 +02:00
Łukasz Magiera
c7de7882a1 Merge branch 'master' of github.com:ppiaas/lotus-ppiaas into fix/stg-fetch-close 2021-04-01 16:38:17 +02:00
yaohcn
3895b2ccca fix reservations 2021-03-31 14:30:24 +08:00
Łukasz Magiera
a7122d424b
Merge pull request #5527 from filcloud/handle-ticket-expire 
fix precommit ticket expiration handling
 2021-03-29 22:55:43 +02:00
Łukasz Magiera
1aea1ebb9d
Merge pull request #5801 from filecoin-project/fix/fsm-input-stuck 
storagefsm: Trigger input processing when below limits
 2021-03-12 19:02:15 +01:00
Łukasz Magiera
0974e02089 storagefsm: common createSector method 2021-03-12 18:42:17 +01:00
Steven Allen
b74b1658b2 test: mock proof side effects on verify as well 2021-03-12 09:06:41 -08:00
Łukasz Magiera
d9923125ff mock sealer: Simulate randomness sideeffects 2021-03-12 17:33:43 +01:00
Łukasz Magiera
e5b5bf1c1b storagefsm: Update sector stats atomically with sector creation 2021-03-12 17:25:24 +01:00
Łukasz Magiera
679f5f7661 storagefsm: Helper stat functions in updateSector 2021-03-12 17:00:20 +01:00
Łukasz Magiera
957d3f0636 storgefsm: Use locks correctly 2021-03-12 15:01:00 +01:00
Łukasz Magiera
5c605d99e2 storgefsm: Fix tests 2021-03-12 14:56:46 +01:00
Łukasz Magiera
f5ed25371b storagefsm: Trigger input processing wheen below limits 2021-03-12 14:54:10 +01:00
Łukasz Magiera
d0243904f7
Merge pull request #5779 from filecoin-project/feat/storagefsm-no-restart-log 
storagefsm: Don't log the SectorRestart event
 2021-03-11 15:18:32 +01:00
Łukasz Magiera
881c722025 storagefsm: Fix double unlock with ready WaitDeals sectors 2021-03-11 15:03:51 +01:00
Łukasz Magiera
a1b6a2f011 storagefsm: Don't log the SectorRestart event 2021-03-11 12:52:34 +01:00
Łukasz Magiera
cc490b983a
Merge pull request #5761 from filecoin-project/fix/testunpadreader-go116 
Fix TestUnpadReader on Go 1.16
 2021-03-10 13:13:27 +01:00
Łukasz Magiera
7fbb4bdd3b Fix TestUnpadReader on Go 1.16 2021-03-10 13:05:25 +01:00
Łukasz Magiera
40fdf6c180
Merge pull request #5624 from filecoin-project/feat/miner-storage-limit 
Configurable storage path storage limit
 2021-03-10 11:15:28 +01:00
Łukasz Magiera
df0a8d9507
Merge pull request #5759 from filecoin-project/asr/terminate-livecheck 
Check liveness of sectors when processing termination batches
 2021-03-10 11:14:11 +01:00
Aayush Rajasekaran
47064f987c Check liveness of sectors when processing termination batches 2021-03-09 23:36:30 -05:00
Łukasz Magiera
fe230f901e Collect worker task metrics 2021-03-05 12:48:17 +01:00
Łukasz Magiera
29c9fa3137
Merge pull request #5612 from filecoin-project/feat/cc-fsm-cleanup 
storagefsm: Cleanup CC sector creation
 2021-02-26 11:39:28 +01:00
Łukasz Magiera
e49a412f6d
Merge pull request #5375 from filecoin-project/feat/refactor-fsm-input 
storagefsm: Rewrite input handling
 2021-02-25 14:28:04 +01:00
Łukasz Magiera
2b9ed9fa3d Remove sectors with all deals expired in RecoverDealIDs 2021-02-22 21:03:51 +01:00
Łukasz Magiera
35759fa07e
Merge pull request #5635 from filecoin-project/deps/cbg-soft-map 
Update markets, cbor-gen with soft map decoding
 2021-02-22 18:27:38 +01:00
Aayush Rajasekaran
6732758d0f Tweak mock ProveCommits to respect proof sizes 2021-02-19 17:22:59 -05:00
Łukasz Magiera
26399dba70 Update markets, cbor-gen with soft map decoding 2021-02-19 20:11:43 +01:00
Łukasz Magiera
37b1fb97fb Configurable storage path storage limit 2021-02-18 16:44:34 +01:00
yaohcn
3f66e731fa fix drop sector 2021-02-18 12:00:28 +08:00
Łukasz Magiera
90b8612cf4 Merge remote-tracking branch 'origin/master' into feat/refactor-fsm-input 2021-02-16 22:39:06 +01:00
Łukasz Magiera
9ebeafb174 Update ffi 2021-02-16 22:04:08 +01:00
Łukasz Magiera
f719765069 storageminer: exit PledgeSector after sectors enter sealing pipeline 2021-02-16 17:41:58 +01:00
Łukasz Magiera
fc5e243c92 storagefsm: Cleanup CC sector creation 2021-02-16 17:14:59 +01:00
Łukasz Magiera
dd82729f60 Merge remote-tracking branch 'origin/master' into feat/refactor-fsm-input 2021-02-16 13:45:39 +01:00
Aayush Rajasekaran
bb7fabb439 Update FFI 2021-02-15 17:22:25 -05:00
Łukasz Magiera
6907e5879d Fix WaitDeals sector accounting 2021-02-11 13:52:00 +01:00
Łukasz Magiera
e27a530cbc storagefsm: cleanup openSectors better; pendingPieces by pieceCid 2021-02-09 18:44:41 +01:00
He Weidong
0e2e1125d3 fix precommit ticket expiration handling 2021-02-05 10:45:15 +08:00
Łukasz Magiera
6e7fcb7ee6 Merge remote-tracking branch 'origin/master' into feat/refactor-fsm-input 2021-02-03 14:01:56 +01:00
Łukasz Magiera
1e7a107484
Merge pull request #5433 from filecoin-project/feat/keep-unsealed-cfg 
Add miner config to always keep ensealed deal copies
 2021-02-03 12:40:02 +01:00
Łukasz Magiera
94009f247f
Merge pull request #5411 from filecoin-project/feat/sealing-handle-batch-publish 
Handle batch publish storage deals message in sealing recovery
 2021-01-28 19:33:49 +01:00
Łukasz Magiera
aebf3cd64c Update ffi with more portable blst 2021-01-27 22:36:51 +01:00
Łukasz Magiera
d4c7b63aac Add miner config to always keep ensealed deal copies 2021-01-26 17:50:31 +01:00
Łukasz Magiera
fabcbb621d Merge remote-tracking branch 'origin/master' into next 2021-01-26 00:47:22 +01:00
Łukasz Magiera
67780346e2 storagemgr: Always unseal full sectors 2021-01-25 21:39:15 +01:00
Łukasz Magiera
e4f4fa2c7d Update FFI more, fix zero aggregate sigs 2021-01-25 20:05:03 +01:00
Dirk McCormick
1b494acc9e feat: handle batch publish storage deals message in sealing recovery 2021-01-25 14:10:05 +01:00
Łukasz Magiera
9b62e224d5 Use filecoin-ffi master 2021-01-25 13:57:15 +01:00
Steven Allen
03cd3760bb correctly pick the seal type based on the network version 
Of course, we should really just run all of our tests post actors v2.
 2021-01-21 15:21:20 -08:00
Aayush Rajasekaran
ab90a3b2bc Actors update: MinerInfo.SealProofType has been removed 2021-01-21 15:21:19 -08:00
Łukasz Magiera
1070ad2289 storagefsm: Drop unused TargetWaitDealsSectors 2021-01-21 22:20:16 +01:00
Łukasz Magiera
ec4deb7e28 storagefsm: Fix unlocking in handleWaitDeals 2021-01-21 19:59:18 +01:00
Łukasz Magiera
1336d8855d storagefsm: Drop addpiece wait after fixing storageadapter 2021-01-21 17:40:50 +01:00
Łukasz Magiera
df14f156e1 storagefsm: More logging for deal test debugging 2021-01-21 17:40:50 +01:00
Łukasz Magiera
b9a9f23204 storagefsm: Add stub AddPieceFailed state 2021-01-21 17:40:50 +01:00
Łukasz Magiera
f96f12c836 storagefsm: Add rest of checks in WaitDeals 2021-01-21 17:40:50 +01:00
Łukasz Magiera
069766ecc4 storagefsm: Don't persist piece assignment queue 2021-01-21 17:40:50 +01:00
Łukasz Magiera
270f2935a9 storagefsm: Check per-sector deal limits 2021-01-21 17:40:50 +01:00
Łukasz Magiera
fd67a41c75 storagefsm: Change sector CreationTime to unix ts 2021-01-21 17:40:50 +01:00
Łukasz Magiera
9857ad8378 storagefsm: Fix some deadlock cases 2021-01-21 17:40:50 +01:00
Łukasz Magiera
e5814dac4f cbor-gen 2021-01-21 17:40:50 +01:00