Łukasz Magiera
8454abcf45
storage: Use 1M buffers for Tar transfers
2021-11-24 20:08:37 +01:00
Łukasz Magiera
2a1505b364
storage: Test StorageFindSector with groups
2021-11-23 16:11:04 +01:00
Łukasz Magiera
8b548ac02f
storage: Check allowlists in StorageFindSector
2021-11-23 16:11:04 +01:00
Łukasz Magiera
5c77c25747
storage: Add Group tags to StorageInfo
2021-11-23 16:11:04 +01:00
Łukasz Magiera
d1a63e4173
remote store: Remove debug printf
2021-11-22 17:50:12 +01:00
Aayush Rajasekaran
8665e32221
Update deps
2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
07f1be0ff6
Plug in the FFI call
2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
1449644c34
Integrate v7 actors
2021-11-17 15:55:14 -05:00
Łukasz Magiera
e508055dc1
make gen
2021-10-19 11:13:23 +02:00
Łukasz Magiera
080aa3356a
Fix locks in worker-tracked
2021-10-18 20:19:21 +02:00
Łukasz Magiera
70589e4406
Block work in tracked worker before it is started
2021-10-18 18:44:56 +02:00
Łukasz Magiera
261238e157
Show prepared tasks in sealing jobs
2021-10-18 18:44:56 +02:00
Łukasz Magiera
11d738eee0
Track prepared work
2021-10-18 18:44:56 +02:00
Marten Seemann
03806f7063
add missing build constraint to statfs_unix.go
2021-10-18 16:29:01 +02:00
Łukasz Magiera
f352c18290
Don't remove sector data when moving data into a shared path
2021-10-11 21:11:38 +02:00
Łukasz Magiera
9af82f2d68
sched: Fix taskDone chan deadlock
2021-10-03 17:09:43 +02:00
Łukasz Magiera
b87142ec8e
wip improve scheduling of ready work
2021-10-03 10:38:08 +02:00
Łukasz Magiera
d7fbd8b67d
Update proofs to v10.0.0
2021-10-01 18:38:27 +02:00
Łukasz Magiera
a8a9818043
Expose storage states on the metrics endpoint
2021-10-01 14:45:01 +02:00
Łukasz Magiera
ef03314c6d
storagemgr: Cleanup workerLk around worker resources
2021-09-15 16:35:19 +02:00
Łukasz Magiera
3118bd1039
stores: Fix reserved disk usage log spam
2021-08-31 13:36:09 +02:00
Steven Allen
1cf556c3a2
feat: expose ChainGetPath on the gateway
2021-08-30 16:43:21 -07:00
Łukasz Magiera
2293ecd8e8
Reduce lotus-miner startup spam
2021-08-27 19:41:54 +02:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ( #6671 )
...
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.
Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.
When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.
Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.
Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.
Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.
Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.
A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.
Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.
NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:
- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport
At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Steven Allen
18f39be3ba
fix: don't check for t_aux when proving
...
We don't need it.
2021-08-09 11:07:35 -07:00
Anton Evangelatov
16784aa2cc
remove pieceProvider from DI; small refactors
2021-07-12 11:30:26 +02:00
Anton Evangelatov
566a9d5541
fix signature
2021-07-06 18:12:30 +02:00
Anton Evangelatov
7a78527fde
remove not used sa; cleanup
2021-07-06 17:46:21 +02:00
Anton Evangelatov
604be5fc82
Update extern/sector-storage/stores/remote.go
...
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2021-07-06 17:24:56 +02:00
Anton Evangelatov
4be0a7a215
resolve conflicts with master
2021-07-05 13:13:32 +02:00
Łukasz Magiera
8a94ab676e
storage: Fix FinalizeSector with sectors in stoage paths
2021-07-02 19:54:45 +02:00
Anton Evangelatov
21b51328f9
adding TestDealWithMarketAndMinerNode
2021-06-23 12:44:52 +02:00
Anton Evangelatov
6720463799
resolve merge conflicts
2021-06-22 11:28:23 +02:00
Raúl Kripalani
684cce198f
add a unit test.
2021-06-21 20:49:24 +01:00
Raúl Kripalani
59eab2df25
move scheduling filtering logic down.
2021-06-21 20:49:16 +01:00
Raúl Kripalani
83839362c5
fix boolean condition.
2021-06-21 20:35:51 +01:00
Raúl Kripalani
b6147fb27f
extern/storage: retype resource filtering strategy to enum.
2021-06-21 20:28:15 +01:00
Raúl Kripalani
f3b6f8de1a
add ability to ignore worker resources when scheduling.
2021-06-21 20:08:18 +01:00
Łukasz Magiera
b37a66c7c9
Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10
2021-06-21 17:03:03 +02:00
Łukasz Magiera
5516f3492e
ffiwrapper: Assert aggregate length
2021-06-18 12:02:42 +02:00
Łukasz Magiera
44de67cf7d
Merge pull request #6175 from filecoin-project/feat/dynamic-retreival-pricing
...
Dynamic Retrieval pricing
2021-06-17 10:25:48 +02:00
aarshkshah1992
9002fcbe47
local unsealed file wont have the unsealed piece
2021-06-17 09:46:29 +05:30
Anton Evangelatov
d45bb14015
Merge branch 'master' into nonsense/split-market-miner-processes
2021-06-16 18:49:15 +02:00
Aayush Rajasekaran
520a0091f5
Merge branch 'releases' into release/v1.10.0
2021-06-15 17:40:36 -04:00
Łukasz Magiera
c7c593c74e
TestMulticoreSDR: Setup rust logger after envvar check
2021-06-15 17:59:46 +02:00
Łukasz Magiera
6b0aed9317
Setup rust logger in the test
2021-06-15 17:53:26 +02:00
Łukasz Magiera
265afd696e
Run TestMulticoreSDR on Circle
2021-06-15 17:46:06 +02:00
Łukasz Magiera
9ae780902a
Test multicore SDR support
2021-06-15 17:36:10 +02:00
Anton Evangelatov
beb5bffaf6
fix log.Error; rename RemoteStore
2021-06-09 13:05:54 +02:00
Anton Evangelatov
30ba9a2751
remove double http handler decl
2021-06-08 15:36:54 +02:00
Anton Evangelatov
4dd093b160
fix piece provider test
2021-06-08 14:57:45 +02:00
Anton Evangelatov
9f3ec82cc1
initial resolution of conflicts
2021-06-08 13:01:22 +02:00
aarshkshah1992
4e9bb16532
changes as per review
2021-06-08 10:09:27 +05:30
aarshkshah1992
212d0fc264
fix remote store diff
2021-06-08 09:54:45 +05:30
aarshkshah1992
e4e60d7af4
fix mock
2021-06-08 09:37:12 +05:30
aarshkshah1992
16166504f4
merged master
2021-06-08 09:17:40 +05:30
aarshkshah1992
ddd9bf610e
fix CI
2021-06-07 16:45:52 +05:30
Anton Evangelatov
b0cb0c1a4a
do not depend on filecoin-ffi in api package
2021-06-07 11:51:25 +02:00
aarshkshah1992
8d9cef17af
changes as per review
2021-06-07 15:03:09 +05:30
aarshkshah1992
2a134887c3
logs to debug read & unseal
2021-06-07 15:03:09 +05:30
aarshkshah1992
ec6a49693f
logs to debug read and unseal
2021-06-07 15:03:09 +05:30
aarshkshah1992
207f0d901a
integration test should remove unsealed files
2021-06-07 15:03:09 +05:30
aarshkshah1992
73f7825fbb
clean up logging
2021-06-07 15:03:09 +05:30
aarshkshah1992
3b792a32c3
better logging
2021-06-07 15:03:09 +05:30
aarshkshah1992
22f36483cb
more logging
2021-06-07 15:03:09 +05:30
aarshkshah1992
acfa3d7370
finish integration tests
2021-06-07 15:03:09 +05:30
aarshkshah1992
ad4b182bfe
remove read task type and run gen and docsgen
2021-06-07 15:03:06 +05:30
aarshkshah1992
35a0dbfa8c
fix go mod
2021-06-07 15:02:52 +05:30
aarshkshah1992
4efaa8d688
unit tests for the remote store Reader
2021-06-07 15:02:52 +05:30
aarshkshah1992
0d88800eb9
use mockgen
2021-06-07 15:02:52 +05:30
aarshkshah1992
77b5e8d045
use an actual worker in the integration tests
2021-06-07 15:02:52 +05:30
Dirk McCormick
6879ae9e6a
feat: TestPieceProviderReadPiece
2021-06-07 15:02:52 +05:30
aarshkshah1992
74372d3e81
fix linting problems
2021-06-07 15:02:52 +05:30
aarshkshah1992
759d8f090b
test http handler
2021-06-07 15:02:52 +05:30
aarshkshah1992
db5c88196d
address review comments
2021-06-07 15:02:52 +05:30
Aarsh Shah
c853350bdf
Apply suggestions from code review
...
Co-authored-by: dirkmc <dirkmdev@gmail.com>
2021-06-07 15:02:52 +05:30
aarshkshah1992
65eb610ec3
docs, logs and green ci
2021-06-07 15:02:49 +05:30
aarshkshah1992
670835fca0
bypass task scheduler for reading unsealed pieces
2021-06-07 15:02:04 +05:30
Anton Evangelatov
8bd09e39ca
resolve merge conflicts
2021-06-04 16:17:00 +02:00
Anton Evangelatov
88756f3ebf
fix TestAPIDeal tests
2021-06-04 15:41:38 +02:00
aarshkshah1992
05ba3de5cc
changes as per review
2021-06-04 10:27:43 +05:30
Łukasz Magiera
565bb4f589
mock: Log debug info on bad aggregates
2021-06-03 17:35:10 +02:00
aarshkshah1992
43453bb633
fix broken test
2021-06-03 10:15:23 +05:30
Łukasz Magiera
40cc29d723
Skip FD check in TestDownloadParams
2021-06-02 19:50:17 +02:00
Anton Evangelatov
d195a12565
Merge branch 'master' into nonsense/split-market-miner-processes
2021-05-31 11:26:11 +02:00
Łukasz Magiera
1e4456138e
Merge master into feat/nv13
2021-05-27 12:28:20 +02:00
Łukasz Magiera
fc76a09b19
mock: Use real aggregate lengths
2021-05-26 10:06:41 +02:00
Łukasz Magiera
04658e1cae
fix lint
2021-05-25 16:26:59 +02:00
Łukasz Magiera
f5409845b5
Some review addressing
2021-05-25 16:07:45 +02:00
aarshkshah1992
0f3ff9e06a
fix typo
2021-05-24 15:22:11 +05:30
aarshkshah1992
dc6dbc9a11
dpr changes and test based on new unsealing PR
2021-05-22 22:40:21 +05:30
aarshkshah1992
50e023edd3
changes as per review
2021-05-21 19:15:05 +05:30
aarshkshah1992
2c9f5922b5
logs to debug read & unseal
2021-05-21 19:02:37 +05:30
aarshkshah1992
8ff5bce7a3
logs to debug read and unseal
2021-05-21 19:01:17 +05:30
aarshkshah1992
fb29f782df
integration test should remove unsealed files
2021-05-21 16:15:08 +05:30
aarshkshah1992
536d7c4275
clean up logging
2021-05-21 15:50:25 +05:30
aarshkshah1992
40642b2cad
better logging
2021-05-21 15:16:20 +05:30
aarshkshah1992
85f2ac8789
more logging
2021-05-21 14:56:37 +05:30
aarshkshah1992
78a0458ada
finish integration tests
2021-05-21 11:00:17 +05:30
aarshkshah1992
c17300dc1f
remove read task type and run gen and docsgen
2021-05-20 15:25:46 -06:00
aarshkshah1992
50d7acfa0c
fix go mod
2021-05-20 15:25:46 -06:00
aarshkshah1992
bd9959070e
unit tests for the remote store Reader
2021-05-20 15:25:46 -06:00
aarshkshah1992
9b34494501
use mockgen
2021-05-20 15:25:46 -06:00
aarshkshah1992
31a5f68dfa
use an actual worker in the integration tests
2021-05-20 15:25:46 -06:00
Dirk McCormick
ec85a973b1
feat: TestPieceProviderReadPiece
2021-05-20 15:25:46 -06:00
aarshkshah1992
8c4c26ca4c
fix linting problems
2021-05-20 15:25:46 -06:00
aarshkshah1992
dd9c9fd4fd
test http handler
2021-05-20 15:25:46 -06:00
aarshkshah1992
c58048d16a
address review comments
2021-05-20 15:25:46 -06:00
Aarsh Shah
d33d426692
Apply suggestions from code review
...
Co-authored-by: dirkmc <dirkmdev@gmail.com>
2021-05-20 15:25:46 -06:00
aarshkshah1992
73613ee883
docs, logs and green ci
2021-05-20 15:25:46 -06:00
aarshkshah1992
2a40c802ea
bypass task scheduler for reading unsealed pieces
2021-05-20 15:25:46 -06:00
Anton Evangelatov
9a7b0b657e
comment
2021-05-20 13:18:56 +02:00
Anton Evangelatov
19bd5beb96
update remoteGetSector
2021-05-20 13:15:57 +02:00
Anton Evangelatov
1a9b5760a7
add remoteGetAllocated http handler
2021-05-20 13:14:25 +02:00
Anton Evangelatov
76bb424de0
update StorageMinerAPI
2021-05-20 13:01:14 +02:00
Anton Evangelatov
cb603c62d9
update retrievaladapter ; add piece_provider
2021-05-20 12:49:53 +02:00
Anton Evangelatov
c12d802811
update reader to use ReadAtLeast
2021-05-20 12:38:38 +02:00
Anton Evangelatov
2aad7b6979
update sectorstorage.New (Manager) interface
2021-05-20 12:32:29 +02:00
Łukasz Magiera
e088c71b9a
marketadapter: Handle batch sealing messages
2021-05-19 20:07:20 +02:00
Łukasz Magiera
a5677d1b7a
ffiwrapper: Separate Prover interface
2021-05-19 15:20:23 +02:00
Anton Evangelatov
55401116bb
add UnpaddedByteIndex.Valid()
2021-05-19 13:16:29 +02:00
Anton Evangelatov
e275b54f55
export PartialFile and OpenPartialFile
2021-05-19 13:14:16 +02:00
Anton Evangelatov
a989f60e27
add SectorAddPieceToAny and SectorUnsealPiece to StorageMiner iface; model moved to api package - PieceDealInfo, DealSchedule
2021-05-19 13:05:07 +02:00
Łukasz Magiera
dd393b470f
Fix aggregation inputs
2021-05-18 20:36:47 +02:00
Łukasz Magiera
e400bdf87a
Order proofs before aggregation
2021-05-18 18:58:41 +02:00
Łukasz Magiera
9690bc882c
Test to trigger batching logic
2021-05-18 18:41:42 +02:00
Łukasz Magiera
fe9311e435
update ffi
2021-05-18 13:24:10 +02:00
Łukasz Magiera
6278bdc69a
Make things build
2021-05-17 20:47:41 +02:00
Łukasz Magiera
506f39b294
WIP: Integrate FIP0013
2021-05-11 22:10:29 -04:00
Anton Evangelatov
eb13c74dce
panic on unknown pathType
2021-05-11 18:14:01 +02:00
Anton Evangelatov
e07438417c
consider storiface.PathStorage when calculating storage requirements
2021-05-11 13:19:26 +02:00
whyrusleeping
ac77c51d5e
address nit
2021-04-29 10:24:16 -07:00
whyrusleeping
48feb52cbf
add a test for adding padded pieces
2021-04-27 16:28:25 -07:00
whyrusleeping
6daaf6ac2e
attempt to do better padding on pieces being written into sectors
2021-04-27 11:39:06 -07:00
Łukasz Magiera
a24a73bb52
Merge tag 'v1.8.0' into release/v1.9.0
2021-04-27 10:30:33 +02:00
Łukasz Magiera
c4137a438d
Merge commit 'f1ded63d608e1b191d79c780b8065f8cff367c98' into feat/nv12-1.11
2021-04-27 08:22:55 +02:00
Aayush Rajasekaran
f1ded63d60
v4 specs-actors integration, nv12 migration
2021-04-27 02:02:44 -04:00
Dirk McCormick
61344644a4
feat: add more debug logging for unsealing
2021-04-22 11:33:28 +02:00
yaohcn
844d1d364d
fix health report
2021-04-11 14:47:04 +08:00
Łukasz Magiera
9d4935a736
Merge pull request #5976 from filecoin-project/fix/storage-demote-stat-err
...
localstorage: Demote reservation stat error to debug
2021-04-09 21:08:08 +02:00
Łukasz Magiera
bed5ec81bd
localstorage: Demote reservation stat error to debug
2021-04-06 16:06:47 +02:00
Peter Rabbitson
25a77d905d
Centralize everything on ipfs/go-log/v2
...
I am not entirely sure this is right, but everything seems to build...
2021-04-06 15:04:32 +02:00
Łukasz Magiera
f372964a60
fix lint
2021-04-01 16:40:11 +02:00
Łukasz Magiera
c7de7882a1
Merge branch 'master' of github.com:ppiaas/lotus-ppiaas into fix/stg-fetch-close
2021-04-01 16:38:17 +02:00
yaohcn
3895b2ccca
fix reservations
2021-03-31 14:30:24 +08:00
Steven Allen
b74b1658b2
test: mock proof side effects on verify as well
2021-03-12 09:06:41 -08:00
Łukasz Magiera
d9923125ff
mock sealer: Simulate randomness sideeffects
2021-03-12 17:33:43 +01:00
Łukasz Magiera
cc490b983a
Merge pull request #5761 from filecoin-project/fix/testunpadreader-go116
...
Fix TestUnpadReader on Go 1.16
2021-03-10 13:13:27 +01:00
Łukasz Magiera
7fbb4bdd3b
Fix TestUnpadReader on Go 1.16
2021-03-10 13:05:25 +01:00
Łukasz Magiera
40fdf6c180
Merge pull request #5624 from filecoin-project/feat/miner-storage-limit
...
Configurable storage path storage limit
2021-03-10 11:15:28 +01:00
