Łukasz Magiera
d019853687
review: Cleanup some comments
...
Co-authored-by: Peter Rabbitson <ribasushi@protocol.ai>
2021-11-29 21:29:00 +01:00
Łukasz Magiera
5b5e6b9e44
retrieval: DagSpec.MatchPath -> ExportMerkleProof
2021-11-29 21:14:00 +01:00
Łukasz Magiera
61791b90ea
retrieval: Only output matching nodes, MatchPath dagspec
2021-11-29 20:40:55 +01:00
zenground0
7d2b3f05db
WIP sector storage and integration test
2021-11-29 10:24:00 -05:00
Peter Rabbitson
d5b4d93575
Cleanup partial retrieval codepaths ( zero functional changes )
...
- Adjust the ExportRef comment to reflect change from #7673
- Simplify control flow in parseDagSpec() ( read diff under -w )
2021-11-25 10:35:05 +01:00
Łukasz Magiera
407c2ed114
retrieval: Drop the RootSelector hack
2021-11-23 17:42:43 +01:00
Łukasz Magiera
bdac11ade7
retrieval: Update some API comments
2021-11-22 14:19:03 +01:00
Łukasz Magiera
763659b8a3
retrieval: Update lotus-soup
2021-11-22 12:46:29 +01:00
Łukasz Magiera
ec2bfb99bb
make gen
2021-11-22 12:46:25 +01:00
Łukasz Magiera
b83a9b902a
gofmt
2021-11-22 12:46:17 +01:00
Łukasz Magiera
b26906963b
retrieval: Support multi-root export
2021-11-22 12:46:17 +01:00
Łukasz Magiera
b0c043cc2f
make gen
2021-11-22 12:46:15 +01:00
Łukasz Magiera
b868769ec8
more retrieval api work
2021-11-22 12:46:02 +01:00
Łukasz Magiera
89138bab4d
Simplify retrieval APIs
2021-11-22 12:45:42 +01:00
Travis Person
17cb5117f4
add additional methods to lotus gateway
2021-11-22 12:19:27 +01:00
Aayush Rajasekaran
32fc03886d
CLI: Add a lotus multisig cancel command
2021-11-18 19:21:17 -05:00
Anton Evangelatov
5454aebf13
CLI tools for the DAGStore inverted index ( #7361 )
...
* cli commands for dagstore
* address comments from Dirk
* rename inverted index to piece index
2021-11-17 15:39:45 +02:00
Dirk McCormick
501308239f
feat: CLI command to announce deal to indexers
2021-11-17 12:16:22 +01:00
Jennifer Wang
724dfc6a4d
tidy
2021-10-26 20:10:18 -04:00
Anton Evangelatov
6e5ccc87cf
cli: add retry for deals stuck in Publish with no funds
2021-10-11 14:03:55 +02:00
Peter Rabbitson
c4a7de9d37
Remove dead example code + dep
2021-10-07 09:44:50 +02:00
Peter Rabbitson
2a1094fc17
Merge banch feat/go-fil-markets-v1.13.0 into feat/datamodel-selector-retrieval
2021-10-04 21:56:11 +02:00
Aayush Rajasekaran
1d6db306ec
Randomness: Move getters from ChainAPI to StateAPI
2021-09-15 00:03:13 +02:00
Peter Rabbitson
0444435589
Expose basic text-based datamodel selector on retrieval
...
Syntaxt of selection is located at
https://pkg.go.dev/github.com/ipld/go-ipld-selector-text-lite#SelectorSpecFromPath
Example use, assuming that:
- The root of the deal is a plain dag-pb unixfs directory
- The directory is not sharded
- The user wants to retrieve the first entry in that directory
lotus client retrieve --miner f0XXXXX --datamodel-path-selector 'Links/0/Hash' bafyROOTCID ~/output
For a much more elaborate example see the top of ./itests/deals_partial_retrieval_test.go
2021-09-10 09:44:11 +02:00
Łukasz Magiera
e4044151f0
Show deal sizes is sealing sectors
2021-09-07 18:49:53 +02:00
hannahhoward
91804d5746
feat(deps): update go-graphsync v0.9.0
2021-09-03 16:22:51 +02:00
Steven Allen
1cf556c3a2
feat: expose ChainGetPath on the gateway
2021-08-30 16:43:21 -07:00
Łukasz Magiera
e2ba650a8c
cli for checking alerts
2021-08-26 15:45:17 +02:00
Łukasz Magiera
5366821144
api for getting alert states
2021-08-26 15:44:45 +02:00
Łukasz Magiera
12875a9664
api/command for encoding actor params
2021-08-20 17:32:57 +02:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ( #6671 )
...
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.
Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.
When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.
Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.
Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.
Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.
Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.
A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.
Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.
NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:
- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport
At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
ZenGround0
b914e95f34
revert pricelist by version to pricelist by epoch
2021-08-11 09:49:23 -04:00
Steven Allen
57bf5c2143
feat: add ChainGetTipSetAfterHeight
...
This is identical to ChainGetTipSetByHeight, but returns the tipset
following any null tipsets. This is what the user usually wants anyways.
(and I need it for another PR)
2021-08-05 12:07:14 -07:00
Raúl Kripalani
07f40b9488
fix docs and nits.
2021-07-28 20:03:25 +01:00
Anton Evangelatov
de4a847078
add RuntimeSubsystems API method; use it in lotus-miner info
2021-07-28 16:02:05 +03:00
Mike Greenberg
76a9f4241b
feat: Graceful error when api impl is nil
2021-07-26 13:51:37 +02:00
vyzo
1f6935f8c4
make gen
2021-07-26 12:15:01 +03:00
vyzo
3c409d5023
require admin for running checks on the splitstore.
2021-07-26 12:13:41 +03:00
vyzo
77604db716
make gen
2021-07-26 08:33:25 +03:00
vyzo
a0d6fdba33
add ChainBlockstoreInfo APIv1 endpoint
2021-07-26 08:30:07 +03:00
vyzo
21bb2bda09
make gen
2021-07-25 11:25:29 +03:00
vyzo
5b2e4d8ad4
add permission tag to ChainCheckBlockstore API
2021-07-25 11:24:06 +03:00
vyzo
3d2ae433ee
add ChainCheckBlockstore API
2021-07-25 11:14:48 +03:00
ZenGround0
fe5e1b23d5
Update api version to account for new network version 6.5 + new underlying repr
2021-07-22 09:49:47 -04:00
Łukasz Magiera
660829703a
Merge remote-tracking branch 'origin/master' into feat/split-net-api
2021-07-22 15:38:06 +02:00
whyrusleeping
d89e2f065c
add StateReadState to gateway api
2021-07-21 17:21:28 -07:00
Łukasz Magiera
d5c9cccf21
address net interface split review
2021-07-15 15:51:47 +02:00
Łukasz Magiera
b20581bac2
api: check that only the top-level internal struct gets picked up
2021-07-15 15:45:41 +02:00
Łukasz Magiera
49e26cce7d
api: Separate the Net interface from Common
2021-07-15 11:41:30 +02:00
hunjixin
bdaa73a413
add ChainGetMessagesInTipset api
2021-07-15 14:52:09 +08:00
Anton Evangelatov
d89ddb9315
resolve conflicts
2021-07-12 11:34:37 +02:00
Łukasz Magiera
ee97aa0a8a
dealpublisher: Use address selector
2021-07-07 19:12:03 +02:00
Łukasz Magiera
8f5c0c60f5
Config for deal publishing control addresses
2021-07-07 18:00:54 +02:00
Anton Evangelatov
4f0a96c9c7
resolved conflicts
2021-06-30 13:16:52 +02:00
Łukasz Magiera
c1303f1eac
gateway: Add support for Version method
2021-06-28 19:05:27 +02:00
Łukasz Magiera
69e0dff56c
api: Fix Wrap for nested proxy structs
2021-06-28 19:00:37 +02:00
Anton Evangelatov
d45bb14015
Merge branch 'master' into nonsense/split-market-miner-processes
2021-06-16 18:49:15 +02:00
Aayush Rajasekaran
bee548face
Add utils to use multisigs as miner owners
2021-06-15 16:57:52 -04:00
Anton Evangelatov
090a16316c
bump miner version apiv0
2021-06-15 13:59:51 +02:00
Anton Evangelatov
31e6fb154e
resolve conflicts
2021-06-14 11:27:54 +02:00
Anton Evangelatov
9f3ec82cc1
initial resolution of conflicts
2021-06-08 13:01:22 +02:00
Raúl Kripalani
a274fcc533
Merge branch 'master' into raulk/itests
2021-06-07 23:44:22 +01:00
Raúl Kripalani
db1a61852d
Merge branch 'master' into raulk/itests
2021-06-07 23:17:44 +01:00
Łukasz Magiera
8625da3479
Bump miner/worker api versions
2021-06-07 16:57:24 +02:00
aarshkshah1992
ddd9bf610e
fix CI
2021-06-07 16:45:52 +05:30
aarshkshah1992
21e6b50294
finished rebasing PR
2021-06-07 16:02:15 +05:30
aarshkshah1992
ad4b182bfe
remove read task type and run gen and docsgen
2021-06-07 15:03:06 +05:30
Dirk McCormick
52b90e4c9d
test: offline deals
2021-06-04 22:42:07 +02:00
Łukasz Magiera
ba654285f0
Merge pull request #6397 from filecoin-project/fix/some-flaky-tests
...
Fix some flaky tests
2021-06-04 22:00:29 +02:00
Łukasz Magiera
a7746961fb
Merge remote-tracking branch 'origin/master' into feat/list-retrievals
2021-06-04 20:49:01 +02:00
Anton Evangelatov
8bd09e39ca
resolve merge conflicts
2021-06-04 16:17:00 +02:00
Łukasz Magiera
fd2d2d3ff4
Fix TestDeadlineToggling flakiness
2021-06-03 17:08:05 +02:00
Aayush Rajasekaran
cf18709100
Fix TestDeadlineToggling
2021-06-01 17:50:06 -04:00
Aayush Rajasekaran
f00cf70df0
Magik shouldn't write code at midnight
2021-06-01 17:31:44 -04:00
Łukasz Magiera
2f0a9f6c40
Improve AddVerifiedClient test
2021-06-01 21:28:48 +02:00
Łukasz Magiera
a676da8a7e
Merge remote-tracking branch 'origin/master' into feat/nv13-1.11
2021-06-01 21:07:04 +02:00
Łukasz Magiera
dc642d0b7b
Merge remote-tracking branch 'origin/feat/nv13' into feat/nv13-1.11
2021-06-01 21:06:58 +02:00
Łukasz Magiera
5c5b8866c7
gofmt
2021-06-01 19:14:08 +02:00
Łukasz Magiera
8072573292
Fix TestWindowPostDispute
2021-06-01 19:11:32 +02:00
Łukasz Magiera
5a7f97210f
Merge pull request #6317 from filecoin-project/jen/test-datacap-topup
...
Add test for AddVerifiedClient
2021-06-01 14:41:05 +02:00
Łukasz Magiera
482e1110c2
precommit batcher: Improve error propagation
2021-06-01 14:35:30 +02:00
Łukasz Magiera
9fcb564bef
Make commit batcher more robust
2021-06-01 11:56:19 +02:00
Jennifer Wang
fcfc214ed1
Use MockSbBuilder
2021-05-31 20:14:11 -04:00
Jennifer
ac2887c01b
Update api/test/verifreg.go
...
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2021-05-31 16:03:52 -04:00
Łukasz Magiera
ffa47659a1
Merge remote-tracking branch 'origin/feat/nv13' into feat/nv13-1.11
2021-05-31 21:38:34 +02:00
Łukasz Magiera
c3e8eddb9b
Merge remote-tracking branch 'origin/master' into feat/nv13-1.11
2021-05-31 21:24:56 +02:00
Łukasz Magiera
2d6a159dcc
Merge pull request #6041 from filecoin-project/fix/batch-deal-packing
...
storagefsm: Fix batch deal packing behavior
2021-05-31 20:45:12 +02:00
Aayush Rajasekaran
7fca1c1ee7
Implement FIP-0015
2021-05-31 14:40:54 -04:00
Łukasz Magiera
7d1ae7daf5
lotus-wallet: Support permissioned api
2021-05-31 13:56:35 +02:00
Anton Evangelatov
d195a12565
Merge branch 'master' into nonsense/split-market-miner-processes
2021-05-31 11:26:11 +02:00
Łukasz Magiera
670913261c
Self-review cleanup
2021-05-30 19:37:33 +02:00
Łukasz Magiera
9475079b97
Make batch deal input test less flaky
2021-05-30 17:46:22 +02:00
Łukasz Magiera
c8d603557b
storagefsm: Fix batch deal packing behavior
2021-05-30 17:46:22 +02:00
Jennifer Wang
2b08e9f3a6
Add test AddVerifiedClient
2021-05-28 15:41:57 -04:00
hannahhoward
3fbe2b320d
feat(v0api): add list-retrievals to v0
2021-05-27 15:00:31 -07:00
hannahhoward
19b6dc8d1e
feat(cli): add a list retrievals command
...
Currently, there is no way to inspect retrievals on a client. This adds said command, allow with
corresponding APIs
2021-05-27 11:48:25 -07:00
Łukasz Magiera
1e4456138e
Merge master into feat/nv13
2021-05-27 12:28:20 +02:00
Łukasz Magiera
4c6c9a0edb
Fix windowpost/deadline tests
2021-05-25 21:27:32 +02:00
Łukasz Magiera
f5409845b5
Some review addressing
2021-05-25 16:07:45 +02:00
Łukasz Magiera
13ff6ed462
Test pledging before nv13 upgrade
2021-05-20 18:38:33 +02:00
Łukasz Magiera
f3fda4ae7f
Better asserts in TestPledgeBatching
2021-05-20 18:25:47 +02:00
Łukasz Magiera
d71334cc24
Address self-review
2021-05-20 18:04:07 +02:00
Łukasz Magiera
678812f35b
gofmt, lint
2021-05-20 12:05:04 +02:00
Łukasz Magiera
44bf9bf903
tests: Better state logging in pledgeSectors
2021-05-20 10:39:00 +02:00
Łukasz Magiera
e088c71b9a
marketadapter: Handle batch sealing messages
2021-05-19 20:07:20 +02:00
Raúl Kripalani
c46d4ae529
wip
2021-05-19 17:30:43 +01:00
Łukasz Magiera
eafaf6d236
Don't block on batching in node tests
2021-05-19 14:33:15 +02:00
Anton Evangelatov
87ed228fd3
extract pushUrl and fix return type for worker RPC v0
2021-05-19 13:10:56 +02:00
Anton Evangelatov
a989f60e27
add SectorAddPieceToAny and SectorUnsealPiece to StorageMiner iface; model moved to api package - PieceDealInfo, DealSchedule
2021-05-19 13:05:07 +02:00
Łukasz Magiera
dd393b470f
Fix aggregation inputs
2021-05-18 20:36:47 +02:00
Łukasz Magiera
9690bc882c
Test to trigger batching logic
2021-05-18 18:41:42 +02:00
Łukasz Magiera
c7ba083fa4
Import precommit batcher
2021-05-18 16:51:06 +02:00
Łukasz Magiera
ba2032c642
Fix some aggregation bugs
2021-05-17 22:51:29 +02:00
Łukasz Magiera
506f39b294
WIP: Integrate FIP0013
2021-05-11 22:10:29 -04:00
Peter Rabbitson
d54ed1b0c6
Merge remote-tracking branch 'origin/master' into feat/stateless-offline-dealflow
2021-05-11 04:54:22 +02:00
Aayush Rajasekaran
bf10b051ff
Merge pull request #6108 from filecoin-project/feat/actors-codegen
...
wip actor wrapper codegen
2021-05-10 20:12:57 -04:00
Łukasz Magiera
a80259d986
Actor upgrade checklist
2021-05-10 19:46:44 -04:00
Aayush Rajasekaran
b5da2655dc
Introduce v5 actors
2021-05-10 19:44:28 -04:00
Łukasz Magiera
63189cd81d
Generate wrappers for new actor versions
2021-05-09 21:04:50 -04:00
Jakub Sztandera
18cbdcfc81
Disable checks API on Lotus Lite
...
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-05-07 17:19:17 +02:00
Jakub Sztandera
8d75da1629
Use MessagePrototype for check API
...
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-05-07 15:30:05 +02:00
vyzo
e2d0047a2a
introduce message prototypes
...
This introduces message prototypes to applicable API endpoints, which
allows us to invert control of message sending and give the user a
chance to intervene with an interactive ui.
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-05-07 15:30:05 +02:00
vyzo
d782250aba
implement MessagePool.CheckReplaceMessages
...
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-05-07 15:30:04 +02:00
vyzo
91e774063e
implement MessagePool.CheckMessages
...
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-05-07 15:30:04 +02:00
vyzo
ed61642b3a
implement NodeStatus API
...
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-05-07 15:30:04 +02:00
Peter Rabbitson
e648b72d8d
Merge remote-tracking branch 'origin/master' into feat/stateless-offline-dealflow
2021-05-06 15:57:10 +02:00
Łukasz Magiera
a24a73bb52
Merge tag 'v1.8.0' into release/v1.9.0
2021-04-27 10:30:33 +02:00
Łukasz Magiera
c4137a438d
Merge commit 'f1ded63d608e1b191d79c780b8065f8cff367c98' into feat/nv12-1.11
2021-04-27 08:22:55 +02:00
Aayush Rajasekaran
f1ded63d60
v4 specs-actors integration, nv12 migration
2021-04-27 02:02:44 -04:00
Łukasz Magiera
bfa332ca7d
api: Document API change process
2021-04-26 20:36:20 +02:00
Cory Schwartz
fbd003a22b
add walletbalance to v0api
2021-04-21 09:46:44 -07:00
Cory Schwartz
39e49f1700
implement WalletBalance on gateway
2021-04-20 21:55:47 -07:00
Łukasz Magiera
508e2d5c49
gateway: Fix api getter
2021-04-20 18:42:12 +02:00
Łukasz Magiera
c7a67cf0e1
Merge pull request #6059 from filecoin-project/frrist/expose-filReserveDisbursed
...
polish(api): expose filReserveDisbursed via CirculatingSupply API
2021-04-18 16:31:03 +02:00
frrist
c118415b12
polish(api): expose filReserveDisbursed via CirculatingSupply API
...
- motivated by: https://github.com/filecoin-project/sentinel-visor/issues/462
2021-04-16 15:15:38 -07:00
Łukasz Magiera
4436c184ed
Fix v0/v1 API versions
2021-04-16 00:20:13 +02:00
Łukasz Magiera
e8f28d7b9f
Fix tests
2021-04-06 12:24:58 +02:00
Łukasz Magiera
81bd27911f
Propagate StateMsg api changes
2021-04-05 19:56:53 +02:00
Peter Rabbitson
b79be2a2c2
Merge remote-tracking branch 'origin/master' into feat/stateless-offline-dealflow
2021-04-05 16:06:44 +02:00
Łukasz Magiera
d8bff4d19f
Make gateway work with v1 api
2021-04-05 13:47:10 +02:00
Łukasz Magiera
deb2b90b6a
Fix lotus/miner build
2021-04-05 13:23:46 +02:00
Peter Rabbitson
7fddbb528d
Introduce stateless offline dealflow, bypassing the FSM/deallists
...
This is aproposal for an additional flag --manual-stateless-deal and a
corresponding API endpoint ClientStatelessDeal. This allows firing off
an offline-style deal against a miner without keeping further track of
it locally.
Not keeping any local state introduces the limitation of requiring free
storage deals, as there is nothing to tie the payment channel setup to.
Rationale/need for this type of flow is the case of incredibly large
sets of data nd deals, where the client and providers have prearranged
payment ahead of time, and the client has a separate-from-lotus database
of deal inventory. This way the client can use their lotus node merely
as a network gateway, without running into any limitations currently
present in both lotus as a whole and go-fil-markets in particular.
Specific context for this work is filecoin-discover, where the requirement
is to onboard ~ 12,000,000 individual deals against a pool of miners
with whom the client has prearranged a relationship.
2021-04-05 13:11:10 +02:00
Łukasz Magiera
09c374cdde
api stub codegen
2021-04-03 13:12:59 +02:00
Łukasz Magiera
0103d2f621
v1 api: Cleanup message wait/search apis
2021-04-02 13:52:24 +02:00
Łukasz Magiera
c9866da688
Merge remote-tracking branch 'origin/master' into feat/v1-api
2021-04-02 13:15:33 +02:00
Łukasz Magiera
4538550999
Merge remote-tracking branch 'origin/master' into feat/local-retrieval
2021-04-02 12:36:28 +02:00
Anton Evangelatov
df003d44d2
include Stages and DealStages only on ClientDealInfo, not on ClientListDeals
2021-04-01 17:08:01 +03:00
Łukasz Magiera
1a5c4d70f4
Merge remote-tracking branch 'origin/master' into feat/v1-api
2021-04-01 15:33:37 +02:00
Łukasz Magiera
ba49c6206e
cli: Default to v0 api for now
2021-04-01 14:17:22 +02:00
Anton Evangelatov
967fbf52c6
resolve conflicts
2021-03-31 23:14:29 +03:00