Aarsh Shah
e3f5b7c59f
update dagstore top level index db
2022-02-18 14:00:01 +04:00
Masih H. Derkani
031bfaf120
Update to the head of go-fil-markets#673
...
Update to the head of the PR that introduces indexing integration in
`go-fil-markets` so that failure to connect to full node is logged only
instead of crashing markets process.
Relates to:
- https://github.com/filecoin-project/go-fil-markets/pull/673
- https://github.com/filecoin-project/index-provider/issues/177
2022-02-15 12:37:13 +00:00
vyzo
b260c849f7
deps: update go-libp2p-resource-manager to v0.1.4
2022-02-14 19:43:07 +02:00
Łukasz Magiera
a1098f6dc2
Cleanup go.mod
2022-02-10 17:37:09 -05:00
Łukasz Magiera
263de5b0c1
Use tagged specs-storage
2022-02-10 17:29:57 -05:00
Łukasz Magiera
212f5ddb4f
wip FinalizeReplicaUpdate
2022-02-10 17:24:26 -05:00
Will
296eab3045
Merge pull request #8045 from gammazero/feat/cid-to-piece-idx
...
Add indexer pubsub message authentication and rate limiting
2022-02-10 13:49:58 -08:00
Masih H. Derkani
f0accce286
Upgrade index-provider
and go-fil-markets to latest
...
See: https://github.com/filecoin-project/go-fil-markets/pull/680
2022-02-10 14:04:11 +00:00
gammazero
9481fa0a4b
Use new indexer pubsub message encoding
2022-02-09 16:40:27 -08:00
Łukasz Magiera
e7038d286d
Merge pull request #8018 from filecoin-project/feat/fru
...
FinalizeReplicaUpdate
2022-02-09 17:40:47 +00:00
Łukasz Magiera
67e3f182f5
Use tagged specs-storage
2022-02-09 11:28:48 +01:00
Aayush Rajasekaran
316836f43b
deps: update go-paramfetch to v0.0.4
2022-02-08 12:00:55 -05:00
Aayush Rajasekaran
9df19ae436
deps: update go-paramfetch to v0.0.4
2022-02-08 12:00:19 -05:00
Łukasz Magiera
142ba6660a
wip FinalizeReplicaUpdate
2022-02-08 17:22:41 +01:00
Masih H. Derkani
cc61650f86
Upgrade to latest index-provider
and set miner ID as extra gossip data
...
Upgrade to the latest `index-provider` which upgrades the go-legs
protocol to allow the inclusion of extra gossip data that may be used
for gossip validation purposes. In the case of lotus gossip message
validators the miner ID is used to verify the sender's peer ID on chain.
Relates to:
- https://github.com/filecoin-project/lotus/pull/8045
2022-02-08 14:32:03 +00:00
vyzo
80e9f76261
update libp2p to v0.18.0-rc4
2022-02-08 11:51:14 +02:00
Aarsh Shah
399a1afd94
Merge remote-tracking branch 'origin/master' into feat/cid-to-piece-idx
2022-02-08 12:37:57 +04:00
vyzo
e78c4ab9b3
update go-libp2p to v0.18.0-rc3
2022-02-03 19:32:30 +02:00
Anton Evangelatov
b6b5046cce
upgrade go-fil-markets
2022-02-03 16:26:04 +01:00
Anton Evangelatov
595ad44ee7
refactor indexprovider libp2p host connection to fullnode with meshcreator
2022-02-03 15:44:18 +01:00
Anton Evangelatov
c3d41d4e17
upgrade go-fil-markets
2022-02-02 16:14:44 +01:00
Anton Evangelatov
03dd33dbbe
upgrade go-fil-markets
2022-02-02 14:07:41 +01:00
Anton Evangelatov
2790cf35af
change fullnodeApi to listen addrs
2022-02-02 13:06:53 +01:00
Masih H. Derkani
ad348c8e2f
Upgrade to the latest index provider
...
Upgrade to the latest index provider which rolls out a more compact ad
signature mechanism along with dependency updates.
2022-02-01 09:53:57 +00:00
Masih H. Derkani
5d72ebe0cc
Upgrade index-provider
and update topic name
...
Update the topic name to be consistent with default
configured in storetheindex.
Upgrade to the latest `index-provider`.
2022-01-27 09:48:10 +00:00
Aarsh Shah
211ad166af
update markets deps
2022-01-26 15:53:27 +04:00
Aarsh Shah
4dd93e5bbb
update deps
2022-01-26 10:58:44 +04:00
Masih H. Derkani
5da2995bcd
Upgrade to index-provider
v0.2.3
...
Upgrade index provider to latest so that on engine restart the head
advertisement is published.
2022-01-25 09:37:44 +00:00
Aarsh Shah
8d062bc880
update index provider
2022-01-24 14:26:17 +04:00
Aarsh Shah
9e2d46b33f
Merge remote-tracking branch 'origin/master' into feat/cid-to-piece-idx
2022-01-23 10:32:54 +04:00
vyzo
6e5f521463
update go-libp2p to v0.18.0-rc2
2022-01-21 10:45:45 +02:00
Aarsh Shah
b8bf2af3e3
upgrade on master
2022-01-20 16:34:13 +04:00
Dirk McCormick
8a47a71772
feat: update to go-fil-markets v1.19.0
2022-01-20 11:46:17 +01:00
vyzo
2e5474e864
update go-fil-markets@v0.19.0-rc1
2022-01-20 11:37:08 +02:00
vyzo
6260271b78
go mod: update go-libp2p-resource-manager@v0.1.2
2022-01-20 11:36:11 +02:00
vyzo
83427c478f
go mod: update go-yamux@v3.0.2
2022-01-20 11:36:11 +02:00
vyzo
aa599bfaf9
go mod: update go-libp2p-pubsub
2022-01-20 11:36:11 +02:00
vyzo
62c0f35b5a
go mod: go-libp2p-resource-manager@v0.1.1
2022-01-20 11:36:11 +02:00
vyzo
c394fbdfc3
go mod: go-libp2p-resource-manager
2022-01-20 11:36:11 +02:00
vyzo
2fb5837a55
go mod: go-libp2p@v0.18.0-rc1
2022-01-20 11:36:11 +02:00
Łukasz Magiera
7dba12fc81
Merge pull request #7967 from filecoin-project/chore/mod-comments
...
chore: Cleanup go.mod
2022-01-18 15:18:19 +01:00
Dirk McCormick
1ce1dbbc50
feat: update go-fil-markets to tagged release
2022-01-18 14:29:59 +01:00
Łukasz Magiera
d677826312
Cleanup go.mod
2022-01-18 10:24:43 +01:00
hannahhoward
92d56d82f4
feat(deps): update markets stack
...
update go-fil-markets, go-data-transfer 1.13.0, go-graphsync 0.12.0
2022-01-14 17:21:04 -08:00
Aayush
2b29ceab83
Use go-libp2p-connmgr v0.3.1
2022-01-14 17:38:56 -05:00
Aayush Rajasekaran
b9474cf0a2
Implement an autobatcher
2022-01-14 17:15:20 -05:00
Aayush Rajasekaran
074a4cc3e4
Update to actors v7.0.0-rc1
2022-01-14 17:15:20 -05:00
Aayush Rajasekaran
790ba155b8
Fast migration for v15
2022-01-14 17:15:15 -05:00
zenground0
d6aa17e21f
Snap Deals Integration
...
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
ceb11bfe77
Update go-paramfetch
2022-01-14 17:14:32 -05:00
zenground0
d1480c36c0
RemoveData and Decode
...
- Unsealing replica update with sector key works and tested
- Sector key generation added and tested
2022-01-14 17:14:32 -05:00
zenground0
93656e65f8
WIP sector storage and integration test
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
c0bb554d8d
Update actors
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
393d8541e2
Update deps
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
1ef780d96f
Plug in the FFI call
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
a8cb027c08
Integrate v7 actors
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
25768a291e
Implement an autobatcher
2022-01-11 17:20:10 -05:00
Aayush Rajasekaran
19bd9cf945
Update to actors v7.0.0-rc1
2022-01-11 17:20:08 -05:00
Aayush Rajasekaran
8aabe1b488
Fast migration for v15
2022-01-11 17:19:38 -05:00
Aayush Rajasekaran
207d33eaba
Merge pull request #7810 from filecoin-project/feat/snap-deals
...
Snap Deals Lotus Integration: FSM Posting and integration test
2022-01-11 12:46:55 -05:00
Aayush Rajasekaran
9614f6db8d
Update go-paramfetch
2022-01-10 19:03:26 -05:00
zenground0
33f2d24f54
Snap Deals Integration
...
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
2022-01-10 15:39:38 +05:30
hannahhoward
cddf63efe9
feat(storageminer): add api for transfer diagnostics
...
Add API + CLI for inspecting in depth diagnostics on graphsync transfers with a given peer
2021-12-22 13:41:29 -08:00
Dirk McCormick
6db8a862bb
feat: update to go-fil-markets v1.14.1
2021-12-21 16:36:47 +01:00
Dirk McCormick
6311e53a76
feat: update to go-fil-markets v1.14.0
2021-12-21 11:23:38 +01:00
Dirk McCormick
839fc8df7d
fix: when retrying Add Piece, first seek to start of reader
2021-12-17 14:44:51 +01:00
Łukasz Magiera
bc384c01e3
Merge remote-tracking branch 'origin/master' into deps/update-ctx-dsbs
2021-12-17 13:01:22 +01:00
Aayush Rajasekaran
62de84d5b6
Deps: Update drand to 1.3.0
2021-12-15 18:34:21 -05:00
Aayush Rajasekaran
0e2278cc76
Merge branch 'master' into deps/update-ctx-dsbs
2021-12-14 18:08:29 -05:00
Aayush Rajasekaran
bdd56d2516
Deps: Update go-car to v0.3.3
2021-12-14 18:06:28 -05:00
Aayush Rajasekaran
02b00c025a
Deps: Update go-fil-markets to v1.13.5
2021-12-14 17:55:55 -05:00
Aayush Rajasekaran
8935c4386c
Deps: Update go-ipld-cbor to v0.0.6
2021-12-14 17:53:11 -05:00
Aayush Rajasekaran
bcd2c71031
Deps: Update go-storedcounter to v0.1.0
2021-12-14 17:51:49 -05:00
Aayush Rajasekaran
69495f5b59
Deps: Update dagstore to v0.4.4
2021-12-14 17:50:18 -05:00
Aayush Rajasekaran
3f703ff8a4
Deps: Update go-car/v2 to v2.1.1
2021-12-14 16:54:22 -05:00
Aayush Rajasekaran
46d8ae47dc
Deps: Update go-ipld-prime to v0.14.3
2021-12-14 16:41:34 -05:00
Aayush Rajasekaran
5bc4ee257f
Deps: Update go-ds-badger2 to v0.1.2
2021-12-14 16:33:38 -05:00
Aayush Rajasekaran
aac6d88f72
Deps: Update actors v2 to v2.3.6
2021-12-14 12:49:18 -05:00
Aayush Rajasekaran
a3ea3ad911
Deps: Update actors v2
2021-12-14 11:39:06 -05:00
vyzo
73ca65a6ff
update drand
2021-12-14 15:55:00 +02:00
vyzo
773f896bcd
update go-libp2p-connmgr to v0.3
2021-12-14 15:27:34 +02:00
vyzo
0001a6bad0
update go-libp2p to v0.17
2021-12-14 15:26:15 +02:00
Aayush Rajasekaran
80d5e52923
Merge branch 'master' into next
2021-12-13 13:24:28 -05:00
vyzo
73c7e9edf1
go mod tidy
2021-12-13 14:47:01 +02:00
vyzo
7ed40c1ec7
go get drand@release/v1.3
2021-12-13 14:17:05 +02:00
vyzo
97a91f23f1
update go-ds-measure and go-filestore
2021-12-13 14:12:14 +02:00
Aayush Rajasekaran
e5895af4a2
Update go-ipld-cbor
2021-12-11 18:15:35 -05:00
Aayush Rajasekaran
dfb65ed89f
Plumb contexts through
2021-12-11 17:04:00 -05:00
Aayush Rajasekaran
6c31cecc7d
Update markets
2021-12-11 14:37:37 -05:00
Aayush Rajasekaran
095361bead
Update carv2
2021-12-10 19:15:53 -05:00
Aayush Rajasekaran
702a7815b9
Update go-car
2021-12-09 22:46:30 -05:00
Aayush Rajasekaran
276eabd4e5
Update go-storedcounter and go-ds-versioning
2021-12-09 22:31:46 -05:00
Jennifer Wang
29cdee761b
update to ipld-legacy to v0.1.1
2021-12-09 21:06:56 -05:00
Jennifer Wang
f7a49e7be7
update to ipld-legacy to v0.1.1
2021-12-09 21:02:24 -05:00
vyzo
a9e22df765
update deps
2021-12-09 15:14:48 +02:00
Aayush Rajasekaran
b9b4a2659d
Update go-data-transfer
2021-12-09 14:12:25 +02:00
whyrusleeping
072297e661
WIP: updating to new datastore/blockstore code with contexts
2021-12-09 14:12:25 +02:00
Łukasz Magiera
b4bdb89f65
CARv2 v2.1.0
2021-12-08 01:52:00 -05:00
Łukasz Magiera
84c48de5d9
CARv2 v2.1.0
2021-12-06 15:39:03 +01:00
zenground0
a5be80828a
RemoveData and Decode
...
- Unsealing replica update with sector key works and tested
- Sector key generation added and tested
2021-12-03 15:21:06 -05:00
hannahhoward
f3b691d618
feat(tracing): switch to OpenTelemetry
...
Switch the underlying tracing library to OpenTelemetry, with a bridge to OpenCensus for
compatibility
2021-12-01 16:19:47 -08:00
Clint Armstrong
e2a1ca7caa
Use cgroup limits in worker memory calculations
...
Worker processes may have memory limitations imposed by Systemd. But
/proc/meminfo shows the entire system memory regardless of these limits.
This results in the scheduler believing the worker has the entire system
memory avaliable and the worker being allocated too many tasks.
This change attempts to read cgroup memory limits for the worker
process. It supports cgroups v1 and v2, and compares cgroup limits
against the system memory and returns the most conservative values to
prevent the worker from being allocated too many tasks and potentially
triggering an OOM event.
2021-11-30 02:06:58 +01:00
hannahhoward
d4074e45cf
feat(deps): update go-graphsync v0.10.6
2021-11-29 13:29:12 -08:00
Łukasz Magiera
f39942283b
Merge pull request #7706 from filecoin-project/feat/ret-matchding-cars
...
retrieval: Only output matching nodes, MatchPath dagspec
2021-11-29 22:14:25 +01:00
Łukasz Magiera
4d51980cb5
deps: Use tagged go-ipld-selector-text-lite
2021-11-29 21:22:30 +01:00
Łukasz Magiera
61791b90ea
retrieval: Only output matching nodes, MatchPath dagspec
2021-11-29 20:40:55 +01:00
zenground0
7d2b3f05db
WIP sector storage and integration test
2021-11-29 10:24:00 -05:00
Marten Seemann
cf0faf58dc
remove muxer config via LIBP2P_MUX_PREFS env
2021-11-26 11:01:54 +04:00
Aayush Rajasekaran
a5847fd06f
Update actors
2021-11-17 20:33:18 -05:00
Aayush Rajasekaran
8665e32221
Update deps
2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
07f1be0ff6
Plug in the FFI call
2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
1449644c34
Integrate v7 actors
2021-11-17 15:55:14 -05:00
Matija Petrunić
df17ea06f5
Merge branch 'master' into libp2p-pubsub-tracer
2021-11-10 12:47:09 +00:00
Aayush Rajasekaran
8cb9c6e21f
Update go-state-types
2021-11-02 15:39:46 +00:00
vyzo
758b54f36d
update go-libp2p-pubsub to v0.5.6
2021-10-29 13:53:55 +03:00
Jennifer Wang
47b82bdaea
update to actor v6.0.1 to make the logs less noisy
2021-10-26 19:53:53 -04:00
Łukasz Magiera
97e4921a4a
Merge pull request #7542 from filecoin-project/feat/gs-metrics
...
Collect and expose graphsync metrics
2021-10-20 15:22:03 +01:00
Łukasz Magiera
8182ffcf9f
Merge pull request #7532 from filecoin-project/chore/unpin-yamux
...
unpin the yamux dependency
2021-10-20 14:58:11 +01:00
Łukasz Magiera
32ce12acfc
Update go-graphsync to v0.10.4
2021-10-19 19:03:38 +02:00
Dirk McCormick
116ed01197
feat: update to go-fil-markets v1.13.3
2021-10-19 10:03:07 +02:00
Steven Allen
3b2e8c439a
unpin the yamux dependency
...
This has been pinned for a while under suspicion that it was causing
graphsync stalls. However, the libp2p team has been unable to reliably
reproduce the issue and believes that it's likely a graphsync issue.
Libp2p issue: https://github.com/libp2p/go-yamux/issues/61
2021-10-18 10:44:32 -07:00
jennijuju
dd998037cf
Merge branch 'releases' into jen/12tomaster
2021-10-12 20:06:55 -04:00
jennijuju
fa64cacfa7
integrate actor v6
2021-10-12 15:11:06 -04:00
Jennifer Wang
d1e15e61d0
Merge branch 'release/v1.12.0' into jen/12-master
2021-10-11 14:37:32 -04:00
Dirk McCormick
8957213c56
feat: update to go-fil-markets v1.13.2
2021-10-11 15:29:59 +02:00
dirkmc
f9433811c7
Merge pull request #7454 from filecoin-project/nonsense/retry-in-publish-deal
...
itests: retry deal when control addr is out of funds
2021-10-11 15:14:27 +02:00
Anton Evangelatov
6e5ccc87cf
cli: add retry for deals stuck in Publish with no funds
2021-10-11 14:03:55 +02:00
Peter Rabbitson
6184e8a943
peerstore@v0.2.9 was withdrawn, let's not depend on it directly
2021-10-08 22:47:51 +02:00
Jennifer Wang
e62aaa3d3d
update to actor v6-rc2
2021-10-08 15:09:57 -04:00
Peter Rabbitson
8b01725129
Dep upgrade pass
...
Touching just the seemingly non-controversial ones
2021-10-08 12:37:40 +02:00
Łukasz Magiera
c95940acac
Merge pull request #7466 from filecoin-project/chore/remove-dead-example
...
Remove dead example code + dep
2021-10-07 17:42:21 +01:00
Łukasz Magiera
3215eba0eb
Merge pull request #7464 from filecoin-project/rvagg/tag-go-ipld-selector-text-lite
...
chore(deps): use tagged github.com/ipld/go-ipld-selector-text-lite
2021-10-07 17:41:41 +01:00
Peter Rabbitson
2c10dc6c57
Stop indirectly depending on deprecated github.com/prometheus/common parts
2021-10-07 18:23:13 +02:00
Peter Rabbitson
c4a7de9d37
Remove dead example code + dep
2021-10-07 09:44:50 +02:00
Rod Vagg
9e76aba3d1
chore(deps): use tagged github.com/ipld/go-ipld-selector-text-lite
...
Ref: https://github.com/filecoin-project/lotus/pull/6393#discussion_r722866111
2021-10-07 17:34:28 +11:00
hannahhoward
7e6c3a330a
feat(deps): update go-graphsync v0.10.1
2021-10-06 17:21:38 -07:00
Peter Rabbitson
5e6aceeb0e
Merge remote-tracking branch 'origin/master' into feat/datamodel-selector-retrieval
2021-10-05 18:15:51 +02:00
Jiaying Wang
cbb147de23
Merge pull request #7405 from filecoin-project/feat/update-graphsync-0.10.0
...
Update to graphsync to v0.10.0, enable seperate storage and retrieval transfer limits
2021-10-05 11:50:59 -04:00
Dirk McCormick
a4e2fce9ca
feat: update to go-fil-markets v1.13.1
2021-10-05 10:40:01 +02:00
hannahhoward
368d72ebfe
feat(graphsync): update to v0.10.0-rc1
...
also add config changes
2021-10-05 14:13:58 +11:00
Peter Rabbitson
2a1094fc17
Merge banch feat/go-fil-markets-v1.13.0 into feat/datamodel-selector-retrieval
2021-10-04 21:56:11 +02:00
Dirk McCormick
f70ec02a6c
feat: go-fil-markets-v1.13.0
2021-10-04 17:35:32 +02:00
Dirk McCormick
16973bc2e8
feat: use go-fil-markets@v1.13.0
2021-10-04 15:41:40 +02:00
Jennifer Wang
38e01cba83
Merge branch 'release/v1.12.0' into jen/12tomaster
2021-10-04 02:38:00 -04:00
Jennifer Wang
5f8c44125b
Merge branch 'releases' into jen/12rc1prep
2021-10-01 20:31:29 -04:00
Aayush Rajasekaran
50ce475701
Upgrade to actors v6-rc1
2021-10-01 17:57:32 -04:00
Aayush Rajasekaran
a335cb3767
Update to latest actors
2021-10-01 17:11:05 -04:00
Łukasz Magiera
95e8b59367
Merge pull request #7341 from filecoin-project/mg/feat/panic-reporter
...
feat: Catch panic to generate report and reraise
2021-10-01 10:50:19 +01:00
Aayush Rajasekaran
dc8de20b9a
Update to latest actors
2021-09-29 12:16:01 -04:00
Łukasz Magiera
8e0314b3a3
Merge pull request #7362 from filecoin-project/chore/update-libp2p
...
Update go-libp2p to v0.15.0
2021-09-23 14:24:12 +01:00
Aarsh Shah
44ed3dc46b
update go mod
2021-09-22 10:44:42 +04:00
hannahhoward
f77dc42ef0
fix(deps): use go-graphsync v0.9.3 with hotfix
2021-09-21 22:14:20 -04:00
hannahhoward
490f70f071
feat(deps): update go-graphsync v0.9.2
2021-09-21 22:14:04 -04:00
Aayush Rajasekaran
7b4c657e90
Add v6 actors
2021-09-21 14:23:14 -04:00
hannahhoward
e70cefaa28
fix(deps): use go-graphsync v0.9.3 with hotfix
2021-09-21 04:26:02 -07:00
Aarsh Shah
89ac8b09b5
update libp2p
2021-09-21 15:10:04 +04:00
hannahhoward
ee7b6f2470
feat(deps): update go-graphsync v0.9.2
2021-09-20 16:12:58 -07:00
Mike Greenberg
595b51ecdd
feat: Catch panic to generate report and reraise
2021-09-16 22:51:10 -04:00
Mak Muftic
bb85ca031d
Add elasticsearch transport
2021-09-15 14:50:27 +02:00
Mike Greenberg
b681078b06
feat(lotus): Ensure vk and srs keys are checked despite TRUST_PARAMS
2021-09-14 12:09:33 -04:00
Peter Rabbitson
0444435589
Expose basic text-based datamodel selector on retrieval
...
Syntaxt of selection is located at
https://pkg.go.dev/github.com/ipld/go-ipld-selector-text-lite#SelectorSpecFromPath
Example use, assuming that:
- The root of the deal is a plain dag-pb unixfs directory
- The directory is not sharded
- The user wants to retrieve the first entry in that directory
lotus client retrieve --miner f0XXXXX --datamodel-path-selector 'Links/0/Hash' bafyROOTCID ~/output
For a much more elaborate example see the top of ./itests/deals_partial_retrieval_test.go
2021-09-10 09:44:11 +02:00
hannahhoward
a63bc10d26
feat(deps): update go-graphsync v0.9.1
2021-09-07 16:24:06 +02:00
Dirk McCormick
dc1f482a7d
feat: update to markets v1.12.0
2021-09-03 16:23:36 +02:00
hannahhoward
91804d5746
feat(deps): update go-graphsync v0.9.0
2021-09-03 16:22:51 +02:00
hannahhoward
b4e9bc50f8
feat(deps): update go-graphsync v0.8.0
...
Update to go-graphsync v0.8.0 with go-ipld-prime linksystem branch & trusted store.
2021-09-03 16:21:55 +02:00
Dirk McCormick
dbee23c895
feat: update to go-fil-markets v1.11.0
2021-09-02 14:06:14 +02:00
Łukasz Magiera
3e450b8818
Merge pull request #7208 from filecoin-project/raulk/fix-deal-cancellation
...
upgrade go-data-transfer; propagate deal cancellations.
2021-08-30 14:06:14 +02:00
Dirk McCormick
7b7a5b0b21
revert: changes to OnDealExpiredOrChanged in #5431 #7201
2021-08-30 10:42:41 +02:00
Raúl Kripalani
8f2d6ac042
upgrade go-data-transfer; propagate deal cancellations.
2021-08-27 20:50:16 +01:00
Dirk McCormick
ac51b8eabc
feat: go-fil-markets v1.9.0
2021-08-27 09:22:44 +02:00
Steven Allen
95d4483593
chore: remove unecessary replace
2021-08-23 15:18:01 -07:00
Aarsh Shah
b096887b16
update deps
2021-08-23 14:45:17 +05:30
Aarsh Shah
b3e73e4dad
update markets, dt and graphsync
2021-08-23 11:23:11 +05:30
Raúl Kripalani
d494778435
upgrade upstream dependencies.
2021-08-18 11:13:11 +01:00
Aarsh Shah
88f5bcdb63
Remove replace directive for multihash dep ( #7113 )
2021-08-18 11:00:29 +01:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ( #6671 )
...
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.
Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.
When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.
Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.
Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.
Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.
Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.
A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.
Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.
NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:
- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport
At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
ZenGround0
bf41240ee1
Update deps to revert version representation
2021-08-11 09:49:23 -04:00
vyzo
8d51c5a971
update go-libp2p-pubsub to v0.5.4
2021-07-30 23:15:24 +03:00
Aarsh Shah
9549087ba9
update deps for logging
2021-07-29 20:04:50 +05:30
aarshkshah1992
973476930e
more logging in data transfer and markets
2021-07-29 15:28:08 +05:30
hannahhoward
5a3b4ab34c
feat(deps): update to branches with improved logging
...
update sub repos with improved logging around data transfer processing
2021-07-28 14:46:13 -07:00
Aayush Rajasekaran
b397955f24
Merge branch 'master' into deps/update-pubsub
2021-07-28 17:15:38 -04:00
Łukasz Magiera
3b617f4528
mod tidy
2021-07-28 17:59:37 +02:00
vyzo
66e793531d
update go-libp2p-pubsub to v0.5.3
2021-07-28 15:59:16 +03:00
Jennifer Wang
d24b2acc63
update to go-fil-market v1.6.0
2021-07-27 04:34:35 -04:00
Peter Rabbitson
20c5dd5bdc
Merge remote-tracking branch 'origin/master' into chore/upgrade_go-multihash
2021-07-26 21:26:10 +02:00
Jiaying Wang
98c8a2abe5
Merge branch 'master' into feat/allow_padding_redux
2021-07-26 13:59:33 -04:00
Peter Rabbitson
ed387b43cc
Bump go-multihash, adjust test for supported version
2021-07-26 17:50:19 +02:00
Mike Greenberg
76a9f4241b
feat: Graceful error when api impl is nil
2021-07-26 13:51:37 +02:00
whyrusleeping
78f94c4c6b
This pulls in forgotten parts properly implementing PR#5988
...
( previous testing focused exclusively on offline dealflow .cars )
Allows a workflow of:
~$ dd if=/dev/urandom bs=1M count=1 | ~/go-ipfs/cmd/ipfs/ipfs add --pin=false
added QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2 QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2
~$ ~/go-ipfs/cmd/ipfs/ipfs dag export QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2 > test_mib.car
~$ lotus client import --car ~/test_mib.car
Import 2, Root QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2
~$ ~/go/bin/stream-commp -p $(( 256 * 1024 * 1024 )) < test_mib.car
CommP: 54e4e75ddc3fffa8fd33d3ededc06e564603ac0fe62543ec6463d51b553be40b
CommPCid: baga6ea4seaqfjzhhlxod775i7uz5h3pnybxfmrqdvqh6mjkd5rsghvi3ku56icy
Raw bytes: 1049073 bytes
Unpadded piece: 266338304 bytes
Padded piece: 268435456 bytes
CARv1 detected in stream:
Blocks: 5
Roots: 1
1: QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2
~$ curl http://127.0.0.1:1234/rpc/v0 -X POST -H "Authorization: Bearer $(cat ~/.lotus/token)" -H "Content-Type: application/json" --data '
{ "jsonrpc": "2.0", "id":1, "method": "Filecoin.ClientStartDeal", "params": [
{
"Wallet":"t01004",
"Miner":"t01005",
"EpochPrice":"0",
"MinBlocksDuration":518400,
"Data": {
"Root":{ "/":"QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2" },
"PieceCid":{ "/":"baga6ea4seaqfjzhhlxod775i7uz5h3pnybxfmrqdvqh6mjkd5rsghvi3ku56icy" },
"PieceSize": 266338304
}
}
] }
'
~$ ~/go/bin/stream-commp -p $(( 128 * 1024 * 1024 )) < test_mib.car
CommP: ed904105399ed346f6b03844abc14710a1748854c2781824d6bd1100e63b1807
CommPCid: baga6ea4seaqo3ecbau4z5u2g62ydqrflyfdrbilurbkme6ayetll2eia4y5rqby
Raw bytes: 1049073 bytes
Unpadded piece: 133169152 bytes
Padded piece: 134217728 bytes
CARv1 detected in stream:
Blocks: 5
Roots: 1
1: QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2
~$ curl http://127.0.0.1:1234/rpc/v0 -X POST -H "Authorization: Bearer $(cat ~/.lotus/token)" -H "Content-Type: application/json" --data '
{ "jsonrpc": "2.0", "id":1, "method": "Filecoin.ClientStatelessDeal", "params": [
{
"Wallet":"t01004",
"Miner":"t01005",
"EpochPrice":"0",
"ProviderCollateral":"0",
"MinBlocksDuration":518400,
"Data": {
"TransferType": "manual",
"Root":{ "/":"QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2" },
"PieceCid":{ "/":"baga6ea4seaqo3ecbau4z5u2g62ydqrflyfdrbilurbkme6ayetll2eia4y5rqby" },
"PieceSize": 133169152
}
}
] }
'
{"jsonrpc":"2.0","result":{"/":"bafyreianhjvev3w6q5lteap3h7tkxbe2jaobwlsi7vzbcoobjpicg3foqi"},"id":1}
~$ lotus-miner storage-deals import-data bafyreianhjvev3w6q5lteap3h7tkxbe2jaobwlsi7vzbcoobjpicg3foqi ~/test_mib.car
2021-07-24 18:17:13 +02:00
Peter Rabbitson
8d873e3edc
Tests for online and offline dealpadding
...
Also bump the times on several flaky tests that can not complete in time
on a typical laptop ( and fail half the time on CircleCI )
2021-07-24 17:46:04 +02:00
Raúl Kripalani
21f9dc8096
update graphsync to an actual release.
2021-07-22 23:47:47 +01:00
raulk
4ffbb03a82
Merge branch 'master' into raulk/merge-gs-fix
2021-07-22 23:37:17 +01:00
ZenGround0
3155609304
Finalize deps + fix after rebase
2021-07-22 09:49:56 -04:00
ZenGround0
56d71a190f
Update versions to have more digits
2021-07-22 09:49:47 -04:00
ZenGround0
f49a8248f0
PriceListByVersion
2021-07-22 09:49:47 -04:00
Raúl Kripalani
c1b304b633
update graphsync.
2021-07-20 15:43:44 +01:00
Raúl Kripalani
9042429abd
Merge branch 'master' into raulk/merge-gs-fix
2021-07-16 18:19:50 +01:00
Raúl Kripalani
aeb8e57eef
use https://github.com/ipfs/go-graphsync/pull/176 .
2021-07-16 18:17:34 +01:00
Raúl Kripalani
b2feb7e960
upgrade go-fil-markets, go-data-transfer; add logging.
2021-07-16 17:30:45 +01:00
vyzo
1d9dcef878
update go-libp2p-pubsub to v0.5.0
2021-07-15 15:07:14 +03:00
Łukasz Magiera
c37401a1a3
Merge pull request #6474 from filecoin-project/feat/splitstore-redux
...
Splitstore Enhanchements
2021-07-13 12:43:57 +02:00
Raúl Kripalani
71ddf73f44
check if merging ipfs/go-graphsync#177 fixes deal concurrency.
2021-07-10 10:48:18 +01:00
vyzo
ec586a852a
remove bbolt dependency from go.mod
2021-07-07 16:41:11 +03:00
Peter Rabbitson
dcfbb1998f
Stop outputing ANSI color on non-TTY
...
No more stuff like: https://circleci.com/gh/filecoin-project/lotus/189136
2021-07-07 11:06:20 +02:00
Peter Rabbitson
5c643d6141
Remove leftover from the great de-submoduling
...
This is no longer used anywhere in lotus
2021-07-04 09:40:47 +02:00
Raúl Kripalani
ec69ac0b2c
downgrade libp2p/go-libp2p-yamux to v0.5.1.
2021-06-25 22:39:26 +01:00
Łukasz Magiera
ebd746cdec
Merge pull request #6406 from filecoin-project/feat/lotus-sim
2021-06-24 18:20:43 +02:00
Łukasz Magiera
c3480dc0e8
Merge remote-tracking branch 'origin/releases' into chore/merge-release
2021-06-24 09:39:11 +02:00
Jennifer Wang
85f2fcf2c6
pull actor v5.0.1
2021-06-23 18:51:20 -04:00
Jakub Sztandera
7dd58efb84
Add quantiles and histogram
...
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-06-18 15:44:36 -07:00
Łukasz Magiera
44de67cf7d
Merge pull request #6175 from filecoin-project/feat/dynamic-retreival-pricing
...
Dynamic Retrieval pricing
2021-06-17 10:25:48 +02:00
Łukasz Magiera
20c4fe9bba
Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10
2021-06-15 16:50:23 +02:00
Łukasz Magiera
b00cbcaf5e
Update to go-praamfetch with fslocks
2021-06-14 18:54:59 +02:00
aarshkshah1992
fed5afa704
merge master
2021-06-11 09:35:20 +05:30
Aayush Rajasekaran
3766980331
Merge branch 'release/v1.10.0' into asr/merge-release
2021-06-09 18:41:10 -04:00
Aayush Rajasekaran
29ebdc07c9
Update to specs-actors v5-rc-3
2021-06-09 18:07:46 -04:00
aarshkshah1992
16166504f4
merged master
2021-06-08 09:17:40 +05:30
Steven Allen
5ef5680b98
chore: update to go-libp2p v0.14.2
...
This should fix test flakes due to a race in the go-libp2p mock network.
2021-06-07 12:14:01 -07:00
Jakub Sztandera
f9acd07987
Update libp2p to 0.14.1
...
This does not resolve the CI issue but prepares us for it after https://github.com/libp2p/go-libp2p/pull/1116
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-06-07 18:51:48 +02:00
aarshkshah1992
0d88800eb9
use mockgen
2021-06-07 15:02:52 +05:30
Aayush Rajasekaran
cd4505dd63
Update to specs-actors v5-rc-2
2021-06-02 19:11:47 -04:00
Aayush Rajasekaran
6854cc5944
Update to specs-actors v5-rc-2
2021-06-02 14:51:49 -04:00
Łukasz Magiera
6d9779ecae
Merge remote-tracking branch 'origin/master' into feat/nv13-1.11
2021-06-02 15:07:13 +02:00
Łukasz Magiera
3ae817d549
Update required golang version to 1.16
2021-06-02 12:29:49 +02:00
Cory Schwartz
e13dea7da8
use go:embed
2021-06-02 12:16:09 +02:00
Cory Schwartz
a7f7350c51
switch to go rice embedded
2021-06-02 12:16:09 +02:00
Dirk McCormick
06dbca24b4
feat: update to markets v1.4.0
2021-06-01 13:45:04 -06:00
Dirk McCormick
8997ed508b
feat: update to markets-v1.4.0
2021-06-01 10:16:12 -06:00
Łukasz Magiera
ffa47659a1
Merge remote-tracking branch 'origin/feat/nv13' into feat/nv13-1.11
2021-05-31 21:38:34 +02:00
Aayush Rajasekaran
61554cf3e0
Update to latest actors
2021-05-31 14:47:41 -04:00
Łukasz Magiera
1e4456138e
Merge master into feat/nv13
2021-05-27 12:28:20 +02:00
Aayush Rajasekaran
d42eda4336
Use new actor tags
2021-05-25 13:02:49 -04:00
aarshkshah1992
78255eac54
test unsealing prices for default pricing strategy
2021-05-24 15:11:50 +05:30
aarshkshah1992
dc6dbc9a11
dpr changes and test based on new unsealing PR
2021-05-22 22:40:21 +05:30
aarshkshah1992
9b34494501
use mockgen
2021-05-20 15:25:46 -06:00
Dirk McCormick
885564fe24
Revert "chore: update go-libp2p"
...
This reverts commit f7fbaef361
.
2021-05-20 11:22:19 -06:00
Łukasz Magiera
3a99f95f8e
Merge pull request #6149 from filecoin-project/feat/markets-1.3
...
Update to markets 1.3
2021-05-19 20:51:33 +02:00
Aayush Rajasekaran
5f8c80533a
Update to latest actors and FFI
2021-05-17 13:15:07 -04:00
Łukasz Magiera
506f39b294
WIP: Integrate FIP0013
2021-05-11 22:10:29 -04:00
Steven Allen
f7fbaef361
chore: update go-libp2p
...
From 0.12.0 to 0.14.0.
Headline for lotus: faster yamux.
2021-05-10 19:41:57 -07:00
Aayush Rajasekaran
b5da2655dc
Introduce v5 actors
2021-05-10 19:44:28 -04:00
Jakub Sztandera
7535c5bb53
Add mpool manage
command
...
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-05-07 15:30:05 +02:00
Jakub Sztandera
86e90dc6f1
Message sending UI
...
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-05-07 15:30:04 +02:00
vyzo
91e774063e
implement MessagePool.CheckMessages
...
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-05-07 15:30:04 +02:00
Dirk McCormick
312ad4abdc
feat: update to markets v1.3.0
2021-05-06 10:58:23 +02:00
Dirk McCormick
5b1fab8ffa
feat: markets v1.3
2021-05-06 09:49:44 +02:00
whyrusleeping
7e4d906f06
update commp utils import
2021-04-29 10:59:28 -07:00
whyrusleeping
48feb52cbf
add a test for adding padded pieces
2021-04-27 16:28:25 -07:00
Łukasz Magiera
a24a73bb52
Merge tag 'v1.8.0' into release/v1.9.0
2021-04-27 10:30:33 +02:00