cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
14,490 Commits 24 Branches 342 Tags 314 MiB
f148397e1b
Commit Graph

1222 Commits

Author SHA1 Message Date
Aayush Rajasekaran
095361bead Update carv2 2021-12-10 19:15:53 -05:00
Aayush Rajasekaran
702a7815b9 Update go-car 2021-12-09 22:46:30 -05:00
Aayush Rajasekaran
276eabd4e5 Update go-storedcounter and go-ds-versioning 2021-12-09 22:31:46 -05:00
Jennifer Wang
f7a49e7be7 update to ipld-legacy to v0.1.1 2021-12-09 21:02:24 -05:00
vyzo
a9e22df765 update deps 2021-12-09 15:14:48 +02:00
Aayush Rajasekaran
b9b4a2659d Update go-data-transfer 2021-12-09 14:12:25 +02:00
whyrusleeping
072297e661 WIP: updating to new datastore/blockstore code with contexts 2021-12-09 14:12:25 +02:00
Łukasz Magiera
84c48de5d9 CARv2 v2.1.0 2021-12-06 15:39:03 +01:00
zenground0
a5be80828a RemoveData and Decode 
- Unsealing replica update with sector key works and tested
- Sector key generation added and tested
 2021-12-03 15:21:06 -05:00
hannahhoward
f3b691d618 feat(tracing): switch to OpenTelemetry 
Switch the underlying tracing library to OpenTelemetry, with a bridge to OpenCensus for
compatibility
 2021-12-01 16:19:47 -08:00
Clint Armstrong
e2a1ca7caa Use cgroup limits in worker memory calculations 
Worker processes may have memory limitations imposed by Systemd. But
/proc/meminfo shows the entire system memory regardless of these limits.
This results in the scheduler believing the worker has the entire system
memory avaliable and the worker being allocated too many tasks.

This change attempts to read cgroup memory limits for the worker
process. It supports cgroups v1 and v2, and compares cgroup limits
against the system memory and returns the most conservative values to
prevent the worker from being allocated too many tasks and potentially
triggering an OOM event.
 2021-11-30 02:06:58 +01:00
hannahhoward
d4074e45cf feat(deps): update go-graphsync v0.10.6 2021-11-29 13:29:12 -08:00
Łukasz Magiera
f39942283b
Merge pull request #7706 from filecoin-project/feat/ret-matchding-cars 
retrieval: Only output matching nodes, MatchPath dagspec
 2021-11-29 22:14:25 +01:00
Łukasz Magiera
4d51980cb5 deps: Use tagged go-ipld-selector-text-lite 2021-11-29 21:22:30 +01:00
Łukasz Magiera
61791b90ea retrieval: Only output matching nodes, MatchPath dagspec 2021-11-29 20:40:55 +01:00
zenground0
7d2b3f05db WIP sector storage and integration test 2021-11-29 10:24:00 -05:00
Marten Seemann
cf0faf58dc remove muxer config via LIBP2P_MUX_PREFS env 2021-11-26 11:01:54 +04:00
Aayush Rajasekaran
a5847fd06f Update actors 2021-11-17 20:33:18 -05:00
Aayush Rajasekaran
8665e32221 Update deps 2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
07f1be0ff6 Plug in the FFI call 2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
1449644c34 Integrate v7 actors 2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
8cb9c6e21f Update go-state-types 2021-11-02 15:39:46 +00:00
vyzo
758b54f36d update go-libp2p-pubsub to v0.5.6 2021-10-29 13:53:55 +03:00
Jennifer Wang
47b82bdaea update to actor v6.0.1 to make the logs less noisy 2021-10-26 19:53:53 -04:00
Łukasz Magiera
97e4921a4a
Merge pull request #7542 from filecoin-project/feat/gs-metrics 
Collect and expose graphsync metrics
 2021-10-20 15:22:03 +01:00
Łukasz Magiera
8182ffcf9f
Merge pull request #7532 from filecoin-project/chore/unpin-yamux 
unpin the yamux dependency
 2021-10-20 14:58:11 +01:00
Łukasz Magiera
32ce12acfc Update go-graphsync to v0.10.4 2021-10-19 19:03:38 +02:00
Dirk McCormick
116ed01197 feat: update to go-fil-markets v1.13.3 2021-10-19 10:03:07 +02:00
Steven Allen
3b2e8c439a unpin the yamux dependency 
This has been pinned for a while under suspicion that it was causing
graphsync stalls. However, the libp2p team has been unable to reliably
reproduce the issue and believes that it's likely a graphsync issue.

Libp2p issue: https://github.com/libp2p/go-yamux/issues/61
 2021-10-18 10:44:32 -07:00
jennijuju
dd998037cf Merge branch 'releases' into jen/12tomaster 2021-10-12 20:06:55 -04:00
jennijuju
fa64cacfa7 integrate actor v6 2021-10-12 15:11:06 -04:00
Jennifer Wang
d1e15e61d0 Merge branch 'release/v1.12.0' into jen/12-master 2021-10-11 14:37:32 -04:00
Dirk McCormick
8957213c56 feat: update to go-fil-markets v1.13.2 2021-10-11 15:29:59 +02:00
dirkmc
f9433811c7
Merge pull request #7454 from filecoin-project/nonsense/retry-in-publish-deal 
itests: retry deal when control addr is out of funds
 2021-10-11 15:14:27 +02:00
Anton Evangelatov
6e5ccc87cf cli: add retry for deals stuck in Publish with no funds 2021-10-11 14:03:55 +02:00
Peter Rabbitson
6184e8a943 peerstore@v0.2.9 was withdrawn, let's not depend on it directly 2021-10-08 22:47:51 +02:00
Jennifer Wang
e62aaa3d3d update to actor v6-rc2 2021-10-08 15:09:57 -04:00
Peter Rabbitson
8b01725129 Dep upgrade pass 
Touching just the seemingly non-controversial ones
 2021-10-08 12:37:40 +02:00
Łukasz Magiera
c95940acac
Merge pull request #7466 from filecoin-project/chore/remove-dead-example 
Remove dead example code + dep
 2021-10-07 17:42:21 +01:00
Łukasz Magiera
3215eba0eb
Merge pull request #7464 from filecoin-project/rvagg/tag-go-ipld-selector-text-lite 
chore(deps): use tagged github.com/ipld/go-ipld-selector-text-lite
 2021-10-07 17:41:41 +01:00
Peter Rabbitson
2c10dc6c57 Stop indirectly depending on deprecated github.com/prometheus/common parts 2021-10-07 18:23:13 +02:00
Peter Rabbitson
c4a7de9d37 Remove dead example code + dep 2021-10-07 09:44:50 +02:00
Rod Vagg
9e76aba3d1
chore(deps): use tagged github.com/ipld/go-ipld-selector-text-lite 
Ref: https://github.com/filecoin-project/lotus/pull/6393#discussion_r722866111
 2021-10-07 17:34:28 +11:00
hannahhoward
7e6c3a330a feat(deps): update go-graphsync v0.10.1 2021-10-06 17:21:38 -07:00
Peter Rabbitson
5e6aceeb0e Merge remote-tracking branch 'origin/master' into feat/datamodel-selector-retrieval 2021-10-05 18:15:51 +02:00
Jiaying Wang
cbb147de23
Merge pull request #7405 from filecoin-project/feat/update-graphsync-0.10.0 
Update to graphsync to v0.10.0, enable seperate storage and retrieval transfer limits
 2021-10-05 11:50:59 -04:00
Dirk McCormick
a4e2fce9ca feat: update to go-fil-markets v1.13.1 2021-10-05 10:40:01 +02:00
hannahhoward
368d72ebfe
feat(graphsync): update to v0.10.0-rc1 
also add config changes
 2021-10-05 14:13:58 +11:00
Peter Rabbitson
2a1094fc17 Merge banch feat/go-fil-markets-v1.13.0 into feat/datamodel-selector-retrieval 2021-10-04 21:56:11 +02:00
Dirk McCormick
f70ec02a6c feat: go-fil-markets-v1.13.0 2021-10-04 17:35:32 +02:00
Dirk McCormick
16973bc2e8 feat: use go-fil-markets@v1.13.0 2021-10-04 15:41:40 +02:00
Jennifer Wang
38e01cba83 Merge branch 'release/v1.12.0' into jen/12tomaster 2021-10-04 02:38:00 -04:00
Jennifer Wang
5f8c44125b Merge branch 'releases' into jen/12rc1prep 2021-10-01 20:31:29 -04:00
Aayush Rajasekaran
50ce475701 Upgrade to actors v6-rc1 2021-10-01 17:57:32 -04:00
Aayush Rajasekaran
a335cb3767 Update to latest actors 2021-10-01 17:11:05 -04:00
Łukasz Magiera
95e8b59367
Merge pull request #7341 from filecoin-project/mg/feat/panic-reporter 
feat: Catch panic to generate report and reraise
 2021-10-01 10:50:19 +01:00
Aayush Rajasekaran
dc8de20b9a Update to latest actors 2021-09-29 12:16:01 -04:00
Łukasz Magiera
8e0314b3a3
Merge pull request #7362 from filecoin-project/chore/update-libp2p 
Update go-libp2p to v0.15.0
 2021-09-23 14:24:12 +01:00
Aarsh Shah
44ed3dc46b update go mod 2021-09-22 10:44:42 +04:00
hannahhoward
f77dc42ef0 fix(deps): use go-graphsync v0.9.3 with hotfix 2021-09-21 22:14:20 -04:00
hannahhoward
490f70f071 feat(deps): update go-graphsync v0.9.2 2021-09-21 22:14:04 -04:00
Aayush Rajasekaran
7b4c657e90 Add v6 actors 2021-09-21 14:23:14 -04:00
hannahhoward
e70cefaa28 fix(deps): use go-graphsync v0.9.3 with hotfix 2021-09-21 04:26:02 -07:00
Aarsh Shah
89ac8b09b5 update libp2p 2021-09-21 15:10:04 +04:00
hannahhoward
ee7b6f2470 feat(deps): update go-graphsync v0.9.2 2021-09-20 16:12:58 -07:00
Mike Greenberg
595b51ecdd feat: Catch panic to generate report and reraise 2021-09-16 22:51:10 -04:00
Mike Greenberg
b681078b06 feat(lotus): Ensure vk and srs keys are checked despite TRUST_PARAMS 2021-09-14 12:09:33 -04:00
Peter Rabbitson
0444435589 Expose basic text-based datamodel selector on retrieval 
Syntaxt of selection is located at
https://pkg.go.dev/github.com/ipld/go-ipld-selector-text-lite#SelectorSpecFromPath

Example use, assuming that:
  - The root of the deal is a plain dag-pb unixfs directory
  - The directory is not sharded
  - The user wants to retrieve the first entry in that directory

  lotus client retrieve --miner f0XXXXX --datamodel-path-selector 'Links/0/Hash' bafyROOTCID ~/output

For a much more elaborate example see the top of ./itests/deals_partial_retrieval_test.go
 2021-09-10 09:44:11 +02:00
hannahhoward
a63bc10d26 feat(deps): update go-graphsync v0.9.1 2021-09-07 16:24:06 +02:00
Dirk McCormick
dc1f482a7d feat: update to markets v1.12.0 2021-09-03 16:23:36 +02:00
hannahhoward
91804d5746 feat(deps): update go-graphsync v0.9.0 2021-09-03 16:22:51 +02:00
hannahhoward
b4e9bc50f8 feat(deps): update go-graphsync v0.8.0 
Update to go-graphsync v0.8.0 with go-ipld-prime linksystem branch & trusted store.
 2021-09-03 16:21:55 +02:00
Dirk McCormick
dbee23c895 feat: update to go-fil-markets v1.11.0 2021-09-02 14:06:14 +02:00
Łukasz Magiera
3e450b8818
Merge pull request #7208 from filecoin-project/raulk/fix-deal-cancellation 
upgrade go-data-transfer; propagate deal cancellations.
 2021-08-30 14:06:14 +02:00
Dirk McCormick
7b7a5b0b21 revert: changes to OnDealExpiredOrChanged in #5431 #7201 2021-08-30 10:42:41 +02:00
Raúl Kripalani
8f2d6ac042 upgrade go-data-transfer; propagate deal cancellations. 2021-08-27 20:50:16 +01:00
Dirk McCormick
ac51b8eabc feat: go-fil-markets v1.9.0 2021-08-27 09:22:44 +02:00
Steven Allen
95d4483593 chore: remove unecessary replace 2021-08-23 15:18:01 -07:00
Aarsh Shah
b096887b16 update deps 2021-08-23 14:45:17 +05:30
Aarsh Shah
b3e73e4dad update markets, dt and graphsync 2021-08-23 11:23:11 +05:30
Raúl Kripalani
d494778435 upgrade upstream dependencies. 2021-08-18 11:13:11 +01:00
Aarsh Shah
88f5bcdb63
Remove replace directive for multihash dep (#7113) 2021-08-18 11:00:29 +01:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671) 
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
 2021-08-16 23:34:32 +01:00
ZenGround0
bf41240ee1 Update deps to revert version representation 2021-08-11 09:49:23 -04:00
vyzo
8d51c5a971 update go-libp2p-pubsub to v0.5.4 2021-07-30 23:15:24 +03:00
Aarsh Shah
9549087ba9 update deps for logging 2021-07-29 20:04:50 +05:30
aarshkshah1992
973476930e more logging in data transfer and markets 2021-07-29 15:28:08 +05:30
hannahhoward
5a3b4ab34c feat(deps): update to branches with improved logging 
update sub repos with improved logging around data transfer processing
 2021-07-28 14:46:13 -07:00
Aayush Rajasekaran
b397955f24
Merge branch 'master' into deps/update-pubsub 2021-07-28 17:15:38 -04:00
Łukasz Magiera
3b617f4528 mod tidy 2021-07-28 17:59:37 +02:00
vyzo
66e793531d update go-libp2p-pubsub to v0.5.3 2021-07-28 15:59:16 +03:00
Jennifer Wang
d24b2acc63 update to go-fil-market v1.6.0 2021-07-27 04:34:35 -04:00
Peter Rabbitson
20c5dd5bdc Merge remote-tracking branch 'origin/master' into chore/upgrade_go-multihash 2021-07-26 21:26:10 +02:00
Jiaying Wang
98c8a2abe5
Merge branch 'master' into feat/allow_padding_redux 2021-07-26 13:59:33 -04:00
Peter Rabbitson
ed387b43cc Bump go-multihash, adjust test for supported version 2021-07-26 17:50:19 +02:00
Mike Greenberg
76a9f4241b feat: Graceful error when api impl is nil 2021-07-26 13:51:37 +02:00
whyrusleeping
78f94c4c6b This pulls in forgotten parts properly implementing PR#5988 
( previous testing focused exclusively on offline dealflow .cars )

Allows a workflow of:

~$ dd if=/dev/urandom bs=1M count=1 | ~/go-ipfs/cmd/ipfs/ipfs add --pin=false
added QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2 QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2

~$ ~/go-ipfs/cmd/ipfs/ipfs dag export QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2 > test_mib.car

~$ lotus client import --car ~/test_mib.car
Import 2, Root QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2

~$ ~/go/bin/stream-commp -p $(( 256 * 1024 * 1024 )) < test_mib.car
CommP:    54e4e75ddc3fffa8fd33d3ededc06e564603ac0fe62543ec6463d51b553be40b
CommPCid: baga6ea4seaqfjzhhlxod775i7uz5h3pnybxfmrqdvqh6mjkd5rsghvi3ku56icy
Raw bytes:           1049073 bytes
Unpadded piece:    266338304 bytes
Padded piece:      268435456 bytes
CARv1 detected in stream:
Blocks:         5
Roots:          1
    1: QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2

~$ curl http://127.0.0.1:1234/rpc/v0 -X POST -H "Authorization: Bearer $(cat ~/.lotus/token)" -H "Content-Type: application/json" --data '
  { "jsonrpc": "2.0", "id":1, "method": "Filecoin.ClientStartDeal", "params": [
    {
      "Wallet":"t01004",
      "Miner":"t01005",
      "EpochPrice":"0",
      "MinBlocksDuration":518400,
      "Data": {
        "Root":{ "/":"QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2" },
        "PieceCid":{ "/":"baga6ea4seaqfjzhhlxod775i7uz5h3pnybxfmrqdvqh6mjkd5rsghvi3ku56icy" },
        "PieceSize": 266338304
      }
    }
  ] }
'

~$ ~/go/bin/stream-commp -p $(( 128 * 1024 * 1024 )) < test_mib.car
CommP:    ed904105399ed346f6b03844abc14710a1748854c2781824d6bd1100e63b1807
CommPCid: baga6ea4seaqo3ecbau4z5u2g62ydqrflyfdrbilurbkme6ayetll2eia4y5rqby
Raw bytes:           1049073 bytes
Unpadded piece:    133169152 bytes
Padded piece:      134217728 bytes
CARv1 detected in stream:
Blocks:         5
Roots:          1
    1: QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2

~$ curl http://127.0.0.1:1234/rpc/v0 -X POST -H "Authorization: Bearer $(cat ~/.lotus/token)" -H "Content-Type: application/json" --data '
  { "jsonrpc": "2.0", "id":1, "method": "Filecoin.ClientStatelessDeal", "params": [
    {
      "Wallet":"t01004",
      "Miner":"t01005",
      "EpochPrice":"0",
      "ProviderCollateral":"0",
      "MinBlocksDuration":518400,
      "Data": {
        "TransferType": "manual",
        "Root":{ "/":"QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2" },
        "PieceCid":{ "/":"baga6ea4seaqo3ecbau4z5u2g62ydqrflyfdrbilurbkme6ayetll2eia4y5rqby" },
        "PieceSize": 133169152
      }
    }
  ] }
'
{"jsonrpc":"2.0","result":{"/":"bafyreianhjvev3w6q5lteap3h7tkxbe2jaobwlsi7vzbcoobjpicg3foqi"},"id":1}

~$ lotus-miner storage-deals import-data bafyreianhjvev3w6q5lteap3h7tkxbe2jaobwlsi7vzbcoobjpicg3foqi ~/test_mib.car
 2021-07-24 18:17:13 +02:00
Peter Rabbitson
8d873e3edc Tests for online and offline dealpadding 
Also bump the times on several flaky tests that can not complete in time
on a typical laptop ( and fail half the time on CircleCI )
 2021-07-24 17:46:04 +02:00
Raúl Kripalani
21f9dc8096 update graphsync to an actual release. 2021-07-22 23:47:47 +01:00
raulk
4ffbb03a82
Merge branch 'master' into raulk/merge-gs-fix 2021-07-22 23:37:17 +01:00