Commit Graph

240 Commits

Author SHA1 Message Date
hannahhoward
71289b58ff feat(markets): update to support car files
Upgrades lotus version of go-fil-markets to its head, which supports car files and handles tracking
of pieces, reducing the size of sectorBlocks
2020-01-28 14:08:13 -08:00
Łukasz Magiera
d4c50b8ea0 sealing: gofmt 2020-01-16 03:54:57 +01:00
Łukasz Magiera
604bf64ef6 Move miner sealing logic into a separate package 2020-01-15 21:49:11 +01:00
Łukasz Magiera
2ca0c152ba Deal tests on mock sectorbuilder 2020-01-14 03:05:46 +01:00
Łukasz Magiera
a2bcc1fec2 Mostly functional mock sectorbuilder 2020-01-14 03:05:35 +01:00
whyrusleeping
fefd21ad9c remove private network and put dht into a sub namespace 2020-01-13 15:29:06 -08:00
Łukasz Magiera
a896b30039 gofmt 2020-01-10 20:55:58 +01:00
Łukasz Magiera
50bac00329 Merge branch 'master' into feat/testnet2 2020-01-10 20:55:40 +01:00
hannahhoward
0ce1bf706b refactor(markets): reorg file structure 2020-01-10 10:33:26 -08:00
hannahhoward
56653eeca6 fix(storagemarket): fix compile errors 2020-01-10 10:01:48 -08:00
hannahhoward
65ecb33630 feat(storagemarket): extract storage market
remove all shared components, point at go-fil-components repo
2020-01-10 09:49:07 -08:00
hannahhoward
0a4eeb7cb1 refactor(retrievalmarket): switch to go-fil-markets repo
Switch go-fil-components to go-fil-markets
2020-01-10 09:13:12 -08:00
hannahhoward
ccf359d057 feat(retrieval): extract retrievalmarket
Extract retrieval market and modify shared types
2020-01-09 19:46:07 -08:00
hannahhoward
da4528932a feat(storagemarket): initial extraction
Types for storage market

Modify deals.Provider to implement storagemarket.StorageProvider

Inject storagemarket.StorageProvider

Storage Provider interfaces

Storage Client interfaces

Add ValidatePublishedDeal to ClientNodeAdapter

Remove FundManager from client

Remove Wallet from client

Remove StateManager, Events, Wallet from client

Rebasing

- Copy types.BigInt, use TokenAmount/BigInt for token amounts
- Remove auto-imported log package
- Move `checkAskSignature` to a client file.
- Plumb contexts through

fix(storagemarket): use publish cids

Switch back to publish message cids to reduce the dependency surface area
2020-01-10 03:29:46 +01:00
Łukasz Magiera
5f460b380d
Merge pull request #1045 from filecoin-project/feat/mock-sectorbuilder
WIP: implement mock sectorbuilder for easier storage miner testing
2020-01-10 01:33:43 +01:00
Łukasz Magiera
f0f8d83980
Merge pull request #826 from filecoin-project/feat/retrieval-market-skeleton
Retrieval Market Cleanup W/ Node Interfaces
2020-01-09 22:06:46 +01:00
Łukasz Magiera
fac47bfc54 Pass correct datastore into sectorbuilder 2020-01-09 14:23:44 +01:00
whyrusleeping
cc56389dbb attempt to fix the DI loading, still not working 2020-01-08 20:51:18 -08:00
Łukasz Magiera
8cbcbbd7a4 Merge branch 'feat/testnet2' of github.com:filecoin-project/go-lotus into feat/testnet2 2020-01-08 22:44:54 +01:00
hannahhoward
5b74a71dd3 feat(retrievalmarket): extract skeleton interfaces
Define all types to spec, modify interfaces, wrap old code

fix(builder): use client blockstore for retrieval

feat(retrieval): add node implementations

add node adapters for client & provider so that retrieval can be extracted
2020-01-08 12:45:43 -08:00
Łukasz Magiera
da38822dfd paramfetch: bring parameters.json back to this repo 2020-01-08 21:31:35 +01:00
Jakub Sztandera
1ed62628a7
Update go-log to v2
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-01-08 20:22:31 +01:00
hannahhoward
5312340bf9 refactor(datatransfer): use independent repo
Use independent repo for go-data-transfer
2020-01-08 14:20:31 +01:00
hannahhoward
61e2568b8d feat(datatransfer): implement and extract
feat(datatransfer): setup implementation path

Sets up a path to implementation, offering both the dagservice implementation and a future graphsync
implement, establishes message interfaces and network layer, and isolates the datatransfer module
from the app

WIP using CBOR encoding for dataxfermsg

* Bring cbor-gen stuff into datatransfer package
* make transferRequest private struct
* add transferResponse + funcs
* Rename VoucherID to VoucherType
* more tests passing

WIP trying out some stuff
* Embed request/response in message so all the interfaces work AND the CBOR unmarshaling works: this is more like the spec anyway
* get rid of pb stuff

all message tests passing, some others in datatransfer

Some cleanup for PR

Cleanup for PR, clarifying and additional comments

mod tidy

Respond to PR comments:
* Make DataTransferRequest/Response be returned in from Net
* Regenerate cbor_gen and fix the generator caller so it works better
* Please the linters

Fix tests

Initiate push and pull requests (#536)

* add issue link for data TransferID generation
* comment out failing but not relevant tests
* finish voucher rename from Identifier --> Type

tests passing

cleanup for PR

remove unused fmt import in graphsync_test

a better reflection

send data transfer response

other tests passing

feat(datatransfer): milestone 2 infrastructure

Setup test path for all tickets for milestone 2

responses alert subscribers when request is not accepted (#607)

Graphsync response is scheduled when a valid push request is received (#625)

fix(datatransfer): fix tests

fix an error with read buffers in tests

fix(deps): fix go.sum

Feat/dt graphsync pullreqs (#627)

* graphsync responses to pull requests

Feat/dt initiator cleanup (#645)

* ChannelID.To --> ChannelID.Initiator
* We now store our peer ID (from host.ID()) so it can be used when creating ChannelIDs.
* InProgressChannels returns all of impl.channels, currently just for testing
* Implements go-data-transfer issue
* Some assertions were changed based on the above.
* Renamed some variables and added some assertions based on the new understanding
* Updated SHA for graphsync module
* Updated fakeGraphSync test structs to use new interfaces from new SHA above

Techdebt/dt split graphsync impl receiver (#651)

* Split up graphsyncImpl and graphsyncReceiver
* rename graphsync to utils

DTM sends data over graphsync for validated push requests (#665)

* create channels when a request is received. register push request hook with graphsync. fix tests.
* better NewReaders
* use mutex lock around impl.channels access
* fix(datatransfer): fix test uncertainty
* fix a data race and also don't use random bytes in basic block which can fail
* privatize 3 funcs

with @hannahhoward

Feat/dt gs pullrequests (#693)

* Implements DTM Sends Data Over Graphsync For Validated Pull Requests
* rename a field in a test struct
* refactor a couple of private functions (one was refactored out of existence)

Feat/dt subscribe, file Xfer round trip (#720)

Implements the rest of Subscriber Is Notified When Request Completed #24:
* send a graphsync message within a go func and consume responses until error or transfer is complete.
* notify subscribers of results.
* Rename datatransfer.Event to EventCode.
* datatransfer.Event is now a struct that includes a message and a timestamp as well as the Event.Code int, formerly Event, update all uses
* Add extension data to graphsync request hook, gsReq
* rename sendRequest to sendDtRequest, to distinguish it from sendGsRequest, where Dt = datatransfer, Gs = graphsync
* use a mutex lock for last transfer ID
* obey the linter

Don't respond with error in gsReqRcdHook when we can't find the datatransfer extension. (#754)

* update to correct graphsync version, update tests & code to call the new graphsync hooks
* getExtensionData returns an empty struct + nil if we can't find our extension
* Don't respond with error when we can't find the extension.
* Test for same
* mod tidy

minor fix to go.sum

feat(datatransfer): switch to graphsync implementation

Move over to real graphsync implementation of data transfer, add constructors for graphsync
instances on client and miner side

fix(datatransfer): Fix validators

Validators were checking payload cid against commP -- which are not the same any more. Added a
payloadCid to client deal to maintain the record, fixed validator logic

Feat/dt extraction use go-fil-components/datatransfer (#770)

* Initial commit after changing to go-fil-components/datatransfer
* blow away the datatransfer dir
* use go-fil-components master after its PR #1 was merged
* go mod tidy

use a package

updates after rebase with master
2020-01-08 14:19:28 +01:00
laser
20321bb476 handle rebase 2020-01-07 08:18:35 -08:00
acruikshank
7873213b11 switch sectorbuilder from lib to external module
point to correct version of sectorbuilder + ensure go-datastore stays at 0.1.1

replace local address type with go-address

consume paramfetch.GetParams instead of local paramfetch.go

remove constants now defined in sectorbuilder
2020-01-07 07:48:43 -08:00
hannahhoward
8418464d91 refactor(address): use extracted address library
Switch to using extracted address library
2020-01-07 14:53:27 +01:00
laser
6a740a6d67 consume newly-extracted paramfetch 2020-01-02 11:08:49 -08:00
whyrusleeping
e5899f614a Tag peers who send us good blocks over pubsub 2019-12-19 12:47:55 -08:00
Łukasz Magiera
a4521955d5 libp2p: enable ping 2019-12-18 20:33:13 +01:00
whyrusleeping
619e6ffdd7 fix master build 2019-12-17 14:20:37 -08:00
whyrusleeping
ff5b086f58 mark bootstrap peers as protected 2019-12-17 11:18:19 -08:00
whyrusleeping
ca7e2e76c2 drop miner tagger logic as it doesnt do what we want it to 2019-12-17 10:16:10 -08:00
Łukasz Magiera
8284436eae Setup PeerTagger as Invoke 2019-12-17 17:28:02 +01:00
Łukasz Magiera
241c7c7ecc Wire up ConnMgr config 2019-12-17 17:09:43 +01:00
whyrusleeping
7dcc528cd1 tag peers who propagate good blocks to us 2019-12-16 23:06:48 -08:00
whyrusleeping
022581d50b add routine to tag miners peer IDs as high value in connection manager 2019-12-16 22:51:41 -08:00
whyrusleeping
28b470cb0d blocksync tagging 2019-12-16 22:15:06 -08:00
whyrusleeping
eb4077b1af allow specifying some protected peers in the config file 2019-12-16 21:37:31 -08:00
Łukasz Magiera
825e3a0774 sectorbuilder: check free space before creating sectors 2019-12-16 17:16:08 +01:00
whyrusleeping
d6bfbe5a26 allow setting genesis timestamp when initializing network 2019-12-11 15:36:39 +01:00
Łukasz Magiera
ce6912047d Change pnet key 2019-12-10 23:11:45 +01:00
whyrusleeping
b444771427 setup enough fake peer IDs for miner generation 2019-12-09 19:56:55 +01:00
Łukasz Magiera
c49dc28a04 Move api struct to a seprate pkg 2019-12-09 18:22:11 +01:00
Łukasz Magiera
83924e6b97
sectorbuilder: Allow to restrict task types
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-12-07 19:05:15 +01:00
Łukasz Magiera
6f024bee58
Merge pull request #736 from filecoin-project/feat/params-v20
Params v20
2019-12-05 14:03:37 +01:00
whyrusleeping
b7a89b56b3 more statticcheck! 2019-12-04 21:14:19 -08:00
whyrusleeping
df069af5e5 some errcheck discoveries 2019-12-04 21:02:14 -08:00
Łukasz Magiera
3daf9103a8 paramfetch: Only pull necessary params 2019-12-04 20:44:15 +01:00
Łukasz Magiera
160e11ce8c Merge branch 'master' into feat/remote-workers 2019-12-03 03:24:00 +01:00
whyrusleeping
68e25b36f1 rename to avoid stuttering 2019-12-02 14:33:11 -08:00
whyrusleeping
b58e7344e8 pull messagepool into separate package 2019-12-02 14:33:11 -08:00
whyrusleeping
8da3cc875e Fix message nonce tracking during head changes 2019-12-02 14:33:11 -08:00
Łukasz Magiera
923748e551 Expand paths in node init 2019-12-02 23:10:22 +01:00
Łukasz Magiera
d5b94884c8 Buch of lint fixes 2019-12-02 13:51:16 +01:00
Łukasz Magiera
a47fbfa799 Merge branch 'devnet/9' into feat/election-post 2019-12-02 12:25:23 +01:00
Łukasz Magiera
a59d0f0f8c Fix unsealing, sector based data refs 2019-12-01 18:58:31 +01:00
Łukasz Magiera
ed9279cf0c Some fixes and dev utils 2019-11-30 10:25:31 +01:00
whyrusleeping
3b533ed76c Genesis miner config actually sets the created miner address now 2019-11-29 22:31:26 -06:00
Łukasz Magiera
cb7d15e67e genesis setup improvements 2019-11-28 23:50:58 +01:00
Łukasz Magiera
16d5eea9b1
Merge pull request #684 from gguoss/master
Delete died code
2019-11-28 14:21:21 +01:00
郭光华
a8093c2962
Delete died code
Lotus used kad dht as discovery protocol, not mdns .
2019-11-28 18:06:00 +08:00
Łukasz Magiera
23e0008b81 Merge remote-tracking branch 'origin/master' into feat/election-post 2019-11-27 13:31:44 +01:00
Łukasz Magiera
d5a632f644 sectorbuilder: Drop stateful sectorbuilder refs 2019-11-26 23:13:01 +01:00
whyrusleeping
a0588d513d Add lotus-gen, rewire genesis mining 2019-11-24 22:45:13 -06:00
Łukasz Magiera
43a5172944 mpool: persist local messages 2019-11-23 20:05:18 +01:00
Łukasz Magiera
2deae35dcc
Merge pull request #672 from filecoin-project/feat/no-pkg-error
Swtich to xerrors
2019-11-22 12:40:59 -06:00
Jakub Sztandera
36b7c5a32a
Swtich to xerrors
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-11-22 10:20:56 -06:00
Łukasz Magiera
c7c422abbd Use config for pubsub tracing 2019-11-20 21:31:00 +01:00
whyrusleeping
f8eabd3db4 Refactor mining logic to match specs ticket chain 2019-11-20 10:52:36 -06:00
vyzo
3a8100fd3c hardcode pubsub tracer 2019-11-20 01:31:55 +02:00
whyrusleeping
7aa76d21d1 It works! 2019-11-15 13:35:29 -08:00
Jakub Sztandera
2bc5ccab04
Add repub for local messages
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-11-13 23:43:16 +01:00
anorth
84a4766d8f Add TipSetKey as a concatenation of block CIDs, and use if for ChainGetTipSet, ChainGetRandomness. 2019-11-13 15:23:37 +11:00
hannahhoward
4b3bab371b refactor(deals): move type instantiation to modules
Move all type instantiation to dtypes & modules for any type unique to provider/client
2019-11-11 12:51:28 -08:00
hannahhoward
ccfc3c5b19 refactor(datatransfer): move registrations to builder
move registration function for validators to builder code
2019-11-11 12:25:19 -08:00
whyrusleeping
904fdad4c6 extract blocksync into its own package 2019-11-09 15:00:22 -08:00
Łukasz Magiera
3d360167df sectorbuilder: Call destroy in DI module 2019-11-08 21:30:50 +01:00
Łukasz Magiera
2d26a4edf7 Sector storage refactor 2019-11-07 19:22:59 +01:00
whyrusleeping
9863942fe2 WIP: trying to write a test to reproduce the storage deal error 2019-11-07 00:39:34 +01:00
Łukasz Magiera
ed5ccfd750 Merge remote-tracking branch 'origin/devnet/7' into feat/interactive-porep 2019-11-05 18:53:19 +01:00
Łukasz Magiera
13da5a5966 Put WorkerThreads on sectorbuilder.Config 2019-11-04 17:47:08 +01:00
Łukasz Magiera
3bc4a5dddf More progress on fixing storing garbage data 2019-11-02 00:43:54 +01:00
Łukasz Magiera
82344649d3 Use state store for sectors 2019-11-01 15:06:49 +01:00
whyrusleeping
fc9091cc89 Get interactive porep sector sealing mostly working 2019-10-31 20:57:10 -07:00
whyrusleeping
a954ebc6b6 Add command to unregister miner from full node manually 2019-10-31 15:04:13 -07:00
whyrusleeping
759094198c Some more progress on interactive porep 2019-10-30 18:22:50 -07:00
Łukasz Magiera
7fa3e2db7a WIP Interactive PoRep 2019-10-30 19:10:29 +01:00
Łukasz Magiera
46d782b30b storageminer: Drop commitment tracker 2019-10-30 18:37:38 +01:00
Łukasz Magiera
ac98c8f451 Use spec complaiant SealRandomnessLookback 2019-10-28 18:14:24 +01:00
Łukasz Magiera
c1076a0c7d set correct block height in SealTicket 2019-10-27 10:24:20 +01:00
Łukasz Magiera
117ec636c5 paramfetch: check checksums in parallel 2019-10-27 10:18:27 +01:00
Łukasz Magiera
874be79958 very basic sector seal scheduling 2019-10-27 09:56:53 +01:00
whyrusleeping
4890f577f5 give miners the ability to select different sector sizes 2019-10-25 17:26:15 +02:00
Łukasz Magiera
eeca3d86df
Merge pull request #415 from filecoin-project/feat/deals-on-chain
On-Chain deals
2019-10-25 17:03:25 +02:00
Łukasz Magiera
5175d8541e peer manager: Handle bootstrap in peermgr 2019-10-23 13:11:18 +02:00
whyrusleeping
fc7c7ddd97 Initial pass at a peer manager 2019-10-23 12:47:22 +02:00
Łukasz Magiera
46a0333c9c on chain deals: Wip porting deal systems to storagemarket 2019-10-23 12:44:00 +02:00
Jakub Sztandera
1bf713cb0a
Cleanup imports after rename
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-10-18 13:47:41 +09:00
whyrusleeping
d818e20df5 Wallets record default address in keystore 2019-10-17 19:18:40 +09:00