Commit Graph

141 Commits

Author SHA1 Message Date
hannahhoward
5312340bf9 refactor(datatransfer): use independent repo
Use independent repo for go-data-transfer
2020-01-08 14:20:31 +01:00
hannahhoward
61e2568b8d feat(datatransfer): implement and extract
feat(datatransfer): setup implementation path

Sets up a path to implementation, offering both the dagservice implementation and a future graphsync
implement, establishes message interfaces and network layer, and isolates the datatransfer module
from the app

WIP using CBOR encoding for dataxfermsg

* Bring cbor-gen stuff into datatransfer package
* make transferRequest private struct
* add transferResponse + funcs
* Rename VoucherID to VoucherType
* more tests passing

WIP trying out some stuff
* Embed request/response in message so all the interfaces work AND the CBOR unmarshaling works: this is more like the spec anyway
* get rid of pb stuff

all message tests passing, some others in datatransfer

Some cleanup for PR

Cleanup for PR, clarifying and additional comments

mod tidy

Respond to PR comments:
* Make DataTransferRequest/Response be returned in from Net
* Regenerate cbor_gen and fix the generator caller so it works better
* Please the linters

Fix tests

Initiate push and pull requests ()

* add issue link for data TransferID generation
* comment out failing but not relevant tests
* finish voucher rename from Identifier --> Type

tests passing

cleanup for PR

remove unused fmt import in graphsync_test

a better reflection

send data transfer response

other tests passing

feat(datatransfer): milestone 2 infrastructure

Setup test path for all tickets for milestone 2

responses alert subscribers when request is not accepted ()

Graphsync response is scheduled when a valid push request is received ()

fix(datatransfer): fix tests

fix an error with read buffers in tests

fix(deps): fix go.sum

Feat/dt graphsync pullreqs ()

* graphsync responses to pull requests

Feat/dt initiator cleanup ()

* ChannelID.To --> ChannelID.Initiator
* We now store our peer ID (from host.ID()) so it can be used when creating ChannelIDs.
* InProgressChannels returns all of impl.channels, currently just for testing
* Implements go-data-transfer issue
* Some assertions were changed based on the above.
* Renamed some variables and added some assertions based on the new understanding
* Updated SHA for graphsync module
* Updated fakeGraphSync test structs to use new interfaces from new SHA above

Techdebt/dt split graphsync impl receiver ()

* Split up graphsyncImpl and graphsyncReceiver
* rename graphsync to utils

DTM sends data over graphsync for validated push requests ()

* create channels when a request is received. register push request hook with graphsync. fix tests.
* better NewReaders
* use mutex lock around impl.channels access
* fix(datatransfer): fix test uncertainty
* fix a data race and also don't use random bytes in basic block which can fail
* privatize 3 funcs

with @hannahhoward

Feat/dt gs pullrequests ()

* Implements DTM Sends Data Over Graphsync For Validated Pull Requests
* rename a field in a test struct
* refactor a couple of private functions (one was refactored out of existence)

Feat/dt subscribe, file Xfer round trip ()

Implements the rest of Subscriber Is Notified When Request Completed :
* send a graphsync message within a go func and consume responses until error or transfer is complete.
* notify subscribers of results.
* Rename datatransfer.Event to EventCode.
* datatransfer.Event is now a struct that includes a message and a timestamp as well as the Event.Code int, formerly Event, update all uses
* Add extension data to graphsync request hook, gsReq
* rename sendRequest to sendDtRequest, to distinguish it from sendGsRequest, where Dt = datatransfer, Gs = graphsync
* use a mutex lock for last transfer ID
* obey the linter

Don't respond with error in gsReqRcdHook when we can't find the datatransfer extension. ()

* update to correct graphsync version, update tests & code to call the new graphsync hooks
* getExtensionData returns an empty struct + nil if we can't find our extension
* Don't respond with error when we can't find the extension.
* Test for same
* mod tidy

minor fix to go.sum

feat(datatransfer): switch to graphsync implementation

Move over to real graphsync implementation of data transfer, add constructors for graphsync
instances on client and miner side

fix(datatransfer): Fix validators

Validators were checking payload cid against commP -- which are not the same any more. Added a
payloadCid to client deal to maintain the record, fixed validator logic

Feat/dt extraction use go-fil-components/datatransfer ()

* Initial commit after changing to go-fil-components/datatransfer
* blow away the datatransfer dir
* use go-fil-components master after its PR  was merged
* go mod tidy

use a package

updates after rebase with master
2020-01-08 14:19:28 +01:00
laser
841f9c3627 reorganize imports to pass linter 2020-01-07 08:23:12 -08:00
laser
20321bb476 handle rebase 2020-01-07 08:18:35 -08:00
acruikshank
7873213b11 switch sectorbuilder from lib to external module
point to correct version of sectorbuilder + ensure go-datastore stays at 0.1.1

replace local address type with go-address

consume paramfetch.GetParams instead of local paramfetch.go

remove constants now defined in sectorbuilder
2020-01-07 07:48:43 -08:00
Łukasz Magiera
1e7f10e387 gofmt 2020-01-07 15:00:10 +01:00
hannahhoward
718acb0a86 refactor(cborutil): extract cborutil library
Use extracted cborutil library
2020-01-07 14:59:35 +01:00
hannahhoward
8418464d91 refactor(address): use extracted address library
Switch to using extracted address library
2020-01-07 14:53:27 +01:00
whyrusleeping
fda3811dcf Update cbor-gen to latest master 2019-12-17 19:06:06 -08:00
whyrusleeping
99eae286c2 clean up some log spam 2019-12-16 09:14:21 -08:00
whyrusleeping
28a325efe6 soft fail messages that cause precomitted sectors map to become too large 2019-12-13 00:00:20 +01:00
Jakub Sztandera
c37ec227e9
Fix lint warning
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-12-10 14:16:01 +01:00
Łukasz Magiera
d750723405 Merge remote-tracking branch 'origin/testnet-staging' into feat/cleanup-sectorstate-structs 2019-12-09 21:27:20 +01:00
Łukasz Magiera
94df2c656e Use map encoders for some structs 2019-12-09 17:40:15 +01:00
whyrusleeping
eba3f4af86 address PR review 2019-12-08 23:39:08 +01:00
whyrusleeping
f0653decf3 Clean up storage deal structures 2019-12-07 21:29:37 +01:00
whyrusleeping
b7a89b56b3 more statticcheck! 2019-12-04 21:14:19 -08:00
Łukasz Magiera
e239fe41d4 Fix Failing tests 2019-12-02 12:34:01 +01:00
Łukasz Magiera
fd8f65248b deals: Correctly set deal ID in provider states 2019-12-01 21:07:57 +01:00
Łukasz Magiera
a59d0f0f8c Fix unsealing, sector based data refs 2019-12-01 18:58:31 +01:00
Łukasz Magiera
ad978949db import sector/deal into state machines on init 2019-11-29 20:11:42 +01:00
Łukasz Magiera
0a6a0e5e39 Address review 2019-11-20 18:00:05 +01:00
Łukasz Magiera
ddf2e05dd0 post: More correct 'wait' logic 2019-11-20 18:00:05 +01:00
Łukasz Magiera
32aa574b8f Fix build 2019-11-19 17:17:26 +01:00
Łukasz Magiera
e48949b41f chain: cbor-gen 2019-11-14 00:04:13 +01:00
hannahhoward
311556346e refactor(datatransfer): implement style fixes
implement style fixes suggested by Magic6k
2019-11-11 13:02:49 -08:00
hannahhoward
4b3bab371b refactor(deals): move type instantiation to modules
Move all type instantiation to dtypes & modules for any type unique to provider/client
2019-11-11 12:51:28 -08:00
hannahhoward
ccfc3c5b19 refactor(datatransfer): move registrations to builder
move registration function for validators to builder code
2019-11-11 12:25:19 -08:00
Hannah Howard
b9ea6354fa
Update chain/deals/types.go
Co-Authored-By: Łukasz Magiera <magik6k@users.noreply.github.com>
2019-11-11 12:08:15 -08:00
hannahhoward
1532533712 style(formatting): run go fmt 2019-11-11 11:55:25 -08:00
hannahhoward
dcb35d90df fix(datatransfer): rebase fixes
Fixes after rebase on interactive porep
2019-11-11 11:07:15 -08:00
hannahhoward
ca5032937c refactor(datatransfer): xerrors, cbor-gen, tweaks
Various refactors per PR comments: use xerrors, add context to dt calls, use cbor-gen, move files
into seperate modules, etc
2019-11-11 11:07:15 -08:00
hannahhoward
d0b705705f refactor(datatransfer): add comments, renames
minor name changes and additional comments to clarify how data transfer works
2019-11-11 11:07:15 -08:00
hannahhoward
905259e192 feat(datatransfer): integration w/ simple merkledag
Integrates the data transfer module with a mock version of the module that just calls the dag
service
2019-11-11 11:07:15 -08:00
hannahhoward
98cf0769ff feat(datatransfer): request valdiators for storage
Add request validators that will work by default for storage deals
2019-11-11 11:07:15 -08:00
Łukasz Magiera
aaa073bdf8 deals: Inactive deal in client.sealing is an error 2019-11-08 21:18:16 +01:00
Łukasz Magiera
8c39486736 some import fixes 2019-11-08 21:11:56 +01:00
Łukasz Magiera
be2e58a2fb Simple market fund manager 2019-11-08 18:15:56 +01:00
whyrusleeping
af2789c3d8 fix non-deterministic map serialization 2019-11-07 14:55:24 -08:00
Łukasz Magiera
2d26a4edf7 Sector storage refactor 2019-11-07 19:22:59 +01:00
Łukasz Magiera
3fbe0abb75 cborrpc -> cborutil 2019-11-07 15:11:39 +01:00
Łukasz Magiera
58472afa3a deals: Verify network message signatures 2019-11-07 15:09:11 +01:00
Łukasz Magiera
2588a6a7eb deals: check msg params in ProveCommitSector event handler 2019-11-07 14:41:41 +01:00
Łukasz Magiera
8ec37e8728 deals: cleanup client state machine 2019-11-07 14:29:43 +01:00
Łukasz Magiera
79c9fb719e deals: disconnect early, rely on chain 2019-11-07 13:57:40 +01:00
Łukasz Magiera
be0d07e143 pass deal IDs in SectorPreCommitInfo 2019-11-07 13:03:18 +01:00
Łukasz Magiera
12161fc607 deals: Finish client sealing state handler 2019-11-07 10:13:42 +01:00
whyrusleeping
54722a0d38 WIP: fixing the tests by waiting for chain activity 2019-11-06 23:57:10 -08:00
Łukasz Magiera
69d7a9910e deals: remove outdated comment 2019-11-07 01:32:10 +01:00
Łukasz Magiera
d45544c90c Correct proposal cid on provider side 2019-11-07 01:01:05 +01:00