Commit Graph
53 Commits
Author SHA1 Message Date
Steven Allen 5733c71c50 Lint everything
We were ignoring quite a few error cases, and had one case where we weren't
actually updating state where we wanted to. Unfortunately, if the linter doesn't
pass, nobody has any reason to actually check lint failures in CI.

There are three remaining XXXs marked in the code for lint.
2020-08-20 20:46:36 -07:00
hannahhoward 6b5525b8d2 feat(markets): update markets 0.5.6
update markets to 0.5.6, removing use of validators that moved inside markets
2020-08-14 11:14:03 -07:00
hannahhoward a89c290f62 feat(markets): update to 0.5.4 2020-08-05 15:35:59 -07:00
hannahhoward 3cac1080cc feat(retrievalstoremgr): add retrievalstoremgr
add manager for retrievals to handle different cases for IPFS/non-ipfs
2020-07-31 14:16:18 -07:00
Łukasz Magiera b6c9169a87 client cli: Interactive deal command 2020-07-31 18:23:36 +02:00
hannahhoward e201408d61 feat(markets): better event logging 2020-07-31 12:55:45 +02:00
Łukasz Magiera 4d328cad54 Get things to mostly work 2020-07-30 14:31:50 +02:00
Łukasz Magiera 09b90773d8 Merge remote-tracking branch 'origin/next' into feat/cid-builder 2020-07-28 16:06:06 +02:00
hannahhoward 5a623cccb5 feat(markets): update markets mulitple deal stores 2020-07-27 23:13:28 -07:00
hannahhoward dd885b7302 feat(multistore): extract multistore code to repo
Extract multistore + multiread blockstore to a seperate repo
2020-07-24 14:47:22 -07:00
Steven Allen b7a4dbb07f Support inline CIDs
And use the new CidBuilder from the spec actors.

This patch does not switch over to inline CIDs by default, but paves the way.
2020-07-23 23:12:32 -07:00
Jakub Sztandera cc1d23a94c Use single multi ds
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-07-17 22:14:37 +02:00
hannahhoward 61f120222f feat(markets): update markets with new retrieval 2020-07-15 14:04:31 -07:00
Łukasz Magiera 66237415c5 Merge remote-tracking branch 'origin/next' into feat/client-multi-bstore 2020-07-08 22:48:47 +02:00
Łukasz Magiera ba0f7d703e Set lower polling interval for deal client 2020-07-08 12:52:37 +02:00
hannahhoward 24e3a4218b feat(markets): upgrade to latest
upgrade to latest markets with fast retrieval, underlying data transfer upgrade
2020-07-08 01:35:50 -07:00
Łukasz Magiera 55f089d997 client: Unbreak retrieval 2020-07-07 14:35:02 +02:00
Łukasz Magiera 47f0898ce9 Wire up client import manager 2020-07-07 10:52:19 +02:00
Łukasz Magiera 8942967223 Client Import manager 2020-07-07 10:52:04 +02:00
hannahhoward 9262661135 chore(deps): update to fil-markets 0.3.0
Updates to latest fil markets with resumability and other fixes
2020-06-15 15:43:47 -07:00
hannahhoward 3264a9faad feat(deps): update to latest go-fil-markets
update to go-fil-markets v0.2.7 with latest changes, modify constructors as needed to accomodate new
DS Prefix customization
2020-05-20 16:01:03 -07:00
shannonwells 652c257763 passing w/ new namespace wraps 2020-05-20 15:58:59 -07:00
shannonwells e68ab286f6 upgrade lotus to go-fil-markets v0.2.5 2020-05-20 15:56:58 -07:00
dtynn 30e2642f4d use abs path to construct FileManager 2020-05-16 09:36:38 +08:00
hannahhoward 581e7e6fde chore(deps): update to fil-markets 0.2.1
Update fil-markets to 0.2.1 that no longer keeps streams open
2020-05-08 14:11:40 -07:00
hannahhoward a9866c757e feat(deps): update go-fil-markets
Update to latest 0.2.0 release of go-fil-markets with less-blocking node io
2020-05-04 18:36:51 -07:00
Jeromy 6a27c65201 updating to specs actors v0.2.0 2020-04-28 13:52:07 -07:00
Travis PersonandTravis Person c807469a5a bitswap: disable provider (#1620)
Co-authored-by: Travis Person <travisperson@users.noreply.github.com>
2020-04-27 15:54:16 -07:00
laser 366db2f38c adjust import order as per PR feedback 2020-03-18 12:43:06 -07:00
laser f59bf6fab1 wire up dtypes.ClientDatastore 2020-03-18 11:57:22 -07:00
hannahhoward f259bc6a09 feat(graphsync): unified graphsync instance
setup a single graphsync that loads from both the chainstore & client blockstore
2020-03-17 17:25:12 -07:00
whyrusleeping df6e3e83bf make it all build finally 2020-02-27 13:45:31 -08:00
Łukasz Magiera e1a593c22e Rename paych -> paychmgr to fix cbor-gen 2020-02-13 01:28:23 +01:00
hannahhoward d2c5d81e59 fix(deps): update go-fil-markets
update go-fil-markets to latest to solve an issue with retrieval
2020-02-05 18:48:47 -08:00
hannahhoward 71289b58ff feat(markets): update to support car files
Upgrades lotus version of go-fil-markets to its head, which supports car files and handles tracking
of pieces, reducing the size of sectorBlocks
2020-01-28 14:08:13 -08:00
Łukasz Magiera 2ca0c152ba Deal tests on mock sectorbuilder 2020-01-14 03:05:46 +01:00
Łukasz Magiera a896b30039 gofmt 2020-01-10 20:55:58 +01:00
hannahhoward 0ce1bf706b refactor(markets): reorg file structure 2020-01-10 10:33:26 -08:00
hannahhoward 65ecb33630 feat(storagemarket): extract storage market
remove all shared components, point at go-fil-components repo
2020-01-10 09:49:07 -08:00
hannahhoward 0a4eeb7cb1 refactor(retrievalmarket): switch to go-fil-markets repo
Switch go-fil-components to go-fil-markets
2020-01-10 09:13:12 -08:00
hannahhoward ccf359d057 feat(retrieval): extract retrievalmarket
Extract retrieval market and modify shared types
2020-01-09 19:46:07 -08:00
hannahhoward 5b74a71dd3 feat(retrievalmarket): extract skeleton interfaces
Define all types to spec, modify interfaces, wrap old code

fix(builder): use client blockstore for retrieval

feat(retrieval): add node implementations

add node adapters for client & provider so that retrieval can be extracted
2020-01-08 12:45:43 -08:00
hannahhoward 5312340bf9 refactor(datatransfer): use independent repo
Use independent repo for go-data-transfer
2020-01-08 14:20:31 +01:00
hannahhoward 61e2568b8d feat(datatransfer): implement and extract
feat(datatransfer): setup implementation path

Sets up a path to implementation, offering both the dagservice implementation and a future graphsync
implement, establishes message interfaces and network layer, and isolates the datatransfer module
from the app

WIP using CBOR encoding for dataxfermsg

* Bring cbor-gen stuff into datatransfer package
* make transferRequest private struct
* add transferResponse + funcs
* Rename VoucherID to VoucherType
* more tests passing

WIP trying out some stuff
* Embed request/response in message so all the interfaces work AND the CBOR unmarshaling works: this is more like the spec anyway
* get rid of pb stuff

all message tests passing, some others in datatransfer

Some cleanup for PR

Cleanup for PR, clarifying and additional comments

mod tidy

Respond to PR comments:
* Make DataTransferRequest/Response be returned in from Net
* Regenerate cbor_gen and fix the generator caller so it works better
* Please the linters

Fix tests

Initiate push and pull requests (#536)

* add issue link for data TransferID generation
* comment out failing but not relevant tests
* finish voucher rename from Identifier --> Type

tests passing

cleanup for PR

remove unused fmt import in graphsync_test

a better reflection

send data transfer response

other tests passing

feat(datatransfer): milestone 2 infrastructure

Setup test path for all tickets for milestone 2

responses alert subscribers when request is not accepted (#607)

Graphsync response is scheduled when a valid push request is received (#625)

fix(datatransfer): fix tests

fix an error with read buffers in tests

fix(deps): fix go.sum

Feat/dt graphsync pullreqs (#627)

* graphsync responses to pull requests

Feat/dt initiator cleanup (#645)

* ChannelID.To --> ChannelID.Initiator
* We now store our peer ID (from host.ID()) so it can be used when creating ChannelIDs.
* InProgressChannels returns all of impl.channels, currently just for testing
* Implements go-data-transfer issue
* Some assertions were changed based on the above.
* Renamed some variables and added some assertions based on the new understanding
* Updated SHA for graphsync module
* Updated fakeGraphSync test structs to use new interfaces from new SHA above

Techdebt/dt split graphsync impl receiver (#651)

* Split up graphsyncImpl and graphsyncReceiver
* rename graphsync to utils

DTM sends data over graphsync for validated push requests (#665)

* create channels when a request is received. register push request hook with graphsync. fix tests.
* better NewReaders
* use mutex lock around impl.channels access
* fix(datatransfer): fix test uncertainty
* fix a data race and also don't use random bytes in basic block which can fail
* privatize 3 funcs

with @hannahhoward

Feat/dt gs pullrequests (#693)

* Implements DTM Sends Data Over Graphsync For Validated Pull Requests
* rename a field in a test struct
* refactor a couple of private functions (one was refactored out of existence)

Feat/dt subscribe, file Xfer round trip (#720)

Implements the rest of Subscriber Is Notified When Request Completed #24:
* send a graphsync message within a go func and consume responses until error or transfer is complete.
* notify subscribers of results.
* Rename datatransfer.Event to EventCode.
* datatransfer.Event is now a struct that includes a message and a timestamp as well as the Event.Code int, formerly Event, update all uses
* Add extension data to graphsync request hook, gsReq
* rename sendRequest to sendDtRequest, to distinguish it from sendGsRequest, where Dt = datatransfer, Gs = graphsync
* use a mutex lock for last transfer ID
* obey the linter

Don't respond with error in gsReqRcdHook when we can't find the datatransfer extension. (#754)

* update to correct graphsync version, update tests & code to call the new graphsync hooks
* getExtensionData returns an empty struct + nil if we can't find our extension
* Don't respond with error when we can't find the extension.
* Test for same
* mod tidy

minor fix to go.sum

feat(datatransfer): switch to graphsync implementation

Move over to real graphsync implementation of data transfer, add constructors for graphsync
instances on client and miner side

fix(datatransfer): Fix validators

Validators were checking payload cid against commP -- which are not the same any more. Added a
payloadCid to client deal to maintain the record, fixed validator logic

Feat/dt extraction use go-fil-components/datatransfer (#770)

* Initial commit after changing to go-fil-components/datatransfer
* blow away the datatransfer dir
* use go-fil-components master after its PR #1 was merged
* go mod tidy

use a package

updates after rebase with master
2020-01-08 14:19:28 +01:00
acruikshank 7873213b11 switch sectorbuilder from lib to external module
point to correct version of sectorbuilder + ensure go-datastore stays at 0.1.1

replace local address type with go-address

consume paramfetch.GetParams instead of local paramfetch.go

remove constants now defined in sectorbuilder
2020-01-07 07:48:43 -08:00
whyrusleeping df069af5e5 some errcheck discoveries 2019-12-04 21:02:14 -08:00
hannahhoward 4b3bab371b refactor(deals): move type instantiation to modules
Move all type instantiation to dtypes & modules for any type unique to provider/client
2019-11-11 12:51:28 -08:00
hannahhoward ccfc3c5b19 refactor(datatransfer): move registrations to builder
move registration function for validators to builder code
2019-11-11 12:25:19 -08:00
Jakub Sztandera 1bf713cb0a Cleanup imports after rename
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-10-18 13:47:41 +09:00
Łukasz Magiera 28d3eb38eb basic retrieval content discovery 2019-08-29 17:48:19 +02:00