Commit Graph

55 Commits

Author SHA1 Message Date
hannahhoward
3cac1080cc feat(retrievalstoremgr): add retrievalstoremgr
add manager for retrievals to handle different cases for IPFS/non-ipfs
2020-07-31 14:16:18 -07:00
Łukasz Magiera
fc3c91b738 Support external deal decision logic 2020-07-31 20:29:16 +02:00
Łukasz Magiera
4cb4e7c2c2 Merge remote-tracking branch 'origin/master' into next 2020-07-31 14:02:04 +02:00
vyzo
d8ca29dd52 support extended pubsub peer scores 2020-07-31 11:27:22 +03:00
Łukasz Magiera
09b90773d8 Merge remote-tracking branch 'origin/next' into feat/cid-builder 2020-07-28 16:06:06 +02:00
hannahhoward
5a623cccb5 feat(markets): update markets mulitple deal stores 2020-07-27 23:13:28 -07:00
hannahhoward
dd885b7302 feat(multistore): extract multistore code to repo
Extract multistore + multiread blockstore to a seperate repo
2020-07-24 14:47:22 -07:00
Steven Allen
b7a4dbb07f Support inline CIDs
And use the new CidBuilder from the spec actors.

This patch does not switch over to inline CIDs by default, but paves the way.
2020-07-23 23:12:32 -07:00
Jakub Sztandera
ff5ff4d294
Add multiple drand configs
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-07-23 18:45:08 +02:00
Aayush Rajasekaran
a5ef629cef Add expected seal duration to custom deal logic, reject deals that start too early 2020-07-15 17:31:50 -04:00
Frank
14ec9a2068 update storage miner and seal worker document 2020-07-11 16:55:13 +08:00
Łukasz Magiera
66237415c5 Merge remote-tracking branch 'origin/next' into feat/client-multi-bstore 2020-07-08 22:48:47 +02:00
Aayush Rajasekaran
411c82ae8f Update storage-FSM, add API to set sector seal delay 2020-07-07 15:23:23 -04:00
Łukasz Magiera
47f0898ce9 Wire up client import manager 2020-07-07 10:52:19 +02:00
Łukasz Magiera
8942967223 Client Import manager 2020-07-07 10:52:04 +02:00
laser
b9180a99d4 toggle consideration of offline storage/retrieval deal proposals 2020-06-26 12:27:41 -07:00
laser
e910a045d0 big rename to allow for later introducing the "consider offline" config 2020-06-26 10:50:54 -07:00
laser
4c760ec1b1 lint: update comments to match type declarations 2020-06-24 08:08:10 -07:00
laser
ef0abf2b98 disable/enable retrieval deal consideration via config 2020-06-24 08:07:12 -07:00
Yusef Napora
b448de422e improve DrandConfig dependency injection 2020-06-23 16:01:10 -04:00
laser
fbeaab466a obey the linter 2020-06-19 09:20:20 -07:00
laser
de7d6c255c blacklist -> blocklist 2020-06-18 15:51:01 -07:00
laser
0c8d648998 specify which CID is being blacklisted (it's the piece) 2020-06-18 14:02:22 -07:00
laser
b0edf924b4 add commands for manipulating storage deal CID blacklist 2020-06-18 13:15:18 -07:00
laser
36b327b57b various symbol renames 2020-06-11 13:18:18 -07:00
laser
7587e6c08b get and set storage deal acceptance through CLI 2020-06-11 12:59:50 -07:00
laser
d6b2519843 config-driven IsAcceptingStorageDeals flag
Makes incremental progress towards #1920.
2020-06-11 11:29:59 -07:00
Jakub Sztandera
7762cd0a68
Assign positive scores to drand bootstrappers
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-06-08 11:34:46 +02:00
Jakub Sztandera
7bee5b3abc
Merge remote-tracking branch 'origin/master' into next
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-06-05 00:33:43 +02:00
Jakub Sztandera
7f962e19c2
Fix lint
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-06-03 03:19:18 +02:00
Jakub Sztandera
186fd4da74
Add lotus net scores for pubsub score visibility
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-06-03 03:13:49 +02:00
laser
bea9cd2fe2 rename to better convey intent + document 2020-06-02 12:35:11 -07:00
laser
38d714a28a first take at "daemon stop" command
Fixes #1827

TODO: plumb the daemon-stopping into lotus-storage-miner, too
2020-06-02 12:29:58 -07:00
hannahhoward
3264a9faad feat(deps): update to latest go-fil-markets
update to go-fil-markets v0.2.7 with latest changes, modify constructors as needed to accomodate new
DS Prefix customization
2020-05-20 16:01:03 -07:00
Jeromy
d3eb808ff4 add profile for bootstrappers 2020-05-14 09:27:07 -07:00
hannahhoward
00cd89750d feat(deps): update fil-markets, graphsync
Updates dependencies for graphsync, fil-markets, data-transfer. Moves to new graphsync blockstore
swapping capabilities, and also locks down graphsync impl so it does not accept arbitrary requests
2020-04-07 23:25:29 -07:00
Łukasz Magiera
13f2c21e1d Get network name from genesis 2020-04-01 01:15:49 +02:00
Łukasz Magiera
1ed7779701 Merge remote-tracking branch 'origin/testnet/3' into feat/new-workers 2020-03-19 03:28:55 +01:00
laser
366db2f38c adjust import order as per PR feedback 2020-03-18 12:43:06 -07:00
laser
f59bf6fab1 wire up dtypes.ClientDatastore 2020-03-18 11:57:22 -07:00
Łukasz Magiera
5f2caffb51 Fix tests after specs-storage changes 2020-03-18 02:08:11 +01:00
hannahhoward
f259bc6a09 feat(graphsync): unified graphsync instance
setup a single graphsync that loads from both the chainstore & client blockstore
2020-03-17 17:25:12 -07:00
Łukasz Magiera
7e997e40f3 workers: Simple storage diagnostics 2020-03-16 18:50:07 +01:00
Jakub Sztandera
8af80169c0 Add graphsync
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-02-17 18:15:30 +01:00
hannahhoward
71289b58ff feat(markets): update to support car files
Upgrades lotus version of go-fil-markets to its head, which supports car files and handles tracking
of pieces, reducing the size of sectorBlocks
2020-01-28 14:08:13 -08:00
hannahhoward
5312340bf9 refactor(datatransfer): use independent repo
Use independent repo for go-data-transfer
2020-01-08 14:20:31 +01:00
hannahhoward
61e2568b8d feat(datatransfer): implement and extract
feat(datatransfer): setup implementation path

Sets up a path to implementation, offering both the dagservice implementation and a future graphsync
implement, establishes message interfaces and network layer, and isolates the datatransfer module
from the app

WIP using CBOR encoding for dataxfermsg

* Bring cbor-gen stuff into datatransfer package
* make transferRequest private struct
* add transferResponse + funcs
* Rename VoucherID to VoucherType
* more tests passing

WIP trying out some stuff
* Embed request/response in message so all the interfaces work AND the CBOR unmarshaling works: this is more like the spec anyway
* get rid of pb stuff

all message tests passing, some others in datatransfer

Some cleanup for PR

Cleanup for PR, clarifying and additional comments

mod tidy

Respond to PR comments:
* Make DataTransferRequest/Response be returned in from Net
* Regenerate cbor_gen and fix the generator caller so it works better
* Please the linters

Fix tests

Initiate push and pull requests (#536)

* add issue link for data TransferID generation
* comment out failing but not relevant tests
* finish voucher rename from Identifier --> Type

tests passing

cleanup for PR

remove unused fmt import in graphsync_test

a better reflection

send data transfer response

other tests passing

feat(datatransfer): milestone 2 infrastructure

Setup test path for all tickets for milestone 2

responses alert subscribers when request is not accepted (#607)

Graphsync response is scheduled when a valid push request is received (#625)

fix(datatransfer): fix tests

fix an error with read buffers in tests

fix(deps): fix go.sum

Feat/dt graphsync pullreqs (#627)

* graphsync responses to pull requests

Feat/dt initiator cleanup (#645)

* ChannelID.To --> ChannelID.Initiator
* We now store our peer ID (from host.ID()) so it can be used when creating ChannelIDs.
* InProgressChannels returns all of impl.channels, currently just for testing
* Implements go-data-transfer issue
* Some assertions were changed based on the above.
* Renamed some variables and added some assertions based on the new understanding
* Updated SHA for graphsync module
* Updated fakeGraphSync test structs to use new interfaces from new SHA above

Techdebt/dt split graphsync impl receiver (#651)

* Split up graphsyncImpl and graphsyncReceiver
* rename graphsync to utils

DTM sends data over graphsync for validated push requests (#665)

* create channels when a request is received. register push request hook with graphsync. fix tests.
* better NewReaders
* use mutex lock around impl.channels access
* fix(datatransfer): fix test uncertainty
* fix a data race and also don't use random bytes in basic block which can fail
* privatize 3 funcs

with @hannahhoward

Feat/dt gs pullrequests (#693)

* Implements DTM Sends Data Over Graphsync For Validated Pull Requests
* rename a field in a test struct
* refactor a couple of private functions (one was refactored out of existence)

Feat/dt subscribe, file Xfer round trip (#720)

Implements the rest of Subscriber Is Notified When Request Completed #24:
* send a graphsync message within a go func and consume responses until error or transfer is complete.
* notify subscribers of results.
* Rename datatransfer.Event to EventCode.
* datatransfer.Event is now a struct that includes a message and a timestamp as well as the Event.Code int, formerly Event, update all uses
* Add extension data to graphsync request hook, gsReq
* rename sendRequest to sendDtRequest, to distinguish it from sendGsRequest, where Dt = datatransfer, Gs = graphsync
* use a mutex lock for last transfer ID
* obey the linter

Don't respond with error in gsReqRcdHook when we can't find the datatransfer extension. (#754)

* update to correct graphsync version, update tests & code to call the new graphsync hooks
* getExtensionData returns an empty struct + nil if we can't find our extension
* Don't respond with error when we can't find the extension.
* Test for same
* mod tidy

minor fix to go.sum

feat(datatransfer): switch to graphsync implementation

Move over to real graphsync implementation of data transfer, add constructors for graphsync
instances on client and miner side

fix(datatransfer): Fix validators

Validators were checking payload cid against commP -- which are not the same any more. Added a
payloadCid to client deal to maintain the record, fixed validator logic

Feat/dt extraction use go-fil-components/datatransfer (#770)

* Initial commit after changing to go-fil-components/datatransfer
* blow away the datatransfer dir
* use go-fil-components master after its PR #1 was merged
* go mod tidy

use a package

updates after rebase with master
2020-01-08 14:19:28 +01:00
acruikshank
7873213b11 switch sectorbuilder from lib to external module
point to correct version of sectorbuilder + ensure go-datastore stays at 0.1.1

replace local address type with go-address

consume paramfetch.GetParams instead of local paramfetch.go

remove constants now defined in sectorbuilder
2020-01-07 07:48:43 -08:00
hannahhoward
4b3bab371b refactor(deals): move type instantiation to modules
Move all type instantiation to dtypes & modules for any type unique to provider/client
2019-11-11 12:51:28 -08:00
Łukasz Magiera
9a536e7cc0 config: Allow overriding bootstrap peers 2019-10-11 05:16:12 +02:00