Commit Graph

377 Commits

Author SHA1 Message Date
Łukasz Magiera
5f460b380d
Merge pull request from filecoin-project/feat/mock-sectorbuilder
WIP: implement mock sectorbuilder for easier storage miner testing
2020-01-10 01:33:43 +01:00
whyrusleeping
d08898ef37 make storage miner acccept an interface to the sectorbuilder 2020-01-08 17:10:17 -08:00
hannahhoward
5b74a71dd3 feat(retrievalmarket): extract skeleton interfaces
Define all types to spec, modify interfaces, wrap old code

fix(builder): use client blockstore for retrieval

feat(retrieval): add node implementations

add node adapters for client & provider so that retrieval can be extracted
2020-01-08 12:45:43 -08:00
hannahhoward
61e2568b8d feat(datatransfer): implement and extract
feat(datatransfer): setup implementation path

Sets up a path to implementation, offering both the dagservice implementation and a future graphsync
implement, establishes message interfaces and network layer, and isolates the datatransfer module
from the app

WIP using CBOR encoding for dataxfermsg

* Bring cbor-gen stuff into datatransfer package
* make transferRequest private struct
* add transferResponse + funcs
* Rename VoucherID to VoucherType
* more tests passing

WIP trying out some stuff
* Embed request/response in message so all the interfaces work AND the CBOR unmarshaling works: this is more like the spec anyway
* get rid of pb stuff

all message tests passing, some others in datatransfer

Some cleanup for PR

Cleanup for PR, clarifying and additional comments

mod tidy

Respond to PR comments:
* Make DataTransferRequest/Response be returned in from Net
* Regenerate cbor_gen and fix the generator caller so it works better
* Please the linters

Fix tests

Initiate push and pull requests ()

* add issue link for data TransferID generation
* comment out failing but not relevant tests
* finish voucher rename from Identifier --> Type

tests passing

cleanup for PR

remove unused fmt import in graphsync_test

a better reflection

send data transfer response

other tests passing

feat(datatransfer): milestone 2 infrastructure

Setup test path for all tickets for milestone 2

responses alert subscribers when request is not accepted ()

Graphsync response is scheduled when a valid push request is received ()

fix(datatransfer): fix tests

fix an error with read buffers in tests

fix(deps): fix go.sum

Feat/dt graphsync pullreqs ()

* graphsync responses to pull requests

Feat/dt initiator cleanup ()

* ChannelID.To --> ChannelID.Initiator
* We now store our peer ID (from host.ID()) so it can be used when creating ChannelIDs.
* InProgressChannels returns all of impl.channels, currently just for testing
* Implements go-data-transfer issue
* Some assertions were changed based on the above.
* Renamed some variables and added some assertions based on the new understanding
* Updated SHA for graphsync module
* Updated fakeGraphSync test structs to use new interfaces from new SHA above

Techdebt/dt split graphsync impl receiver ()

* Split up graphsyncImpl and graphsyncReceiver
* rename graphsync to utils

DTM sends data over graphsync for validated push requests ()

* create channels when a request is received. register push request hook with graphsync. fix tests.
* better NewReaders
* use mutex lock around impl.channels access
* fix(datatransfer): fix test uncertainty
* fix a data race and also don't use random bytes in basic block which can fail
* privatize 3 funcs

with @hannahhoward

Feat/dt gs pullrequests ()

* Implements DTM Sends Data Over Graphsync For Validated Pull Requests
* rename a field in a test struct
* refactor a couple of private functions (one was refactored out of existence)

Feat/dt subscribe, file Xfer round trip ()

Implements the rest of Subscriber Is Notified When Request Completed :
* send a graphsync message within a go func and consume responses until error or transfer is complete.
* notify subscribers of results.
* Rename datatransfer.Event to EventCode.
* datatransfer.Event is now a struct that includes a message and a timestamp as well as the Event.Code int, formerly Event, update all uses
* Add extension data to graphsync request hook, gsReq
* rename sendRequest to sendDtRequest, to distinguish it from sendGsRequest, where Dt = datatransfer, Gs = graphsync
* use a mutex lock for last transfer ID
* obey the linter

Don't respond with error in gsReqRcdHook when we can't find the datatransfer extension. ()

* update to correct graphsync version, update tests & code to call the new graphsync hooks
* getExtensionData returns an empty struct + nil if we can't find our extension
* Don't respond with error when we can't find the extension.
* Test for same
* mod tidy

minor fix to go.sum

feat(datatransfer): switch to graphsync implementation

Move over to real graphsync implementation of data transfer, add constructors for graphsync
instances on client and miner side

fix(datatransfer): Fix validators

Validators were checking payload cid against commP -- which are not the same any more. Added a
payloadCid to client deal to maintain the record, fixed validator logic

Feat/dt extraction use go-fil-components/datatransfer ()

* Initial commit after changing to go-fil-components/datatransfer
* blow away the datatransfer dir
* use go-fil-components master after its PR  was merged
* go mod tidy

use a package

updates after rebase with master
2020-01-08 14:19:28 +01:00
acruikshank
7873213b11 switch sectorbuilder from lib to external module
point to correct version of sectorbuilder + ensure go-datastore stays at 0.1.1

replace local address type with go-address

consume paramfetch.GetParams instead of local paramfetch.go

remove constants now defined in sectorbuilder
2020-01-07 07:48:43 -08:00
whyrusleeping
ca7e2e76c2 drop miner tagger logic as it doesnt do what we want it to 2019-12-17 10:16:10 -08:00
Łukasz Magiera
8284436eae Setup PeerTagger as Invoke 2019-12-17 17:28:02 +01:00
Łukasz Magiera
241c7c7ecc Wire up ConnMgr config 2019-12-17 17:09:43 +01:00
whyrusleeping
022581d50b add routine to tag miners peer IDs as high value in connection manager 2019-12-16 22:51:41 -08:00
whyrusleeping
28b470cb0d blocksync tagging 2019-12-16 22:15:06 -08:00
Łukasz Magiera
edd40895b8 disable head notifs by default 2019-12-11 16:08:50 +01:00
Łukasz Magiera
83924e6b97
sectorbuilder: Allow to restrict task types
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-12-07 19:05:15 +01:00
Łukasz Magiera
3daf9103a8 paramfetch: Only pull necessary params 2019-12-04 20:44:15 +01:00
whyrusleeping
b58e7344e8 pull messagepool into separate package 2019-12-02 14:33:11 -08:00
Łukasz Magiera
23e0008b81 Merge remote-tracking branch 'origin/master' into feat/election-post 2019-11-27 13:31:44 +01:00
whyrusleeping
a0588d513d Add lotus-gen, rewire genesis mining 2019-11-24 22:45:13 -06:00
Łukasz Magiera
c7c422abbd Use config for pubsub tracing 2019-11-20 21:31:00 +01:00
whyrusleeping
7aa76d21d1 It works! 2019-11-15 13:35:29 -08:00
Jakub Sztandera
2bc5ccab04
Add repub for local messages
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-11-13 23:43:16 +01:00
Jakub Sztandera
a58e165339
Restore api flags
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-11-13 16:37:40 +01:00
Łukasz Magiera
73ab6c0c66 WorkerCount on storageminer config 2019-11-12 18:59:38 +01:00
hannahhoward
4b3bab371b refactor(deals): move type instantiation to modules
Move all type instantiation to dtypes & modules for any type unique to provider/client
2019-11-11 12:51:28 -08:00
hannahhoward
ccfc3c5b19 refactor(datatransfer): move registrations to builder
move registration function for validators to builder code
2019-11-11 12:25:19 -08:00
hannahhoward
905259e192 feat(datatransfer): integration w/ simple merkledag
Integrates the data transfer module with a mock version of the module that just calls the dag
service
2019-11-11 11:07:15 -08:00
whyrusleeping
904fdad4c6 extract blocksync into its own package 2019-11-09 15:00:22 -08:00
Łukasz Magiera
fa67ccb5f3 Fix storageminer sectorblocks constructor 2019-11-08 22:00:01 +01:00
Łukasz Magiera
3d360167df sectorbuilder: Call destroy in DI module 2019-11-08 21:30:50 +01:00
Łukasz Magiera
8c39486736 some import fixes 2019-11-08 21:11:56 +01:00
Łukasz Magiera
be2e58a2fb Simple market fund manager 2019-11-08 18:15:56 +01:00
Łukasz Magiera
2d26a4edf7 Sector storage refactor 2019-11-07 19:22:59 +01:00
Łukasz Magiera
ed5ccfd750 Merge remote-tracking branch 'origin/devnet/7' into feat/interactive-porep 2019-11-05 18:53:19 +01:00
whyrusleeping
fc9091cc89 Get interactive porep sector sealing mostly working 2019-10-31 20:57:10 -07:00
Jakub Sztandera
407183b258
Fix double locking
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-10-30 23:33:31 +01:00
Łukasz Magiera
46d782b30b storageminer: Drop commitment tracker 2019-10-30 18:37:38 +01:00
Jakub Sztandera
3ea0997c93
Add support for different configs
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-10-30 17:38:39 +01:00
Łukasz Magiera
874be79958 very basic sector seal scheduling 2019-10-27 09:56:53 +01:00
Łukasz Magiera
eeca3d86df
Merge pull request from filecoin-project/feat/deals-on-chain
On-Chain deals
2019-10-25 17:03:25 +02:00
Łukasz Magiera
1d1f468c98 peer manager: Disable in tests 2019-10-23 13:02:00 +02:00
whyrusleeping
fc7c7ddd97 Initial pass at a peer manager 2019-10-23 12:47:22 +02:00
Łukasz Magiera
46a0333c9c on chain deals: Wip porting deal systems to storagemarket 2019-10-23 12:44:00 +02:00
Jakub Sztandera
1bf713cb0a
Cleanup imports after rename
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-10-18 13:47:41 +09:00
Łukasz Magiera
9a536e7cc0 config: Allow overriding bootstrap peers 2019-10-11 05:16:12 +02:00
Łukasz Magiera
387adae115 townhall: Node names 2019-10-11 04:45:45 +02:00
Łukasz Magiera
40267e15b2
Merge pull request from filecoin-project/feat/periodic-bootstrap
Periodic bootstrap
2019-10-11 04:09:01 +02:00
Łukasz Magiera
c0179bd4fa Periodic bootstrap 2019-10-11 02:31:26 +02:00
Łukasz Magiera
4c5157d270 Lotus Townhall 2019-10-10 14:02:43 +02:00
Łukasz Magiera
752654ab03 Devnet 3; Builtin bootstrap; NAT Port Map 2019-10-09 05:16:35 +02:00
Łukasz Magiera
1534536590 Store libp2p key in keystore 2019-10-03 02:11:31 +02:00
Łukasz Magiera
83f1a336a6 node: Basic graceful shutdown 2019-09-17 16:23:08 +02:00
Łukasz Magiera
eda72468ce Sector Commitment tracker 2019-09-16 20:08:24 +02:00
whyrusleeping
ed45d1c2b4 refactor state utilities into StateManager package, implement proper election proofs 2019-09-09 19:14:00 -07:00
Łukasz Magiera
fe6d5ff3a8 retrieval: resolve some TODOs 2019-08-29 17:51:52 +02:00
Łukasz Magiera
28d3eb38eb basic retrieval content discovery 2019-08-29 17:48:19 +02:00
Łukasz Magiera
cad3efb9ba Command to list sealed blocks 2019-08-29 17:48:19 +02:00
Łukasz Magiera
e0dc17bc1a Correctly construct sealbstore 2019-08-29 17:48:19 +02:00
Łukasz Magiera
eda26faf21 Register miner address from storageminer process 2019-08-20 19:19:24 +02:00
Łukasz Magiera
4431dffc39 miner: Create miner in DI 2019-08-20 18:50:17 +02:00
Łukasz Magiera
e6493afd46 move poller to sector store 2019-08-14 23:34:43 +02:00
Łukasz Magiera
399f91940b Sector store 2019-08-14 22:27:32 +02:00
Whyrusleeping
e050d56594
Merge pull request from filecoin-project/feat/paych-manager
Implement payment channel manager
2019-08-13 12:13:23 -07:00
Łukasz Magiera
9733e1f011 deals: use GetWorkerAddr 2019-08-13 01:54:53 +02:00
whyrusleeping
6d52abcb2e WIP: wiring up the payment channel manager to the api 2019-08-12 10:54:17 -07:00
Łukasz Magiera
c79cb7bf31 deals: Almost sealing client data 2019-08-07 20:01:51 -07:00
Łukasz Magiera
b65041cac1 deals: Sending initial proposal works 2019-08-07 20:01:22 -07:00
Łukasz Magiera
ad9e433232 Start implementing storage deals 2019-08-07 20:00:49 -07:00
whyrusleeping
5c32725510 sector commitments make it all the way to the chain 2019-08-07 16:22:35 -07:00
Łukasz Magiera
9ae450620a Cleanup node/modules 2019-08-01 16:19:53 +02:00
Łukasz Magiera
6a4b9a6515 More separation for storage types in di modules 2019-08-01 16:14:16 +02:00
Łukasz Magiera
0f2334f513 chain sync: rebase 'fixes' 2019-07-31 10:43:14 -07:00
Łukasz Magiera
cdf0e0c858 chain: Test 'manual' sync 2019-07-31 10:43:13 -07:00
Łukasz Magiera
9ef5e1266e chain: Test basinc sync on generated chain 2019-07-31 10:43:13 -07:00
whyrusleeping
ee224e5b21 Implement 'storage miner' module, wire up a few bits it needs to start 2019-07-29 17:46:56 -07:00
whyrusleeping
259b38eb55 move sectorbuilder DI stuff to Online option 2019-07-29 12:11:55 -07:00
whyrusleeping
4fb4835b35 can we store it? YES WE CAN 2019-07-29 12:08:47 -07:00
whyrusleeping
f1432826d5 start integrating sector builder 2019-07-29 12:08:05 -07:00
whyrusleeping
0c67d66198 Make message wait work and use it for the storage-miner init 2019-07-29 11:58:30 -07:00
whyrusleeping
71baa5cbfe implement chain generator 2019-07-26 13:47:29 -07:00
whyrusleeping
78da356752 pull more things apart 2019-07-26 13:49:30 +02:00
Łukasz Magiera
8d58c0a2fd Only set genesis once 2019-07-25 01:12:22 +02:00
whyrusleeping
69d21d39a0 fix chainstore default option 2019-07-24 15:04:31 -07:00
whyrusleeping
4b76a06224 address code review 2019-07-24 14:46:54 -07:00
Łukasz Magiera
33dc14c9f2 Move API implementations to a separate package 2019-07-24 03:13:56 +02:00
Łukasz Magiera
772dd6c549 Implement storage miner API in node 2019-07-24 02:58:31 +02:00
Łukasz Magiera
f0e807dabb Fix jsonrpc client after splitting apis 2019-07-24 02:40:19 +02:00
Łukasz Magiera
eda03095b0 Separate API for storageminer 2019-07-24 02:09:34 +02:00
Łukasz Magiera
d0cbf02d36 Don't build full API in storage miner 2019-07-24 01:48:59 +02:00
Łukasz Magiera
8d529d1ae7 gofmt 2019-07-24 01:47:36 +02:00
Łukasz Magiera
fe147ce90d StorageMiner node type 2019-07-24 01:47:36 +02:00
Łukasz Magiera
0aadddb6c8 auth: Load JWT secret once 2019-07-24 00:38:52 +02:00
Łukasz Magiera
309ecc4052 auth: Put auth methods in API 2019-07-24 00:37:05 +02:00
Łukasz Magiera
fa4bf5178a auth: Restructure for getting tokens form keystore 2019-07-24 00:37:05 +02:00
Jakub Sztandera
935e4e8da7 Move KeyStore into separate module
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>
2019-07-18 18:10:53 +02:00
Jakub Sztandera
5ee7ba8420 Fix tests
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>
2019-07-18 17:52:48 +02:00
Jakub Sztandera
710f87fdf4 Integrate keystore with chain/wallet
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>
2019-07-18 17:41:30 +02:00
Łukasz Magiera
0fa47c6dfa Lint fixes 2019-07-16 18:07:08 +02:00
Łukasz Magiera
d30d9a30bc Fix test node setup 2019-07-16 18:02:51 +02:00
Łukasz Magiera
041598dbb6 Use extracted go-filestore 2019-07-15 17:08:18 +02:00
Łukasz Magiera
1f6629e978 Basic ClientListImports 2019-07-15 16:18:47 +02:00
Łukasz Magiera
97c61d4ce5 Client import 2019-07-15 16:17:34 +02:00
Whyrusleeping
3f61488e3c
Merge pull request from filecoin-project/feat/mining-1
Implement basic mining logic
2019-07-11 10:09:15 -07:00
whyrusleeping
13ba6d938b Remove unused variable 2019-07-10 21:56:39 -07:00
whyrusleeping
d381025ccc Implement initial mining logic 2019-07-10 19:36:43 -07:00
whyrusleeping
e09a379c3b add mpool pending command 2019-07-10 12:20:16 -07:00
Łukasz Magiera
1f97ad2c2e Use repo libp2p identiny 2019-07-10 19:43:19 +02:00
Łukasz Magiera
e639670195 cli: use repo api endpoint 2019-07-10 19:28:49 +02:00
Łukasz Magiera
14d515da3b Wire up memrepo 2019-07-10 17:38:35 +02:00
Łukasz Magiera
5ed695a84a Expose some node settings 2019-07-10 15:06:04 +02:00
Łukasz Magiera
4797e6a473 Don't import mocknet in builder.go 2019-07-10 15:03:40 +02:00
Łukasz Magiera
0a9ef94da5 Test connecting two nodes 2019-07-09 19:03:36 +02:00
Łukasz Magiera
4ac8eba59e Test API through jsonrpc 2019-07-09 18:36:40 +02:00
Łukasz Magiera
ba846e9bfb Wire up node API to tests 2019-07-09 18:27:07 +02:00
Łukasz Magiera
271c268e28 disable natPortMap for now 2019-07-09 16:30:54 +02:00
Łukasz Magiera
d852b3f7ef API struct instead of DI magic 2019-07-09 12:58:13 +02:00
Łukasz Magiera
cdca2ff2c7 Some net commands / apis 2019-07-08 21:07:16 +02:00
Łukasz Magiera
d1eb9073d1 Pubsub for blocks/messages 2019-07-08 16:07:09 +02:00
Łukasz Magiera
e162b5395c Construct enough to run hello 2019-07-08 15:36:53 +02:00
Jakub Sztandera
71dfa38032 Update linter
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>
2019-07-05 13:21:54 +02:00
Łukasz Magiera
fd7daf4a31 Fix import ordering 2019-07-05 12:06:28 +02:00
Łukasz Magiera
e78c378021 constructor: Add doc strings 2019-07-04 23:09:49 +02:00
Łukasz Magiera
de604065fb Rewrite constructor to functional opts 2019-07-04 19:34:06 +02:00
Jakub Sztandera
f08263662f Use config for listen addresses in libp2p
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>
2019-07-04 14:04:39 +02:00
Łukasz Magiera
4fcdd4a400 Address some lint warnings 2019-07-02 15:05:43 +02:00
Łukasz Magiera
7fdd369283 Address review 2019-07-02 14:40:25 +02:00
Jakub Sztandera
5736ac1d9d go fmt
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>
2019-07-01 23:11:34 +02:00
Łukasz Magiera
088c1e7860 Confgure PNet
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>
2019-07-01 23:11:33 +02:00
Łukasz Magiera
0779e7dc00 Build libp2p node
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>
2019-07-01 23:11:33 +02:00
Łukasz Magiera
795621ed27 Node builder structure
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>
2019-07-01 23:11:33 +02:00