Commit Graph

3010 Commits

Author SHA1 Message Date
Łukasz Magiera
9fd183e74f
Merge pull request #500 from filecoin-project/feat/dt-implementation
Extract Data Transfer Implementation
2020-01-08 14:52:01 +01:00
Łukasz Magiera
31173ff4b4 Set logging levels in a single place 2020-01-08 14:49:34 +01:00
Łukasz Magiera
bf9aa4465b mod tidy 2020-01-08 14:20:43 +01:00
hannahhoward
5312340bf9 refactor(datatransfer): use independent repo
Use independent repo for go-data-transfer
2020-01-08 14:20:31 +01:00
hannahhoward
61e2568b8d feat(datatransfer): implement and extract
feat(datatransfer): setup implementation path

Sets up a path to implementation, offering both the dagservice implementation and a future graphsync
implement, establishes message interfaces and network layer, and isolates the datatransfer module
from the app

WIP using CBOR encoding for dataxfermsg

* Bring cbor-gen stuff into datatransfer package
* make transferRequest private struct
* add transferResponse + funcs
* Rename VoucherID to VoucherType
* more tests passing

WIP trying out some stuff
* Embed request/response in message so all the interfaces work AND the CBOR unmarshaling works: this is more like the spec anyway
* get rid of pb stuff

all message tests passing, some others in datatransfer

Some cleanup for PR

Cleanup for PR, clarifying and additional comments

mod tidy

Respond to PR comments:
* Make DataTransferRequest/Response be returned in from Net
* Regenerate cbor_gen and fix the generator caller so it works better
* Please the linters

Fix tests

Initiate push and pull requests (#536)

* add issue link for data TransferID generation
* comment out failing but not relevant tests
* finish voucher rename from Identifier --> Type

tests passing

cleanup for PR

remove unused fmt import in graphsync_test

a better reflection

send data transfer response

other tests passing

feat(datatransfer): milestone 2 infrastructure

Setup test path for all tickets for milestone 2

responses alert subscribers when request is not accepted (#607)

Graphsync response is scheduled when a valid push request is received (#625)

fix(datatransfer): fix tests

fix an error with read buffers in tests

fix(deps): fix go.sum

Feat/dt graphsync pullreqs (#627)

* graphsync responses to pull requests

Feat/dt initiator cleanup (#645)

* ChannelID.To --> ChannelID.Initiator
* We now store our peer ID (from host.ID()) so it can be used when creating ChannelIDs.
* InProgressChannels returns all of impl.channels, currently just for testing
* Implements go-data-transfer issue
* Some assertions were changed based on the above.
* Renamed some variables and added some assertions based on the new understanding
* Updated SHA for graphsync module
* Updated fakeGraphSync test structs to use new interfaces from new SHA above

Techdebt/dt split graphsync impl receiver (#651)

* Split up graphsyncImpl and graphsyncReceiver
* rename graphsync to utils

DTM sends data over graphsync for validated push requests (#665)

* create channels when a request is received. register push request hook with graphsync. fix tests.
* better NewReaders
* use mutex lock around impl.channels access
* fix(datatransfer): fix test uncertainty
* fix a data race and also don't use random bytes in basic block which can fail
* privatize 3 funcs

with @hannahhoward

Feat/dt gs pullrequests (#693)

* Implements DTM Sends Data Over Graphsync For Validated Pull Requests
* rename a field in a test struct
* refactor a couple of private functions (one was refactored out of existence)

Feat/dt subscribe, file Xfer round trip (#720)

Implements the rest of Subscriber Is Notified When Request Completed #24:
* send a graphsync message within a go func and consume responses until error or transfer is complete.
* notify subscribers of results.
* Rename datatransfer.Event to EventCode.
* datatransfer.Event is now a struct that includes a message and a timestamp as well as the Event.Code int, formerly Event, update all uses
* Add extension data to graphsync request hook, gsReq
* rename sendRequest to sendDtRequest, to distinguish it from sendGsRequest, where Dt = datatransfer, Gs = graphsync
* use a mutex lock for last transfer ID
* obey the linter

Don't respond with error in gsReqRcdHook when we can't find the datatransfer extension. (#754)

* update to correct graphsync version, update tests & code to call the new graphsync hooks
* getExtensionData returns an empty struct + nil if we can't find our extension
* Don't respond with error when we can't find the extension.
* Test for same
* mod tidy

minor fix to go.sum

feat(datatransfer): switch to graphsync implementation

Move over to real graphsync implementation of data transfer, add constructors for graphsync
instances on client and miner side

fix(datatransfer): Fix validators

Validators were checking payload cid against commP -- which are not the same any more. Added a
payloadCid to client deal to maintain the record, fixed validator logic

Feat/dt extraction use go-fil-components/datatransfer (#770)

* Initial commit after changing to go-fil-components/datatransfer
* blow away the datatransfer dir
* use go-fil-components master after its PR #1 was merged
* go mod tidy

use a package

updates after rebase with master
2020-01-08 14:19:28 +01:00
Łukasz Magiera
1392dcc273 Merge remote-tracking branch 'origin/master' into feat/testnet2 2020-01-08 13:53:01 +01:00
Łukasz Magiera
117bea4e4e
Merge pull request #1030 from filecoin-project/feat/chain-list-messages
implement list-messages command
2020-01-08 13:50:55 +01:00
Łukasz Magiera
86a22a3896
Merge pull request #1029 from filecoin-project/feat/mpoolpush-cid
mpool: Return CID in MpoolPush
2020-01-08 13:43:03 +01:00
whyrusleeping
95a89ad192 implement list-messages command 2020-01-08 13:41:23 +01:00
Łukasz Magiera
5153f67a2e
Merge pull request #1033 from filecoin-project/feat/sync-mark-bad
Add command to mark given blocks as bad
2020-01-08 13:35:34 +01:00
Łukasz Magiera
4de01190a1
api: Require admin perm for SyncMarkBad 2020-01-08 13:34:53 +01:00
Jakub Sztandera
d213bd1fee
Merge pull request #1025 from filecoin-project/fix/faults-counting
Fix faults handling
2020-01-08 11:03:14 +01:00
whyrusleeping
80c19df4bb go mod tidy 2020-01-07 21:45:10 -08:00
whyrusleeping
897cf72933 Add command to mark given blocks as bad 2020-01-07 21:42:15 -08:00
Whyrusleeping
11bfab67f9
Merge pull request #986 from filecoin-project/feat/update-stats-to-golog
Use golog in stats tool, and better sync control
2020-01-07 21:38:56 -08:00
whyrusleeping
62db7a6dda fix build 2020-01-07 21:10:43 -08:00
Łukasz Magiera
a11e8093e5 mpool: Return CID in MpoolPush 2020-01-07 21:10:30 -08:00
whyrusleeping
3965e1053e fix build 2020-01-07 21:05:45 -08:00
whyrusleeping
e41dbdad90 reduce most vm errors to non-fatal, explicitly mark disk issues as fatal 2020-01-07 19:46:55 -08:00
Jakub Sztandera
64496690bb
go mod tidy
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-01-07 23:09:46 +01:00
Jakub Sztandera
f572cdea47
Fix logic, improve tests
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-01-07 23:03:58 +01:00
Jakub Sztandera
5e775929be
Cleanup handing and add tests
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-01-07 23:03:58 +01:00
Jakub Sztandera
65724133ea
Add tests
,
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-01-07 23:03:58 +01:00
Jakub Sztandera
a63c34a902
Handle faults in ss and pss correctly
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-01-07 23:03:18 +01:00
Jakub Sztandera
dbc729d049
Validate length of RLE
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-01-07 22:54:43 +01:00
Łukasz Magiera
7b134b8317
Merge pull request #1031 from filecoin-project/feat/block-message-limit
introduce a limit on the number of messages a block can have
2020-01-07 21:48:57 +01:00
Łukasz Magiera
22e9815928 Merge remote-tracking branch 'origin/master' into feat/testnet2 2020-01-07 21:45:30 +01:00
whyrusleeping
a109ae1490 introduce a limit on the number of messages a block can have 2020-01-07 12:41:26 -08:00
Łukasz Magiera
f019b80af0
Merge pull request #998 from filecoin-project/feat/move_sectorbuilder_to_module
replace sectorbuilder with go-sectorbuilder
2020-01-07 18:03:58 +01:00
laser
0ffec0a9ad go mod tidy 2020-01-07 08:27:08 -08:00
laser
841f9c3627 reorganize imports to pass linter 2020-01-07 08:23:12 -08:00
laser
20321bb476 handle rebase 2020-01-07 08:18:35 -08:00
laser
4bbc693d9c go mod tidy 2020-01-07 07:52:19 -08:00
laser
3ae1bd5b78 add paramfetch import to lotus-bench 2020-01-07 07:52:06 -08:00
laser
ce837da935 point go-sectorbuilder to head of master 2020-01-07 07:48:53 -08:00
laser
1b4ddefe27 manage Groth parameter cache keys by hand
I've moved paramfetch.go and parameters.json into the go-sectorbuilder module, so the CircleCI checksum-calculating functions won't work as-written. We'll need to manage the cache key manually.
2020-01-07 07:48:44 -08:00
laser
abf6acc211 moved logger from paramfetch.go (no longer in lotus) to genesis.go 2020-01-07 07:48:44 -08:00
acruikshank
7873213b11 switch sectorbuilder from lib to external module
point to correct version of sectorbuilder + ensure go-datastore stays at 0.1.1

replace local address type with go-address

consume paramfetch.GetParams instead of local paramfetch.go

remove constants now defined in sectorbuilder
2020-01-07 07:48:43 -08:00
Łukasz Magiera
97ae184d0a Merge remote-tracking branch 'origin/master' into feat/testnet2 2020-01-07 15:06:35 +01:00
Łukasz Magiera
8f0c02043b
Merge pull request #992 from filecoin-project/feat/extract-cborutil
Extract cborutil library
2020-01-07 15:04:08 +01:00
Łukasz Magiera
1e7f10e387 gofmt 2020-01-07 15:00:10 +01:00
hannahhoward
718acb0a86 refactor(cborutil): extract cborutil library
Use extracted cborutil library
2020-01-07 14:59:35 +01:00
Łukasz Magiera
85097a871f
Merge pull request #991 from filecoin-project/feat/extract-address
Extract address type & crypto utils
2020-01-07 14:57:39 +01:00
hannahhoward
14ecd1929b refactor(crypto): use extracted crypto library 2020-01-07 14:53:30 +01:00
hannahhoward
8418464d91 refactor(address): use extracted address library
Switch to using extracted address library
2020-01-07 14:53:27 +01:00
Łukasz Magiera
f6b96480d1
Merge pull request #1008 from steven004/EnhanceRle
Enhance rleplus testing
2020-01-07 14:44:02 +01:00
Łukasz Magiera
29af953cec
Merge pull request #1013 from adam-hanna/master
adds fedora installation instructions
2020-01-07 14:34:51 +01:00
Łukasz Magiera
8e407acd76
Merge pull request #1020 from filecoin-project/feat/consume-extracted-paramfetch
consume newly-extracted paramfetch
2020-01-07 14:33:15 +01:00
Łukasz Magiera
2635028933
Merge pull request #1024 from filecoin-project/fix/sizestr-overflow
Fix SizeStr index out of band
2020-01-07 14:30:15 +01:00
Łukasz Magiera
605562467d Flatten fork code for testnet2 2020-01-06 19:39:57 +01:00