Commit Graph

1298 Commits

Author SHA1 Message Date
Ian Norden
d79cc90cb2 unit test for ipfs fetcher 2019-12-02 13:24:58 -06:00
Ian Norden
069646905e converter unit test 2019-12-02 13:24:58 -06:00
Ian Norden
89faa85871 fix transaction signing, also removes need for rpc call to do it 2019-12-02 13:24:58 -06:00
Ian Norden
e72995605d possible fix for travis error 2019-12-02 13:24:58 -06:00
Ian Norden
90d685a763 switch from dep to go modules 2019-12-02 13:24:58 -06:00
Ian Norden
0bbb7a30d1 review fixes 2019-12-02 13:24:58 -06:00
Ian Norden
2244d1869f process receipts from payload instead of retreiving over additional rpc call 2019-12-02 13:24:58 -06:00
Ian Norden
13d09e15c1 update deps 2019-12-02 13:24:58 -06:00
Ian Norden
905585b212 backfill concurrently, one block at a time, while normal streaming continues 2019-12-02 13:24:58 -06:00
Ian Norden
f2efbb5d01 worker pool for handling concurrent ipld putting and cid indexing without overloading Postgres connections 2019-12-02 13:24:58 -06:00
Ian Norden
3fa33fb767 index receipts by the contract address 2019-12-02 13:24:58 -06:00
Ian Norden
23a21c14f3 make subscription config rlp encodable, group subs of the
same type using a hash of their config, process only once for each sub
type instead of for every sub.
2019-12-02 13:24:58 -06:00
Ian Norden
1d4d0cbc15 fixing dep bug 2019-12-02 13:24:58 -06:00
Ian Norden
b76bdc51f2 fixing dockerfile and start up script 2019-12-02 13:24:54 -06:00
Ian Norden
1235ec1f97 postgres table to hold ipfs data in same database as vulcanizedb data;
shared SeedNodeTransformer interface
2019-12-02 13:24:54 -06:00
Ian Norden
230e782e6c update to work with go-ipfs fork that doesn't use go modules and so can play nice with our forked dependencies; update documentation and dockerfile 2019-12-02 13:24:54 -06:00
Ian Norden
5356cd50bb seed node documentation 2019-12-02 13:24:51 -06:00
Ian Norden
4c81ca4d54 rename blocks table to eth_blocks so that we don't clash with the ipfs blocks table; dockerfile and startup_script for the seed node 2019-12-02 13:24:51 -06:00
Ian Norden
e7cdd6247e update to use ReceiptForStorage; expose rpc server over ws 2019-12-02 13:24:51 -06:00
Ian Norden
723c7c6244 fix backfill operations and dependency issue; hopefully travis will work now 2019-12-02 13:24:51 -06:00
Ian Norden
4baea2923c make db fks deferrable so that we can commit entire cid payload in single transaction; adjust buffer sizes to optimize performane and stability 2019-12-02 13:24:51 -06:00
Ian Norden
b1bb646ad5 goimports + streamSubscribe command for raw access to the seed node data 2019-12-02 13:24:51 -06:00
Ian Norden
8ccdfd4835 fix streamFilters issue 2019-12-02 13:24:51 -06:00
Ian Norden
b5099a5051 demo commands 2019-12-02 13:24:51 -06:00
Ian Norden
2db0ce971d vulcanizedb seed node streamer and syncPublishScreenAndServe command 2019-12-02 13:24:51 -06:00
Ian Norden
1dc46640cf request handler and response screener for seed node api service 2019-12-02 13:24:51 -06:00
Ian Norden
34393ffb3f fixes for issues uncovered in integration 2019-12-02 13:24:51 -06:00
Ian Norden
b83c0371d9 pub sub service and api 2019-12-02 13:24:50 -06:00
Ian Norden
5ebe2243d8 goimports -w 2019-12-02 13:24:50 -06:00
Ian Norden
671a324294 update to work with updated state diffing code 2019-12-02 13:24:50 -06:00
Ian Norden
3108957e5f ipfs fetcher 2019-12-02 13:24:50 -06:00
Ian Norden
15e044403d work on mocks and unit tests 2019-12-02 13:24:50 -06:00
Ian Norden
d702cb720c update dependencies to work with update eth-block-extractor 2019-12-02 13:24:49 -06:00
Ian Norden
31a9017c4f index all cids in a payload in a single atomic tx; misc fixes; comment additions 2019-12-02 13:24:46 -06:00
Ian Norden
79efaeb089 syncAndPublish command 2019-12-02 13:24:46 -06:00
Ian Norden
e033eabc28 cid database table migrations 2019-12-02 13:24:46 -06:00
Ian Norden
1921c06b02 repository - repository for indexing CIDs 2019-12-02 13:24:46 -06:00
Ian Norden
1fba4d0858 publisher - interface for publishing IPLD objects 2019-12-02 13:24:46 -06:00
Ian Norden
d248a1050e syncer - interface for syncing with a statediff subscription 2019-12-02 13:24:46 -06:00
Ian Norden
a31c7e9ced payload converter - interface for converting state diff payload into payload for ipld publishing 2019-12-02 13:24:46 -06:00
Ian Norden
35719b5180 indexer - top level interface for syncAndPublish 2019-12-02 13:24:46 -06:00
Ian Norden
c6c070005b subscribe method for the rpc client 2019-12-02 13:24:46 -06:00
Ian Norden
bc59aa4ed6 readme additions and vendor updates 2019-12-02 13:24:46 -06:00
Ian Norden
6434a7279d integrate backfill into storage watcher; documentation for storage backfill 2019-12-02 13:24:45 -06:00
Ian Norden
7f32c907fd integrate backfill into storage watcher; documentation for storage backfill 2019-12-02 13:24:45 -06:00
Ian Norden
1a208546d1
Merge pull request #157 from vulcanize/statediff_at
(VDB-933) Backfill for storage diff transformer
2019-12-02 12:56:26 -06:00
Ian Norden
313c5fe427 adjustments after rebase 2019-12-02 11:35:16 -06:00
Ian Norden
db0f024088 review fixes 2019-12-02 11:26:44 -06:00
Ian Norden
a834e55b9f command edits; external pkg for finding min deployment block; gofmt 2019-12-02 11:22:18 -06:00
Ian Norden
8562abd180 split backfill range up into smaller bins and process them concurrently; improve tests; review fixes 2019-12-02 11:20:49 -06:00