Ian Norden
5f85a7ecf6
more descriptive comments/method names for super node service; send msg
...
when backfill is finished
2020-02-23 17:15:26 -06:00
Ian Norden
48fb5bcd27
separate super node instances for different chains need to be within the same process to avoid contention over ipfs repo lock
2020-02-20 16:14:17 -06:00
Ian Norden
ef3b043f97
emulate btc data streamer over http; misc fixes
2020-02-20 16:14:17 -06:00
Ian Norden
5173edf563
fix broken go-ipld-eth trie node dag putters
2020-02-20 16:14:17 -06:00
Ian Norden
ca273a026d
major refactor part 2: remove cold import, full sync, generalize node table
2020-02-20 16:14:17 -06:00
Ian Norden
0f765df12c
finish constructors
2020-02-20 16:14:17 -06:00
Ian Norden
5d46e02509
tx inputs directly reference tx ouputs
2020-02-20 16:14:17 -06:00
Ian Norden
642e08a04b
btc converter, publisher, and indexer unit tests
2020-02-20 16:14:17 -06:00
Ian Norden
8643a7f3b6
btc ipld fetcher, paylaod fetcher, and ipld resolver
2020-02-20 16:14:17 -06:00
Ian Norden
aa2f8bd0a5
btc retriever and filterer
2020-02-20 16:14:17 -06:00
Ian Norden
f33cc3f34b
TxOutputs: extract and index pkscript metadata (script type, addresses, #required sigs); TxInputs: outpoint_hash => outpoint_tx_id that references transaction_cids.id
2020-02-20 16:14:17 -06:00
Ian Norden
48f70d4ddf
btc mocks; reorg
2020-02-20 16:14:17 -06:00
Ian Norden
808f1b5662
btc cid indexer
2020-02-20 16:14:17 -06:00
Ian Norden
5094b975fc
postgres schema and tables for btc
2020-02-20 16:14:17 -06:00
Ian Norden
9018e551ba
move eth tables under eth schema
2020-02-20 16:14:17 -06:00
Ian Norden
4dde90447e
wrap generic top-level interfaces with loose types
2020-02-20 16:14:17 -06:00
Ian Norden
68d8e68144
btc ipld publisher
2020-02-20 16:14:17 -06:00
Ian Norden
8099f726c3
internalized ipld types and dag_putters; adjust converters/publishers to work with them
2020-02-20 16:14:17 -06:00
Ian Norden
2e81f8d31a
btc payload converter
2020-02-20 16:14:17 -06:00
Ian Norden
076903b174
btc payload streamer
2020-02-20 16:14:17 -06:00
Ian Norden
da844b0b83
reorg pkg/ to prepare to support chains other than ethereumm
2020-02-20 16:14:16 -06:00
Ian Norden
33ac5978f5
split uncles into their own table; add tx index to the tx index (lol); continue work on geth api endpoints
2020-01-27 14:14:03 -06:00
Ian Norden
e52284b9ba
update documentation, bug/error fixes, work on eth api for drawbridge
2020-01-24 15:37:52 -06:00
Ian Norden
7843312815
major refactoring of super_node to make it easier to support other chains
2020-01-24 15:37:52 -06:00
Ian Norden
5ff05225c2
begin eth api recapitulation
2020-01-24 15:37:52 -06:00
Ian Norden
0785507a7d
support for total difficulty (needed to support some eth endpoints)
2020-01-24 15:37:52 -06:00
Ian Norden
358575335b
expand retriever, fetcher, and resolver interfaces in prep for eth api mimick
2020-01-24 15:37:52 -06:00
Rob Mulholand
56ce8bdb41
(VDB-950) Write raw diffs before transforming
...
- Raw field we can reference by FK for related data
- Enables replay for unwatched or mistransformed diffs
2019-12-03 14:51:17 -06:00
Ian Norden
74a90fd124
use Debugf instead of strconv
2019-12-02 23:37:01 -06:00
yaoandrew
5a1f599a4d
Clean up files after rebase
2019-12-02 23:36:00 -06:00
yaoandrew
614ea3bba1
Adds logging for storage and RPC
2019-12-02 23:10:32 -06:00
yaoandrew
0b9f29c300
Removes import alias for logrus
2019-12-02 23:08:48 -06:00
Ian Norden
4036d9d6a0
review fixes
2019-12-02 13:49:27 -06:00
Ian Norden
5be205ffa6
super node backfill breaks batch call into smaller bins; retrieve gap
...
test
2019-12-02 13:24:58 -06:00
Ian Norden
c16ac026db
logWithCommand; rebase fixes; config for testing super node subscription
2019-12-02 13:24:58 -06:00
Ian Norden
67df8dea77
header_cids.final => header_cids.uncle
2019-12-02 13:24:58 -06:00
Ian Norden
3a666df294
backfiller refactoring; explicity errs; golint
2019-12-02 13:24:58 -06:00
Ian Norden
40c3aff597
seed => super; port 80 => port 8080; backfill process for the super_node
2019-12-02 13:24:58 -06:00
Ian Norden
83fd76bc8a
fix flaky test
2019-12-02 13:24:58 -06:00
Ian Norden
b789ab69e6
adjustments after rebase
2019-12-02 13:24:58 -06:00
Ian Norden
8fe273fb7b
getter rpc method for the seed node's geth info
2019-12-02 13:24:58 -06:00
Ian Norden
2c997921cb
update dockerfile and documentation
2019-12-02 13:24:58 -06:00
Ian Norden
8b31d12716
fixes after rebase
2019-12-02 13:24:58 -06:00
Ian Norden
d2db5d0ac9
filterer unit test; that should be the last of them
2019-12-02 13:24:58 -06:00
Ian Norden
03dbe5dd22
resolver unit test
2019-12-02 13:24:58 -06:00
Ian Norden
825b0762cd
retriever unit test
2019-12-02 13:24:58 -06:00
Ian Norden
9ebc785d98
unit test for repository/indexer
2019-12-02 13:24:58 -06:00
Ian Norden
35c8f3561a
publisher unit test
2019-12-02 13:24:58 -06:00
Ian Norden
d79cc90cb2
unit test for ipfs fetcher
2019-12-02 13:24:58 -06:00
Ian Norden
069646905e
converter unit test
2019-12-02 13:24:58 -06:00
Ian Norden
89faa85871
fix transaction signing, also removes need for rpc call to do it
2019-12-02 13:24:58 -06:00
Ian Norden
e72995605d
possible fix for travis error
2019-12-02 13:24:58 -06:00
Ian Norden
0bbb7a30d1
review fixes
2019-12-02 13:24:58 -06:00
Ian Norden
2244d1869f
process receipts from payload instead of retreiving over additional rpc call
2019-12-02 13:24:58 -06:00
Ian Norden
13d09e15c1
update deps
2019-12-02 13:24:58 -06:00
Ian Norden
905585b212
backfill concurrently, one block at a time, while normal streaming continues
2019-12-02 13:24:58 -06:00
Ian Norden
f2efbb5d01
worker pool for handling concurrent ipld putting and cid indexing without overloading Postgres connections
2019-12-02 13:24:58 -06:00
Ian Norden
3fa33fb767
index receipts by the contract address
2019-12-02 13:24:58 -06:00
Ian Norden
23a21c14f3
make subscription config rlp encodable, group subs of the
...
same type using a hash of their config, process only once for each sub
type instead of for every sub.
2019-12-02 13:24:58 -06:00
Ian Norden
1235ec1f97
postgres table to hold ipfs data in same database as vulcanizedb data;
...
shared SeedNodeTransformer interface
2019-12-02 13:24:54 -06:00
Ian Norden
5356cd50bb
seed node documentation
2019-12-02 13:24:51 -06:00
Ian Norden
4c81ca4d54
rename blocks table to eth_blocks so that we don't clash with the ipfs blocks table; dockerfile and startup_script for the seed node
2019-12-02 13:24:51 -06:00
Ian Norden
e7cdd6247e
update to use ReceiptForStorage; expose rpc server over ws
2019-12-02 13:24:51 -06:00
Ian Norden
723c7c6244
fix backfill operations and dependency issue; hopefully travis will work now
2019-12-02 13:24:51 -06:00
Ian Norden
4baea2923c
make db fks deferrable so that we can commit entire cid payload in single transaction; adjust buffer sizes to optimize performane and stability
2019-12-02 13:24:51 -06:00
Ian Norden
b1bb646ad5
goimports + streamSubscribe command for raw access to the seed node data
2019-12-02 13:24:51 -06:00
Ian Norden
8ccdfd4835
fix streamFilters issue
2019-12-02 13:24:51 -06:00
Ian Norden
1dc46640cf
request handler and response screener for seed node api service
2019-12-02 13:24:51 -06:00
Ian Norden
34393ffb3f
fixes for issues uncovered in integration
2019-12-02 13:24:51 -06:00
Ian Norden
b83c0371d9
pub sub service and api
2019-12-02 13:24:50 -06:00
Ian Norden
5ebe2243d8
goimports -w
2019-12-02 13:24:50 -06:00
Ian Norden
671a324294
update to work with updated state diffing code
2019-12-02 13:24:50 -06:00
Ian Norden
3108957e5f
ipfs fetcher
2019-12-02 13:24:50 -06:00
Ian Norden
15e044403d
work on mocks and unit tests
2019-12-02 13:24:50 -06:00
Ian Norden
d702cb720c
update dependencies to work with update eth-block-extractor
2019-12-02 13:24:49 -06:00
Ian Norden
31a9017c4f
index all cids in a payload in a single atomic tx; misc fixes; comment additions
2019-12-02 13:24:46 -06:00
Ian Norden
1921c06b02
repository - repository for indexing CIDs
2019-12-02 13:24:46 -06:00
Ian Norden
1fba4d0858
publisher - interface for publishing IPLD objects
2019-12-02 13:24:46 -06:00
Ian Norden
d248a1050e
syncer - interface for syncing with a statediff subscription
2019-12-02 13:24:46 -06:00
Ian Norden
a31c7e9ced
payload converter - interface for converting state diff payload into payload for ipld publishing
2019-12-02 13:24:46 -06:00
Ian Norden
35719b5180
indexer - top level interface for syncAndPublish
2019-12-02 13:24:46 -06:00
Ian Norden
c6c070005b
subscribe method for the rpc client
2019-12-02 13:24:46 -06:00
Ian Norden
bc59aa4ed6
readme additions and vendor updates
2019-12-02 13:24:46 -06:00
Ian Norden
db0f024088
review fixes
2019-12-02 11:26:44 -06:00
Ian Norden
a834e55b9f
command edits; external pkg for finding min deployment block; gofmt
2019-12-02 11:22:18 -06:00
Ian Norden
563832422c
add err logging to eth/node
2019-12-02 11:07:43 -06:00
Rob Mulholand
4a98b7942f
Don't return unexported types
2019-10-31 13:42:19 -05:00
Rob Mulholand
e1236b4072
Prefer all caps for initialisms and acronyms
2019-10-31 13:42:19 -05:00
Rob Mulholand
eba868ff00
Drop else when if statement includes return
2019-10-31 13:33:39 -05:00
Rob Mulholand
0ded7bcd62
Prefer fmt.Errorf(s) to errors.New(fmt.Sprintf(s))
2019-10-31 13:33:39 -05:00
Rob Mulholand
234fff83c5
Use testing.toml and vulcanize_testing in tests
...
- remove legacy references to infura.toml and vulcanize_private
2019-10-31 13:10:25 -05:00
Rob Mulholand
e252229b8a
Add constraint to prevent duplicate headers
...
- Disallow inserts of headers with the same number, hash, and node
fingerprint, since it will enable duplicate log fetching for the
same header
2019-10-28 14:57:13 -05:00
Ian Norden
a2d249ca9d
review fixes
2019-10-28 11:40:32 -05:00
Ian Norden
4fbde836d4
log sql.ErrNoRows which I suspect is what is leading to the flaky test
2019-10-28 09:37:21 -05:00
Ian Norden
65808998b3
goimports -w; golinting, remove some unused code
2019-10-28 09:37:21 -05:00
Ian Norden
11b5efbfe3
fix for issue #146 ; mark header checked for contract if it doesnt have
...
any logs at that header but other contracts do; test
2019-10-28 09:34:42 -05:00
Edvard Hübinette
3fff2896aa
Rename geth to eth, signifying client independence ( #161 )
2019-10-28 12:30:24 +01:00
Edvard Hübinette
f7c4a6736d
VDB-919 Generalise converter ( #152 )
...
* Generalise transformer stack to use InsertionModel
* Add tests for event repository
* Restrict accepted values in InsertionModel
* Add call to repository.SetDB
* Improve error propagation/clarity on GetABI()
* Remove maker references in example
* Please golint
* refactor rollback error handling in repository
* Cleaner errors in repository, refactor tests
2019-10-28 11:48:31 +01:00
Rob Mulholand
2800e6df36
Enable contractWatcher without prior headerSync
...
- Previous setup would fail if there were no headers in the db. This
makes sense because we need headers that haven't been checked for
logs to exist so that we can fetch logs for those headers. But it
also prevents us from kicking off the headerSync and contractWatcher
processes concurrently. These changes enable kicking off both
processes at the same time with the idea that we will have unchecked
headers upon transformer execution.
2019-10-04 16:00:13 -05:00
Elizabeth
f6ab9382b2
Address repo updates ( #134 )
...
* Factor out get or create address into one sql string
* Factor out getChecksumAddress method in address repo
* Update address repo methods to not need a receiver
* Move address repository to libraries/shared
2019-10-03 11:17:08 -05:00