cerc-io/ipld-eth-server
18
0
Fork 1
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 37 Wiki Activity
1,775 Commits 12 Branches 52 Tags 80 MiB
ba5cdecf93
Commit Graph

916 Commits

Author SHA1 Message Date
Ian Norden
1e2b09045a fix state and storage iplds/dag_putters; add support for eth rct trie and eth and btc tx tries 2020-03-17 11:18:20 -05:00
Ian Norden
e7225317d4 split btc and eth super node processes into serpate containers; dockerfile for Postgraphile with plugins 2020-03-17 08:37:00 -05:00
Ian Norden
36b714b6df resync config initializer 2020-03-17 07:51:00 -05:00
Ian Norden
b934f6c963 btc and eth unit tests for db cleaning 2020-03-17 07:50:38 -05:00
Ian Norden
f814b64543 bug fix 2020-03-12 13:59:23 -05:00
Ian Norden
77b1c25bc1 resync service 2020-03-12 13:52:23 -05:00
Ian Norden
b4005eff39 db cleaners to eth and btc 2020-03-12 13:51:58 -05:00
Ian Norden
e5c5422edc adhjustments to work with statediffing geth v1.10-alpha.2 2020-03-11 13:41:59 -05:00
Ian Norden
be875c0100 db comments to resolve postgraphile type conflicts 2020-03-10 09:38:07 -05:00
Ian Norden
aad318c67b improved logging in super node service 2020-03-08 11:47:49 -05:00
Ian Norden
8c2a71b16f finish wasm watcher engine 2020-02-27 15:09:20 -06:00
Ian Norden
330a083749 watcher configuration; super node versioning 2020-02-25 16:38:46 -06:00
Ian Norden
25aa4634e9 work on wasm watchers 2020-02-24 12:54:10 -06:00
Ian Norden
e3e8700d34 api test; return full ipld models to subscribers 2020-02-23 17:15:26 -06:00
Ian Norden
fb360d8562 changes to super node to improve compatibility with watcher 2020-02-23 17:15:26 -06:00
Ian Norden
f0c5ff8077 begin wasm watcher engine 2020-02-23 17:15:26 -06:00
Ian Norden
94aefafd7c more explicity payload err and msg 2020-02-23 17:15:26 -06:00
Ian Norden
2fbf97da9d wasm instantiator 2020-02-23 17:15:26 -06:00
Ian Norden
5f85a7ecf6 more descriptive comments/method names for super node service; send msg 
when backfill is finished
 2020-02-23 17:15:26 -06:00
Ian Norden
48fb5bcd27 separate super node instances for different chains need to be within the same process to avoid contention over ipfs repo lock 2020-02-20 16:14:17 -06:00
Ian Norden
ef3b043f97 emulate btc data streamer over http; misc fixes 2020-02-20 16:14:17 -06:00
Ian Norden
5173edf563 fix broken go-ipld-eth trie node dag putters 2020-02-20 16:14:17 -06:00
Ian Norden
ca273a026d major refactor part 2: remove cold import, full sync, generalize node table 2020-02-20 16:14:17 -06:00
Ian Norden
0f765df12c finish constructors 2020-02-20 16:14:17 -06:00
Ian Norden
5d46e02509 tx inputs directly reference tx ouputs 2020-02-20 16:14:17 -06:00
Ian Norden
642e08a04b btc converter, publisher, and indexer unit tests 2020-02-20 16:14:17 -06:00
Ian Norden
8643a7f3b6 btc ipld fetcher, paylaod fetcher, and ipld resolver 2020-02-20 16:14:17 -06:00
Ian Norden
aa2f8bd0a5 btc retriever and filterer 2020-02-20 16:14:17 -06:00
Ian Norden
f33cc3f34b TxOutputs: extract and index pkscript metadata (script type, addresses, #required sigs); TxInputs: outpoint_hash => outpoint_tx_id that references transaction_cids.id 2020-02-20 16:14:17 -06:00
Ian Norden
48f70d4ddf btc mocks; reorg 2020-02-20 16:14:17 -06:00
Ian Norden
808f1b5662 btc cid indexer 2020-02-20 16:14:17 -06:00
Ian Norden
5094b975fc postgres schema and tables for btc 2020-02-20 16:14:17 -06:00
Ian Norden
9018e551ba move eth tables under eth schema 2020-02-20 16:14:17 -06:00
Ian Norden
4dde90447e wrap generic top-level interfaces with loose types 2020-02-20 16:14:17 -06:00
Ian Norden
68d8e68144 btc ipld publisher 2020-02-20 16:14:17 -06:00
Ian Norden
8099f726c3 internalized ipld types and dag_putters; adjust converters/publishers to work with them 2020-02-20 16:14:17 -06:00
Ian Norden
2e81f8d31a btc payload converter 2020-02-20 16:14:17 -06:00
Ian Norden
076903b174 btc payload streamer 2020-02-20 16:14:17 -06:00
Ian Norden
da844b0b83 reorg pkg/ to prepare to support chains other than ethereumm 2020-02-20 16:14:16 -06:00
Ian Norden
33ac5978f5 split uncles into their own table; add tx index to the tx index (lol); continue work on geth api endpoints 2020-01-27 14:14:03 -06:00
Ian Norden
e52284b9ba update documentation, bug/error fixes, work on eth api for drawbridge 2020-01-24 15:37:52 -06:00
Ian Norden
7843312815 major refactoring of super_node to make it easier to support other chains 2020-01-24 15:37:52 -06:00
Ian Norden
5ff05225c2 begin eth api recapitulation 2020-01-24 15:37:52 -06:00
Ian Norden
0785507a7d support for total difficulty (needed to support some eth endpoints) 2020-01-24 15:37:52 -06:00
Ian Norden
358575335b expand retriever, fetcher, and resolver interfaces in prep for eth api mimick 2020-01-24 15:37:52 -06:00
Rob Mulholand
56ce8bdb41 (VDB-950) Write raw diffs before transforming 
- Raw field we can reference by FK for related data
- Enables replay for unwatched or mistransformed diffs
 2019-12-03 14:51:17 -06:00
Ian Norden
74a90fd124 use Debugf instead of strconv 2019-12-02 23:37:01 -06:00
yaoandrew
5a1f599a4d Clean up files after rebase 2019-12-02 23:36:00 -06:00
yaoandrew
614ea3bba1 Adds logging for storage and RPC 2019-12-02 23:10:32 -06:00
yaoandrew
0b9f29c300 Removes import alias for logrus 2019-12-02 23:08:48 -06:00
Ian Norden
4036d9d6a0 review fixes 2019-12-02 13:49:27 -06:00
Ian Norden
5be205ffa6 super node backfill breaks batch call into smaller bins; retrieve gap 
test
 2019-12-02 13:24:58 -06:00
Ian Norden
c16ac026db logWithCommand; rebase fixes; config for testing super node subscription 2019-12-02 13:24:58 -06:00
Ian Norden
67df8dea77 header_cids.final => header_cids.uncle 2019-12-02 13:24:58 -06:00
Ian Norden
3a666df294 backfiller refactoring; explicity errs; golint 2019-12-02 13:24:58 -06:00
Ian Norden
40c3aff597 seed => super; port 80 => port 8080; backfill process for the super_node 2019-12-02 13:24:58 -06:00
Ian Norden
83fd76bc8a fix flaky test 2019-12-02 13:24:58 -06:00
Ian Norden
b789ab69e6 adjustments after rebase 2019-12-02 13:24:58 -06:00
Ian Norden
8fe273fb7b getter rpc method for the seed node's geth info 2019-12-02 13:24:58 -06:00
Ian Norden
2c997921cb update dockerfile and documentation 2019-12-02 13:24:58 -06:00
Ian Norden
8b31d12716 fixes after rebase 2019-12-02 13:24:58 -06:00
Ian Norden
d2db5d0ac9 filterer unit test; that should be the last of them 2019-12-02 13:24:58 -06:00
Ian Norden
03dbe5dd22 resolver unit test 2019-12-02 13:24:58 -06:00
Ian Norden
825b0762cd retriever unit test 2019-12-02 13:24:58 -06:00
Ian Norden
9ebc785d98 unit test for repository/indexer 2019-12-02 13:24:58 -06:00
Ian Norden
35c8f3561a publisher unit test 2019-12-02 13:24:58 -06:00
Ian Norden
d79cc90cb2 unit test for ipfs fetcher 2019-12-02 13:24:58 -06:00
Ian Norden
069646905e converter unit test 2019-12-02 13:24:58 -06:00
Ian Norden
89faa85871 fix transaction signing, also removes need for rpc call to do it 2019-12-02 13:24:58 -06:00
Ian Norden
e72995605d possible fix for travis error 2019-12-02 13:24:58 -06:00
Ian Norden
0bbb7a30d1 review fixes 2019-12-02 13:24:58 -06:00
Ian Norden
2244d1869f process receipts from payload instead of retreiving over additional rpc call 2019-12-02 13:24:58 -06:00
Ian Norden
13d09e15c1 update deps 2019-12-02 13:24:58 -06:00
Ian Norden
905585b212 backfill concurrently, one block at a time, while normal streaming continues 2019-12-02 13:24:58 -06:00
Ian Norden
f2efbb5d01 worker pool for handling concurrent ipld putting and cid indexing without overloading Postgres connections 2019-12-02 13:24:58 -06:00
Ian Norden
3fa33fb767 index receipts by the contract address 2019-12-02 13:24:58 -06:00
Ian Norden
23a21c14f3 make subscription config rlp encodable, group subs of the 
same type using a hash of their config, process only once for each sub
type instead of for every sub.
 2019-12-02 13:24:58 -06:00
Ian Norden
1235ec1f97 postgres table to hold ipfs data in same database as vulcanizedb data; 
shared SeedNodeTransformer interface
 2019-12-02 13:24:54 -06:00
Ian Norden
5356cd50bb seed node documentation 2019-12-02 13:24:51 -06:00
Ian Norden
4c81ca4d54 rename blocks table to eth_blocks so that we don't clash with the ipfs blocks table; dockerfile and startup_script for the seed node 2019-12-02 13:24:51 -06:00
Ian Norden
e7cdd6247e update to use ReceiptForStorage; expose rpc server over ws 2019-12-02 13:24:51 -06:00
Ian Norden
723c7c6244 fix backfill operations and dependency issue; hopefully travis will work now 2019-12-02 13:24:51 -06:00
Ian Norden
4baea2923c make db fks deferrable so that we can commit entire cid payload in single transaction; adjust buffer sizes to optimize performane and stability 2019-12-02 13:24:51 -06:00
Ian Norden
b1bb646ad5 goimports + streamSubscribe command for raw access to the seed node data 2019-12-02 13:24:51 -06:00
Ian Norden
8ccdfd4835 fix streamFilters issue 2019-12-02 13:24:51 -06:00
Ian Norden
1dc46640cf request handler and response screener for seed node api service 2019-12-02 13:24:51 -06:00
Ian Norden
34393ffb3f fixes for issues uncovered in integration 2019-12-02 13:24:51 -06:00
Ian Norden
b83c0371d9 pub sub service and api 2019-12-02 13:24:50 -06:00
Ian Norden
5ebe2243d8 goimports -w 2019-12-02 13:24:50 -06:00
Ian Norden
671a324294 update to work with updated state diffing code 2019-12-02 13:24:50 -06:00
Ian Norden
3108957e5f ipfs fetcher 2019-12-02 13:24:50 -06:00
Ian Norden
15e044403d work on mocks and unit tests 2019-12-02 13:24:50 -06:00
Ian Norden
d702cb720c update dependencies to work with update eth-block-extractor 2019-12-02 13:24:49 -06:00
Ian Norden
31a9017c4f index all cids in a payload in a single atomic tx; misc fixes; comment additions 2019-12-02 13:24:46 -06:00
Ian Norden
1921c06b02 repository - repository for indexing CIDs 2019-12-02 13:24:46 -06:00
Ian Norden
1fba4d0858 publisher - interface for publishing IPLD objects 2019-12-02 13:24:46 -06:00
Ian Norden
d248a1050e syncer - interface for syncing with a statediff subscription 2019-12-02 13:24:46 -06:00
Ian Norden
a31c7e9ced payload converter - interface for converting state diff payload into payload for ipld publishing 2019-12-02 13:24:46 -06:00
Ian Norden
35719b5180 indexer - top level interface for syncAndPublish 2019-12-02 13:24:46 -06:00
Ian Norden
c6c070005b subscribe method for the rpc client 2019-12-02 13:24:46 -06:00
Ian Norden
bc59aa4ed6 readme additions and vendor updates 2019-12-02 13:24:46 -06:00
Ian Norden
db0f024088 review fixes 2019-12-02 11:26:44 -06:00
Ian Norden
a834e55b9f command edits; external pkg for finding min deployment block; gofmt 2019-12-02 11:22:18 -06:00
Ian Norden
563832422c add err logging to eth/node 2019-12-02 11:07:43 -06:00
Rob Mulholand
4a98b7942f Don't return unexported types 2019-10-31 13:42:19 -05:00
Rob Mulholand
e1236b4072 Prefer all caps for initialisms and acronyms 2019-10-31 13:42:19 -05:00
Rob Mulholand
eba868ff00 Drop else when if statement includes return 2019-10-31 13:33:39 -05:00
Rob Mulholand
0ded7bcd62 Prefer fmt.Errorf(s) to errors.New(fmt.Sprintf(s)) 2019-10-31 13:33:39 -05:00
Rob Mulholand
234fff83c5 Use testing.toml and vulcanize_testing in tests 
- remove legacy references to infura.toml and vulcanize_private
 2019-10-31 13:10:25 -05:00
Rob Mulholand
e252229b8a Add constraint to prevent duplicate headers 
- Disallow inserts of headers with the same number, hash, and node
  fingerprint, since it will enable duplicate log fetching for the
  same header
 2019-10-28 14:57:13 -05:00
Ian Norden
a2d249ca9d review fixes 2019-10-28 11:40:32 -05:00
Ian Norden
4fbde836d4 log sql.ErrNoRows which I suspect is what is leading to the flaky test 2019-10-28 09:37:21 -05:00
Ian Norden
65808998b3 goimports -w; golinting, remove some unused code 2019-10-28 09:37:21 -05:00
Ian Norden
11b5efbfe3 fix for issue #146; mark header checked for contract if it doesnt have 
any logs at that header but other contracts do; test
 2019-10-28 09:34:42 -05:00
Edvard Hübinette
3fff2896aa
Rename geth to eth, signifying client independence (#161) 2019-10-28 12:30:24 +01:00
Edvard Hübinette
f7c4a6736d
VDB-919 Generalise converter (#152) 
* Generalise transformer stack to use InsertionModel

* Add tests for event repository

* Restrict accepted values in InsertionModel

* Add call to repository.SetDB

* Improve error propagation/clarity on GetABI()

* Remove maker references in example

* Please golint

* refactor rollback error handling in repository

* Cleaner errors in repository, refactor tests
 2019-10-28 11:48:31 +01:00
Rob Mulholand
2800e6df36 Enable contractWatcher without prior headerSync 
- Previous setup would fail if there were no headers in the db. This
  makes sense because we need headers that haven't been checked for
  logs to exist so that we can fetch logs for those headers. But it
  also prevents us from kicking off the headerSync and contractWatcher
  processes concurrently. These changes enable kicking off both
  processes at the same time with the idea that we will have unchecked
  headers upon transformer execution.
 2019-10-04 16:00:13 -05:00
Elizabeth
f6ab9382b2
Address repo updates (#134) 
* Factor out get or create address into one sql string

* Factor out getChecksumAddress method in address repo

* Update address repo methods to not need a receiver

* Move address repository to libraries/shared
 2019-10-03 11:17:08 -05:00
Rob Mulholand
4505382590 Increase logging in contract watcher 
- Focus on header mode
- Add context to errors, trace guard clauses, warn on non-returned
  errors
- Give errors distinct names so compiler will recognize if unchecked
- Remove redundant type declarations/fix typos
 2019-10-02 15:14:42 -05:00
Rob Mulholand
2ff88de859 Extract helper for converting hex to keccak256 hash 
- Also prefer crypto.Keccak256Hash(x) to common.BytesToHash(crypto.Keccak256(x))
 2019-09-25 16:36:08 -05:00
Elizabeth Engelman
8e166f2a56 Update after rebasing with staging 2019-09-25 16:32:31 -05:00
Elizabeth Engelman
ee244ac6f5 Apply go fmt 2019-09-25 16:32:27 -05:00
Elizabeth Engelman
34f8e16c11 Add a Subscribe method to rpc client interface 2019-09-25 16:32:27 -05:00
Elizabeth Engelman
36533f7c3f Update vendor directory and make necessary code changes 
Fixes for new geth version
 2019-09-25 16:32:27 -05:00
Matt Griswold
20ce0ab852 Revert "option to write the plugin .go file and run the plugin migrations without building the .so file so that it can be done from the shell instead to afford complete error messages" 
This reverts commit 9bb2f27a69.
 2019-09-24 23:54:25 +00:00
Ian Norden
9bb2f27a69 option to write the plugin .go file and run the plugin migrations without building the .so file so that it can be done from the shell instead to afford complete error messages 2019-09-24 18:44:04 -05:00
Rob Mulholand
c21c069b69 Replace magic numbers with variables 2019-09-23 16:13:45 -05:00
Rob Mulholand
eea0b0bdca Return missing headers after starting block number 
- Previously, we required that the first missing header match the
  configured starting block number. This helps guarantee that we have
  the necessary data for method polling in memory, but prevents the
  process from moving forward if restarted after the starting block
  has already been checked.
 2019-09-23 16:13:45 -05:00
Rob Mulholand
37e2673a8d Do nothing on duplicate log insert 
- Prevents duplicate key constraint violation from blocking the process
  from moving forward on restart.
- If header_id, log_idx, and tx_idx are the same, we can safely do
  nothing since it's definitely the same log - a reorg would cause the
  original header to be replaced with a new ID.
 2019-09-23 16:02:07 -05:00
Rob Mulholand
3466a51b4d Correctly parse uint8 values in contract watcher 
- calling string(uint8) => insert errors on numeric columns
 2019-09-20 21:37:59 -05:00
Rob Mulholand
749db40b96 Parse bytes32 vals in Contract Watcher 2019-09-20 13:48:43 -05:00
Rob Mulholand
f83e996ab8 Update comments in checked headers repository 2019-09-18 21:39:34 -05:00
Rob Mulholand
2b798e00e0 Cap random fake timestamp 2019-09-13 11:20:13 -05:00
Rob Mulholand
3f9b034c4c Rename checked_logs => watched_logs 
- We're logging that a given log has been included in any fetch calls
  for checked headers, rather than that we have already checked for
  that log
 2019-09-10 21:22:14 -05:00
Rob Mulholand
13d503b851 Distinguish between missing and unchecked headers 
- Missing == not in DB
- Unchecked == logs haven't been fetched
 2019-09-10 14:39:45 -05:00
Rob Mulholand
ce91b0d9e6 Simplify checked header repository tests 
- Use assertions instead of comments to document expectations
- Also randomize the fake timestamp in test data
 2019-09-10 13:47:43 -05:00
Rob Mulholand
b9f3b9f946 Reference header sync logs address via foreign key 2019-08-28 10:41:34 -05:00
Rob Mulholand
5ac76eee74 Make check_count a column on public.headers 
- Don't need to maintain it on public.checked_headers if we're not
  adding additional columns to that table
 2019-08-28 09:50:17 -05:00
Rob Mulholand
666ea1c325 Update checked headers for new transformers 
- If a header was marked as checked before a transformer was added to
  the watcher, mark all headers since the new transformer's starting
  block number as unchecked.
 2019-08-28 09:25:14 -05:00
Rob Mulholand
63dabbb051 Extract and delegate logs concurrently 2019-08-28 09:25:13 -05:00
Rob Mulholand
d496dad33c Decouple log extraction from transformer delegation 
- limit missing headers results set to 100 so that extraction doesn't
  excessively block delegation
- wrap checked headers functions in repository struct
- move storage repository to factory, to correspond with event
  repository path
- remove unused files
- reformat sql
- remove line breaks in imports
 2019-08-28 09:25:13 -05:00
Rob Mulholand
cb819fa9a6 Write event logs to database before transforming 
- enables decoupling event extraction/persistence from transformation
- modifies event transformer, converter, and log chunker to accept
  payload that includes internal log database ID with log data
- remove alias for transformer pkg as shared_t
- remove unused mock watcher repository
 2019-08-28 09:13:44 -05:00
Rob Mulholand
3693ed905f Rename logs to full sync logs 
- Enable creating new table for logs used in event watching based on
  header sync
 2019-08-28 09:11:06 -05:00
Elizabeth Engelman
edc0bdf668 Address more PR feedback 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
1b3786338f Address PR feedback 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
58c23c6632 Update Copyrights 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
38f6847ff2 Create a HeaderSyncReceiptRepository 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
7e38764618 Rename ReceiptRepository -> FullSyncReceiptRepository 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
4e40e892d2 Updating header_sync_receipts to have FK reference to addresses 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
bcd6d14fcd Updating full_sync_receipts to have FK reference to addresses 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
258035833b Add GetAddressById 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
4b61c87b55 Get or create address record in a transaction 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
1373fe83a1 Pass a db into GetOrCreateAddress 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
b13eefbce7 Add AddressRepository 2019-08-23 10:11:37 -05:00
Rob Mulholand
987abd4b2e Update Geth to 1.9.0 2019-07-23 15:26:18 -05:00
Andrew J Yao
ee77fc6521 Remove injection of SubCommand for logs 2019-07-23 13:11:20 -07:00
Andrew J Yao
92d153b010 Add subCommand name to log fields 2019-07-23 08:09:57 -07:00
Edvard Hübinette
2c092e8d04
[VDB-751 VDB-754] Bugfix null pointer panic and improve logging (#119) 
* VDB-751 VDB-754 Bugfix null pointer panic and improve logging

* Fix typo
 2019-07-18 09:21:40 +02:00
Rob Mulholand
0f95267e84 Use goose library for running plugin migrations 
- Removes dependency on goose binary existing in plugin execution
  environment
 2019-06-27 13:58:58 -05:00
Rob Mulholand
f0bcf930f4 Separate tmp migrations dir from home in join 
- enable separator between home and migrations dir
 2019-06-27 13:58:58 -05:00
Rob Mulholand
0558792282 Log error on failure to setup plugin migrations 
- include specific error
 2019-06-27 13:58:58 -05:00
Andrew J Yao
11a65c525c Adds ability to set log level from toml or CLI args 2019-06-27 11:14:11 -07:00
Elizabeth Engelman
fa03716cb2 Address small PR comments 2019-05-10 11:54:15 -05:00
Elizabeth Engelman
e1a0d894a2 Update blockchain method GetHeaderByNumbers -> GetHeadersByNumbers 2019-05-08 13:42:43 -05:00
Elizabeth Engelman
ba81766f6c Small spelling fix 2019-05-08 13:42:43 -05:00
Rob Mulholand
76ab914bdc Add license 2019-05-01 12:32:39 -05:00
Rob Mulholand
2d684c5aec Extract storage diff fetching behind an interface 
- Replaces directly reading from a CSV
- Simplifies testing
- Should hopefully make it easier to plug in other sources for storage
  diffs (e.g. differently formatted CSVs, JSON RPC, etc)
 2019-05-01 12:30:36 -05:00
Gabe Laughlin
a344432156
(VDB-560) Rename lightSync to headerSync 2019-05-01 12:12:55 -05:00
Rob Mulholand
3fd6269b78 Don't lookup transactions if no log events 
- Prevents EOF error on transactions lookup
 2019-04-18 09:53:18 -05:00
ana0
964fc8bd22 Merge branch 'staging' into databaseConfig 2019-04-11 14:19:57 -04:00
anon
bd12149594 Database config 2019-04-10 17:56:31 -04:00
Gabe Laughlin
8903b6fb2d Add timestamp customization for fake header 2019-04-09 17:08:15 -05:00
Ian Norden
9befc76fc6 remove LIMIT 100; method to check continuity of headers returned but doesn't require 
the set to start at a specific block number; config for
account_transformer; review fixes; update schema
 2019-04-05 17:21:59 -05:00
Ian Norden
c1940c3e58 full_sync and light_sync receipt tables and light receipt repository methods and test 2019-04-05 16:36:48 -05:00
Ian Norden
04ad7b7696 Merge pull request #47 from vulcanize/track_uncle_rewards 
Track uncle rewards + direct balance polling
 2019-04-04 15:55:40 -05:00
Ian Norden
5dcf534b2c review fixes 2019-04-04 15:21:39 -05:00
Ian Norden
72c73c25b7 update vendor dep; constrain to revision of geth with UnpackIntoMap; updates to match 2019-04-03 00:14:02 -05:00
Ian Norden
197f98c93d going ahead and indexing the entire uncle blocks (one of the issues open on public); finish tests 2019-04-02 14:32:30 -05:00
Ian Norden
fd407825c1 review fixes 2019-04-02 10:10:17 -05:00
Ian Norden
185f4c0e93 adjust block/uncle reward tests and add methods to pkg/geth blockchain and ethclient for direct fetching of eth balances 2019-04-02 10:10:17 -05:00
Ian Norden
9030cff2bd keep track of who is receiving uncle rewards 2019-04-02 10:10:17 -05:00
Rob Mulholand
f5b32a11b0 Improve transaction syncing test coverage 2019-03-28 14:31:17 -05:00
Rob Mulholand
81dfd12665 Address PR comments 2019-03-28 14:31:17 -05:00
Rob Mulholand
54d46638a8 updates for light sync transactions 2019-03-28 14:31:17 -05:00
Rob Mulholand
79e011aad2 Add light sync transactions table 
- Foreign key to header instead of block
- Can use same Transaction struct for both
 2019-03-28 14:31:17 -05:00
Rob Mulholand
d93006321b Rename transactions to full_sync_transactions 
- Table has foreign key to blocks
- Add transaction RLP and transaction index to table
- Enables other tables with standalone transactions or transactions
  associated with other data structures (e.g. headers)
 2019-03-28 14:31:17 -05:00
Ian Norden
1aa849bcb4 PR fixes; remove all infura token references and setup travis to use encrypted env 
variable; rest of the ethjson_rpc dependent tests extracted to integration_test
 2019-03-21 18:36:51 -05:00
Ian Norden
d3e172d9ab remove references to infura token except for in infura.toml so that tests will continue to pass on CI 2019-03-21 18:36:49 -05:00
Ian Norden
37fc038605 fixes for review comments 2019-03-21 18:36:49 -05:00
Ian Norden
37e581c7ec PR#72 from public repo- https://github.com/vulcanize/vulcanizedb/pull/72- also needed to finish pluggin in ENS record transformer 2019-03-21 18:33:56 -05:00
Ian Norden
24879a85aa mv pkg/omni pkg/contract_watcher 2019-03-21 18:33:56 -05:00
Ian Norden
073378de6d update © to 2019 2019-03-21 18:33:56 -05:00
Ian Norden
e4e092f542 changes to plugin and commands to accomodate changes 2019-03-21 18:33:56 -05:00
Ian Norden
8174ce4aee refactor omni code; rename omniWatcher => contractWatcher; use config instead of excessive number of CLI flags 2019-03-21 18:30:48 -05:00
Ian Norden
ff55e3ba7a TransformerInitializer => EventTransformerInitializer reorganization 2019-03-14 11:59:39 -05:00
Ian Norden
9f8c50e3ab tests for migration path ordering and errors 2019-03-13 11:42:30 -05:00
Ian Norden
f37c458992 the part that actually runs the migrations in order 2019-03-13 11:42:29 -05:00
Ian Norden
84aa0a7eba plugin migration order specified in config 2019-03-13 11:42:29 -05:00
Rob Mulholand
1414779d52 Use *sqlx.Tx instead of *sql.Tx 
- requires using db.Beginx() instead of db.Begin()
- enables calling tx.Get()
 2019-03-06 12:43:39 -06:00
Rob Mulholand
7ed4dcd1fa Remove Tic logic 2019-03-06 12:43:39 -06:00
Rob Mulholand
b6bef49318 Add tests for plugin helpers 
- Prevents code analysis from showing functions as unused
- Also address other code analysis issues
 2019-03-06 12:43:39 -06:00
Rob Mulholand
a806cb2818 Extract slow tests from omni full transformer 2019-03-06 12:43:06 -06:00
Rob Mulholand
34c96654ce Extract slow tests from omni light transformer 2019-03-06 12:43:06 -06:00
Ian Norden
8bd397e2eb Merge pull request #29 from vulcanize/fix 
Fix for Travis issue at https://github.com/vulcanize/mcd_transformers/pull/6
 2019-03-06 11:27:43 -06:00
Ian Norden
e1a317c3f6 different errors; readme edit 2019-03-06 00:14:37 -06:00
Ian Norden
5fd9a59027 fix for issue with Travis building plugin from plugin repo 2019-03-05 08:57:54 -06:00
Edvard Hübinette
55b6ad9381 Optimise missing block numbers query (#23) 2019-03-05 11:52:38 +01:00
Ian Norden
83f7daf37d compose and execute as separate commands; update README and add git checkout vendor/github.com/ethereum/go-ethereum/accounts/abi to make build 2019-02-28 18:15:16 -06:00
Ian Norden
708425c4d6 rebase; extract factories and the mocks they are dependent on to 
libraries/shared; adjust omni test_helpers to drop and recreate
checked_headers table to avoid reaching 1600 column limit after repeated
tests (dropping columns doesn't actually remove them from contributing
to that limit)
 2019-02-25 01:34:38 -06:00
Ian Norden
17f4d3dfa5 move maker plugin generator test to mcd_transformers 2019-02-24 21:53:04 -06:00
Ian Norden
3d34a9e7c9 remove maker migrations and convert back to timestamps and fix bug in 
composeAndExecute command
 2019-02-24 21:52:23 -06:00
Ian Norden
b449193b16 addressing review comments; still need to reorg migrations and add helper tests 2019-02-24 21:38:48 -06:00
Ian Norden
2868cf2f73 goose changes; update plugin readme; use rel paths for transformers in config; more commenting in pkg/plugin; fix PluginWriter to work with different transformer types; reopen generator tests 2019-02-24 21:38:48 -06:00
Ian Norden
03a7379617 work on porting storage watcher; watcher type is defined for 
transformers in the config, storage transformers are exported from
plugin like event transformers
 2019-02-24 21:38:47 -06:00
Ian Norden
b51dcb55de comment out plugin test til mcd_transformers is updated correctly 2019-02-24 21:38:47 -06:00
Ian Norden
a322372713 refactoring plugin generating code 2019-02-24 21:38:47 -06:00
Ian Norden
decc2a3caf remove pkg/transformers; extract shared files needed to 
libraries/shared; work on automated db migration management
 2019-02-24 21:38:47 -06:00
Ian Norden
55fa9b8364 auto management of dependencies temporarily needed to build plugin and transformer 
migrations; new test showing it is working with external transformers runs
properly when migrations were manually performed but still need
to test automated migration management
 2019-02-24 15:23:35 -06:00
Ian Norden
6c2d895023 composeAndExecute command that loads and executes over arbitrary transformer set exported from a go plugin generated according to config file; test for plugin generation, loading, and execution; work on plugins README 2019-02-24 15:23:35 -06:00
Edvard Hübinette
fa4abe4e5c Remove spammy and unnecessary error log (#21) 2019-02-22 12:56:06 +01:00
Rob Mulholand
745d0e8eda Use storage value metadata getter in pit repository tests 2019-02-21 15:25:16 -06:00
Rob Mulholand
a5121913f4 Error on missing keys in pit repository 
- If ilk missing for ilk-related value, fail fast
 2019-02-21 15:03:22 -06:00
Rob Mulholand
3d9341ea1d Reference ilks foreign key from Cat storage repository 2019-02-21 14:33:59 -06:00
Rob Mulholand
dbdd16d3a2 (VDB-380) Create separate table for ilks 
- reference ilk by foreign key every place it's used
 2019-02-21 11:23:00 -06:00
Rob Mulholand
11dd641a84 (VDB-354) Queue unrecognized storage diffs 
- If we recognize a storage diff as coming from a watched contract but
  don't recognize the key, queue it for retrying later (after we've seen
  an event that might help us recognize the key)
- Remove unused errs and args
- Panic on unrecognized types (should not happen)
 2019-02-20 15:22:39 -06:00
Edvard
fb47a747f6 Fix confusion between nflip and flip 2019-02-20 14:37:55 +01:00
Edvard
56ca45509f DRY up storage diff repo tests 2019-02-20 14:37:55 +01:00
Edvard
5746de7c97 Add tests for Cat storage diff transformer 
Update schema after rebase migrations

Fix small issues from review
 2019-02-20 14:37:46 +01:00
Edvard
cd6611d2ec Add Cat storage transformer 2019-02-20 14:37:46 +01:00
Edvard Hübinette
53789c82ab Merge pull request #12 from vulcanize/VDB-392-fix-io-error-propagation 
VDB-392 Fix IO error propagation
 2019-02-20 13:58:00 +01:00
Edvard
1a2f46b308 Fix small issues from review 2019-02-20 13:10:55 +01:00
Ian Norden
2ecc524224 Merge pull request #9 from vulcanize/VDB322-Vow-storage-diff 
Vdb322 vow storage diff
 2019-02-15 11:04:22 -06:00
Taka Goto
64c7ae1bb1 uppercase Woe and Ash 2019-02-15 06:15:47 -06:00
Rob Mulholand
df521eea7f Merge pull request #11 from vulcanize/storage-transformers-readme 
Add storage transformers readme
 2019-02-14 14:24:01 -06:00
Edvard
13eacf2081 Update tests for error propagation 2019-02-14 16:04:12 +01:00
Edvard
6cd4e5ea95 Improve I/O error propagation 2019-02-14 16:03:57 +01:00
Rob Mulholand
0378068467 add example contract and instructions for watching its storage 2019-02-13 20:34:30 -06:00
Taka Goto
a8322c7d26 add woe to storage diff vow contract 2019-02-12 21:50:52 -06:00
Rob Mulholand
ee113668a5 Add first draft of storage transformers readme 
- Explain process of creating a new storage transformer
 2019-02-12 15:58:28 -06:00
Rob Mulholand
ac9375ae47 Discard logs created during tests 
- Make sure output is being set to discard for logrus to reduce clutter in the console
 2019-02-12 11:18:39 -06:00
Taka Goto
4fe2ff4104 repository methods: 2019-02-11 16:34:02 -06:00
Taka Goto
c2c21325b5 vow contract storage diffing excluding sin 2019-02-11 15:28:54 -06:00
Edvard Hübinette
f73623789e Merge pull request #6 from 8thlight/VDB-337-Dockerize-vDB 
VDB-337 Dockerize vDB
 2019-02-11 12:37:56 +01:00
Edvard
9b986c4c14 Nuke OpenRC from docker image, update readme's 2019-02-11 11:27:33 +01:00
Edvard
553fdbfa09 Default config starting block to 0 2019-02-11 11:27:33 +01:00
Edvard
e6e66d58dd Add container service files and docker README 2019-02-11 11:26:12 +01:00
Rob Mulholand
c05d3b7836 (VDB-280) Consume vat contract storage diffs 2019-02-08 13:36:30 -06:00
Rob Mulholand
dba4e48a3e Use map for key in storage value metadata 
- Enables inserting multiple keys for one storage row (e.g. ilk and urn)
 2019-02-08 11:41:17 -06:00
Rob Mulholand
09abcb841a Persist Urns in raw bytes32 format 
- Enables simple usage in contract storage lookups
 2019-02-08 11:41:17 -06:00
Takayuki Goto
0a024d429d * update transformer to able to recheck headers (#4) 
* update transformer to able to recheck headers
* put cap on rechecking header
* integration test for recheck headers
* use enum for recheck headers param; make recheck cap configurable
* update integration tests with new test config
* update omni pkg with new recheck header column type
* update migration with new migration tool and final tweaks needed to accommodate changes in omni pkg
 2019-02-08 10:35:46 -06:00
Rob Mulholand
867f92c431 (VDB-298) Consume Pit contract storage diffs 
- Continuously parse storage diffs CSV data to read Pit contract state
- Convert ilks in database to raw bytes32 value for use in generating
  storage keys dynamically
- Persist storage diffs with block number and hash for validation
 2019-02-06 10:31:46 -06:00
Edvard Hübinette
29f2537130 Merge pull request #7 from vulcanize/VDB-347-Squelch-lightSync 
VDB-347 Squelch lightSync in stdout
 2019-02-01 11:02:48 +01:00
Edvard
a1fa21cbec Remove test based on window buffer stuff 2019-01-31 16:39:29 +01:00
Edvard
8eea1b00ed Redirect validation window status to logrus instead of stdout 2019-01-31 15:55:28 +01:00
Ian Norden
8749c94994 update license to AGPL-3.0 2019-01-24 14:41:30 -06:00
Ian Norden
ab371a2024 Merge remote-tracking branch 'maker/staging' into staging 2019-01-24 10:22:22 -06:00
Ian Norden
1f95eb6443 Merge branch 'staging' into pub_merge 2019-01-24 09:47:12 -06:00
Taka Goto
c5d513e96d fix index out of range on lightSync; hopefully the new integration tests will catch them next time :) 2019-01-23 15:25:44 -06:00
Ian Norden
00c2052da3 for some reason an error is thrown when the ipc string is declared in integration_tests_suite_test.go but not when moved to a different file in the integration_tests package; I think this is due to switching to go1.11 2019-01-23 15:23:23 -06:00
Ian Norden
dc87f39889 change omniwatcher database test helpers so that it doesn't interfere with downstream maker tests 2019-01-23 14:55:20 -06:00
Konstantin Zolotarev
65cb6e9092 VDB-205 Fix external configuration values (#121) 
* Updated loading config from /Users/konstantinzolotarev/.vulcanizedb for addreses

* Updated contract addresses loading from env variables.

* Implement getter chain for addresses, configs and transformers

Solves the issue with declaring stuff as vars when the environment file hasn't been loaded yet. Instead, this data is fetched dynamically when the root commands are called.

* Update test data to use constant Kovan addresses

* Decouple integration tests from config file IO

* Add ABI to environment file, convert getter chains for related values

* Decouple tests suites from ABI config file IO

* Add startingBlock to environment and implement getter chain

* Exit when called without config file

* Add missing deployment blocks to staging.toml

* Remove comment
 2019-01-23 19:44:09 +01:00
Ian Norden
46b898f690 merge public vulcanizedb/master 2019-01-23 00:37:26 -06:00
Taka Goto
3595771825 use batchElem from rpc client 2019-01-21 14:52:37 -06:00
Takayuki Goto
54d771cb06 Merge branch 'staging' into batch-historical-headers 2019-01-16 15:56:36 -06:00
Taka Goto
24cf0661b0 lowercase address on map to get transformers 2019-01-16 14:24:00 -06:00
Taka Goto
07da9d720b fix get headers when less than 100 fetched 2019-01-16 10:16:06 -06:00
Taka Goto
074140b9ac iterate for loop by headers instead of blocknumbers 2019-01-15 16:34:20 -06:00
Ian Norden
aa4e698240 move interface_getter.go from ens_watcher to vDB pkging- used to construct a custom ABI based on results from calls to supportsInterface; also moved fetcher.go from examples to pkg 2019-01-15 15:49:18 -06:00
Takayuki Goto
f1ba21b505 Merge branch 'staging' into batch-historical-headers 2019-01-13 21:29:58 -06:00
yaoandrew
e2b864baa8 Add rollback if GetTicInTx fails and move GetTicInTx to shared/repository.go 2019-01-11 10:07:00 -08:00
Andrew J Yao
c8aa3dc275 Replace standard log with logrus import 2019-01-11 10:07:00 -08:00
yaoandrew
bc8b7a101c refactor error handling in vat, cat, vow 2019-01-11 10:07:00 -08:00
yaoandrew
525fada3d7 refactor error handling in frob, pit, price, tend 2019-01-11 10:07:00 -08:00
yaoandrew
657aaccff3 refactor error handling in drip, flap, flop 2019-01-11 10:07:00 -08:00
yaoandrew
05351a8ef4 refactor error handling in dent 2019-01-11 10:07:00 -08:00
yaoandrew
08ac7f18a5 refactor error handling in flip_deal 2019-01-11 10:07:00 -08:00
yaoandrew
f655997fde refactor error handling in pit_vow 2019-01-11 10:07:00 -08:00
yaoandrew
843d30c88e refactor error handling in flip 2019-01-11 10:07:00 -08:00
yaoandrew
513513e93b refactor error handling in chop_lump 2019-01-11 10:07:00 -08:00
yaoandrew
78b302eab6 refactor error handling in bite transformer 2019-01-11 10:07:00 -08:00
Edvard Hübinette
ebca338b1e VDB-302 Sleep when no missing blocks in lightSync (#129) 
* Sleep when no missing blocks in lightSync

* Fix tests and error propagation

* Correct geth.log to logrus
 2019-01-11 10:58:41 +01:00
Takayuki Goto
ac430ddf54 Merge branch 'staging' into batch-historical-headers 2019-01-08 19:06:53 -06:00
Taka Goto
aa99c861a8 batch headers by 100s 2019-01-08 16:53:26 -06:00
Ian Norden
59cdaa05e6 moving some memory allocation to init function rather than within 
execution cycles and fixing test db tear down so that it drops
checked_headers table instead of individual columns (droping columns
only hides them, they remain in memory and continue to contribute to the
1600 column limit which causes us to eventually exceed it)
 2019-01-07 13:07:31 -06:00
Edvard
2292c94e99 Merge branch 'staging' into VDB-125-aggregate-log-fetching 2019-01-07 15:25:37 +01:00
Edvard
50cc0c2459 Update tests for watcher and repository 2019-01-07 15:20:05 +01:00
Edvard
833dde62cf Remove injection of fetcher and repository to watcher 2019-01-07 15:19:31 +01:00
Ian Norden
2cbe6e7a70 method piping 2018-12-21 11:14:54 -06:00
Ian Norden
456c735087 fetch headers and logs in batches across all contracts and events 2018-12-21 10:33:31 -06:00
Ian Norden
d188329661 fetch event logs in batches- all events for a contract address at a given block height are fetched together, different contract events still fetched separately 2018-12-21 10:33:31 -06:00
Ian Norden
e390a97502 finish method polling with hash or address type argument values 
collected from watched events; 'MissingMethodsCheckedEventsIntersection'
method to find headers which have been checked for each event of
interest but methods have not yet been polled at that header; tests for
new features; travis ci go version 1.9 -> 1.11 ; consolidate omniWatcher and lightOmniWatcher into single command with light as default
 2018-12-21 10:33:31 -06:00
Ian Norden
0a59f06cac begin work on: Add checked_headers column for methods that are polled so taht we don’t duplicate; Add batching of method polling so that we arent generating a rediculously large account address list before using it to poll methods (or persist the list in pg?); User passed ABI and other ways to get ABI; Add ability to collect []byte and hashes from events and use them in method polling same manner as addresses; Event filter addrs => only those event’s addresses/hashes are used for polling; Option to persist seen address/hash/bytes lists into pg; Only generate lists of addresses, []byte, or hashes if a method will use them later 2018-12-21 10:33:31 -06:00
Edvard
e54699c039 Remove todo on chunker non-match (it's basically a filter) 2018-12-14 15:22:46 +01:00
Edvard Hübinette
a3e8633aff VDB-104 improve lightsync ui (#125) 
* Return PopulateMissingHeaders early if the sync is at the head of the chain

* Squelch logging if no blocks to sync

* Fix broken test

* Refactor repository tests
 2018-12-14 12:11:48 +01:00
Edvard
7cf694f5e7 Update transformer documentation 2018-12-14 10:30:08 +01:00
Edvard
35f7f719e6 Swap strings.Builder for bytes.Buffer for go 1.9 support 2018-12-13 16:08:57 +01:00
Edvard
129964f3bc Add tests for now shared MissingHeaders 2018-12-13 13:58:16 +01:00
Edvard
e5db740ae4 Move MockTransformer to test_data 2018-12-13 13:03:51 +01:00
Edvard
d30fcfed80 Use new config getter on shared.Transformer <3 2018-12-13 12:39:57 +01:00
Edvard
882a6dd7a2 Update tests with new adding of transformers 2018-12-12 15:41:44 +01:00