cerc-io/ipld-eth-server
18
0
Fork 1
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 37 Wiki Activity
1,304 Commits 12 Branches 52 Tags 80 MiB
bb14c01529
Commit Graph

701 Commits

Author SHA1 Message Date
Ian Norden
35c8f3561a publisher unit test 2019-12-02 13:24:58 -06:00
Ian Norden
d79cc90cb2 unit test for ipfs fetcher 2019-12-02 13:24:58 -06:00
Ian Norden
069646905e converter unit test 2019-12-02 13:24:58 -06:00
Ian Norden
89faa85871 fix transaction signing, also removes need for rpc call to do it 2019-12-02 13:24:58 -06:00
Ian Norden
e72995605d possible fix for travis error 2019-12-02 13:24:58 -06:00
Ian Norden
0bbb7a30d1 review fixes 2019-12-02 13:24:58 -06:00
Ian Norden
2244d1869f process receipts from payload instead of retreiving over additional rpc call 2019-12-02 13:24:58 -06:00
Ian Norden
13d09e15c1 update deps 2019-12-02 13:24:58 -06:00
Ian Norden
905585b212 backfill concurrently, one block at a time, while normal streaming continues 2019-12-02 13:24:58 -06:00
Ian Norden
f2efbb5d01 worker pool for handling concurrent ipld putting and cid indexing without overloading Postgres connections 2019-12-02 13:24:58 -06:00
Ian Norden
3fa33fb767 index receipts by the contract address 2019-12-02 13:24:58 -06:00
Ian Norden
23a21c14f3 make subscription config rlp encodable, group subs of the 
same type using a hash of their config, process only once for each sub
type instead of for every sub.
 2019-12-02 13:24:58 -06:00
Ian Norden
1235ec1f97 postgres table to hold ipfs data in same database as vulcanizedb data; 
shared SeedNodeTransformer interface
 2019-12-02 13:24:54 -06:00
Ian Norden
5356cd50bb seed node documentation 2019-12-02 13:24:51 -06:00
Ian Norden
4c81ca4d54 rename blocks table to eth_blocks so that we don't clash with the ipfs blocks table; dockerfile and startup_script for the seed node 2019-12-02 13:24:51 -06:00
Ian Norden
e7cdd6247e update to use ReceiptForStorage; expose rpc server over ws 2019-12-02 13:24:51 -06:00
Ian Norden
723c7c6244 fix backfill operations and dependency issue; hopefully travis will work now 2019-12-02 13:24:51 -06:00
Ian Norden
4baea2923c make db fks deferrable so that we can commit entire cid payload in single transaction; adjust buffer sizes to optimize performane and stability 2019-12-02 13:24:51 -06:00
Ian Norden
b1bb646ad5 goimports + streamSubscribe command for raw access to the seed node data 2019-12-02 13:24:51 -06:00
Ian Norden
8ccdfd4835 fix streamFilters issue 2019-12-02 13:24:51 -06:00
Ian Norden
1dc46640cf request handler and response screener for seed node api service 2019-12-02 13:24:51 -06:00
Ian Norden
34393ffb3f fixes for issues uncovered in integration 2019-12-02 13:24:51 -06:00
Ian Norden
b83c0371d9 pub sub service and api 2019-12-02 13:24:50 -06:00
Ian Norden
5ebe2243d8 goimports -w 2019-12-02 13:24:50 -06:00
Ian Norden
671a324294 update to work with updated state diffing code 2019-12-02 13:24:50 -06:00
Ian Norden
3108957e5f ipfs fetcher 2019-12-02 13:24:50 -06:00
Ian Norden
15e044403d work on mocks and unit tests 2019-12-02 13:24:50 -06:00
Ian Norden
d702cb720c update dependencies to work with update eth-block-extractor 2019-12-02 13:24:49 -06:00
Ian Norden
31a9017c4f index all cids in a payload in a single atomic tx; misc fixes; comment additions 2019-12-02 13:24:46 -06:00
Ian Norden
1921c06b02 repository - repository for indexing CIDs 2019-12-02 13:24:46 -06:00
Ian Norden
1fba4d0858 publisher - interface for publishing IPLD objects 2019-12-02 13:24:46 -06:00
Ian Norden
d248a1050e syncer - interface for syncing with a statediff subscription 2019-12-02 13:24:46 -06:00
Ian Norden
a31c7e9ced payload converter - interface for converting state diff payload into payload for ipld publishing 2019-12-02 13:24:46 -06:00
Ian Norden
35719b5180 indexer - top level interface for syncAndPublish 2019-12-02 13:24:46 -06:00
Ian Norden
c6c070005b subscribe method for the rpc client 2019-12-02 13:24:46 -06:00
Ian Norden
bc59aa4ed6 readme additions and vendor updates 2019-12-02 13:24:46 -06:00
Ian Norden
db0f024088 review fixes 2019-12-02 11:26:44 -06:00
Ian Norden
a834e55b9f command edits; external pkg for finding min deployment block; gofmt 2019-12-02 11:22:18 -06:00
Ian Norden
563832422c add err logging to eth/node 2019-12-02 11:07:43 -06:00
Rob Mulholand
4a98b7942f Don't return unexported types 2019-10-31 13:42:19 -05:00
Rob Mulholand
e1236b4072 Prefer all caps for initialisms and acronyms 2019-10-31 13:42:19 -05:00
Rob Mulholand
eba868ff00 Drop else when if statement includes return 2019-10-31 13:33:39 -05:00
Rob Mulholand
0ded7bcd62 Prefer fmt.Errorf(s) to errors.New(fmt.Sprintf(s)) 2019-10-31 13:33:39 -05:00
Rob Mulholand
234fff83c5 Use testing.toml and vulcanize_testing in tests 
- remove legacy references to infura.toml and vulcanize_private
 2019-10-31 13:10:25 -05:00
Rob Mulholand
e252229b8a Add constraint to prevent duplicate headers 
- Disallow inserts of headers with the same number, hash, and node
  fingerprint, since it will enable duplicate log fetching for the
  same header
 2019-10-28 14:57:13 -05:00
Ian Norden
a2d249ca9d review fixes 2019-10-28 11:40:32 -05:00
Ian Norden
4fbde836d4 log sql.ErrNoRows which I suspect is what is leading to the flaky test 2019-10-28 09:37:21 -05:00
Ian Norden
65808998b3 goimports -w; golinting, remove some unused code 2019-10-28 09:37:21 -05:00
Ian Norden
11b5efbfe3 fix for issue #146; mark header checked for contract if it doesnt have 
any logs at that header but other contracts do; test
 2019-10-28 09:34:42 -05:00
Edvard Hübinette
3fff2896aa
Rename geth to eth, signifying client independence (#161) 2019-10-28 12:30:24 +01:00
Edvard Hübinette
f7c4a6736d
VDB-919 Generalise converter (#152) 
* Generalise transformer stack to use InsertionModel

* Add tests for event repository

* Restrict accepted values in InsertionModel

* Add call to repository.SetDB

* Improve error propagation/clarity on GetABI()

* Remove maker references in example

* Please golint

* refactor rollback error handling in repository

* Cleaner errors in repository, refactor tests
 2019-10-28 11:48:31 +01:00
Rob Mulholand
2800e6df36 Enable contractWatcher without prior headerSync 
- Previous setup would fail if there were no headers in the db. This
  makes sense because we need headers that haven't been checked for
  logs to exist so that we can fetch logs for those headers. But it
  also prevents us from kicking off the headerSync and contractWatcher
  processes concurrently. These changes enable kicking off both
  processes at the same time with the idea that we will have unchecked
  headers upon transformer execution.
 2019-10-04 16:00:13 -05:00
Elizabeth
f6ab9382b2
Address repo updates (#134) 
* Factor out get or create address into one sql string

* Factor out getChecksumAddress method in address repo

* Update address repo methods to not need a receiver

* Move address repository to libraries/shared
 2019-10-03 11:17:08 -05:00
Rob Mulholand
4505382590 Increase logging in contract watcher 
- Focus on header mode
- Add context to errors, trace guard clauses, warn on non-returned
  errors
- Give errors distinct names so compiler will recognize if unchecked
- Remove redundant type declarations/fix typos
 2019-10-02 15:14:42 -05:00
Rob Mulholand
2ff88de859 Extract helper for converting hex to keccak256 hash 
- Also prefer crypto.Keccak256Hash(x) to common.BytesToHash(crypto.Keccak256(x))
 2019-09-25 16:36:08 -05:00
Elizabeth Engelman
8e166f2a56 Update after rebasing with staging 2019-09-25 16:32:31 -05:00
Elizabeth Engelman
ee244ac6f5 Apply go fmt 2019-09-25 16:32:27 -05:00
Elizabeth Engelman
34f8e16c11 Add a Subscribe method to rpc client interface 2019-09-25 16:32:27 -05:00
Elizabeth Engelman
36533f7c3f Update vendor directory and make necessary code changes 
Fixes for new geth version
 2019-09-25 16:32:27 -05:00
Matt Griswold
20ce0ab852 Revert "option to write the plugin .go file and run the plugin migrations without building the .so file so that it can be done from the shell instead to afford complete error messages" 
This reverts commit 9bb2f27a69.
 2019-09-24 23:54:25 +00:00
Ian Norden
9bb2f27a69 option to write the plugin .go file and run the plugin migrations without building the .so file so that it can be done from the shell instead to afford complete error messages 2019-09-24 18:44:04 -05:00
Rob Mulholand
c21c069b69 Replace magic numbers with variables 2019-09-23 16:13:45 -05:00
Rob Mulholand
eea0b0bdca Return missing headers after starting block number 
- Previously, we required that the first missing header match the
  configured starting block number. This helps guarantee that we have
  the necessary data for method polling in memory, but prevents the
  process from moving forward if restarted after the starting block
  has already been checked.
 2019-09-23 16:13:45 -05:00
Rob Mulholand
37e2673a8d Do nothing on duplicate log insert 
- Prevents duplicate key constraint violation from blocking the process
  from moving forward on restart.
- If header_id, log_idx, and tx_idx are the same, we can safely do
  nothing since it's definitely the same log - a reorg would cause the
  original header to be replaced with a new ID.
 2019-09-23 16:02:07 -05:00
Rob Mulholand
3466a51b4d Correctly parse uint8 values in contract watcher 
- calling string(uint8) => insert errors on numeric columns
 2019-09-20 21:37:59 -05:00
Rob Mulholand
749db40b96 Parse bytes32 vals in Contract Watcher 2019-09-20 13:48:43 -05:00
Rob Mulholand
f83e996ab8 Update comments in checked headers repository 2019-09-18 21:39:34 -05:00
Rob Mulholand
2b798e00e0 Cap random fake timestamp 2019-09-13 11:20:13 -05:00
Rob Mulholand
3f9b034c4c Rename checked_logs => watched_logs 
- We're logging that a given log has been included in any fetch calls
  for checked headers, rather than that we have already checked for
  that log
 2019-09-10 21:22:14 -05:00
Rob Mulholand
13d503b851 Distinguish between missing and unchecked headers 
- Missing == not in DB
- Unchecked == logs haven't been fetched
 2019-09-10 14:39:45 -05:00
Rob Mulholand
ce91b0d9e6 Simplify checked header repository tests 
- Use assertions instead of comments to document expectations
- Also randomize the fake timestamp in test data
 2019-09-10 13:47:43 -05:00
Rob Mulholand
b9f3b9f946 Reference header sync logs address via foreign key 2019-08-28 10:41:34 -05:00
Rob Mulholand
5ac76eee74 Make check_count a column on public.headers 
- Don't need to maintain it on public.checked_headers if we're not
  adding additional columns to that table
 2019-08-28 09:50:17 -05:00
Rob Mulholand
666ea1c325 Update checked headers for new transformers 
- If a header was marked as checked before a transformer was added to
  the watcher, mark all headers since the new transformer's starting
  block number as unchecked.
 2019-08-28 09:25:14 -05:00
Rob Mulholand
63dabbb051 Extract and delegate logs concurrently 2019-08-28 09:25:13 -05:00
Rob Mulholand
d496dad33c Decouple log extraction from transformer delegation 
- limit missing headers results set to 100 so that extraction doesn't
  excessively block delegation
- wrap checked headers functions in repository struct
- move storage repository to factory, to correspond with event
  repository path
- remove unused files
- reformat sql
- remove line breaks in imports
 2019-08-28 09:25:13 -05:00
Rob Mulholand
cb819fa9a6 Write event logs to database before transforming 
- enables decoupling event extraction/persistence from transformation
- modifies event transformer, converter, and log chunker to accept
  payload that includes internal log database ID with log data
- remove alias for transformer pkg as shared_t
- remove unused mock watcher repository
 2019-08-28 09:13:44 -05:00
Rob Mulholand
3693ed905f Rename logs to full sync logs 
- Enable creating new table for logs used in event watching based on
  header sync
 2019-08-28 09:11:06 -05:00
Elizabeth Engelman
edc0bdf668 Address more PR feedback 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
1b3786338f Address PR feedback 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
58c23c6632 Update Copyrights 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
38f6847ff2 Create a HeaderSyncReceiptRepository 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
7e38764618 Rename ReceiptRepository -> FullSyncReceiptRepository 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
4e40e892d2 Updating header_sync_receipts to have FK reference to addresses 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
bcd6d14fcd Updating full_sync_receipts to have FK reference to addresses 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
258035833b Add GetAddressById 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
4b61c87b55 Get or create address record in a transaction 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
1373fe83a1 Pass a db into GetOrCreateAddress 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
b13eefbce7 Add AddressRepository 2019-08-23 10:11:37 -05:00
Rob Mulholand
987abd4b2e Update Geth to 1.9.0 2019-07-23 15:26:18 -05:00
Andrew J Yao
ee77fc6521 Remove injection of SubCommand for logs 2019-07-23 13:11:20 -07:00
Andrew J Yao
92d153b010 Add subCommand name to log fields 2019-07-23 08:09:57 -07:00
Edvard Hübinette
2c092e8d04
[VDB-751 VDB-754] Bugfix null pointer panic and improve logging (#119) 
* VDB-751 VDB-754 Bugfix null pointer panic and improve logging

* Fix typo
 2019-07-18 09:21:40 +02:00
Rob Mulholand
0f95267e84 Use goose library for running plugin migrations 
- Removes dependency on goose binary existing in plugin execution
  environment
 2019-06-27 13:58:58 -05:00
Rob Mulholand
f0bcf930f4 Separate tmp migrations dir from home in join 
- enable separator between home and migrations dir
 2019-06-27 13:58:58 -05:00
Rob Mulholand
0558792282 Log error on failure to setup plugin migrations 
- include specific error
 2019-06-27 13:58:58 -05:00
Andrew J Yao
11a65c525c Adds ability to set log level from toml or CLI args 2019-06-27 11:14:11 -07:00
Elizabeth Engelman
fa03716cb2 Address small PR comments 2019-05-10 11:54:15 -05:00
Elizabeth Engelman
e1a0d894a2 Update blockchain method GetHeaderByNumbers -> GetHeadersByNumbers 2019-05-08 13:42:43 -05:00
Elizabeth Engelman
ba81766f6c Small spelling fix 2019-05-08 13:42:43 -05:00
Rob Mulholand
76ab914bdc Add license 2019-05-01 12:32:39 -05:00
Rob Mulholand
2d684c5aec Extract storage diff fetching behind an interface 
- Replaces directly reading from a CSV
- Simplifies testing
- Should hopefully make it easier to plug in other sources for storage
  diffs (e.g. differently formatted CSVs, JSON RPC, etc)
 2019-05-01 12:30:36 -05:00
Gabe Laughlin
a344432156
(VDB-560) Rename lightSync to headerSync 2019-05-01 12:12:55 -05:00
Rob Mulholand
3fd6269b78 Don't lookup transactions if no log events 
- Prevents EOF error on transactions lookup
 2019-04-18 09:53:18 -05:00
ana0
964fc8bd22 Merge branch 'staging' into databaseConfig 2019-04-11 14:19:57 -04:00
anon
bd12149594 Database config 2019-04-10 17:56:31 -04:00
Gabe Laughlin
8903b6fb2d Add timestamp customization for fake header 2019-04-09 17:08:15 -05:00
Ian Norden
9befc76fc6 remove LIMIT 100; method to check continuity of headers returned but doesn't require 
the set to start at a specific block number; config for
account_transformer; review fixes; update schema
 2019-04-05 17:21:59 -05:00
Ian Norden
c1940c3e58 full_sync and light_sync receipt tables and light receipt repository methods and test 2019-04-05 16:36:48 -05:00
Ian Norden
04ad7b7696 Merge pull request #47 from vulcanize/track_uncle_rewards 
Track uncle rewards + direct balance polling
 2019-04-04 15:55:40 -05:00
Ian Norden
5dcf534b2c review fixes 2019-04-04 15:21:39 -05:00
Ian Norden
72c73c25b7 update vendor dep; constrain to revision of geth with UnpackIntoMap; updates to match 2019-04-03 00:14:02 -05:00
Ian Norden
197f98c93d going ahead and indexing the entire uncle blocks (one of the issues open on public); finish tests 2019-04-02 14:32:30 -05:00
Ian Norden
fd407825c1 review fixes 2019-04-02 10:10:17 -05:00
Ian Norden
185f4c0e93 adjust block/uncle reward tests and add methods to pkg/geth blockchain and ethclient for direct fetching of eth balances 2019-04-02 10:10:17 -05:00
Ian Norden
9030cff2bd keep track of who is receiving uncle rewards 2019-04-02 10:10:17 -05:00
Rob Mulholand
f5b32a11b0 Improve transaction syncing test coverage 2019-03-28 14:31:17 -05:00
Rob Mulholand
81dfd12665 Address PR comments 2019-03-28 14:31:17 -05:00
Rob Mulholand
54d46638a8 updates for light sync transactions 2019-03-28 14:31:17 -05:00
Rob Mulholand
79e011aad2 Add light sync transactions table 
- Foreign key to header instead of block
- Can use same Transaction struct for both
 2019-03-28 14:31:17 -05:00
Rob Mulholand
d93006321b Rename transactions to full_sync_transactions 
- Table has foreign key to blocks
- Add transaction RLP and transaction index to table
- Enables other tables with standalone transactions or transactions
  associated with other data structures (e.g. headers)
 2019-03-28 14:31:17 -05:00
Ian Norden
1aa849bcb4 PR fixes; remove all infura token references and setup travis to use encrypted env 
variable; rest of the ethjson_rpc dependent tests extracted to integration_test
 2019-03-21 18:36:51 -05:00
Ian Norden
d3e172d9ab remove references to infura token except for in infura.toml so that tests will continue to pass on CI 2019-03-21 18:36:49 -05:00
Ian Norden
37fc038605 fixes for review comments 2019-03-21 18:36:49 -05:00
Ian Norden
37e581c7ec PR#72 from public repo- https://github.com/vulcanize/vulcanizedb/pull/72- also needed to finish pluggin in ENS record transformer 2019-03-21 18:33:56 -05:00
Ian Norden
24879a85aa mv pkg/omni pkg/contract_watcher 2019-03-21 18:33:56 -05:00
Ian Norden
073378de6d update © to 2019 2019-03-21 18:33:56 -05:00
Ian Norden
e4e092f542 changes to plugin and commands to accomodate changes 2019-03-21 18:33:56 -05:00
Ian Norden
8174ce4aee refactor omni code; rename omniWatcher => contractWatcher; use config instead of excessive number of CLI flags 2019-03-21 18:30:48 -05:00
Ian Norden
ff55e3ba7a TransformerInitializer => EventTransformerInitializer reorganization 2019-03-14 11:59:39 -05:00
Ian Norden
9f8c50e3ab tests for migration path ordering and errors 2019-03-13 11:42:30 -05:00
Ian Norden
f37c458992 the part that actually runs the migrations in order 2019-03-13 11:42:29 -05:00
Ian Norden
84aa0a7eba plugin migration order specified in config 2019-03-13 11:42:29 -05:00
Rob Mulholand
1414779d52 Use *sqlx.Tx instead of *sql.Tx 
- requires using db.Beginx() instead of db.Begin()
- enables calling tx.Get()
 2019-03-06 12:43:39 -06:00
Rob Mulholand
7ed4dcd1fa Remove Tic logic 2019-03-06 12:43:39 -06:00
Rob Mulholand
b6bef49318 Add tests for plugin helpers 
- Prevents code analysis from showing functions as unused
- Also address other code analysis issues
 2019-03-06 12:43:39 -06:00
Rob Mulholand
a806cb2818 Extract slow tests from omni full transformer 2019-03-06 12:43:06 -06:00
Rob Mulholand
34c96654ce Extract slow tests from omni light transformer 2019-03-06 12:43:06 -06:00
Ian Norden
8bd397e2eb Merge pull request #29 from vulcanize/fix 
Fix for Travis issue at https://github.com/vulcanize/mcd_transformers/pull/6
 2019-03-06 11:27:43 -06:00
Ian Norden
e1a317c3f6 different errors; readme edit 2019-03-06 00:14:37 -06:00
Ian Norden
5fd9a59027 fix for issue with Travis building plugin from plugin repo 2019-03-05 08:57:54 -06:00
Edvard Hübinette
55b6ad9381 Optimise missing block numbers query (#23) 2019-03-05 11:52:38 +01:00
Ian Norden
83f7daf37d compose and execute as separate commands; update README and add git checkout vendor/github.com/ethereum/go-ethereum/accounts/abi to make build 2019-02-28 18:15:16 -06:00
Ian Norden
708425c4d6 rebase; extract factories and the mocks they are dependent on to 
libraries/shared; adjust omni test_helpers to drop and recreate
checked_headers table to avoid reaching 1600 column limit after repeated
tests (dropping columns doesn't actually remove them from contributing
to that limit)
 2019-02-25 01:34:38 -06:00
Ian Norden
17f4d3dfa5 move maker plugin generator test to mcd_transformers 2019-02-24 21:53:04 -06:00
Ian Norden
3d34a9e7c9 remove maker migrations and convert back to timestamps and fix bug in 
composeAndExecute command
 2019-02-24 21:52:23 -06:00
Ian Norden
b449193b16 addressing review comments; still need to reorg migrations and add helper tests 2019-02-24 21:38:48 -06:00
Ian Norden
2868cf2f73 goose changes; update plugin readme; use rel paths for transformers in config; more commenting in pkg/plugin; fix PluginWriter to work with different transformer types; reopen generator tests 2019-02-24 21:38:48 -06:00
Ian Norden
03a7379617 work on porting storage watcher; watcher type is defined for 
transformers in the config, storage transformers are exported from
plugin like event transformers
 2019-02-24 21:38:47 -06:00
Ian Norden
b51dcb55de comment out plugin test til mcd_transformers is updated correctly 2019-02-24 21:38:47 -06:00