Commit Graph

88 Commits

Author SHA1 Message Date
Rob Mulholand
13d503b851 Distinguish between missing and unchecked headers
- Missing == not in DB
- Unchecked == logs haven't been fetched
2019-09-10 14:39:45 -05:00
Rob Mulholand
ce91b0d9e6 Simplify checked header repository tests
- Use assertions instead of comments to document expectations
- Also randomize the fake timestamp in test data
2019-09-10 13:47:43 -05:00
Rob Mulholand
5ac76eee74 Make check_count a column on public.headers
- Don't need to maintain it on public.checked_headers if we're not
  adding additional columns to that table
2019-08-28 09:50:17 -05:00
Rob Mulholand
666ea1c325 Update checked headers for new transformers
- If a header was marked as checked before a transformer was added to
  the watcher, mark all headers since the new transformer's starting
  block number as unchecked.
2019-08-28 09:25:14 -05:00
Rob Mulholand
d496dad33c Decouple log extraction from transformer delegation
- limit missing headers results set to 100 so that extraction doesn't
  excessively block delegation
- wrap checked headers functions in repository struct
- move storage repository to factory, to correspond with event
  repository path
- remove unused files
- reformat sql
- remove line breaks in imports
2019-08-28 09:25:13 -05:00
Elizabeth Engelman
7e38764618 Rename ReceiptRepository -> FullSyncReceiptRepository 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
bcd6d14fcd Updating full_sync_receipts to have FK reference to addresses 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
e1a0d894a2 Update blockchain method GetHeaderByNumbers -> GetHeadersByNumbers 2019-05-08 13:42:43 -05:00
Rob Mulholand
76ab914bdc Add license 2019-05-01 12:32:39 -05:00
Rob Mulholand
2d684c5aec Extract storage diff fetching behind an interface
- Replaces directly reading from a CSV
- Simplifies testing
- Should hopefully make it easier to plug in other sources for storage
  diffs (e.g. differently formatted CSVs, JSON RPC, etc)
2019-05-01 12:30:36 -05:00
Gabe Laughlin
a344432156
(VDB-560) Rename lightSync to headerSync 2019-05-01 12:12:55 -05:00
Gabe Laughlin
8903b6fb2d Add timestamp customization for fake header 2019-04-09 17:08:15 -05:00
Ian Norden
5dcf534b2c review fixes 2019-04-04 15:21:39 -05:00
Ian Norden
197f98c93d going ahead and indexing the entire uncle blocks (one of the issues open on public); finish tests 2019-04-02 14:32:30 -05:00
Ian Norden
185f4c0e93 adjust block/uncle reward tests and add methods to pkg/geth blockchain and ethclient for direct fetching of eth balances 2019-04-02 10:10:17 -05:00
Rob Mulholand
81dfd12665 Address PR comments 2019-03-28 14:31:17 -05:00
Rob Mulholand
54d46638a8 updates for light sync transactions 2019-03-28 14:31:17 -05:00
Rob Mulholand
79e011aad2 Add light sync transactions table
- Foreign key to header instead of block
- Can use same Transaction struct for both
2019-03-28 14:31:17 -05:00
Rob Mulholand
d93006321b Rename transactions to full_sync_transactions
- Table has foreign key to blocks
- Add transaction RLP and transaction index to table
- Enables other tables with standalone transactions or transactions
  associated with other data structures (e.g. headers)
2019-03-28 14:31:17 -05:00
Ian Norden
1aa849bcb4 PR fixes; remove all infura token references and setup travis to use encrypted env
variable; rest of the ethjson_rpc dependent tests extracted to integration_test
2019-03-21 18:36:51 -05:00
Ian Norden
37fc038605 fixes for review comments 2019-03-21 18:36:49 -05:00
Ian Norden
24879a85aa mv pkg/omni pkg/contract_watcher 2019-03-21 18:33:56 -05:00
Ian Norden
9f8c50e3ab tests for migration path ordering and errors 2019-03-13 11:42:30 -05:00
Rob Mulholand
7ed4dcd1fa Remove Tic logic 2019-03-06 12:43:39 -06:00
Rob Mulholand
a806cb2818 Extract slow tests from omni full transformer 2019-03-06 12:43:06 -06:00
Rob Mulholand
34c96654ce Extract slow tests from omni light transformer 2019-03-06 12:43:06 -06:00
Ian Norden
decc2a3caf remove pkg/transformers; extract shared files needed to
libraries/shared; work on automated db migration management
2019-02-24 21:38:47 -06:00
Edvard
13eacf2081 Update tests for error propagation 2019-02-14 16:04:12 +01:00
Rob Mulholand
867f92c431 (VDB-298) Consume Pit contract storage diffs
- Continuously parse storage diffs CSV data to read Pit contract state
- Convert ilks in database to raw bytes32 value for use in generating
  storage keys dynamically
- Persist storage diffs with block number and hash for validation
2019-02-06 10:31:46 -06:00
Ian Norden
dc87f39889 change omniwatcher database test helpers so that it doesn't interfere with downstream maker tests 2019-01-23 14:55:20 -06:00
Ian Norden
46b898f690 merge public vulcanizedb/master 2019-01-23 00:37:26 -06:00
Taka Goto
3595771825 use batchElem from rpc client 2019-01-21 14:52:37 -06:00
Takayuki Goto
f1ba21b505 Merge branch 'staging' into batch-historical-headers 2019-01-13 21:29:58 -06:00
Edvard Hübinette
ebca338b1e VDB-302 Sleep when no missing blocks in lightSync (#129)
* Sleep when no missing blocks in lightSync

* Fix tests and error propagation

* Correct geth.log to logrus
2019-01-11 10:58:41 +01:00
Takayuki Goto
ac430ddf54 Merge branch 'staging' into batch-historical-headers 2019-01-08 19:06:53 -06:00
Taka Goto
aa99c861a8 batch headers by 100s 2019-01-08 16:53:26 -06:00
Edvard Hübinette
a3e8633aff VDB-104 improve lightsync ui (#125)
* Return PopulateMissingHeaders early if the sync is at the head of the chain

* Squelch logging if no blocks to sync

* Fix broken test

* Refactor repository tests
2018-12-14 12:11:48 +01:00
Ian Norden
975f13b969 reorganizing omni directory and beginning light watcher work 2018-11-29 20:33:21 -06:00
Ian Norden
390a60f7f6 port over lightSync updates from maker repo 2018-11-29 20:33:21 -06:00
Ian Norden
417b18ec6a Edits to address PR issues; change license from apache to AGPL; and work
towards generic method polling and reposito;y; config settings to
filterevents/methods by account address; refactoring some stuff out of
repo and into converter; remove fetcher and instead call
blockchain's FetchContractData directly; finishing tests
2018-11-15 12:32:52 -06:00
Edvard Hübinette
40408e8300 VDB-120 include tic in dent and tend (#118)
* Add constants for TTL and tic for fakeHeader

* Add tic to tend transformer

* Update tests for tend

* Fix string conversion bug in fakes

* Fix tend integration tests after staging rebase

* Add tic to dent transformer

* Update dent tests

* Change integration tests to use hardcoded block timestamp
2018-11-15 12:57:29 +01:00
Rob Mulholand
82fd73ba3f (VDB-68) Verify log block hash matches header block hash
- Delete header on conflict to prompt data refresh (cascade deletes all
  data associated with that block)
- Derive header hash from rpc payload rather than computing it from data
  (prevents hash mismatch between blockchain and cache)
2018-11-13 14:51:39 -06:00
Ian Norden
9b41000b88 Moved fetcher to generic directory (methods have to remain public since it is in seperate package now), added FetchHash method, created ERC20 and generic getters which call the fetcher with specific contract methods (GetTotalSupply, GetBalance, GetAllowance for ERC20 getter, and GetOwner, GetStoppedStatus, GetStringName, GetHashName, GetStringSymbol, GetHashSymbol, and GetDecimals for generic getter). Getter tests cover all but GetBalance and GetAllowance, and also cover all of the Fetcher methods- but with only nil methodArgs. GetAllowance and GetBalance tests are not working against infura and these are the only contract method calls with arguments passed in so I suspect this might be where the issue lies. Have tested GetBalance using previous version of FetchContractData without the variadic input to the Pack method and it fails with the same error so I don’t think it is due to those changes. 2018-11-03 13:49:23 -05:00
Ian Norden
ccd21f73c9 adjusted FetchContractData to work with variable number of contract method arguments, changed fetcher's FetchSupply method to more general FetchBigInt method and added FetchBool and FetchAddress methods 2018-11-03 13:49:23 -05:00
Rob Mulholand
ba071ef13f Consolidate test doubles
- Migrate various mocks of core namespaces to shared version in `fakes` pkg
- Err on the side of making test doubles less sophisticated
- Don't pull over mocks of namespaces that are only used in example code
2018-11-03 13:49:23 -05:00
Rob Mulholand
5fe6394406 Add tests for pkg/geth/blockchain
- inject dependencies instead of initializing them in the constructor
2018-11-03 13:49:23 -05:00
Rob Mulholand
05186634bd Add light sync command
- Only syncs block headers (excludes block bodies, transactions, receipts, and logs)
- Modifies validation window to include the most recent block
- Isolates validation window to the variable defined in the cmd directory (blocks
  have a separate variable defined in the block_repository for determining when
  to set a block as final)
2018-11-03 13:49:23 -05:00
Rob Mulholand
d3df685c46 Add timestamp to block header
- Enables quickly deriving timestamp from header for log events
2018-10-12 12:00:02 -05:00
Rob Mulholand
a0ba6ca6bd Return error if RPC returns empty header
- Enables graceful failure if RPC returns empty data with no error
2018-09-25 14:11:35 -05:00
Rob Mulholand
042f1142c0 Handle headers from POA chain
- Fetching headers from Kovan requires custom type without mixHash/Nonce
2018-09-19 11:00:20 -05:00