Commit Graph

238 Commits

Author SHA1 Message Date
Gabe Laughlin
b4cdf76905
(VDB-479) Upgrade Posgres to 11 2019-05-14 16:49:05 -05:00
Gabe Laughlin
a344432156
(VDB-560) Rename lightSync to headerSync 2019-05-01 12:12:55 -05:00
Rob Mulholand
fb753a1468 Update schema to remove uncles 2019-04-08 15:16:43 -05:00
Ian Norden
9befc76fc6 remove LIMIT 100; method to check continuity of headers returned but doesn't require
the set to start at a specific block number; config for
account_transformer; review fixes; update schema
2019-04-05 17:21:59 -05:00
Ian Norden
c1940c3e58 full_sync and light_sync receipt tables and light receipt repository methods and test 2019-04-05 16:36:48 -05:00
Rob Mulholand
fdd870563e Update db schema for latest changes 2019-04-05 14:51:51 -05:00
Ian Norden
5dcf534b2c review fixes 2019-04-04 15:21:39 -05:00
Ian Norden
197f98c93d going ahead and indexing the entire uncle blocks (one of the issues open on public); finish tests 2019-04-02 14:32:30 -05:00
Ian Norden
185f4c0e93 adjust block/uncle reward tests and add methods to pkg/geth blockchain and ethclient for direct fetching of eth balances 2019-04-02 10:10:17 -05:00
Ian Norden
9030cff2bd keep track of who is receiving uncle rewards 2019-04-02 10:10:17 -05:00
Rob Mulholand
54d46638a8 updates for light sync transactions 2019-03-28 14:31:17 -05:00
Rob Mulholand
79e011aad2 Add light sync transactions table
- Foreign key to header instead of block
- Can use same Transaction struct for both
2019-03-28 14:31:17 -05:00
Rob Mulholand
d93006321b Rename transactions to full_sync_transactions
- Table has foreign key to blocks
- Add transaction RLP and transaction index to table
- Enables other tables with standalone transactions or transactions
  associated with other data structures (e.g. headers)
2019-03-28 14:31:17 -05:00
Rob Mulholand
d9e2bece27 Consolidate migrations
- Facilitate modifying migrations without cascading consequences for
  later migrations updating the same tables
2019-03-23 10:57:39 -05:00
Rob Mulholand
5eff2618ed Assorted cleanups
- Remove DataDog
- Remove token_supply table
- Drop from more tables when cleaning DB for tests
2019-03-08 11:35:10 -06:00
Edvard Hübinette
55b6ad9381 Optimise missing block numbers query (#23) 2019-03-05 11:52:38 +01:00
Ian Norden
2c5ddd03ba core repo migrations in fixed form 2019-02-26 00:52:54 -06:00
Ian Norden
708425c4d6 rebase; extract factories and the mocks they are dependent on to
libraries/shared; adjust omni test_helpers to drop and recreate
checked_headers table to avoid reaching 1600 column limit after repeated
tests (dropping columns doesn't actually remove them from contributing
to that limit)
2019-02-25 01:34:38 -06:00
Ian Norden
f2072561a7 removing mcd/maker references; delete test_data; update READMEs; use
logrus
2019-02-24 21:53:04 -06:00
Ian Norden
3d34a9e7c9 remove maker migrations and convert back to timestamps and fix bug in
composeAndExecute command
2019-02-24 21:52:23 -06:00
Rob Mulholand
3d9341ea1d Reference ilks foreign key from Cat storage repository 2019-02-21 14:33:59 -06:00
Rob Mulholand
dbdd16d3a2 (VDB-380) Create separate table for ilks
- reference ilk by foreign key every place it's used
2019-02-21 11:23:00 -06:00
Rob Mulholand
a27aaa7e68 Use timestamps for migrations with numeric indexes
- Reverting the commit that switched away from timestamps did not modify
  migrations created since that change was applied
- Apply timestamps that are incremented by one from previous
2019-02-20 16:01:35 -06:00
Rob Mulholand
925b1b0b1c Revert "Convert existing migrations from timestamp to versioned"
- This reverts commit 4e2536bf421fd9d2c9067d81f30e0f45c6b94fe7.
- Restore timestamped migrations so goose fix can handle ordering
  dynamically with plugins also supplying timestamp based migrations
2019-02-20 16:01:35 -06:00
Rob Mulholand
11dd641a84 (VDB-354) Queue unrecognized storage diffs
- If we recognize a storage diff as coming from a watched contract but
  don't recognize the key, queue it for retrying later (after we've seen
  an event that might help us recognize the key)
- Remove unused errs and args
- Panic on unrecognized types (should not happen)
2019-02-20 15:22:39 -06:00
Edvard
fb47a747f6 Fix confusion between nflip and flip 2019-02-20 14:37:55 +01:00
Edvard
5746de7c97 Add tests for Cat storage diff transformer
Update schema after rebase migrations

Fix small issues from review
2019-02-20 14:37:46 +01:00
Edvard
6e0f033078 Add migrations for cat storage tables 2019-02-20 14:37:46 +01:00
Taka Goto
a8322c7d26 add woe to storage diff vow contract 2019-02-12 21:50:52 -06:00
Taka Goto
764befef69 fix rollback migration 2019-02-11 15:32:38 -06:00
Taka Goto
c2c21325b5 vow contract storage diffing excluding sin 2019-02-11 15:28:54 -06:00
Rob Mulholand
c05d3b7836 (VDB-280) Consume vat contract storage diffs 2019-02-08 13:36:30 -06:00
Takayuki Goto
0a024d429d * update transformer to able to recheck headers (#4)
* update transformer to able to recheck headers
* put cap on rechecking header
* integration test for recheck headers
* use enum for recheck headers param; make recheck cap configurable
* update integration tests with new test config
* update omni pkg with new recheck header column type
* update migration with new migration tool and final tweaks needed to accommodate changes in omni pkg
2019-02-08 10:35:46 -06:00
Rob Mulholand
867f92c431 (VDB-298) Consume Pit contract storage diffs
- Continuously parse storage diffs CSV data to read Pit contract state
- Convert ilks in database to raw bytes32 value for use in generating
  storage keys dynamically
- Persist storage diffs with block number and hash for validation
2019-02-06 10:31:46 -06:00
Edvard Hübinette
b4ded221c3 Merge branch 'staging' into VDB-281-Migrations-remake 2019-02-01 17:29:56 +01:00
Edvard
699dc1143d Update schema with for goose metatable 2019-01-25 11:44:05 +01:00
Edvard
4c87ef44b8 Apply taka's fix to faulty down migration 2019-01-25 11:44:05 +01:00
Edvard
8de6fc45e1 Remove tx from migrations (goose does it automagically) 2019-01-25 11:44:05 +01:00
Edvard
bbb9260d13 Convert existing migrations from timestamp to versioned 2019-01-25 11:44:05 +01:00
Edvard
0a96a366d3 Change migration timestamp from epoch to yyyymmddhhmmss 2019-01-25 11:44:05 +01:00
Ian Norden
1f95eb6443 Merge branch 'staging' into pub_merge 2019-01-24 09:47:12 -06:00
Taka Goto
b6baa106cc bring back transaction_id index when adding back transaction_id column 2019-01-23 10:24:18 -06:00
Ian Norden
46b898f690 merge public vulcanizedb/master 2019-01-23 00:37:26 -06:00
Ian Norden
ce428199eb finishing porting omni watcher to work with light sync; split into full, light,
and shared directories and refactor as much into shared; finish
lightSync omni watcher tests
2018-12-07 09:54:53 -06:00
Ian Norden
390a60f7f6 port over lightSync updates from maker repo 2018-11-29 20:33:21 -06:00
Rob Mulholand
854236fd99 (VDB-267) Remove pit file stability fee
- Method is no longer present on the Pit contract
2018-11-15 12:14:59 -06:00
Rob Mulholand
cb141f7cb7 (VDB-204) Add transformer for Flap kick 2018-11-07 10:52:55 -06:00
Ian Norden
d66e50dad6 Changes to address Rob's comments. Added generic/event_triggered transformer, tests, and repo migrations for Burn and Mint events 2018-11-03 13:49:23 -05:00
Ian Norden
ada872404e Transfer and Approval event repo and repo tests + db migrations for these events. 2018-11-03 13:49:23 -05:00
Ian Norden
44e0a8d303 retriever for generating list of all token holder addresses + updated transformer to use said addresses to populate balanceOf and allowance information and added database migrations for balance and allowance tables 2018-11-03 13:49:23 -05:00
Ian Norden
1d50a0ace0 fixed tests for getting balanceOf and allowance, added balanceOf and allowance to the repo and repo tests and updated sql schema with new token_balance and token_allowance tables to accept this data 2018-11-03 13:49:23 -05:00
Rob Mulholand
05186634bd Add light sync command
- Only syncs block headers (excludes block bodies, transactions, receipts, and logs)
- Modifies validation window to include the most recent block
- Isolates validation window to the variable defined in the cmd directory (blocks
  have a separate variable defined in the block_repository for determining when
  to set a block as final)
2018-11-03 13:49:23 -05:00
Elizabeth Engelman
b6f93e735f Add ERC20 token watcher example
- starting with the totalSupply function
- sets contract config on transformer by passing it into the transformer
initializer
- handles block records with the same number for different nodes for both creating token_supply records, and finding missing blocks
2018-11-03 13:49:23 -05:00
Elizabeth Engelman
929643132a Squash price feed trigger update into original migration 2018-11-01 15:12:04 -05:00
Elizabeth Engelman
99dc12c1fd Squash changes to tend and dent tables 2018-11-01 15:03:08 -05:00
Elizabeth Engelman
21239a986d Checking in new schema for reference 2018-11-01 14:56:36 -05:00
Elizabeth Engelman
8ce6ac9b7c Update vow_flog migration for consistency 2018-11-01 14:55:56 -05:00
Elizabeth Engelman
aedcafe488 Update create_vat_heal down migration 2018-11-01 14:13:07 -05:00
Elizabeth Engelman
504cb0ec2d Change timestamp on create_checked_headers_table migration 2018-11-01 14:12:58 -05:00
Edvard
78ad436228 Rename flog -> vow_flog 2018-10-26 15:42:18 +02:00
Edvard
77c7c9b453 Squash frob checked_headers 2018-10-26 15:26:36 +02:00
Edvard
6610821833 Squash cat_file checked_headers 2018-10-26 15:25:27 +02:00
Edvard
d87371cb11 Squash tend checked_headers 2018-10-26 15:24:03 +02:00
Edvard
058ab85518 Squash flip_kick checked_headers 2018-10-26 15:22:27 +02:00
Edvard
393c5297b2 Squash bite checked_headers 2018-10-26 15:21:04 +02:00
Edvard
000c504603 Remove _table from combined vat_init migration 2018-10-26 15:19:44 +02:00
Edvard
48525203f3 Squash dent checked_headers 2018-10-26 15:17:30 +02:00
Edvard
07a0c4f097 Squash deal checked_headers 2018-10-26 15:16:07 +02:00
Edvard
8eb6767da9 Squash drip_drip checked_headers 2018-10-26 15:14:17 +02:00
Edvard
60fec86881 Squash drip_file checked_headers 2018-10-26 15:12:41 +02:00
Edvard
a865aa1aea Squash pit_file checked_headers 2018-10-26 15:09:16 +02:00
Edvard
46a26f5bd9 Squash vat_init checked_headers 2018-10-26 15:08:58 +02:00
Edvard Hübinette
be249437c2 VDB-117: Add Vow.flog transformer (#80)
* Add DB migrations for flog

* Add Vow addresses and constants

* Add err logging to ABI signature getter, test for flog sig

* Add flog transformer

* Add tests+data for flog transformer

* Prepend DB tables and columns with vow_

* Panic on failure to get method sig
2018-10-25 12:19:46 +02:00
Elizabeth
5682ee988b Add log index to various events (#78)
* Add log_index field to flip kick

* Add log index to flop kick

* Add repo tests for Cat.file

* Add log_index to vat_flux

* Add log_index to vat_slip

* Add log_index to vat_toll

* Add log_index to vat_tune

* Add test to repos for updating checked headers
2018-10-23 16:33:04 -05:00
Rob Mulholand
42d60c3cbd Add log index to Vat move 2018-10-23 15:25:21 -05:00
Rob Mulholand
4ac896c246 Add log index to Vat init 2018-10-23 15:25:21 -05:00
Rob Mulholand
702f90367a Add log index to Vat heal 2018-10-23 15:25:21 -05:00
Rob Mulholand
650bbf5874 Add log index to Vat grab 2018-10-23 15:25:21 -05:00
Rob Mulholand
1e74c73591 Add log index to Vat fold 2018-10-23 15:25:21 -05:00
Rob Mulholand
db52955669 Add log index to Tend 2018-10-23 15:25:21 -05:00
Rob Mulholand
e0253cce55 Add log index to Price Feed events 2018-10-23 15:25:21 -05:00
Rob Mulholand
a0a86871b5 Add log index to Frob 2018-10-23 15:25:21 -05:00
Rob Mulholand
b3d00fe3c5 Add log index to Pit file events 2018-10-23 15:25:21 -05:00
Edvard
7d3bf1f0f6 Fixes after rebase 2018-10-23 10:45:48 +02:00
Elizabeth
954fcf2378 integration tests (#75)
* Deal integration test

* Add LogIndex to CatFileChopLump to update unique constraint

* Add LogIndex to cat_file_pit_vow records

* Add integration tests for Cat.file transformers

* Add log index to Cat.file flip
2018-10-22 13:28:42 -05:00
Rob Mulholand
d87fb14c6b Add log index to Drip file events 2018-10-22 12:14:43 -05:00
Rob Mulholand
de294ad8ea Add log index to Drip drip 2018-10-22 12:14:43 -05:00
Rob Mulholand
e5ba0496a6 Add log index to Dent 2018-10-22 12:14:43 -05:00
Rob Mulholand
1aafb7201e Add log index to Deal 2018-10-22 12:14:43 -05:00
Rob Mulholand
495f39abb8 Add log index to Bite 2018-10-22 12:14:43 -05:00
Rob Mulholand
fdbe0b0b44 Add Vat heal integration test 2018-10-19 10:56:28 -05:00
Rob Mulholand
130d11d0aa Add Vat slip transformer 2018-10-17 13:30:35 -05:00
Taka Goto
4d99e8a324 migration for vat flux 2018-10-17 10:53:36 -05:00
Rob Mulholand
d0226dbce5 Convert Bite event field types to numeric
- Enables arithmetic operations when constructing views
- Also removes application-level references to DB ID
2018-10-16 15:27:13 -05:00
Rob Mulholand
d3df685c46 Add timestamp to block header
- Enables quickly deriving timestamp from header for log events
2018-10-12 12:00:02 -05:00
Rob Mulholand
5f67161f41 Convert raw header to JSON
- Enables parsing values off of header via Postgres migration
2018-10-12 12:00:02 -05:00
Edvard Hübinette
be58dd4ac8 Add vat_move transformer (#47)
* Add vat_move transformer base

* Add vat_move migrations

* Add test data for vat_move

* Add vat_move transformer to initialisers

* Add numeric cast to psql insert of Rad

* Add new db schema

* Dependency update

* Expand abbreviation in repository

* Add test suite for vat_move

* Add header checking to transformer and mock repository

* Remove trailing zero in test data

* Fix minor mishaps

* Go fmt nitpicking

* Refactoring in tests

* Add tests covering checked headers stuff (and fix revealed bugs)

* go fmt fixes

* Implement batching behaviour of transformer

* Small fixes after review

* Go fmt
2018-10-12 16:13:13 +02:00
Elizabeth
8fe92edd26 Remove premature urn view migrations (#59) 2018-10-11 14:44:29 -05:00
Elizabeth
6c77f369d9 Vat heal (#56)
* VatHeal Converter

* Add VatHeal repository

* Add VatHeal transformer

* Add VatHeal to continuousLogSync command

* Mark vat_init_checked as true when creating vat init records

* Update urn and v converting

* Return error if Repository.MarkCheckedHeader fails

* Add deleting vat heal from test cleanup method
2018-10-10 11:56:06 -05:00
David Terry
b1745a6403 Merge branch 'staging' into VDB-101-vat-fold-transformer
Conflicts:
	cmd/continuousLogSync.go
	db/schema.sql
	pkg/transformers/shared/constants.go
	pkg/transformers/transformers.go
	test_config/test_config.go
2018-10-10 14:09:14 +03:00