Commit Graph

248 Commits

Author SHA1 Message Date
Edvard Hübinette
55b6ad9381 Optimise missing block numbers query (#23) 2019-03-05 11:52:38 +01:00
Ian Norden
2c5ddd03ba core repo migrations in fixed form 2019-02-26 00:52:54 -06:00
Ian Norden
708425c4d6 rebase; extract factories and the mocks they are dependent on to
libraries/shared; adjust omni test_helpers to drop and recreate
checked_headers table to avoid reaching 1600 column limit after repeated
tests (dropping columns doesn't actually remove them from contributing
to that limit)
2019-02-25 01:34:38 -06:00
Rob Mulholand
3d9341ea1d Reference ilks foreign key from Cat storage repository 2019-02-21 14:33:59 -06:00
Rob Mulholand
dbdd16d3a2 (VDB-380) Create separate table for ilks
- reference ilk by foreign key every place it's used
2019-02-21 11:23:00 -06:00
Rob Mulholand
a27aaa7e68 Use timestamps for migrations with numeric indexes
- Reverting the commit that switched away from timestamps did not modify
  migrations created since that change was applied
- Apply timestamps that are incremented by one from previous
2019-02-20 16:01:35 -06:00
Rob Mulholand
925b1b0b1c Revert "Convert existing migrations from timestamp to versioned"
- This reverts commit 4e2536bf421fd9d2c9067d81f30e0f45c6b94fe7.
- Restore timestamped migrations so goose fix can handle ordering
  dynamically with plugins also supplying timestamp based migrations
2019-02-20 16:01:35 -06:00
Rob Mulholand
11dd641a84 (VDB-354) Queue unrecognized storage diffs
- If we recognize a storage diff as coming from a watched contract but
  don't recognize the key, queue it for retrying later (after we've seen
  an event that might help us recognize the key)
- Remove unused errs and args
- Panic on unrecognized types (should not happen)
2019-02-20 15:22:39 -06:00
Edvard
fb47a747f6 Fix confusion between nflip and flip 2019-02-20 14:37:55 +01:00
Edvard
6e0f033078 Add migrations for cat storage tables 2019-02-20 14:37:46 +01:00
Taka Goto
a8322c7d26 add woe to storage diff vow contract 2019-02-12 21:50:52 -06:00
Taka Goto
764befef69 fix rollback migration 2019-02-11 15:32:38 -06:00
Taka Goto
c2c21325b5 vow contract storage diffing excluding sin 2019-02-11 15:28:54 -06:00
Rob Mulholand
c05d3b7836 (VDB-280) Consume vat contract storage diffs 2019-02-08 13:36:30 -06:00
Takayuki Goto
0a024d429d * update transformer to able to recheck headers (#4)
* update transformer to able to recheck headers
* put cap on rechecking header
* integration test for recheck headers
* use enum for recheck headers param; make recheck cap configurable
* update integration tests with new test config
* update omni pkg with new recheck header column type
* update migration with new migration tool and final tweaks needed to accommodate changes in omni pkg
2019-02-08 10:35:46 -06:00
Rob Mulholand
867f92c431 (VDB-298) Consume Pit contract storage diffs
- Continuously parse storage diffs CSV data to read Pit contract state
- Convert ilks in database to raw bytes32 value for use in generating
  storage keys dynamically
- Persist storage diffs with block number and hash for validation
2019-02-06 10:31:46 -06:00
Edvard
4c87ef44b8 Apply taka's fix to faulty down migration 2019-01-25 11:44:05 +01:00
Edvard
8de6fc45e1 Remove tx from migrations (goose does it automagically) 2019-01-25 11:44:05 +01:00
Edvard
bbb9260d13 Convert existing migrations from timestamp to versioned 2019-01-25 11:44:05 +01:00
Edvard
0a96a366d3 Change migration timestamp from epoch to yyyymmddhhmmss 2019-01-25 11:44:05 +01:00
Taka Goto
b6baa106cc bring back transaction_id index when adding back transaction_id column 2019-01-23 10:24:18 -06:00
Rob Mulholand
854236fd99 (VDB-267) Remove pit file stability fee
- Method is no longer present on the Pit contract
2018-11-15 12:14:59 -06:00
Rob Mulholand
cb141f7cb7 (VDB-204) Add transformer for Flap kick 2018-11-07 10:52:55 -06:00
Elizabeth Engelman
929643132a Squash price feed trigger update into original migration 2018-11-01 15:12:04 -05:00
Elizabeth Engelman
99dc12c1fd Squash changes to tend and dent tables 2018-11-01 15:03:08 -05:00
Elizabeth Engelman
8ce6ac9b7c Update vow_flog migration for consistency 2018-11-01 14:55:56 -05:00
Elizabeth Engelman
aedcafe488 Update create_vat_heal down migration 2018-11-01 14:13:07 -05:00
Elizabeth Engelman
504cb0ec2d Change timestamp on create_checked_headers_table migration 2018-11-01 14:12:58 -05:00
Edvard
78ad436228 Rename flog -> vow_flog 2018-10-26 15:42:18 +02:00
Edvard
77c7c9b453 Squash frob checked_headers 2018-10-26 15:26:36 +02:00
Edvard
6610821833 Squash cat_file checked_headers 2018-10-26 15:25:27 +02:00
Edvard
d87371cb11 Squash tend checked_headers 2018-10-26 15:24:03 +02:00
Edvard
058ab85518 Squash flip_kick checked_headers 2018-10-26 15:22:27 +02:00
Edvard
393c5297b2 Squash bite checked_headers 2018-10-26 15:21:04 +02:00
Edvard
000c504603 Remove _table from combined vat_init migration 2018-10-26 15:19:44 +02:00
Edvard
48525203f3 Squash dent checked_headers 2018-10-26 15:17:30 +02:00
Edvard
07a0c4f097 Squash deal checked_headers 2018-10-26 15:16:07 +02:00
Edvard
8eb6767da9 Squash drip_drip checked_headers 2018-10-26 15:14:17 +02:00
Edvard
60fec86881 Squash drip_file checked_headers 2018-10-26 15:12:41 +02:00
Edvard
a865aa1aea Squash pit_file checked_headers 2018-10-26 15:09:16 +02:00
Edvard
46a26f5bd9 Squash vat_init checked_headers 2018-10-26 15:08:58 +02:00
Edvard Hübinette
be249437c2 VDB-117: Add Vow.flog transformer (#80)
* Add DB migrations for flog

* Add Vow addresses and constants

* Add err logging to ABI signature getter, test for flog sig

* Add flog transformer

* Add tests+data for flog transformer

* Prepend DB tables and columns with vow_

* Panic on failure to get method sig
2018-10-25 12:19:46 +02:00
Elizabeth
5682ee988b Add log index to various events (#78)
* Add log_index field to flip kick

* Add log index to flop kick

* Add repo tests for Cat.file

* Add log_index to vat_flux

* Add log_index to vat_slip

* Add log_index to vat_toll

* Add log_index to vat_tune

* Add test to repos for updating checked headers
2018-10-23 16:33:04 -05:00
Rob Mulholand
42d60c3cbd Add log index to Vat move 2018-10-23 15:25:21 -05:00
Rob Mulholand
4ac896c246 Add log index to Vat init 2018-10-23 15:25:21 -05:00
Rob Mulholand
702f90367a Add log index to Vat heal 2018-10-23 15:25:21 -05:00
Rob Mulholand
650bbf5874 Add log index to Vat grab 2018-10-23 15:25:21 -05:00
Rob Mulholand
1e74c73591 Add log index to Vat fold 2018-10-23 15:25:21 -05:00
Rob Mulholand
db52955669 Add log index to Tend 2018-10-23 15:25:21 -05:00
Rob Mulholand
e0253cce55 Add log index to Price Feed events 2018-10-23 15:25:21 -05:00
Rob Mulholand
a0a86871b5 Add log index to Frob 2018-10-23 15:25:21 -05:00
Rob Mulholand
b3d00fe3c5 Add log index to Pit file events 2018-10-23 15:25:21 -05:00
Elizabeth
954fcf2378 integration tests (#75)
* Deal integration test

* Add LogIndex to CatFileChopLump to update unique constraint

* Add LogIndex to cat_file_pit_vow records

* Add integration tests for Cat.file transformers

* Add log index to Cat.file flip
2018-10-22 13:28:42 -05:00
Rob Mulholand
d87fb14c6b Add log index to Drip file events 2018-10-22 12:14:43 -05:00
Rob Mulholand
de294ad8ea Add log index to Drip drip 2018-10-22 12:14:43 -05:00
Rob Mulholand
e5ba0496a6 Add log index to Dent 2018-10-22 12:14:43 -05:00
Rob Mulholand
1aafb7201e Add log index to Deal 2018-10-22 12:14:43 -05:00
Rob Mulholand
495f39abb8 Add log index to Bite 2018-10-22 12:14:43 -05:00
Rob Mulholand
fdbe0b0b44 Add Vat heal integration test 2018-10-19 10:56:28 -05:00
Rob Mulholand
130d11d0aa Add Vat slip transformer 2018-10-17 13:30:35 -05:00
Taka Goto
4d99e8a324 migration for vat flux 2018-10-17 10:53:36 -05:00
Rob Mulholand
d0226dbce5 Convert Bite event field types to numeric
- Enables arithmetic operations when constructing views
- Also removes application-level references to DB ID
2018-10-16 15:27:13 -05:00
Rob Mulholand
d3df685c46 Add timestamp to block header
- Enables quickly deriving timestamp from header for log events
2018-10-12 12:00:02 -05:00
Rob Mulholand
5f67161f41 Convert raw header to JSON
- Enables parsing values off of header via Postgres migration
2018-10-12 12:00:02 -05:00
Edvard Hübinette
be58dd4ac8 Add vat_move transformer (#47)
* Add vat_move transformer base

* Add vat_move migrations

* Add test data for vat_move

* Add vat_move transformer to initialisers

* Add numeric cast to psql insert of Rad

* Add new db schema

* Dependency update

* Expand abbreviation in repository

* Add test suite for vat_move

* Add header checking to transformer and mock repository

* Remove trailing zero in test data

* Fix minor mishaps

* Go fmt nitpicking

* Refactoring in tests

* Add tests covering checked headers stuff (and fix revealed bugs)

* go fmt fixes

* Implement batching behaviour of transformer

* Small fixes after review

* Go fmt
2018-10-12 16:13:13 +02:00
Elizabeth
8fe92edd26 Remove premature urn view migrations (#59) 2018-10-11 14:44:29 -05:00
Elizabeth
6c77f369d9 Vat heal (#56)
* VatHeal Converter

* Add VatHeal repository

* Add VatHeal transformer

* Add VatHeal to continuousLogSync command

* Mark vat_init_checked as true when creating vat init records

* Update urn and v converting

* Return error if Repository.MarkCheckedHeader fails

* Add deleting vat heal from test cleanup method
2018-10-10 11:56:06 -05:00
David Terry
b1745a6403 Merge branch 'staging' into VDB-101-vat-fold-transformer
Conflicts:
	cmd/continuousLogSync.go
	db/schema.sql
	pkg/transformers/shared/constants.go
	pkg/transformers/transformers.go
	test_config/test_config.go
2018-10-10 14:09:14 +03:00
David Terry
f05f2e3ae9 vat.fold: correct table name in down migration 2018-10-10 13:37:31 +03:00
Rob Mulholand
e935f2f553 Add Vat grab transformer 2018-10-09 16:48:19 -05:00
Rob Mulholand
61dca314c4 Add Vat tune transformer 2018-10-09 14:37:26 -05:00
Elizabeth
ed1ed979b3 Remove bid_id unique constraint for Tend and Dent (#48) 2018-10-09 14:32:23 -05:00
Rob Mulholand
8ee106e14d Add Vat toll transformer 2018-10-09 11:40:35 -05:00
David Terry
09770b2e50 Merge branch 'staging' into VDB-101-vat-fold-transformer 2018-10-09 14:15:07 +03:00
Taka Goto
347a339687 update frob and bite converters and repository 2018-10-07 22:02:14 -05:00
Taka Goto
7080c0a74f migration to update columns 2018-10-04 13:56:44 -05:00
David Terry
b3e2218eb5 Merge branch 'staging' into VDB-101-vat-fold-transformer 2018-10-04 18:49:38 +03:00
David Terry
f79a017b48 Vat.fold: store ilk and urn as text instead of bytea 2018-10-04 18:48:06 +03:00
David Terry
2feb7ea546 Vat.fold: add migrations 2018-10-03 12:48:11 +02:00
Rob Mulholand
bf44ec614a Log checked headers for MCD events (#43)
* Log checked headers for Bite events

* Log checked headers for Drip drip events

* Log checked headers for Drip file events

- mark headers checked for 3 implementations of file on the Drip contract

* Log checked headers for Frob frob events

* Log checked headers for Pit file events

- mark headers checked for 3 implementations of file on the Pit contract

* Log checked headers for Vat init events
2018-10-02 17:25:38 -05:00
Rob Mulholand
58f17a24ae Add transformers for Cat file events
- transform logs from 3 implementations of file on the Cat contract
2018-09-27 10:32:08 -05:00
Elizabeth
b0cd852aa3 Flop deal (#39)
*Refactor deal transformer to take in several contract addresses
* Add flop address to deal transformer
2018-09-27 10:12:29 -05:00
Taka Goto
aaec3adf9a update price feed trigger to return new price feed 2018-09-26 11:32:24 -05:00
Elizabeth
31516ea87e Flop kick (#28)
* FlopKick transformer
2018-09-26 09:42:52 -05:00
Rob Mulholand
ac362650db Log checked headers for tend events 2018-09-25 14:11:35 -05:00
Rob Mulholand
009cbb8aa0 Log checked headers for flip kick events 2018-09-25 14:11:35 -05:00
Rob Mulholand
d8b1a61ff5 Log checked headers for dent events 2018-09-25 14:11:35 -05:00
Rob Mulholand
c3f1fcf796 Log checked headers for deal events
- mark header as checked if no matching logs
- mark header as checked if all matching logs persisted successfully
- batch convert and insert matching logs to enable transactional persist
- enable skipping previously checked block headers when restarting a sync
2018-09-25 14:11:35 -05:00
Elizabeth Engelman
6a112e29c4 Update migration names to use unix timestamp 2018-09-19 13:42:15 -05:00
Elizabeth
60901d9095 Update tend, flipkick and dent tables for consistency (#24) 2018-09-18 09:27:14 -05:00
Rob Mulholand
06881db350 Sync unchecked headers for price feeds
- Indicate that a header has been checked for logs if no logs returned
- Indicate that a header has been checked for logs when persisting a log
- Fetch headers that need to be checked based on absence of the above
2018-09-17 14:14:52 -05:00
Takayuki Goto
a6cf173fe7 Merge pull request #11 from 8thlight/add_postgraphile_subscription
Add Postgraphile subscription exposure for price feed
2018-09-17 11:53:47 -05:00
Rob Mulholand
f5595bee78 Add Drip drip transformer 2018-09-13 16:35:10 -05:00
Elizabeth
939ead0c82 Deal transformer (#19) 2018-09-13 09:06:26 -05:00
Rob Mulholand
d69063636e Add raw log to price feeds 2018-09-12 11:35:32 -05:00
Rob Mulholand
b2ba7ee1e3 Add transaction index to flip kick 2018-09-12 11:35:32 -05:00
Rob Mulholand
a843de5eb7 Add drip file transformers
- for all three file functions on the drip contract
2018-09-12 11:35:32 -05:00
Takayuki Goto
c8b002deaa Merge branch 'master' into add_postgraphile_subscription 2018-09-12 09:48:00 -05:00
Taka Goto
13b19c0870 add price feed trigger 2018-09-12 09:43:23 -05:00
Rob Mulholand
ff110592bd Add Vat init transformer 2018-09-11 16:30:29 -05:00
Elizabeth
9abe3ffa68 Dent (#10)
Convert and persist LogNote events as dent records
2018-09-11 14:01:45 -05:00
Rob Mulholand
d48cf38070 Add Pit file debt ceiling event 2018-09-05 13:51:18 -05:00
Rob Mulholand
e1f609528d Add Pit file stability fee event
- file event on the pit contract is overloaded and each implementation
  logs a note
- existing Pit file transformer now specified as dealing with ilks
2018-09-05 11:36:13 -05:00
Rob Mulholand
8f5bc0a956 Add raw log to Frob record in DB
- Enables parsing additional values off the log if necessary
- Also update Frob to reflect DSS changes (lad => urn)
- Move Frob constants to shared directory
2018-09-04 16:11:54 -05:00
Rob Mulholand
db4b891398 Backfill Pit.file log events 2018-09-04 16:07:35 -05:00
Elizabeth
53635ec362 Update flip kick and tend transformers due to contract changes
* Update FlipperAddress to new local ganache address

* Update flip_kick table

* Update flipkick transformer to handle new signature and abi

* Update tend table

* Update tend converter
2018-09-04 15:50:29 -05:00
Takayuki Goto
807b7accd6 Merge pull request #12 from 8thlight/bite-event
bite event transformer
2018-09-04 09:58:51 -05:00
Taka Goto
adc7c47244 update tests and import order per PR feedback 2018-08-30 15:06:14 -05:00
Taka Goto
985fa49178 migration file to create bite; create bite repository
add transaction index and raw log to bite table
work on converter for bite event
update bite repository, replace guy with 32byte lad; create bite converter to entity
update field type for bite event; start on bite transformer
finish bite event transformer
2018-08-30 08:38:43 -05:00
Rob Mulholand
1916d585fa Incorporate DSS updates (#8)
* Incorporate DSS updates
- Lad renamed to Guy
- Dink and Dart added to Frob
* update test chain data
* Remove Mom field from FlipKick
* Update Flip ABI and sample flip kick data
* Incorporate updates to Frob event
- Guy renamed to Lad
- Era and Gem removed, iArt added
- Also turn off integration tests that read from test chain while
events are actively under development
2018-08-28 14:16:39 -05:00
Elizabeth
66ad7e3021 Transform and persist Tend log events 2018-08-16 16:36:35 -05:00
Rob Mulholand
634604d0b5 Combine price feed transformers
- fetches logs from all three price feeds in one query
- assumes eth/usd price feed will be updated to include LogValue event
- updates transformers to run separate from header sync
2018-08-16 11:22:16 -05:00
Rob Mulholand
6e68dc4a92 Add Rep price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand
2949996d22 Add Pip price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand
0551147453 Add Pep price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand
ac63e43543 Backfill Frob log events 2018-08-14 10:47:43 -05:00
Elizabeth
c617cd9c9d Add FlipKick log events transformer 2018-08-07 10:51:34 -05:00
Rob Mulholand
1355271011 Add light sync command
- Only syncs block headers (excludes block bodies, transactions, receipts, and logs)
- Modifies validation window to include the most recent block
- Isolates validation window to the variable defined in the cmd directory (blocks
  have a separate variable defined in the block_repository for determining when
  to set a block as final)
2018-07-18 16:34:12 -05:00
Elizabeth Engelman
7ee253b2a3 Add ERC20 token watcher example
- starting with the totalSupply function
- sets contract config on transformer by passing it into the transformer
initializer
- handles block records with the same number for different nodes for both creating token_supply records, and finding missing blocks
2018-06-27 14:41:51 -05:00
Rob Mulholand
d5c2ab33fc Sync only missing blocks on cold import
= Add eth_node_fingerprint to block that can be imitated by both hot and cold imports
- Only sync missing blocks (blocks that are missing or don't share a fingerprint) on cold import
- Set block is_final status after import
2018-05-07 16:38:04 -05:00
Rob Mulholand
462f94d84a Associate receipts with blocks instead of transactions
- Allows us to fetch receipts by block and persist even when
associated transaction is unknown. (The associated transaction
can still be derived from the tx_hash column at query time, but
is an expensive operation to require for inserts).
2018-05-04 10:52:28 -05:00
Matt Krump
1b4e57a5b4 Rename node table and node column 2018-03-21 13:24:13 -05:00
Matt K
5a652190d9 Allow Parity as ingest node (#36)
* Upgrade go-ethereum to v1.8

* Add Node Info for parity nodes

* Upgrade start_private_blockchain to use v1.8
2018-03-07 15:29:21 -06:00
Matt K
605b0a96ae Add graphql server (#27)
* Add graphql server

* Update Makefile

* Update log_filters constraint

* Add GetLogFilter to repo

* Update travis (use Makefile, go fmt, go vet)

* Add logFilter schema and resolvers

* Add GetWatchedEvent to watched_events_repo

* Add watchedEventLog schema and resolvers
2018-02-08 10:12:08 -06:00
Matt K
d5852654bb Update table columns (#26)
* Update block table names

* Update transaction table names
2018-02-02 16:12:14 -06:00
Matt K
aea9c7b5e2 Separate repositories (#25)
* Separate Repository into multiple Repositories

* Use struct scan for transactions

* Use struct scan for blocks

* Remove unused block repo methods

* Update naming

* Rename / Cleanup repository related fields
2018-02-02 15:53:16 -06:00
Matt K
572023cdf5 Commandline (#135)
* Add cmd line tool and Makefile

* Add shared utils pkg

* Add cmdline README

* Update godo for new structure
2018-01-25 13:21:55 -06:00
Matt K
c00b8a5a98 Add Filters (#133)
* Add LogFilter struct

* Add log_filters table

* Add view for events watching

* Add cmd line "add_filter" to mimic eventual endpoint

* Allow multiple filters in config
2018-01-23 12:43:35 -06:00
Matt Krump
6583ce72b8 Remove log_uc
* Logs now are attached to receipt, so removing block + index unique
constraint
2018-01-16 09:44:46 -06:00
Matt Krump
82c39a2c1f Add fk constraint on logs 2018-01-16 08:58:11 -06:00
Matt K
a9bea4f492 Allow for multiple Geth nodes (#128) 2018-01-10 15:54:36 -06:00
Matt K
4fabe3e917 Add receipts (#119)
* Conversion between Geth Receipt and core.Receipt

* Add receipt to DB

* Insert receipts with transactions

* Update Travis CI to use dep for dependencies
2018-01-03 11:23:43 -06:00
Matt Krump
351d315a4c Add tx fields 2017-12-28 17:23:56 -06:00
Matt Krump
8b024bade9 Add block rewards to db 2017-12-28 11:58:26 -06:00
Matt Krump
cb4e745464 Add extra data field 2017-12-27 12:10:08 -06:00
Matt Krump
c992186846 Add Block Miner 2017-12-27 10:50:56 -06:00
Matt K
a786241c8c Add indices for tx_to and tx_from (#112) 2017-12-20 16:58:37 -06:00
Matt K
24bc83a448 Block categorization (#110)
* Add block categorization (is_final=)

* Add godo task for vulcanizeDB (Example of how everything could work together)

* Add unique constraint on block_number and node

* Add index on block_id for transactions_table

* Add node_id index on blocks table

* Sort transactions returned from FindBlock by tx_hash

* lowercase tx_to, tx_from like etherscan
2017-12-20 14:06:22 -06:00
Matt K
266c9587c8 Canonical blocks (#108)
* Update Block w/ newest Block

* Add cascading delete to blocks and transactions tables

* Add handling for new conflicting blocks

* Command line version of sliding window n behind HEAD
2017-12-19 14:14:41 -06:00
Matt K
a68f277066 Save Logs to DB (#100)
* Save logs to database

* Save multiple logs to db

* Add block number parameter to `FindLogs`
2017-12-12 15:55:26 -06:00
Matt K
921bde1089 Update BlockChain to record NodeInfo (#95) 2017-12-07 13:32:16 -06:00
Matt K
18163f970e Get ABI via etherscan API (#96)
- Added ABI request
- Add unique constraint on contract hash for watched contracts
2017-12-07 09:58:06 -06:00
Eric Meyer
fa2766b64d Store contract ABI on watched_contracts 2017-12-04 14:13:15 -06:00
Matt K
f0f086e48d Merge pull request #69 from 8thlight/watch_contract
* Add contracts table / start building out watch contracts
2017-11-13 10:55:55 -06:00
Matt Krump
4ad1d531a8 Update contract naming per Eric PR review 2017-11-13 10:11:27 -06:00
Matt Krump
c7bd6de7da Added From field to transactions 2017-11-09 16:51:22 -06:00
Matt Krump
30fadffb14 Add contracts table / start building out watch contracts 2017-11-09 15:06:01 -06:00
Eric Meyer
aeb1fe8862 Nest migrations under db directory 2017-11-06 13:12:57 -06:00