Commit Graph
42 Commits
Author SHA1 Message Date
Ian Norden 6c2d895023 composeAndExecute command that loads and executes over arbitrary transformer set exported from a go plugin generated according to config file; test for plugin generation, loading, and execution; work on plugins README 2019-02-24 15:23:35 -06:00
Takayuki Goto 0a024d429d * update transformer to able to recheck headers (#4)
* update transformer to able to recheck headers
* put cap on rechecking header
* integration test for recheck headers
* use enum for recheck headers param; make recheck cap configurable
* update integration tests with new test config
* update omni pkg with new recheck header column type
* update migration with new migration tool and final tweaks needed to accommodate changes in omni pkg
2019-02-08 10:35:46 -06:00
Ian Norden 8749c94994 update license to AGPL-3.0 2019-01-24 14:41:30 -06:00
Konstantin Zolotarev 65cb6e9092 VDB-205 Fix external configuration values (#121)
* Updated loading config from /Users/konstantinzolotarev/.vulcanizedb for addreses

* Updated contract addresses loading from env variables.

* Implement getter chain for addresses, configs and transformers

Solves the issue with declaring stuff as vars when the environment file hasn't been loaded yet. Instead, this data is fetched dynamically when the root commands are called.

* Update test data to use constant Kovan addresses

* Decouple integration tests from config file IO

* Add ABI to environment file, convert getter chains for related values

* Decouple tests suites from ABI config file IO

* Add startingBlock to environment and implement getter chain

* Exit when called without config file

* Add missing deployment blocks to staging.toml

* Remove comment
2019-01-23 19:44:09 +01:00
Andrew J Yao c8aa3dc275 Replace standard log with logrus import 2019-01-11 10:07:00 -08:00
yaoandrew 525fada3d7 refactor error handling in frob, pit, price, tend 2019-01-11 10:07:00 -08:00
Edvard 833dde62cf Remove injection of fetcher and repository to watcher 2019-01-07 15:19:31 +01:00
Edvard d93817f346 Interface-ify shared repository 2018-12-11 11:35:13 +01:00
Edvard 47c75d055b Update repository tests for absence of MissingHeaders 2018-12-04 17:53:46 +01:00
Edvard 38c745e8c3 Refactor fetching out from repositories to log_fetcher and watcher 2018-12-04 16:04:13 +01:00
Taka Goto e8be96a4e7 use logrus for logging 2018-11-21 10:14:11 -06:00
Elizabeth 128d20c9bf Vdb 269- fetch logs by hash (#122)
* Upgrade geth from 1.8.15 to 1.8.18

* Update vat_tune to use shared repository methods

* Query blockchain by block hash instead of block number range

* Remove hash validation from repositories

* Fix vow flog integration test

* Update README Travis build sticker

* Update constants formatting per go fmt

* Update EthPublicKeyParser.ParsePublicKey to use discv5.PubkeyID method

* Address PR comments
2018-11-21 09:30:00 -06:00
Edvard 8f4648a9e3 Add support for never-ending transformer 2018-11-14 10:11:16 -06:00
Rob Mulholand 82fd73ba3f (VDB-68) Verify log block hash matches header block hash
- Delete header on conflict to prompt data refresh (cascade deletes all
  data associated with that block)
- Derive header hash from rpc payload rather than computing it from data
  (prevents hash mismatch between blockchain and cache)
2018-11-13 14:51:39 -06:00
Elizabeth c1e10f09fb Vdb 67 convert numbers to ray and wad (#105)
* Add method to convert values to ray or wad units

* Convert data to ray or wad for cat_file_chop_lump

* Use shared convert functions in price feed conversion

* Pull common ray/wad values into vars

* Fix after rebase with staging
2018-11-09 14:53:20 -06:00
Rob Mulholand 0b58efb56a Extract common repository methods
- Inject column names to reduce duplication across common behavior
- Extract checked headers column names to constants
2018-11-08 13:08:59 -06:00
Rob Mulholand 8c168ed332 Extract constants package
- Reduce clutter in the shared package
2018-11-08 13:08:59 -06:00
Rob Mulholand 1c1a9c699a (VDB-209) Refactor price feed repository tests 2018-11-08 13:08:59 -06:00
Rob Mulholand b2a78f1da4 Remove unused transformer config struct 2018-11-07 10:52:55 -06:00
Elizabeth bfe4531d7e Common transformer tests (#97)
* Move chop lump transformer test to a generic log_note_transformer_test

* Add test cases to transformer test

- cat file flip
- cat file pit vow
- deal
- dent

* Move transformer tests over to transformer factor test file

- drip drip
- drip file ilk/repo/vow
- pit file deb ceiling/ilk/stability fee
- price feeds

* Move LogNoteTransformer test into factory package and remove log specific transformer tests

* Move Transformer test into factory package and remove log event specific transformer tests

* Respond to PR comments
2018-11-06 10:30:11 -06:00
Rob Mulholand d1ee501f12 Use transformer factory for Bite (#89)
* Use transformer factory for Bite

- introduces separate transformer factory for non-LogNote events
- converter includes `ToEntities` for events defined via ABI

* Updates after rebasing with staging
2018-10-26 13:26:10 -05:00
Rob Mulholand 47a3c35938 Use transformer factory for Price Feed events 2018-10-26 11:25:16 -05:00
Rob Mulholand e0253cce55 Add log index to Price Feed events 2018-10-23 15:25:21 -05:00
Elizabeth 3baca64903 Update address formatting (#67)
* Add getSignatures command for list of signatures in constants

* Update VatTune converter's formatting of addresses

and added an integration test

* Move VatTune and PriceFeeds integration tests to a common suite

* Update VatGrab converter for address format

* Update frob converter to fix urn and ilk formatting/decoding

* Update formatting

* Update make test command and add integrationtest command

* Add integration test to travis build

* Update creating headers in integration tests to include Raw and Timestamp fields
2018-10-17 13:04:55 -05:00
Rob Mulholand d3df685c46 Add timestamp to block header
- Enables quickly deriving timestamp from header for log events
2018-10-12 12:00:02 -05:00
Rob Mulholand 5f67161f41 Convert raw header to JSON
- Enables parsing values off of header via Postgres migration
2018-10-12 12:00:02 -05:00
David Terry f56c882b9d increase timeout on price feeds integration tests 2018-09-28 14:19:40 +02:00
Rob Mulholand 92525ca575 Make price feed log persistence atomic per block
- Prevents marking header as checked if only one of several price feed
  logs is successfully persisted
2018-09-25 14:11:35 -05:00
Rob Mulholand e138e58120 Set timeout before assertions in integration tests
- Prevent early evaluation from triggering failure
2018-09-21 11:28:17 -05:00
Rob Mulholand 56c45b6841 Add logging for every transformer
- Also add drip file transformer initializer
2018-09-19 11:00:20 -05:00
Rob Mulholand a735698f77 Update config for Kovan 2018-09-19 11:00:20 -05:00
Rob Mulholand 042f1142c0 Handle headers from POA chain
- Fetching headers from Kovan requires custom type without mixHash/Nonce
2018-09-19 11:00:20 -05:00
Rob Mulholand 778517f33f Add integration tests for price feed transformer
- Verify eth/usd, mkr/usd, and rep/usd price feed logs are successfully
  fetched from Kovan and persisted to Postgres
2018-09-19 11:00:20 -05:00
Rob Mulholand 06881db350 Sync unchecked headers for price feeds
- Indicate that a header has been checked for logs if no logs returned
- Indicate that a header has been checked for logs when persisting a log
- Fetch headers that need to be checked based on absence of the above
2018-09-17 14:14:52 -05:00
Rob Mulholand d69063636e Add raw log to price feeds 2018-09-12 11:35:32 -05:00
Rob Mulholand dc9bda7d68 Extract addresses to shared constants file 2018-09-11 16:30:29 -05:00
Rob Mulholand ceeaa0f8d3 Update licenses and imports
- Verify license added to each file
- Order imports by group and separate with line breaks
2018-09-11 16:30:29 -05:00
Rob Mulholand a30e5036fd generate log event signatures from functions 2018-09-04 16:19:03 -05:00
Rob Mulholand 634604d0b5 Combine price feed transformers
- fetches logs from all three price feeds in one query
- assumes eth/usd price feed will be updated to include LogValue event
- updates transformers to run separate from header sync
2018-08-16 11:22:16 -05:00
Rob Mulholand 9231d40369 Extract price feed contract addresses to config 2018-08-16 11:22:16 -05:00
Rob Mulholand 6e68dc4a92 Add Rep price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand 2949996d22 Add Pip price feed 2018-08-16 11:22:16 -05:00