Commit Graph

324 Commits

Author SHA1 Message Date
Edvard
32654ffabc Add transformer name to config for logging in factory transformer 2018-10-23 10:43:03 +02:00
Edvard
46d8ea8d97 Snag transformer topics from config in factory 2018-10-23 10:43:03 +02:00
Edvard
155569a1d8 Update vat_move tests 2018-10-23 10:43:03 +02:00
Edvard
af1dfbc489 Implement vat_move transformer using factory 2018-10-23 10:43:03 +02:00
Edvard
15e33db33d Fix transformer factory 2018-10-23 10:43:03 +02:00
Edvard
1b9e6491da Add transformer factory 2018-10-23 10:43:03 +02:00
Elizabeth
954fcf2378 integration tests (#75)
* Deal integration test

* Add LogIndex to CatFileChopLump to update unique constraint

* Add LogIndex to cat_file_pit_vow records

* Add integration tests for Cat.file transformers

* Add log index to Cat.file flip
2018-10-22 13:28:42 -05:00
Rob Mulholand
d87fb14c6b Add log index to Drip file events 2018-10-22 12:14:43 -05:00
Rob Mulholand
de294ad8ea Add log index to Drip drip 2018-10-22 12:14:43 -05:00
Rob Mulholand
e5ba0496a6 Add log index to Dent 2018-10-22 12:14:43 -05:00
Rob Mulholand
1aafb7201e Add log index to Deal 2018-10-22 12:14:43 -05:00
Rob Mulholand
495f39abb8 Add log index to Bite 2018-10-22 12:14:43 -05:00
Rob Mulholand
ea3de51792 Add Flop kick integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
fa828cf041 Add Tend integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
cbb5ffa3c7 Add Pit file stability fee integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
3449a1e246 Add Pit file debt ceiling integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
a83bf5ed0d Add Pit file ilk integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
677e78891d Add Drip file vow integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
0ce3e3d829 Add Bite integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
ddc7ee5143 Add Drip drip integration test 2018-10-19 10:56:28 -05:00
Rob Mulholand
b067ba565e Add Vat fold integration test 2018-10-19 10:56:28 -05:00
Rob Mulholand
fdbe0b0b44 Add Vat heal integration test 2018-10-19 10:56:28 -05:00
Rob Mulholand
a7b9b825c8 Add Vat init integration test 2018-10-19 10:56:28 -05:00
Rob Mulholand
9b427ea1e9 Add Vat move integration test 2018-10-19 10:56:28 -05:00
Taka Goto
21088eb3a2 integration test for vat flux 2018-10-17 16:25:12 -05:00
Konstantin Zolotarev
b38c29ce43 Merge pull request #64 from 8thlight/contract_addresses
Added configuration for contract addresses
2018-10-17 21:34:25 +03:00
Rob Mulholand
130d11d0aa Add Vat slip transformer 2018-10-17 13:30:35 -05:00
Elizabeth
3baca64903 Update address formatting (#67)
* Add getSignatures command for list of signatures in constants

* Update VatTune converter's formatting of addresses

and added an integration test

* Move VatTune and PriceFeeds integration tests to a common suite

* Update VatGrab converter for address format

* Update frob converter to fix urn and ilk formatting/decoding

* Update formatting

* Update make test command and add integrationtest command

* Add integration test to travis build

* Update creating headers in integration tests to include Raw and Timestamp fields
2018-10-17 13:04:55 -05:00
Taka Goto
4721a91faa use before each for transformer test 2018-10-17 12:15:15 -05:00
Taka Goto
a3ca319e9b remove unnecessary byte slicing 2018-10-17 12:04:11 -05:00
Taka Goto
2f64269ab3 update from pr feedback; update tests and use shared fake header 2018-10-17 11:48:57 -05:00
Taka Goto
f32a4fa2a5 use kovan vat contract address 2018-10-17 10:55:30 -05:00
Taka Goto
2e72a7e940 vat flux transformer 2018-10-17 10:53:53 -05:00
Rob Mulholand
d0226dbce5 Convert Bite event field types to numeric
- Enables arithmetic operations when constructing views
- Also removes application-level references to DB ID
2018-10-16 15:27:13 -05:00
Rob Mulholand
d3df685c46 Add timestamp to block header
- Enables quickly deriving timestamp from header for log events
2018-10-12 12:00:02 -05:00
Rob Mulholand
5f67161f41 Convert raw header to JSON
- Enables parsing values off of header via Postgres migration
2018-10-12 12:00:02 -05:00
Edvard Hübinette
be58dd4ac8 Add vat_move transformer (#47)
* Add vat_move transformer base

* Add vat_move migrations

* Add test data for vat_move

* Add vat_move transformer to initialisers

* Add numeric cast to psql insert of Rad

* Add new db schema

* Dependency update

* Expand abbreviation in repository

* Add test suite for vat_move

* Add header checking to transformer and mock repository

* Remove trailing zero in test data

* Fix minor mishaps

* Go fmt nitpicking

* Refactoring in tests

* Add tests covering checked headers stuff (and fix revealed bugs)

* go fmt fixes

* Implement batching behaviour of transformer

* Small fixes after review

* Go fmt
2018-10-12 16:13:13 +02:00
Konstantin Zolotarev
0b281eab47 Added default addresses to config & rename fallback var 2018-10-12 11:11:31 +03:00
Konstantin Zolotarev
245732b7f4 Added configuration for contract addresses 2018-10-11 19:10:53 +03:00
Elizabeth
6c77f369d9 Vat heal (#56)
* VatHeal Converter

* Add VatHeal repository

* Add VatHeal transformer

* Add VatHeal to continuousLogSync command

* Mark vat_init_checked as true when creating vat init records

* Update urn and v converting

* Return error if Repository.MarkCheckedHeader fails

* Add deleting vat heal from test cleanup method
2018-10-10 11:56:06 -05:00
David Terry
0bb3737027 vat.fold: pull common test setup into BeforeEach blocks 2018-10-10 18:13:10 +03:00
David Terry
b1745a6403 Merge branch 'staging' into VDB-101-vat-fold-transformer
Conflicts:
	cmd/continuousLogSync.go
	db/schema.sql
	pkg/transformers/shared/constants.go
	pkg/transformers/transformers.go
	test_config/test_config.go
2018-10-10 14:09:14 +03:00
David Terry
89ec0a1cab documentation: document config file for transformers 2018-10-10 14:06:00 +03:00
David Terry
95f1c94bf5 vat.fold: check for errors in insert before updating checked_headers 2018-10-10 14:03:37 +03:00
David Terry
1273acb733 vat.fold: add repository tests for MarkHeaderChecked 2018-10-10 13:51:44 +03:00
David Terry
3778d7ac06 vat.fold: add transformer tests for MarkHeaderChecked 2018-10-10 13:40:25 +03:00
David Terry
7e7ad13de6 vat.fold: handle errors 2018-10-10 13:39:33 +03:00
David Terry
dde57178f1 vat.fold: break out of loop in transformer if no logs found 2018-10-10 13:38:57 +03:00
David Terry
724839ce4a documentation: document vat event types 2018-10-10 13:37:59 +03:00
Rob Mulholand
e935f2f553 Add Vat grab transformer 2018-10-09 16:48:19 -05:00
Rob Mulholand
61dca314c4 Add Vat tune transformer 2018-10-09 14:37:26 -05:00
Rob Mulholand
8ee106e14d Add Vat toll transformer 2018-10-09 11:40:35 -05:00
David Terry
ecef1cbcdb transformers: add step by step docs 2018-10-09 14:25:27 +03:00
David Terry
09770b2e50 Merge branch 'staging' into VDB-101-vat-fold-transformer 2018-10-09 14:15:07 +03:00
David Terry
81dec589fc vat.fold: rollback broken test refactoring 2018-10-09 14:11:02 +03:00
David Terry
e040383fa5 vat.fold: add test for signature generation 2018-10-09 14:10:42 +03:00
David Terry
3e9901864b vat.fold: switch to batched interface 2018-10-09 13:34:22 +03:00
Taka Goto
347a339687 update frob and bite converters and repository 2018-10-07 22:02:14 -05:00
Taka Goto
7e6302c662 update bite to take in string for urn and ilk 2018-10-04 16:35:51 -05:00
David Terry
b3e2218eb5 Merge branch 'staging' into VDB-101-vat-fold-transformer 2018-10-04 18:49:38 +03:00
David Terry
53a74c39f7 Vat.fold: correct typo 2018-10-04 17:47:03 +03:00
David Terry
dde57ea70b Vat.fold: run transformer 2018-10-04 16:04:53 +03:00
David Terry
c8cb079da5 Vat.fold: add transformer & mocks 2018-10-04 14:34:30 +03:00
David Terry
46bba30938 Vat.fold: refactor repository tests 2018-10-03 19:38:47 +02:00
David Terry
ef0e3f9e11 Vat.fold: add repository & tests 2018-10-03 18:22:05 +02:00
David Terry
0c58e0ac5b Vat.fold: verify that the event signature is correct 2018-10-03 18:21:27 +02:00
David Terry
96f8279421 Vat.fold: update test data so it matches a real deployed kovan event 2018-10-03 16:03:30 +02:00
David Terry
b3295e3a86 Vat.fold: add converter & tests 2018-10-03 14:08:01 +02:00
David Terry
ce4d928fb8 transformers: tweak docs 2018-10-03 14:07:32 +02:00
David Terry
22fb019d3f docs: add links 2018-10-03 12:50:28 +02:00
David Terry
377c57ce1b transformers: format documentation 2018-10-03 12:44:05 +02:00
Rob Mulholand
bf44ec614a Log checked headers for MCD events (#43)
* Log checked headers for Bite events

* Log checked headers for Drip drip events

* Log checked headers for Drip file events

- mark headers checked for 3 implementations of file on the Drip contract

* Log checked headers for Frob frob events

* Log checked headers for Pit file events

- mark headers checked for 3 implementations of file on the Pit contract

* Log checked headers for Vat init events
2018-10-02 17:25:38 -05:00
David Terry
cf437e4007 VDB-101: Add signature for vat.fold 2018-09-28 16:31:29 +02:00
David Terry
f56c882b9d increase timeout on price feeds integration tests 2018-09-28 14:19:40 +02:00
Rob Mulholand
02853014b2 Fix Pit file converters
- Use topics 2 & 3 instead of topic 1 (call data instead of msg.sender)
- Prefer extracting fields from topics rather than data
2018-09-27 10:32:08 -05:00
Rob Mulholand
c17ad1f826 Add missing Describe statements 2018-09-27 10:32:08 -05:00
Rob Mulholand
58f17a24ae Add transformers for Cat file events
- transform logs from 3 implementations of file on the Cat contract
2018-09-27 10:32:08 -05:00
Elizabeth
b0cd852aa3 Flop deal (#39)
*Refactor deal transformer to take in several contract addresses
* Add flop address to deal transformer
2018-09-27 10:12:29 -05:00
Elizabeth
31516ea87e Flop kick (#28)
* FlopKick transformer
2018-09-26 09:42:52 -05:00
Rob Mulholand
a0ba6ca6bd Return error if RPC returns empty header
- Enables graceful failure if RPC returns empty data with no error
2018-09-25 14:11:35 -05:00
Rob Mulholand
92525ca575 Make price feed log persistence atomic per block
- Prevents marking header as checked if only one of several price feed
  logs is successfully persisted
2018-09-25 14:11:35 -05:00
Rob Mulholand
ac362650db Log checked headers for tend events 2018-09-25 14:11:35 -05:00
Rob Mulholand
009cbb8aa0 Log checked headers for flip kick events 2018-09-25 14:11:35 -05:00
Rob Mulholand
d8b1a61ff5 Log checked headers for dent events 2018-09-25 14:11:35 -05:00
Rob Mulholand
c3f1fcf796 Log checked headers for deal events
- mark header as checked if no matching logs
- mark header as checked if all matching logs persisted successfully
- batch convert and insert matching logs to enable transactional persist
- enable skipping previously checked block headers when restarting a sync
2018-09-25 14:11:35 -05:00
Elizabeth
cee824f7ad Add transformer documentation (#32) 2018-09-24 15:39:00 -05:00
Rob Mulholand
e138e58120 Set timeout before assertions in integration tests
- Prevent early evaluation from triggering failure
2018-09-21 11:28:17 -05:00
Rob Mulholand
56c45b6841 Add logging for every transformer
- Also add drip file transformer initializer
2018-09-19 11:00:20 -05:00
Rob Mulholand
a735698f77 Update config for Kovan 2018-09-19 11:00:20 -05:00
Rob Mulholand
042f1142c0 Handle headers from POA chain
- Fetching headers from Kovan requires custom type without mixHash/Nonce
2018-09-19 11:00:20 -05:00
Rob Mulholand
778517f33f Add integration tests for price feed transformer
- Verify eth/usd, mkr/usd, and rep/usd price feed logs are successfully
  fetched from Kovan and persisted to Postgres
2018-09-19 11:00:20 -05:00
Elizabeth
cfc8773c5d Continuous sync (#23) 2018-09-19 10:14:49 -05:00
Elizabeth
60901d9095 Update tend, flipkick and dent tables for consistency (#24) 2018-09-18 09:27:14 -05:00
Rob Mulholand
06881db350 Sync unchecked headers for price feeds
- Indicate that a header has been checked for logs if no logs returned
- Indicate that a header has been checked for logs when persisting a log
- Fetch headers that need to be checked based on absence of the above
2018-09-17 14:14:52 -05:00
Rob Mulholand
f5595bee78 Add Drip drip transformer 2018-09-13 16:35:10 -05:00
Rob Mulholand
560305f601 Update dependencies
- uses newer version of go-ethereum required for go1.11
2018-09-13 16:14:35 -05:00
Elizabeth
939ead0c82 Deal transformer (#19) 2018-09-13 09:06:26 -05:00
Rob Mulholand
d69063636e Add raw log to price feeds 2018-09-12 11:35:32 -05:00
Rob Mulholand
b2ba7ee1e3 Add transaction index to flip kick 2018-09-12 11:35:32 -05:00
Rob Mulholand
a843de5eb7 Add drip file transformers
- for all three file functions on the drip contract
2018-09-12 11:35:32 -05:00
Elizabeth
faefd620b1 Get method/event signature from ABI (#17) 2018-09-12 08:59:43 -05:00
Rob Mulholand
4b6ef1e58b Verify pit file logs
- assure required topics + data available before parsing
2018-09-11 16:30:29 -05:00
Rob Mulholand
dc9bda7d68 Extract addresses to shared constants file 2018-09-11 16:30:29 -05:00
Rob Mulholand
ceeaa0f8d3 Update licenses and imports
- Verify license added to each file
- Order imports by group and separate with line breaks
2018-09-11 16:30:29 -05:00
Rob Mulholand
ff110592bd Add Vat init transformer 2018-09-11 16:30:29 -05:00
Elizabeth
9abe3ffa68 Dent (#10)
Convert and persist LogNote events as dent records
2018-09-11 14:01:45 -05:00
Rob Mulholand
d48cf38070 Add Pit file debt ceiling event 2018-09-05 13:51:18 -05:00
Rob Mulholand
e1f609528d Add Pit file stability fee event
- file event on the pit contract is overloaded and each implementation
  logs a note
- existing Pit file transformer now specified as dealing with ilks
2018-09-05 11:36:13 -05:00
Rob Mulholand
a30e5036fd generate log event signatures from functions 2018-09-04 16:19:03 -05:00
Rob Mulholand
8f5bc0a956 Add raw log to Frob record in DB
- Enables parsing additional values off the log if necessary
- Also update Frob to reflect DSS changes (lad => urn)
- Move Frob constants to shared directory
2018-09-04 16:11:54 -05:00
Rob Mulholand
db4b891398 Backfill Pit.file log events 2018-09-04 16:07:35 -05:00
Elizabeth
53635ec362 Update flip kick and tend transformers due to contract changes
* Update FlipperAddress to new local ganache address

* Update flip_kick table

* Update flipkick transformer to handle new signature and abi

* Update tend table

* Update tend converter
2018-09-04 15:50:29 -05:00
Takayuki Goto
807b7accd6 Merge pull request #12 from 8thlight/bite-event
bite event transformer
2018-09-04 09:58:51 -05:00
Taka Goto
415ec3d97c update from PR feedback 2018-08-31 10:37:02 -05:00
Taka Goto
adc7c47244 update tests and import order per PR feedback 2018-08-30 15:06:14 -05:00
Taka Goto
4ca7c10f92 fix compile error 2018-08-30 14:27:28 -05:00
Taka Goto
985fa49178 migration file to create bite; create bite repository
add transaction index and raw log to bite table
work on converter for bite event
update bite repository, replace guy with 32byte lad; create bite converter to entity
update field type for bite event; start on bite transformer
finish bite event transformer
2018-08-30 08:38:43 -05:00
Rob Mulholand
1916d585fa Incorporate DSS updates (#8)
* Incorporate DSS updates
- Lad renamed to Guy
- Dink and Dart added to Frob
* update test chain data
* Remove Mom field from FlipKick
* Update Flip ABI and sample flip kick data
* Incorporate updates to Frob event
- Guy renamed to Lad
- Era and Gem removed, iArt added
- Also turn off integration tests that read from test chain while
events are actively under development
2018-08-28 14:16:39 -05:00
Elizabeth
66ad7e3021 Transform and persist Tend log events 2018-08-16 16:36:35 -05:00
Rob Mulholand
634604d0b5 Combine price feed transformers
- fetches logs from all three price feeds in one query
- assumes eth/usd price feed will be updated to include LogValue event
- updates transformers to run separate from header sync
2018-08-16 11:22:16 -05:00
Rob Mulholand
9231d40369 Extract price feed contract addresses to config 2018-08-16 11:22:16 -05:00
Rob Mulholand
72a849a272 Update price feeds if header changed by validator
- currently not validating price feeds if the underlying header already exists
  and is valid, since price feeds should have been added when initial header
  was added
2018-08-16 11:22:16 -05:00
Rob Mulholand
6e68dc4a92 Add Rep price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand
2949996d22 Add Pip price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand
0551147453 Add Pep price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand
9f7a776d70 update chain data for frob event 2018-08-14 10:47:43 -05:00
Rob Mulholand
ac63e43543 Backfill Frob log events 2018-08-14 10:47:43 -05:00
Rob Mulholand
55229cd2eb Refactorings to get tests passing with local Ganache setup
- removes assertions against data with timestamps to facilitate running
the tests against a freshly setup local Ganache instance
- also applies a few `go vet` and `go fmt` changes
2018-08-09 16:55:44 -05:00
Elizabeth
b37324bf3d Refactoring
* Move flip kick files pkg/transformers/flip_kick

* Consolidate test database setup

* Pull ganache ipcPath from config

* Update README to include info about using a Ganache chain
2018-08-07 15:17:29 -05:00
Elizabeth
c617cd9c9d Add FlipKick log events transformer 2018-08-07 10:51:34 -05:00
Rob Mulholand
a683e45855 Consolidate test doubles
- Migrate various mocks of core namespaces to shared version in `fakes` pkg
- Err on the side of making test doubles less sophisticated
- Don't pull over mocks of namespaces that are only used in example code
2018-07-20 11:37:46 -05:00
Rob Mulholand
63434f6bc9 Add tests for pkg/geth/blockchain
- inject dependencies instead of initializing them in the constructor
2018-07-18 16:34:13 -05:00
Rob Mulholand
1355271011 Add light sync command
- Only syncs block headers (excludes block bodies, transactions, receipts, and logs)
- Modifies validation window to include the most recent block
- Isolates validation window to the variable defined in the cmd directory (blocks
  have a separate variable defined in the block_repository for determining when
  to set a block as final)
2018-07-18 16:34:12 -05:00
Maxim Krasilnikov
e9e3864ba9 Added user and password to db config 2018-06-22 18:28:34 +03:00
Rob Mulholand
3e39ccb9bb Fix missing block numbers lookup
- Previously, a block was treated as missing if there was a number in
the given range for which there existing a block that did not match
the node's fingerprint. This meant that in a case where we have a block
that does not match the node's fingerprint and also one that does match,
the block would be treated as missing. This led to errors being thrown
when attempting to add a block that already exists.
- These changes treat a block as missing only if we do not already have
a block that matches the number and node fingerprint.
2018-06-08 11:26:25 -05:00
Rob Mulholand
d5c2ab33fc Sync only missing blocks on cold import
= Add eth_node_fingerprint to block that can be imitated by both hot and cold imports
- Only sync missing blocks (blocks that are missing or don't share a fingerprint) on cold import
- Set block is_final status after import
2018-05-07 16:38:04 -05:00
Rob Mulholand
5a5e08bd13 Get head block number
- Allows us to refuse to sync past current head
- Creates the opportunity to add a flag for syncing all blocks
2018-05-04 11:03:50 -05:00
Rob Mulholand
5f6bf32ec1 Add cold import script 2018-05-04 10:54:00 -05:00
Rob Mulholand
462f94d84a Associate receipts with blocks instead of transactions
- Allows us to fetch receipts by block and persist even when
associated transaction is unknown. (The associated transaction
can still be derived from the tx_hash column at query time, but
is an expensive operation to require for inserts).
2018-05-04 10:52:28 -05:00
Matt K
8a9395819c Get transactions (#45)
* Make transactions requests in parallel

* Update transaction error handling
2018-03-27 16:06:12 -05:00
Matt Krump
88210e436a Remove test and travis deps on private dev network 2018-03-22 10:15:18 -05:00
Matt Krump
1b4e57a5b4 Rename node table and node column 2018-03-21 13:24:13 -05:00
Matt Krump
a11fb709b2 Update watcher to use blockchain instead of contract data fetcher 2018-03-12 16:19:06 -05:00
Matt Krump
76dad443ec Update FetchContractData to allow optional args 2018-03-12 14:21:34 -05:00
Matt Krump
5a2bb04670 Remove unused contract methods 2018-03-12 14:21:34 -05:00
Matt K
5a652190d9 Allow Parity as ingest node (#36)
* Upgrade go-ethereum to v1.8

* Add Node Info for parity nodes

* Upgrade start_private_blockchain to use v1.8
2018-03-07 15:29:21 -06:00
Matt Krump
06f78e0083 Handle events
- Adds interfaces for developers to build handlers that update data in
response to log events
- Resolves #29
2018-03-05 10:01:50 -06:00
Matt K
ed907535e3 Separate DB access into several repos (#28)
* Separate files for InMemory

* Start using separate repos for collaborating objects

* Before Updating schema

* Separate various repos
2018-02-12 10:54:05 -06:00
Matt K
605b0a96ae Add graphql server (#27)
* Add graphql server

* Update Makefile

* Update log_filters constraint

* Add GetLogFilter to repo

* Update travis (use Makefile, go fmt, go vet)

* Add logFilter schema and resolvers

* Add GetWatchedEvent to watched_events_repo

* Add watchedEventLog schema and resolvers
2018-02-08 10:12:08 -06:00
Matt K
d5852654bb Update table columns (#26)
* Update block table names

* Update transaction table names
2018-02-02 16:12:14 -06:00