Commit Graph
216 Commits
Author SHA1 Message Date
Ian Norden e390a97502 finish method polling with hash or address type argument values
collected from watched events; 'MissingMethodsCheckedEventsIntersection'
method to find headers which have been checked for each event of
interest but methods have not yet been polled at that header; tests for
new features; travis ci go version 1.9 -> 1.11 ; consolidate omniWatcher and lightOmniWatcher into single command with light as default
2018-12-21 10:33:31 -06:00
Ian Norden 0a59f06cac begin work on: Add checked_headers column for methods that are polled so taht we don’t duplicate; Add batching of method polling so that we arent generating a rediculously large account address list before using it to poll methods (or persist the list in pg?); User passed ABI and other ways to get ABI; Add ability to collect []byte and hashes from events and use them in method polling same manner as addresses; Event filter addrs => only those event’s addresses/hashes are used for polling; Option to persist seen address/hash/bytes lists into pg; Only generate lists of addresses, []byte, or hashes if a method will use them later 2018-12-21 10:33:31 -06:00
Edvard a3737c46f6 Remove duplicate entry in initializer map 2018-12-13 13:16:49 +01:00
Edvard d30fcfed80 Use new config getter on shared.Transformer <3 2018-12-13 12:39:57 +01:00
Edvard dc43547612 Improve process of adding transformers 2018-12-12 15:41:29 +01:00
Edvard 4e089c363e Fixes after merging staging 2018-12-11 15:19:27 +01:00
Edvard d93817f346 Interface-ify shared repository 2018-12-11 11:35:13 +01:00
Edvard 587d2219d8 Remove bc references and cleanup pointers 2018-12-10 16:50:13 +01:00
Ian Norden e02b33547d finishing porting omni watcher to work with light sync; split into full, light,
and shared directories and refactor as much into shared; finish
lightSync omni watcher tests
2018-11-30 13:33:48 -06:00
Edvard c26736dc9e WIP continue on aggregate fetching architecture 2018-11-30 17:28:52 +01:00
Ian Norden 975f13b969 reorganizing omni directory and beginning light watcher work 2018-11-29 20:33:21 -06:00
Ian Norden 817bd76713 refactoring event converter and repository so that event logs are
persisted as they are converted, preventing build up in memory, and finishing  method polling (for full sync)
2018-11-29 20:33:21 -06:00
Ian Norden 390a60f7f6 port over lightSync updates from maker repo 2018-11-29 20:33:21 -06:00
Edvard b5dab1e83b Nuke erc20 transformer 2018-11-29 11:48:16 +01:00
Taka Goto e8be96a4e7 use logrus for logging 2018-11-21 10:14:11 -06:00
Ian Norden 417b18ec6a Edits to address PR issues; change license from apache to AGPL; and work
towards generic method polling and reposito;y; config settings to
filterevents/methods by account address; refactoring some stuff out of
repo and into converter; remove fetcher and instead call
blockchain's FetchContractData directly; finishing tests
2018-11-15 12:32:52 -06:00
Rob Mulholand 854236fd99 (VDB-267) Remove pit file stability fee
- Method is no longer present on the Pit contract
2018-11-15 12:14:59 -06:00
Rob Mulholand 82fd73ba3f (VDB-68) Verify log block hash matches header block hash
- Delete header on conflict to prompt data refresh (cascade deletes all
  data associated with that block)
- Derive header hash from rpc payload rather than computing it from data
  (prevents hash mismatch between blockchain and cache)
2018-11-13 14:51:39 -06:00
Edvard Hübinette 571f300392 Add rendering of available transformer names to continuousLogSync help (#109) 2018-11-12 14:01:07 +01:00
Rob Mulholand 68464d375a Add missing signatures to getSignatures command 2018-11-08 13:08:59 -06:00
Rob Mulholand 8c168ed332 Extract constants package
- Reduce clutter in the shared package
2018-11-08 13:08:59 -06:00
Rob Mulholand cb141f7cb7 (VDB-204) Add transformer for Flap kick 2018-11-07 10:52:55 -06:00
Ian Norden b459cf35ed beginning work on method polling; first need to generate list of token holder address in a completely generic/contratc agnostic fashion. Created address retriever that can iterate through any given contract's watched events, finding the inputs/arguments with address type, and generate a list from those values. Edit: Contract objects now cache every event emitted address as its event logs are transformed into the repo to grow a list of contract associated addresses as we go 2018-11-04 21:37:31 -06:00
Ian Norden e9dbd771e5 tests and fixes for fetcher, parser, retriever, converter, and repository; update cmd and watcher 2018-11-04 01:49:11 -05:00
Ian Norden 8ce75fe5ad Generic watcher that takes a contract address, grabs the contract abi and starting block number, creates custom event filters, and extracts and transforms event data into postgres. Can configure to look at only a subset of events through CLI flag. Building but needs testing. 2018-11-03 14:00:25 -05:00
Ian Norden 57820ff473 change transformer interface and watcher so that contract config is now fed into AddTransformers such that a single watcher can be loaded with transformers that use different cofigs 2018-11-03 13:49:24 -05:00
Rob Mulholand ba071ef13f Consolidate test doubles
- Migrate various mocks of core namespaces to shared version in `fakes` pkg
- Err on the side of making test doubles less sophisticated
- Don't pull over mocks of namespaces that are only used in example code
2018-11-03 13:49:23 -05:00
Rob Mulholand 5fe6394406 Add tests for pkg/geth/blockchain
- inject dependencies instead of initializing them in the constructor
2018-11-03 13:49:23 -05:00
Rob Mulholand 05186634bd Add light sync command
- Only syncs block headers (excludes block bodies, transactions, receipts, and logs)
- Modifies validation window to include the most recent block
- Isolates validation window to the variable defined in the cmd directory (blocks
  have a separate variable defined in the block_repository for determining when
  to set a block as final)
2018-11-03 13:49:23 -05:00
Elizabeth Engelman b6f93e735f Add ERC20 token watcher example
- starting with the totalSupply function
- sets contract config on transformer by passing it into the transformer
initializer
- handles block records with the same number for different nodes for both creating token_supply records, and finding missing blocks
2018-11-03 13:49:23 -05:00
Maxim Krasilnikov 7b507667f8 Added user and password to db config 2018-11-03 13:49:23 -05:00
Rob Mulholand ed624330f6 Rename Flog pack to Vow flog
- enables differentiation if another contract implements flog
- also deletes flog-specific mocks
2018-10-26 15:35:18 -05:00
Edvard Hübinette be249437c2 VDB-117: Add Vow.flog transformer (#80)
* Add DB migrations for flog

* Add Vow addresses and constants

* Add err logging to ABI signature getter, test for flog sig

* Add flog transformer

* Add tests+data for flog transformer

* Prepend DB tables and columns with vow_

* Panic on failure to get method sig
2018-10-25 12:19:46 +02:00
Edvard a0deb5eb62 Add and use transformer labels in shared constants 2018-10-23 11:23:31 +02:00
Rob Mulholand 9b427ea1e9 Add Vat move integration test 2018-10-19 10:56:28 -05:00
Elizabeth c86120a36d Fix getSignatures should not print during command init (#70) 2018-10-18 14:19:56 -05:00
Rob Mulholand 130d11d0aa Add Vat slip transformer 2018-10-17 13:30:35 -05:00
Elizabeth 3baca64903 Update address formatting (#67)
* Add getSignatures command for list of signatures in constants

* Update VatTune converter's formatting of addresses

and added an integration test

* Move VatTune and PriceFeeds integration tests to a common suite

* Update VatGrab converter for address format

* Update frob converter to fix urn and ilk formatting/decoding

* Update formatting

* Update make test command and add integrationtest command

* Add integration test to travis build

* Update creating headers in integration tests to include Raw and Timestamp fields
2018-10-17 13:04:55 -05:00
Taka Goto 2e72a7e940 vat flux transformer 2018-10-17 10:53:53 -05:00
Edvard Hübinette be58dd4ac8 Add vat_move transformer (#47)
* Add vat_move transformer base

* Add vat_move migrations

* Add test data for vat_move

* Add vat_move transformer to initialisers

* Add numeric cast to psql insert of Rad

* Add new db schema

* Dependency update

* Expand abbreviation in repository

* Add test suite for vat_move

* Add header checking to transformer and mock repository

* Remove trailing zero in test data

* Fix minor mishaps

* Go fmt nitpicking

* Refactoring in tests

* Add tests covering checked headers stuff (and fix revealed bugs)

* go fmt fixes

* Implement batching behaviour of transformer

* Small fixes after review

* Go fmt
2018-10-12 16:13:13 +02:00
Elizabeth 6c77f369d9 Vat heal (#56)
* VatHeal Converter

* Add VatHeal repository

* Add VatHeal transformer

* Add VatHeal to continuousLogSync command

* Mark vat_init_checked as true when creating vat init records

* Update urn and v converting

* Return error if Repository.MarkCheckedHeader fails

* Add deleting vat heal from test cleanup method
2018-10-10 11:56:06 -05:00
David Terry b1745a6403 Merge branch 'staging' into VDB-101-vat-fold-transformer
Conflicts:
	cmd/continuousLogSync.go
	db/schema.sql
	pkg/transformers/shared/constants.go
	pkg/transformers/transformers.go
	test_config/test_config.go
2018-10-10 14:09:14 +03:00
Rob Mulholand e935f2f553 Add Vat grab transformer 2018-10-09 16:48:19 -05:00
Rob Mulholand dcdf5dc292 Add missing transformers to transformer initializer map 2018-10-09 14:37:26 -05:00
Rob Mulholand 61dca314c4 Add Vat tune transformer 2018-10-09 14:37:26 -05:00
Rob Mulholand 8ee106e14d Add Vat toll transformer 2018-10-09 11:40:35 -05:00
David Terry 8fb573351e Vat.fold: add to continuousLogSync 2018-10-04 17:47:13 +03:00
Rob Mulholand 042f1142c0 Handle headers from POA chain
- Fetching headers from Kovan requires custom type without mixHash/Nonce
2018-09-19 11:00:20 -05:00
Elizabeth cfc8773c5d Continuous sync (#23) 2018-09-19 10:14:49 -05:00
Taka Goto cf72eae86c add datadog monitoring for vdb 2018-09-04 15:41:24 -05:00