Commit Graph

121 Commits

Author SHA1 Message Date
Ian Norden
dc87f39889 change omniwatcher database test helpers so that it doesn't interfere with downstream maker tests 2019-01-23 14:55:20 -06:00
Ian Norden
46b898f690 merge public vulcanizedb/master 2019-01-23 00:37:26 -06:00
Edvard Hübinette
ebca338b1e VDB-302 Sleep when no missing blocks in lightSync (#129)
* Sleep when no missing blocks in lightSync

* Fix tests and error propagation

* Correct geth.log to logrus
2019-01-11 10:58:41 +01:00
Ian Norden
59cdaa05e6 moving some memory allocation to init function rather than within
execution cycles and fixing test db tear down so that it drops
checked_headers table instead of individual columns (droping columns
only hides them, they remain in memory and continue to contribute to the
1600 column limit which causes us to eventually exceed it)
2019-01-07 13:07:31 -06:00
Edvard
833dde62cf Remove injection of fetcher and repository to watcher 2019-01-07 15:19:31 +01:00
Ian Norden
e390a97502 finish method polling with hash or address type argument values
collected from watched events; 'MissingMethodsCheckedEventsIntersection'
method to find headers which have been checked for each event of
interest but methods have not yet been polled at that header; tests for
new features; travis ci go version 1.9 -> 1.11 ; consolidate omniWatcher and lightOmniWatcher into single command with light as default
2018-12-21 10:33:31 -06:00
Ian Norden
0a59f06cac begin work on: Add checked_headers column for methods that are polled so taht we don’t duplicate; Add batching of method polling so that we arent generating a rediculously large account address list before using it to poll methods (or persist the list in pg?); User passed ABI and other ways to get ABI; Add ability to collect []byte and hashes from events and use them in method polling same manner as addresses; Event filter addrs => only those event’s addresses/hashes are used for polling; Option to persist seen address/hash/bytes lists into pg; Only generate lists of addresses, []byte, or hashes if a method will use them later 2018-12-21 10:33:31 -06:00
Edvard
a3737c46f6 Remove duplicate entry in initializer map 2018-12-13 13:16:49 +01:00
Edvard
d30fcfed80 Use new config getter on shared.Transformer <3 2018-12-13 12:39:57 +01:00
Edvard
dc43547612 Improve process of adding transformers 2018-12-12 15:41:29 +01:00
Edvard
4e089c363e Fixes after merging staging 2018-12-11 15:19:27 +01:00
Edvard
d93817f346 Interface-ify shared repository 2018-12-11 11:35:13 +01:00
Edvard
587d2219d8 Remove bc references and cleanup pointers 2018-12-10 16:50:13 +01:00
Ian Norden
e02b33547d finishing porting omni watcher to work with light sync; split into full, light,
and shared directories and refactor as much into shared; finish
lightSync omni watcher tests
2018-11-30 13:33:48 -06:00
Edvard
c26736dc9e WIP continue on aggregate fetching architecture 2018-11-30 17:28:52 +01:00
Ian Norden
975f13b969 reorganizing omni directory and beginning light watcher work 2018-11-29 20:33:21 -06:00
Ian Norden
817bd76713 refactoring event converter and repository so that event logs are
persisted as they are converted, preventing build up in memory, and finishing  method polling (for full sync)
2018-11-29 20:33:21 -06:00
Ian Norden
390a60f7f6 port over lightSync updates from maker repo 2018-11-29 20:33:21 -06:00
Edvard
b5dab1e83b Nuke erc20 transformer 2018-11-29 11:48:16 +01:00
Taka Goto
e8be96a4e7 use logrus for logging 2018-11-21 10:14:11 -06:00
Ian Norden
417b18ec6a Edits to address PR issues; change license from apache to AGPL; and work
towards generic method polling and reposito;y; config settings to
filterevents/methods by account address; refactoring some stuff out of
repo and into converter; remove fetcher and instead call
blockchain's FetchContractData directly; finishing tests
2018-11-15 12:32:52 -06:00
Rob Mulholand
854236fd99 (VDB-267) Remove pit file stability fee
- Method is no longer present on the Pit contract
2018-11-15 12:14:59 -06:00
Rob Mulholand
82fd73ba3f (VDB-68) Verify log block hash matches header block hash
- Delete header on conflict to prompt data refresh (cascade deletes all
  data associated with that block)
- Derive header hash from rpc payload rather than computing it from data
  (prevents hash mismatch between blockchain and cache)
2018-11-13 14:51:39 -06:00
Edvard Hübinette
571f300392 Add rendering of available transformer names to continuousLogSync help (#109) 2018-11-12 14:01:07 +01:00
Rob Mulholand
68464d375a Add missing signatures to getSignatures command 2018-11-08 13:08:59 -06:00
Rob Mulholand
8c168ed332 Extract constants package
- Reduce clutter in the shared package
2018-11-08 13:08:59 -06:00
Rob Mulholand
cb141f7cb7 (VDB-204) Add transformer for Flap kick 2018-11-07 10:52:55 -06:00
Ian Norden
b459cf35ed beginning work on method polling; first need to generate list of token holder address in a completely generic/contratc agnostic fashion. Created address retriever that can iterate through any given contract's watched events, finding the inputs/arguments with address type, and generate a list from those values. Edit: Contract objects now cache every event emitted address as its event logs are transformed into the repo to grow a list of contract associated addresses as we go 2018-11-04 21:37:31 -06:00
Ian Norden
e9dbd771e5 tests and fixes for fetcher, parser, retriever, converter, and repository; update cmd and watcher 2018-11-04 01:49:11 -05:00
Ian Norden
8ce75fe5ad Generic watcher that takes a contract address, grabs the contract abi and starting block number, creates custom event filters, and extracts and transforms event data into postgres. Can configure to look at only a subset of events through CLI flag. Building but needs testing. 2018-11-03 14:00:25 -05:00
Ian Norden
57820ff473 change transformer interface and watcher so that contract config is now fed into AddTransformers such that a single watcher can be loaded with transformers that use different cofigs 2018-11-03 13:49:24 -05:00
Rob Mulholand
ba071ef13f Consolidate test doubles
- Migrate various mocks of core namespaces to shared version in `fakes` pkg
- Err on the side of making test doubles less sophisticated
- Don't pull over mocks of namespaces that are only used in example code
2018-11-03 13:49:23 -05:00
Rob Mulholand
5fe6394406 Add tests for pkg/geth/blockchain
- inject dependencies instead of initializing them in the constructor
2018-11-03 13:49:23 -05:00
Rob Mulholand
05186634bd Add light sync command
- Only syncs block headers (excludes block bodies, transactions, receipts, and logs)
- Modifies validation window to include the most recent block
- Isolates validation window to the variable defined in the cmd directory (blocks
  have a separate variable defined in the block_repository for determining when
  to set a block as final)
2018-11-03 13:49:23 -05:00
Elizabeth Engelman
b6f93e735f Add ERC20 token watcher example
- starting with the totalSupply function
- sets contract config on transformer by passing it into the transformer
initializer
- handles block records with the same number for different nodes for both creating token_supply records, and finding missing blocks
2018-11-03 13:49:23 -05:00
Maxim Krasilnikov
7b507667f8 Added user and password to db config 2018-11-03 13:49:23 -05:00
Rob Mulholand
ed624330f6 Rename Flog pack to Vow flog
- enables differentiation if another contract implements flog
- also deletes flog-specific mocks
2018-10-26 15:35:18 -05:00
Edvard Hübinette
be249437c2 VDB-117: Add Vow.flog transformer (#80)
* Add DB migrations for flog

* Add Vow addresses and constants

* Add err logging to ABI signature getter, test for flog sig

* Add flog transformer

* Add tests+data for flog transformer

* Prepend DB tables and columns with vow_

* Panic on failure to get method sig
2018-10-25 12:19:46 +02:00
Edvard
a0deb5eb62 Add and use transformer labels in shared constants 2018-10-23 11:23:31 +02:00
Rob Mulholand
9b427ea1e9 Add Vat move integration test 2018-10-19 10:56:28 -05:00
Elizabeth
c86120a36d Fix getSignatures should not print during command init (#70) 2018-10-18 14:19:56 -05:00
Rob Mulholand
130d11d0aa Add Vat slip transformer 2018-10-17 13:30:35 -05:00
Elizabeth
3baca64903 Update address formatting (#67)
* Add getSignatures command for list of signatures in constants

* Update VatTune converter's formatting of addresses

and added an integration test

* Move VatTune and PriceFeeds integration tests to a common suite

* Update VatGrab converter for address format

* Update frob converter to fix urn and ilk formatting/decoding

* Update formatting

* Update make test command and add integrationtest command

* Add integration test to travis build

* Update creating headers in integration tests to include Raw and Timestamp fields
2018-10-17 13:04:55 -05:00
Taka Goto
2e72a7e940 vat flux transformer 2018-10-17 10:53:53 -05:00
Edvard Hübinette
be58dd4ac8 Add vat_move transformer (#47)
* Add vat_move transformer base

* Add vat_move migrations

* Add test data for vat_move

* Add vat_move transformer to initialisers

* Add numeric cast to psql insert of Rad

* Add new db schema

* Dependency update

* Expand abbreviation in repository

* Add test suite for vat_move

* Add header checking to transformer and mock repository

* Remove trailing zero in test data

* Fix minor mishaps

* Go fmt nitpicking

* Refactoring in tests

* Add tests covering checked headers stuff (and fix revealed bugs)

* go fmt fixes

* Implement batching behaviour of transformer

* Small fixes after review

* Go fmt
2018-10-12 16:13:13 +02:00
Elizabeth
6c77f369d9 Vat heal (#56)
* VatHeal Converter

* Add VatHeal repository

* Add VatHeal transformer

* Add VatHeal to continuousLogSync command

* Mark vat_init_checked as true when creating vat init records

* Update urn and v converting

* Return error if Repository.MarkCheckedHeader fails

* Add deleting vat heal from test cleanup method
2018-10-10 11:56:06 -05:00
David Terry
b1745a6403 Merge branch 'staging' into VDB-101-vat-fold-transformer
Conflicts:
	cmd/continuousLogSync.go
	db/schema.sql
	pkg/transformers/shared/constants.go
	pkg/transformers/transformers.go
	test_config/test_config.go
2018-10-10 14:09:14 +03:00
Rob Mulholand
e935f2f553 Add Vat grab transformer 2018-10-09 16:48:19 -05:00
Rob Mulholand
dcdf5dc292 Add missing transformers to transformer initializer map 2018-10-09 14:37:26 -05:00
Rob Mulholand
61dca314c4 Add Vat tune transformer 2018-10-09 14:37:26 -05:00