Commit Graph

277 Commits

Author SHA1 Message Date
Rob Mulholand
867f92c431 (VDB-298) Consume Pit contract storage diffs
- Continuously parse storage diffs CSV data to read Pit contract state
- Convert ilks in database to raw bytes32 value for use in generating
  storage keys dynamically
- Persist storage diffs with block number and hash for validation
2019-02-06 10:31:46 -06:00
Edvard Hübinette
29f2537130 Merge pull request #7 from vulcanize/VDB-347-Squelch-lightSync
VDB-347 Squelch lightSync in stdout
2019-02-01 11:02:48 +01:00
Edvard
8eea1b00ed Redirect validation window status to logrus instead of stdout 2019-01-31 15:55:28 +01:00
Ian Norden
8749c94994 update license to AGPL-3.0 2019-01-24 14:41:30 -06:00
Ian Norden
1f95eb6443 Merge branch 'staging' into pub_merge 2019-01-24 09:47:12 -06:00
Ian Norden
dc87f39889 change omniwatcher database test helpers so that it doesn't interfere with downstream maker tests 2019-01-23 14:55:20 -06:00
Konstantin Zolotarev
65cb6e9092 VDB-205 Fix external configuration values (#121)
* Updated loading config from /Users/konstantinzolotarev/.vulcanizedb for addreses

* Updated contract addresses loading from env variables.

* Implement getter chain for addresses, configs and transformers

Solves the issue with declaring stuff as vars when the environment file hasn't been loaded yet. Instead, this data is fetched dynamically when the root commands are called.

* Update test data to use constant Kovan addresses

* Decouple integration tests from config file IO

* Add ABI to environment file, convert getter chains for related values

* Decouple tests suites from ABI config file IO

* Add startingBlock to environment and implement getter chain

* Exit when called without config file

* Add missing deployment blocks to staging.toml

* Remove comment
2019-01-23 19:44:09 +01:00
Ian Norden
46b898f690 merge public vulcanizedb/master 2019-01-23 00:37:26 -06:00
Edvard Hübinette
ebca338b1e VDB-302 Sleep when no missing blocks in lightSync (#129)
* Sleep when no missing blocks in lightSync

* Fix tests and error propagation

* Correct geth.log to logrus
2019-01-11 10:58:41 +01:00
Ian Norden
59cdaa05e6 moving some memory allocation to init function rather than within
execution cycles and fixing test db tear down so that it drops
checked_headers table instead of individual columns (droping columns
only hides them, they remain in memory and continue to contribute to the
1600 column limit which causes us to eventually exceed it)
2019-01-07 13:07:31 -06:00
Edvard
833dde62cf Remove injection of fetcher and repository to watcher 2019-01-07 15:19:31 +01:00
Ian Norden
e390a97502 finish method polling with hash or address type argument values
collected from watched events; 'MissingMethodsCheckedEventsIntersection'
method to find headers which have been checked for each event of
interest but methods have not yet been polled at that header; tests for
new features; travis ci go version 1.9 -> 1.11 ; consolidate omniWatcher and lightOmniWatcher into single command with light as default
2018-12-21 10:33:31 -06:00
Ian Norden
0a59f06cac begin work on: Add checked_headers column for methods that are polled so taht we don’t duplicate; Add batching of method polling so that we arent generating a rediculously large account address list before using it to poll methods (or persist the list in pg?); User passed ABI and other ways to get ABI; Add ability to collect []byte and hashes from events and use them in method polling same manner as addresses; Event filter addrs => only those event’s addresses/hashes are used for polling; Option to persist seen address/hash/bytes lists into pg; Only generate lists of addresses, []byte, or hashes if a method will use them later 2018-12-21 10:33:31 -06:00
Edvard
a3737c46f6 Remove duplicate entry in initializer map 2018-12-13 13:16:49 +01:00
Edvard
d30fcfed80 Use new config getter on shared.Transformer <3 2018-12-13 12:39:57 +01:00
Edvard
dc43547612 Improve process of adding transformers 2018-12-12 15:41:29 +01:00
Edvard
4e089c363e Fixes after merging staging 2018-12-11 15:19:27 +01:00
Edvard
d93817f346 Interface-ify shared repository 2018-12-11 11:35:13 +01:00
Edvard
587d2219d8 Remove bc references and cleanup pointers 2018-12-10 16:50:13 +01:00
Ian Norden
e02b33547d finishing porting omni watcher to work with light sync; split into full, light,
and shared directories and refactor as much into shared; finish
lightSync omni watcher tests
2018-11-30 13:33:48 -06:00
Edvard
c26736dc9e WIP continue on aggregate fetching architecture 2018-11-30 17:28:52 +01:00
Ian Norden
975f13b969 reorganizing omni directory and beginning light watcher work 2018-11-29 20:33:21 -06:00
Ian Norden
817bd76713 refactoring event converter and repository so that event logs are
persisted as they are converted, preventing build up in memory, and finishing  method polling (for full sync)
2018-11-29 20:33:21 -06:00
Ian Norden
390a60f7f6 port over lightSync updates from maker repo 2018-11-29 20:33:21 -06:00
Edvard
b5dab1e83b Nuke erc20 transformer 2018-11-29 11:48:16 +01:00
Taka Goto
e8be96a4e7 use logrus for logging 2018-11-21 10:14:11 -06:00
Ian Norden
417b18ec6a Edits to address PR issues; change license from apache to AGPL; and work
towards generic method polling and reposito;y; config settings to
filterevents/methods by account address; refactoring some stuff out of
repo and into converter; remove fetcher and instead call
blockchain's FetchContractData directly; finishing tests
2018-11-15 12:32:52 -06:00
Rob Mulholand
854236fd99 (VDB-267) Remove pit file stability fee
- Method is no longer present on the Pit contract
2018-11-15 12:14:59 -06:00
Rob Mulholand
82fd73ba3f (VDB-68) Verify log block hash matches header block hash
- Delete header on conflict to prompt data refresh (cascade deletes all
  data associated with that block)
- Derive header hash from rpc payload rather than computing it from data
  (prevents hash mismatch between blockchain and cache)
2018-11-13 14:51:39 -06:00
Edvard Hübinette
571f300392 Add rendering of available transformer names to continuousLogSync help (#109) 2018-11-12 14:01:07 +01:00
Rob Mulholand
68464d375a Add missing signatures to getSignatures command 2018-11-08 13:08:59 -06:00
Rob Mulholand
8c168ed332 Extract constants package
- Reduce clutter in the shared package
2018-11-08 13:08:59 -06:00
Rob Mulholand
cb141f7cb7 (VDB-204) Add transformer for Flap kick 2018-11-07 10:52:55 -06:00
Ian Norden
b459cf35ed beginning work on method polling; first need to generate list of token holder address in a completely generic/contratc agnostic fashion. Created address retriever that can iterate through any given contract's watched events, finding the inputs/arguments with address type, and generate a list from those values. Edit: Contract objects now cache every event emitted address as its event logs are transformed into the repo to grow a list of contract associated addresses as we go 2018-11-04 21:37:31 -06:00
Ian Norden
e9dbd771e5 tests and fixes for fetcher, parser, retriever, converter, and repository; update cmd and watcher 2018-11-04 01:49:11 -05:00
Ian Norden
8ce75fe5ad Generic watcher that takes a contract address, grabs the contract abi and starting block number, creates custom event filters, and extracts and transforms event data into postgres. Can configure to look at only a subset of events through CLI flag. Building but needs testing. 2018-11-03 14:00:25 -05:00
Ian Norden
57820ff473 change transformer interface and watcher so that contract config is now fed into AddTransformers such that a single watcher can be loaded with transformers that use different cofigs 2018-11-03 13:49:24 -05:00
Rob Mulholand
ba071ef13f Consolidate test doubles
- Migrate various mocks of core namespaces to shared version in `fakes` pkg
- Err on the side of making test doubles less sophisticated
- Don't pull over mocks of namespaces that are only used in example code
2018-11-03 13:49:23 -05:00
Rob Mulholand
5fe6394406 Add tests for pkg/geth/blockchain
- inject dependencies instead of initializing them in the constructor
2018-11-03 13:49:23 -05:00
Rob Mulholand
05186634bd Add light sync command
- Only syncs block headers (excludes block bodies, transactions, receipts, and logs)
- Modifies validation window to include the most recent block
- Isolates validation window to the variable defined in the cmd directory (blocks
  have a separate variable defined in the block_repository for determining when
  to set a block as final)
2018-11-03 13:49:23 -05:00
Elizabeth Engelman
b6f93e735f Add ERC20 token watcher example
- starting with the totalSupply function
- sets contract config on transformer by passing it into the transformer
initializer
- handles block records with the same number for different nodes for both creating token_supply records, and finding missing blocks
2018-11-03 13:49:23 -05:00
Maxim Krasilnikov
7b507667f8 Added user and password to db config 2018-11-03 13:49:23 -05:00
Rob Mulholand
ed624330f6 Rename Flog pack to Vow flog
- enables differentiation if another contract implements flog
- also deletes flog-specific mocks
2018-10-26 15:35:18 -05:00
Edvard Hübinette
be249437c2 VDB-117: Add Vow.flog transformer (#80)
* Add DB migrations for flog

* Add Vow addresses and constants

* Add err logging to ABI signature getter, test for flog sig

* Add flog transformer

* Add tests+data for flog transformer

* Prepend DB tables and columns with vow_

* Panic on failure to get method sig
2018-10-25 12:19:46 +02:00
Edvard
a0deb5eb62 Add and use transformer labels in shared constants 2018-10-23 11:23:31 +02:00
Rob Mulholand
9b427ea1e9 Add Vat move integration test 2018-10-19 10:56:28 -05:00
Elizabeth
c86120a36d Fix getSignatures should not print during command init (#70) 2018-10-18 14:19:56 -05:00
Rob Mulholand
130d11d0aa Add Vat slip transformer 2018-10-17 13:30:35 -05:00
Elizabeth
3baca64903 Update address formatting (#67)
* Add getSignatures command for list of signatures in constants

* Update VatTune converter's formatting of addresses

and added an integration test

* Move VatTune and PriceFeeds integration tests to a common suite

* Update VatGrab converter for address format

* Update frob converter to fix urn and ilk formatting/decoding

* Update formatting

* Update make test command and add integrationtest command

* Add integration test to travis build

* Update creating headers in integration tests to include Raw and Timestamp fields
2018-10-17 13:04:55 -05:00
Taka Goto
2e72a7e940 vat flux transformer 2018-10-17 10:53:53 -05:00
Edvard Hübinette
be58dd4ac8 Add vat_move transformer (#47)
* Add vat_move transformer base

* Add vat_move migrations

* Add test data for vat_move

* Add vat_move transformer to initialisers

* Add numeric cast to psql insert of Rad

* Add new db schema

* Dependency update

* Expand abbreviation in repository

* Add test suite for vat_move

* Add header checking to transformer and mock repository

* Remove trailing zero in test data

* Fix minor mishaps

* Go fmt nitpicking

* Refactoring in tests

* Add tests covering checked headers stuff (and fix revealed bugs)

* go fmt fixes

* Implement batching behaviour of transformer

* Small fixes after review

* Go fmt
2018-10-12 16:13:13 +02:00
Elizabeth
6c77f369d9 Vat heal (#56)
* VatHeal Converter

* Add VatHeal repository

* Add VatHeal transformer

* Add VatHeal to continuousLogSync command

* Mark vat_init_checked as true when creating vat init records

* Update urn and v converting

* Return error if Repository.MarkCheckedHeader fails

* Add deleting vat heal from test cleanup method
2018-10-10 11:56:06 -05:00
David Terry
b1745a6403 Merge branch 'staging' into VDB-101-vat-fold-transformer
Conflicts:
	cmd/continuousLogSync.go
	db/schema.sql
	pkg/transformers/shared/constants.go
	pkg/transformers/transformers.go
	test_config/test_config.go
2018-10-10 14:09:14 +03:00
Rob Mulholand
e935f2f553 Add Vat grab transformer 2018-10-09 16:48:19 -05:00
Rob Mulholand
dcdf5dc292 Add missing transformers to transformer initializer map 2018-10-09 14:37:26 -05:00
Rob Mulholand
61dca314c4 Add Vat tune transformer 2018-10-09 14:37:26 -05:00
Rob Mulholand
8ee106e14d Add Vat toll transformer 2018-10-09 11:40:35 -05:00
David Terry
8fb573351e Vat.fold: add to continuousLogSync 2018-10-04 17:47:13 +03:00
Rob Mulholand
042f1142c0 Handle headers from POA chain
- Fetching headers from Kovan requires custom type without mixHash/Nonce
2018-09-19 11:00:20 -05:00
Elizabeth
cfc8773c5d Continuous sync (#23) 2018-09-19 10:14:49 -05:00
Taka Goto
cf72eae86c add datadog monitoring for vdb 2018-09-04 15:41:24 -05:00
Rob Mulholand
634604d0b5 Combine price feed transformers
- fetches logs from all three price feeds in one query
- assumes eth/usd price feed will be updated to include LogValue event
- updates transformers to run separate from header sync
2018-08-16 11:22:16 -05:00
Rob Mulholand
9231d40369 Extract price feed contract addresses to config 2018-08-16 11:22:16 -05:00
Rob Mulholand
72a849a272 Update price feeds if header changed by validator
- currently not validating price feeds if the underlying header already exists
  and is valid, since price feeds should have been added when initial header
  was added
2018-08-16 11:22:16 -05:00
Rob Mulholand
6e68dc4a92 Add Rep price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand
2949996d22 Add Pip price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand
0551147453 Add Pep price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand
ac63e43543 Backfill Frob log events 2018-08-14 10:47:43 -05:00
Elizabeth
b37324bf3d Refactoring
* Move flip kick files pkg/transformers/flip_kick

* Consolidate test database setup

* Pull ganache ipcPath from config

* Update README to include info about using a Ganache chain
2018-08-07 15:17:29 -05:00
Elizabeth
c617cd9c9d Add FlipKick log events transformer 2018-08-07 10:51:34 -05:00
Rob Mulholand
a683e45855 Consolidate test doubles
- Migrate various mocks of core namespaces to shared version in `fakes` pkg
- Err on the side of making test doubles less sophisticated
- Don't pull over mocks of namespaces that are only used in example code
2018-07-20 11:37:46 -05:00
Rob Mulholand
63434f6bc9 Add tests for pkg/geth/blockchain
- inject dependencies instead of initializing them in the constructor
2018-07-18 16:34:13 -05:00
Rob Mulholand
1355271011 Add light sync command
- Only syncs block headers (excludes block bodies, transactions, receipts, and logs)
- Modifies validation window to include the most recent block
- Isolates validation window to the variable defined in the cmd directory (blocks
  have a separate variable defined in the block_repository for determining when
  to set a block as final)
2018-07-18 16:34:12 -05:00
Elizabeth
a620fc6a51 Merge pull request #26 from 8thlight/erc20-take-2
Erc20 take 2
2018-06-27 14:54:20 -05:00
Elizabeth Engelman
7ee253b2a3 Add ERC20 token watcher example
- starting with the totalSupply function
- sets contract config on transformer by passing it into the transformer
initializer
- handles block records with the same number for different nodes for both creating token_supply records, and finding missing blocks
2018-06-27 14:41:51 -05:00
Rob Mulholand
74d6046508 Merge pull request #65 from chapsuk/dev_env
Dockerized dev environment, geth rinkeby
2018-06-27 09:13:38 -05:00
Maxim Krasilnikov
e9e3864ba9 Added user and password to db config 2018-06-22 18:28:34 +03:00
Maxim Krasilnikov
abb7f26124 Fixed sync command help typo 2018-06-21 19:12:09 +03:00
Maxim Krasilnikov
51f2a23b74 Removed useless goroutines from sync cmd, backFillAllBlocks always called with go func() 2018-06-21 18:05:49 +03:00
Rob Mulholand
d5c2ab33fc Sync only missing blocks on cold import
= Add eth_node_fingerprint to block that can be imitated by both hot and cold imports
- Only sync missing blocks (blocks that are missing or don't share a fingerprint) on cold import
- Set block is_final status after import
2018-05-07 16:38:04 -05:00
Rob Mulholand
5a5e08bd13 Get head block number
- Allows us to refuse to sync past current head
- Creates the opportunity to add a flag for syncing all blocks
2018-05-04 11:03:50 -05:00
Rob Mulholand
5f6bf32ec1 Add cold import script 2018-05-04 10:54:00 -05:00
Matt Krump
edf3dbe00c Add starting block arg check 2018-03-27 16:16:27 -05:00
Matt K
5a652190d9 Allow Parity as ingest node (#36)
* Upgrade go-ethereum to v1.8

* Add Node Info for parity nodes

* Upgrade start_private_blockchain to use v1.8
2018-03-07 15:29:21 -06:00
Matt Krump
06f78e0083 Handle events
- Adds interfaces for developers to build handlers that update data in
response to log events
- Resolves #29
2018-03-05 10:01:50 -06:00
Matt K
ed907535e3 Separate DB access into several repos (#28)
* Separate files for InMemory

* Start using separate repos for collaborating objects

* Before Updating schema

* Separate various repos
2018-02-12 10:54:05 -06:00
Matt K
605b0a96ae Add graphql server (#27)
* Add graphql server

* Update Makefile

* Update log_filters constraint

* Add GetLogFilter to repo

* Update travis (use Makefile, go fmt, go vet)

* Add logFilter schema and resolvers

* Add GetWatchedEvent to watched_events_repo

* Add watchedEventLog schema and resolvers
2018-02-08 10:12:08 -06:00
Matt K
aea9c7b5e2 Separate repositories (#25)
* Separate Repository into multiple Repositories

* Use struct scan for transactions

* Use struct scan for blocks

* Remove unused block repo methods

* Update naming

* Rename / Cleanup repository related fields
2018-02-02 15:53:16 -06:00
Matt K
3b31e2a342 Update README, add State sync test (#1) 2018-01-26 13:38:14 -06:00
Matt Krump
aac2239097 Merge old private repo into vulcanize 2018-01-25 18:08:26 -06:00
Matt K
d09c2ae9bb Remove godo (#136) 2018-01-25 15:46:55 -06:00
Matt K
572023cdf5 Commandline (#135)
* Add cmd line tool and Makefile

* Add shared utils pkg

* Add cmdline README

* Update godo for new structure
2018-01-25 13:21:55 -06:00
Matt K
c00b8a5a98 Add Filters (#133)
* Add LogFilter struct

* Add log_filters table

* Add view for events watching

* Add cmd line "add_filter" to mimic eventual endpoint

* Allow multiple filters in config
2018-01-23 12:43:35 -06:00
Matt Krump
0ae25aff95 Remove getLogs entrypoint 2018-01-16 09:49:23 -06:00
Matt K
70cfa20c68 Watch contact updates (#127)
* Downcase all arguments for contact watching

* ABI retrieval from test networks
2018-01-08 15:59:47 -06:00
ee4f7c710a replace '8thlight' with 'vulcanize' 2018-01-06 15:31:53 -05:00
Matt K
6decf0b54b Remove pubsub and replace w/ polling head of chain (#122)
* Rename geth package structs to not be prefaced with package name

* No longer need to dump schema since Travis uses migrate

* Rearrange history package

* Removed double request for receipt from block rewards

* Remove Listener + Observers and Replace w/ Polling Head

* Potential Short term Issue w/ Infura (ignore these tests for now)
2018-01-05 11:55:00 -06:00
Matt K
7e5e12f488 Backfill/listen for contract logs (#113) 2017-12-22 11:42:35 -06:00
Matt K
24bc83a448 Block categorization (#110)
* Add block categorization (is_final=)

* Add godo task for vulcanizeDB (Example of how everything could work together)

* Add unique constraint on block_number and node

* Add index on block_id for transactions_table

* Add node_id index on blocks table

* Sort transactions returned from FindBlock by tx_hash

* lowercase tx_to, tx_from like etherscan
2017-12-20 14:06:22 -06:00
Matt K
266c9587c8 Canonical blocks (#108)
* Update Block w/ newest Block

* Add cascading delete to blocks and transactions tables

* Add handling for new conflicting blocks

* Command line version of sliding window n behind HEAD
2017-12-19 14:14:41 -06:00
Matt Krump
84e77f259d Use most recent blockNumber when blockNumber parame is nil 2017-12-14 10:15:36 -06:00
Matt K
0e837e2d03 Refactoring (#101)
* Make naming consistent for watched_contracts

* Update FindContract and FindBlockByNumber to return errors rather than nil
2017-12-13 10:51:11 -06:00
Matt K
a68f277066 Save Logs to DB (#100)
* Save logs to database

* Save multiple logs to db

* Add block number parameter to `FindLogs`
2017-12-12 15:55:26 -06:00
Matt K
5e64283a12 Get logs for a contract (#99)
* Add ability to fetch logs for a contract and a block

* Test contract related code against Infura, so can run on Travis

* Add godo task for getLogs
2017-12-11 15:08:00 -06:00
Matt K
921bde1089 Update BlockChain to record NodeInfo (#95) 2017-12-07 13:32:16 -06:00
Matt K
18163f970e Get ABI via etherscan API (#96)
- Added ABI request
- Add unique constraint on contract hash for watched contracts
2017-12-07 09:58:06 -06:00
Eric Meyer
0439791381 Rename WatchedContract to Contract 2017-12-05 09:39:58 -06:00
Eric Meyer
e432219e20 Rename watched_contracts package to contract_summary 2017-12-04 17:04:06 -06:00
Eric Meyer
a0cd7f773a Require ABI filepath for watching a contract 2017-12-04 15:12:27 -06:00
Eric Meyer
52e3266495 Move WatchedContract to repositories 2017-12-04 13:42:26 -06:00
Matt K
71de8e970d Contract hist (#84)
Add ability to query contract historical state
2017-12-04 12:54:33 -06:00
Eric Meyer
486fdc10e4 Update functions that return error to not return pointer
* Matches Golang convention
2017-12-04 10:34:49 -06:00
Eric Meyer
655d1b1d6f Remove fatal from pkg
* Functions in pkg should return errors
 * If aborting is desired, that behavior should be left to the consumer
2017-12-04 09:54:39 -06:00
Eric Meyer
708ad114ac Add function to get string state attributes for a given contract 2017-11-27 15:18:04 -06:00
Eric Meyer
9c1051bfce Update NewSummary to take the blockchain 2017-11-27 09:21:21 -06:00
Matt Krump
d0602833bb Update contract summary output to include transaction info 2017-11-14 09:57:27 -06:00
Eric Meyer
f50a4d7726 Allow users to watch and print summaries for contracts 2017-11-13 13:51:09 -06:00
Matt K
f0f086e48d Merge pull request #69 from 8thlight/watch_contract
* Add contracts table / start building out watch contracts
2017-11-13 10:55:55 -06:00
Matt Krump
4ad1d531a8 Update contract naming per Eric PR review 2017-11-13 10:11:27 -06:00
Matt Krump
30fadffb14 Add contracts table / start building out watch contracts 2017-11-09 15:06:01 -06:00
Eric Meyer
a655470f18 Refactor run main.go 2017-11-09 14:30:24 -06:00
Eric Meyer
84205a21ea Extract loading config to a shared file 2017-11-09 13:37:10 -06:00
Eric Meyer
58fbeb32fd Refactor postgres to be initialized with config 2017-11-09 12:42:24 -06:00
Eric Meyer
aa52088ba7 Refactor config to return an error instead of aborting 2017-11-09 12:41:02 -06:00
Eric Meyer
4c84173bc0 Add ability to populate missing blocks
* The command populates up to the highest known block number
 * The anticipated use case is that the listener will be running
   in parallel to the populateBlocks command
    * This will mean that the listener is responsible for picking up
      new blocks, and the populateBlocks command is reposible for
      historical blocks
 * Reformat SQL statements
2017-11-08 14:52:38 -06:00
Eric Meyer
f4a603efcb Nest packages under pkg 2017-11-06 13:06:03 -06:00
Eric Meyer
9ef1fda667 Fix issue where godo would not build project
* The solution is to add a main.go and call run
     *  parses the current file contents
2017-11-06 11:40:52 -06:00