cerc-io/ipld-eth-server
18
0
Fork 1
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 37 Wiki Activity
1,830 Commits 12 Branches 52 Tags 80 MiB
f22d919478
Commit Graph

942 Commits

Author SHA1 Message Date
Matt Griswold
20ce0ab852 Revert "option to write the plugin .go file and run the plugin migrations without building the .so file so that it can be done from the shell instead to afford complete error messages" 
This reverts commit 9bb2f27a69.
 2019-09-24 23:54:25 +00:00
Ian Norden
9bb2f27a69 option to write the plugin .go file and run the plugin migrations without building the .so file so that it can be done from the shell instead to afford complete error messages 2019-09-24 18:44:04 -05:00
Rob Mulholand
c21c069b69 Replace magic numbers with variables 2019-09-23 16:13:45 -05:00
Rob Mulholand
eea0b0bdca Return missing headers after starting block number 
- Previously, we required that the first missing header match the
  configured starting block number. This helps guarantee that we have
  the necessary data for method polling in memory, but prevents the
  process from moving forward if restarted after the starting block
  has already been checked.
 2019-09-23 16:13:45 -05:00
Rob Mulholand
37e2673a8d Do nothing on duplicate log insert 
- Prevents duplicate key constraint violation from blocking the process
  from moving forward on restart.
- If header_id, log_idx, and tx_idx are the same, we can safely do
  nothing since it's definitely the same log - a reorg would cause the
  original header to be replaced with a new ID.
 2019-09-23 16:02:07 -05:00
Rob Mulholand
3466a51b4d Correctly parse uint8 values in contract watcher 
- calling string(uint8) => insert errors on numeric columns
 2019-09-20 21:37:59 -05:00
Rob Mulholand
749db40b96 Parse bytes32 vals in Contract Watcher 2019-09-20 13:48:43 -05:00
Rob Mulholand
f83e996ab8 Update comments in checked headers repository 2019-09-18 21:39:34 -05:00
Rob Mulholand
2b798e00e0 Cap random fake timestamp 2019-09-13 11:20:13 -05:00
Rob Mulholand
3f9b034c4c Rename checked_logs => watched_logs 
- We're logging that a given log has been included in any fetch calls
  for checked headers, rather than that we have already checked for
  that log
 2019-09-10 21:22:14 -05:00
Rob Mulholand
13d503b851 Distinguish between missing and unchecked headers 
- Missing == not in DB
- Unchecked == logs haven't been fetched
 2019-09-10 14:39:45 -05:00
Rob Mulholand
ce91b0d9e6 Simplify checked header repository tests 
- Use assertions instead of comments to document expectations
- Also randomize the fake timestamp in test data
 2019-09-10 13:47:43 -05:00
Rob Mulholand
b9f3b9f946 Reference header sync logs address via foreign key 2019-08-28 10:41:34 -05:00
Rob Mulholand
5ac76eee74 Make check_count a column on public.headers 
- Don't need to maintain it on public.checked_headers if we're not
  adding additional columns to that table
 2019-08-28 09:50:17 -05:00
Rob Mulholand
666ea1c325 Update checked headers for new transformers 
- If a header was marked as checked before a transformer was added to
  the watcher, mark all headers since the new transformer's starting
  block number as unchecked.
 2019-08-28 09:25:14 -05:00
Rob Mulholand
63dabbb051 Extract and delegate logs concurrently 2019-08-28 09:25:13 -05:00
Rob Mulholand
d496dad33c Decouple log extraction from transformer delegation 
- limit missing headers results set to 100 so that extraction doesn't
  excessively block delegation
- wrap checked headers functions in repository struct
- move storage repository to factory, to correspond with event
  repository path
- remove unused files
- reformat sql
- remove line breaks in imports
 2019-08-28 09:25:13 -05:00
Rob Mulholand
cb819fa9a6 Write event logs to database before transforming 
- enables decoupling event extraction/persistence from transformation
- modifies event transformer, converter, and log chunker to accept
  payload that includes internal log database ID with log data
- remove alias for transformer pkg as shared_t
- remove unused mock watcher repository
 2019-08-28 09:13:44 -05:00
Rob Mulholand
3693ed905f Rename logs to full sync logs 
- Enable creating new table for logs used in event watching based on
  header sync
 2019-08-28 09:11:06 -05:00
Elizabeth Engelman
edc0bdf668 Address more PR feedback 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
1b3786338f Address PR feedback 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
58c23c6632 Update Copyrights 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
38f6847ff2 Create a HeaderSyncReceiptRepository 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
7e38764618 Rename ReceiptRepository -> FullSyncReceiptRepository 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
4e40e892d2 Updating header_sync_receipts to have FK reference to addresses 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
bcd6d14fcd Updating full_sync_receipts to have FK reference to addresses 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
258035833b Add GetAddressById 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
4b61c87b55 Get or create address record in a transaction 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
1373fe83a1 Pass a db into GetOrCreateAddress 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
b13eefbce7 Add AddressRepository 2019-08-23 10:11:37 -05:00
Rob Mulholand
987abd4b2e Update Geth to 1.9.0 2019-07-23 15:26:18 -05:00
Andrew J Yao
ee77fc6521 Remove injection of SubCommand for logs 2019-07-23 13:11:20 -07:00
Andrew J Yao
92d153b010 Add subCommand name to log fields 2019-07-23 08:09:57 -07:00
Edvard Hübinette
2c092e8d04
[VDB-751 VDB-754] Bugfix null pointer panic and improve logging (#119) 
* VDB-751 VDB-754 Bugfix null pointer panic and improve logging

* Fix typo
 2019-07-18 09:21:40 +02:00
Rob Mulholand
0f95267e84 Use goose library for running plugin migrations 
- Removes dependency on goose binary existing in plugin execution
  environment
 2019-06-27 13:58:58 -05:00
Rob Mulholand
f0bcf930f4 Separate tmp migrations dir from home in join 
- enable separator between home and migrations dir
 2019-06-27 13:58:58 -05:00
Rob Mulholand
0558792282 Log error on failure to setup plugin migrations 
- include specific error
 2019-06-27 13:58:58 -05:00
Andrew J Yao
11a65c525c Adds ability to set log level from toml or CLI args 2019-06-27 11:14:11 -07:00
Elizabeth Engelman
fa03716cb2 Address small PR comments 2019-05-10 11:54:15 -05:00
Elizabeth Engelman
e1a0d894a2 Update blockchain method GetHeaderByNumbers -> GetHeadersByNumbers 2019-05-08 13:42:43 -05:00
Elizabeth Engelman
ba81766f6c Small spelling fix 2019-05-08 13:42:43 -05:00
Rob Mulholand
76ab914bdc Add license 2019-05-01 12:32:39 -05:00
Rob Mulholand
2d684c5aec Extract storage diff fetching behind an interface 
- Replaces directly reading from a CSV
- Simplifies testing
- Should hopefully make it easier to plug in other sources for storage
  diffs (e.g. differently formatted CSVs, JSON RPC, etc)
 2019-05-01 12:30:36 -05:00
Gabe Laughlin
a344432156
(VDB-560) Rename lightSync to headerSync 2019-05-01 12:12:55 -05:00
Rob Mulholand
3fd6269b78 Don't lookup transactions if no log events 
- Prevents EOF error on transactions lookup
 2019-04-18 09:53:18 -05:00
ana0
964fc8bd22 Merge branch 'staging' into databaseConfig 2019-04-11 14:19:57 -04:00
anon
bd12149594 Database config 2019-04-10 17:56:31 -04:00
Gabe Laughlin
8903b6fb2d Add timestamp customization for fake header 2019-04-09 17:08:15 -05:00
Ian Norden
9befc76fc6 remove LIMIT 100; method to check continuity of headers returned but doesn't require 
the set to start at a specific block number; config for
account_transformer; review fixes; update schema
 2019-04-05 17:21:59 -05:00
Ian Norden
c1940c3e58 full_sync and light_sync receipt tables and light receipt repository methods and test 2019-04-05 16:36:48 -05:00
Ian Norden
04ad7b7696 Merge pull request #47 from vulcanize/track_uncle_rewards 
Track uncle rewards + direct balance polling
 2019-04-04 15:55:40 -05:00
Ian Norden
5dcf534b2c review fixes 2019-04-04 15:21:39 -05:00
Ian Norden
72c73c25b7 update vendor dep; constrain to revision of geth with UnpackIntoMap; updates to match 2019-04-03 00:14:02 -05:00
Ian Norden
197f98c93d going ahead and indexing the entire uncle blocks (one of the issues open on public); finish tests 2019-04-02 14:32:30 -05:00
Ian Norden
fd407825c1 review fixes 2019-04-02 10:10:17 -05:00
Ian Norden
185f4c0e93 adjust block/uncle reward tests and add methods to pkg/geth blockchain and ethclient for direct fetching of eth balances 2019-04-02 10:10:17 -05:00
Ian Norden
9030cff2bd keep track of who is receiving uncle rewards 2019-04-02 10:10:17 -05:00
Rob Mulholand
f5b32a11b0 Improve transaction syncing test coverage 2019-03-28 14:31:17 -05:00
Rob Mulholand
81dfd12665 Address PR comments 2019-03-28 14:31:17 -05:00
Rob Mulholand
54d46638a8 updates for light sync transactions 2019-03-28 14:31:17 -05:00
Rob Mulholand
79e011aad2 Add light sync transactions table 
- Foreign key to header instead of block
- Can use same Transaction struct for both
 2019-03-28 14:31:17 -05:00
Rob Mulholand
d93006321b Rename transactions to full_sync_transactions 
- Table has foreign key to blocks
- Add transaction RLP and transaction index to table
- Enables other tables with standalone transactions or transactions
  associated with other data structures (e.g. headers)
 2019-03-28 14:31:17 -05:00
Ian Norden
1aa849bcb4 PR fixes; remove all infura token references and setup travis to use encrypted env 
variable; rest of the ethjson_rpc dependent tests extracted to integration_test
 2019-03-21 18:36:51 -05:00
Ian Norden
d3e172d9ab remove references to infura token except for in infura.toml so that tests will continue to pass on CI 2019-03-21 18:36:49 -05:00
Ian Norden
37fc038605 fixes for review comments 2019-03-21 18:36:49 -05:00
Ian Norden
37e581c7ec PR#72 from public repo- https://github.com/vulcanize/vulcanizedb/pull/72- also needed to finish pluggin in ENS record transformer 2019-03-21 18:33:56 -05:00
Ian Norden
24879a85aa mv pkg/omni pkg/contract_watcher 2019-03-21 18:33:56 -05:00
Ian Norden
073378de6d update © to 2019 2019-03-21 18:33:56 -05:00
Ian Norden
e4e092f542 changes to plugin and commands to accomodate changes 2019-03-21 18:33:56 -05:00
Ian Norden
8174ce4aee refactor omni code; rename omniWatcher => contractWatcher; use config instead of excessive number of CLI flags 2019-03-21 18:30:48 -05:00
Ian Norden
ff55e3ba7a TransformerInitializer => EventTransformerInitializer reorganization 2019-03-14 11:59:39 -05:00
Ian Norden
9f8c50e3ab tests for migration path ordering and errors 2019-03-13 11:42:30 -05:00
Ian Norden
f37c458992 the part that actually runs the migrations in order 2019-03-13 11:42:29 -05:00
Ian Norden
84aa0a7eba plugin migration order specified in config 2019-03-13 11:42:29 -05:00
Rob Mulholand
1414779d52 Use *sqlx.Tx instead of *sql.Tx 
- requires using db.Beginx() instead of db.Begin()
- enables calling tx.Get()
 2019-03-06 12:43:39 -06:00
Rob Mulholand
7ed4dcd1fa Remove Tic logic 2019-03-06 12:43:39 -06:00
Rob Mulholand
b6bef49318 Add tests for plugin helpers 
- Prevents code analysis from showing functions as unused
- Also address other code analysis issues
 2019-03-06 12:43:39 -06:00
Rob Mulholand
a806cb2818 Extract slow tests from omni full transformer 2019-03-06 12:43:06 -06:00
Rob Mulholand
34c96654ce Extract slow tests from omni light transformer 2019-03-06 12:43:06 -06:00
Ian Norden
8bd397e2eb Merge pull request #29 from vulcanize/fix 
Fix for Travis issue at https://github.com/vulcanize/mcd_transformers/pull/6
 2019-03-06 11:27:43 -06:00
Ian Norden
e1a317c3f6 different errors; readme edit 2019-03-06 00:14:37 -06:00
Ian Norden
5fd9a59027 fix for issue with Travis building plugin from plugin repo 2019-03-05 08:57:54 -06:00
Edvard Hübinette
55b6ad9381 Optimise missing block numbers query (#23) 2019-03-05 11:52:38 +01:00
Ian Norden
83f7daf37d compose and execute as separate commands; update README and add git checkout vendor/github.com/ethereum/go-ethereum/accounts/abi to make build 2019-02-28 18:15:16 -06:00
Ian Norden
708425c4d6 rebase; extract factories and the mocks they are dependent on to 
libraries/shared; adjust omni test_helpers to drop and recreate
checked_headers table to avoid reaching 1600 column limit after repeated
tests (dropping columns doesn't actually remove them from contributing
to that limit)
 2019-02-25 01:34:38 -06:00
Ian Norden
17f4d3dfa5 move maker plugin generator test to mcd_transformers 2019-02-24 21:53:04 -06:00
Ian Norden
3d34a9e7c9 remove maker migrations and convert back to timestamps and fix bug in 
composeAndExecute command
 2019-02-24 21:52:23 -06:00
Ian Norden
b449193b16 addressing review comments; still need to reorg migrations and add helper tests 2019-02-24 21:38:48 -06:00
Ian Norden
2868cf2f73 goose changes; update plugin readme; use rel paths for transformers in config; more commenting in pkg/plugin; fix PluginWriter to work with different transformer types; reopen generator tests 2019-02-24 21:38:48 -06:00
Ian Norden
03a7379617 work on porting storage watcher; watcher type is defined for 
transformers in the config, storage transformers are exported from
plugin like event transformers
 2019-02-24 21:38:47 -06:00
Ian Norden
b51dcb55de comment out plugin test til mcd_transformers is updated correctly 2019-02-24 21:38:47 -06:00
Ian Norden
a322372713 refactoring plugin generating code 2019-02-24 21:38:47 -06:00
Ian Norden
decc2a3caf remove pkg/transformers; extract shared files needed to 
libraries/shared; work on automated db migration management
 2019-02-24 21:38:47 -06:00
Ian Norden
55fa9b8364 auto management of dependencies temporarily needed to build plugin and transformer 
migrations; new test showing it is working with external transformers runs
properly when migrations were manually performed but still need
to test automated migration management
 2019-02-24 15:23:35 -06:00
Ian Norden
6c2d895023 composeAndExecute command that loads and executes over arbitrary transformer set exported from a go plugin generated according to config file; test for plugin generation, loading, and execution; work on plugins README 2019-02-24 15:23:35 -06:00
Edvard Hübinette
fa4abe4e5c Remove spammy and unnecessary error log (#21) 2019-02-22 12:56:06 +01:00
Rob Mulholand
745d0e8eda Use storage value metadata getter in pit repository tests 2019-02-21 15:25:16 -06:00
Rob Mulholand
a5121913f4 Error on missing keys in pit repository 
- If ilk missing for ilk-related value, fail fast
 2019-02-21 15:03:22 -06:00
Rob Mulholand
3d9341ea1d Reference ilks foreign key from Cat storage repository 2019-02-21 14:33:59 -06:00
Rob Mulholand
dbdd16d3a2 (VDB-380) Create separate table for ilks 
- reference ilk by foreign key every place it's used
 2019-02-21 11:23:00 -06:00
Rob Mulholand
11dd641a84 (VDB-354) Queue unrecognized storage diffs 
- If we recognize a storage diff as coming from a watched contract but
  don't recognize the key, queue it for retrying later (after we've seen
  an event that might help us recognize the key)
- Remove unused errs and args
- Panic on unrecognized types (should not happen)
 2019-02-20 15:22:39 -06:00
Edvard
fb47a747f6 Fix confusion between nflip and flip 2019-02-20 14:37:55 +01:00
Edvard
56ca45509f DRY up storage diff repo tests 2019-02-20 14:37:55 +01:00
Edvard
5746de7c97 Add tests for Cat storage diff transformer 
Update schema after rebase migrations

Fix small issues from review
 2019-02-20 14:37:46 +01:00
Edvard
cd6611d2ec Add Cat storage transformer 2019-02-20 14:37:46 +01:00
Edvard Hübinette
53789c82ab Merge pull request #12 from vulcanize/VDB-392-fix-io-error-propagation 
VDB-392 Fix IO error propagation
 2019-02-20 13:58:00 +01:00
Edvard
1a2f46b308 Fix small issues from review 2019-02-20 13:10:55 +01:00
Ian Norden
2ecc524224 Merge pull request #9 from vulcanize/VDB322-Vow-storage-diff 
Vdb322 vow storage diff
 2019-02-15 11:04:22 -06:00
Taka Goto
64c7ae1bb1 uppercase Woe and Ash 2019-02-15 06:15:47 -06:00
Rob Mulholand
df521eea7f Merge pull request #11 from vulcanize/storage-transformers-readme 
Add storage transformers readme
 2019-02-14 14:24:01 -06:00
Edvard
13eacf2081 Update tests for error propagation 2019-02-14 16:04:12 +01:00
Edvard
6cd4e5ea95 Improve I/O error propagation 2019-02-14 16:03:57 +01:00
Rob Mulholand
0378068467 add example contract and instructions for watching its storage 2019-02-13 20:34:30 -06:00
Taka Goto
a8322c7d26 add woe to storage diff vow contract 2019-02-12 21:50:52 -06:00
Rob Mulholand
ee113668a5 Add first draft of storage transformers readme 
- Explain process of creating a new storage transformer
 2019-02-12 15:58:28 -06:00
Rob Mulholand
ac9375ae47 Discard logs created during tests 
- Make sure output is being set to discard for logrus to reduce clutter in the console
 2019-02-12 11:18:39 -06:00
Taka Goto
4fe2ff4104 repository methods: 2019-02-11 16:34:02 -06:00
Taka Goto
c2c21325b5 vow contract storage diffing excluding sin 2019-02-11 15:28:54 -06:00
Edvard Hübinette
f73623789e Merge pull request #6 from 8thlight/VDB-337-Dockerize-vDB 
VDB-337 Dockerize vDB
 2019-02-11 12:37:56 +01:00
Edvard
9b986c4c14 Nuke OpenRC from docker image, update readme's 2019-02-11 11:27:33 +01:00
Edvard
553fdbfa09 Default config starting block to 0 2019-02-11 11:27:33 +01:00
Edvard
e6e66d58dd Add container service files and docker README 2019-02-11 11:26:12 +01:00
Rob Mulholand
c05d3b7836 (VDB-280) Consume vat contract storage diffs 2019-02-08 13:36:30 -06:00
Rob Mulholand
dba4e48a3e Use map for key in storage value metadata 
- Enables inserting multiple keys for one storage row (e.g. ilk and urn)
 2019-02-08 11:41:17 -06:00
Rob Mulholand
09abcb841a Persist Urns in raw bytes32 format 
- Enables simple usage in contract storage lookups
 2019-02-08 11:41:17 -06:00
Takayuki Goto
0a024d429d * update transformer to able to recheck headers (#4) 
* update transformer to able to recheck headers
* put cap on rechecking header
* integration test for recheck headers
* use enum for recheck headers param; make recheck cap configurable
* update integration tests with new test config
* update omni pkg with new recheck header column type
* update migration with new migration tool and final tweaks needed to accommodate changes in omni pkg
 2019-02-08 10:35:46 -06:00
Rob Mulholand
867f92c431 (VDB-298) Consume Pit contract storage diffs 
- Continuously parse storage diffs CSV data to read Pit contract state
- Convert ilks in database to raw bytes32 value for use in generating
  storage keys dynamically
- Persist storage diffs with block number and hash for validation
 2019-02-06 10:31:46 -06:00
Edvard Hübinette
29f2537130 Merge pull request #7 from vulcanize/VDB-347-Squelch-lightSync 
VDB-347 Squelch lightSync in stdout
 2019-02-01 11:02:48 +01:00
Edvard
a1fa21cbec Remove test based on window buffer stuff 2019-01-31 16:39:29 +01:00
Edvard
8eea1b00ed Redirect validation window status to logrus instead of stdout 2019-01-31 15:55:28 +01:00
Ian Norden
8749c94994 update license to AGPL-3.0 2019-01-24 14:41:30 -06:00
Ian Norden
ab371a2024 Merge remote-tracking branch 'maker/staging' into staging 2019-01-24 10:22:22 -06:00
Ian Norden
1f95eb6443 Merge branch 'staging' into pub_merge 2019-01-24 09:47:12 -06:00
Taka Goto
c5d513e96d fix index out of range on lightSync; hopefully the new integration tests will catch them next time :) 2019-01-23 15:25:44 -06:00
Ian Norden
00c2052da3 for some reason an error is thrown when the ipc string is declared in integration_tests_suite_test.go but not when moved to a different file in the integration_tests package; I think this is due to switching to go1.11 2019-01-23 15:23:23 -06:00
Ian Norden
dc87f39889 change omniwatcher database test helpers so that it doesn't interfere with downstream maker tests 2019-01-23 14:55:20 -06:00
Konstantin Zolotarev
65cb6e9092 VDB-205 Fix external configuration values (#121) 
* Updated loading config from /Users/konstantinzolotarev/.vulcanizedb for addreses

* Updated contract addresses loading from env variables.

* Implement getter chain for addresses, configs and transformers

Solves the issue with declaring stuff as vars when the environment file hasn't been loaded yet. Instead, this data is fetched dynamically when the root commands are called.

* Update test data to use constant Kovan addresses

* Decouple integration tests from config file IO

* Add ABI to environment file, convert getter chains for related values

* Decouple tests suites from ABI config file IO

* Add startingBlock to environment and implement getter chain

* Exit when called without config file

* Add missing deployment blocks to staging.toml

* Remove comment
 2019-01-23 19:44:09 +01:00
Ian Norden
46b898f690 merge public vulcanizedb/master 2019-01-23 00:37:26 -06:00
Taka Goto
3595771825 use batchElem from rpc client 2019-01-21 14:52:37 -06:00
Takayuki Goto
54d771cb06 Merge branch 'staging' into batch-historical-headers 2019-01-16 15:56:36 -06:00
Taka Goto
24cf0661b0 lowercase address on map to get transformers 2019-01-16 14:24:00 -06:00
Taka Goto
07da9d720b fix get headers when less than 100 fetched 2019-01-16 10:16:06 -06:00
Taka Goto
074140b9ac iterate for loop by headers instead of blocknumbers 2019-01-15 16:34:20 -06:00
Ian Norden
aa4e698240 move interface_getter.go from ens_watcher to vDB pkging- used to construct a custom ABI based on results from calls to supportsInterface; also moved fetcher.go from examples to pkg 2019-01-15 15:49:18 -06:00
Takayuki Goto
f1ba21b505 Merge branch 'staging' into batch-historical-headers 2019-01-13 21:29:58 -06:00
yaoandrew
e2b864baa8 Add rollback if GetTicInTx fails and move GetTicInTx to shared/repository.go 2019-01-11 10:07:00 -08:00
Andrew J Yao
c8aa3dc275 Replace standard log with logrus import 2019-01-11 10:07:00 -08:00
yaoandrew
bc8b7a101c refactor error handling in vat, cat, vow 2019-01-11 10:07:00 -08:00
yaoandrew
525fada3d7 refactor error handling in frob, pit, price, tend 2019-01-11 10:07:00 -08:00
yaoandrew
657aaccff3 refactor error handling in drip, flap, flop 2019-01-11 10:07:00 -08:00
yaoandrew
05351a8ef4 refactor error handling in dent 2019-01-11 10:07:00 -08:00
yaoandrew
08ac7f18a5 refactor error handling in flip_deal 2019-01-11 10:07:00 -08:00
yaoandrew
f655997fde refactor error handling in pit_vow 2019-01-11 10:07:00 -08:00
yaoandrew
843d30c88e refactor error handling in flip 2019-01-11 10:07:00 -08:00
yaoandrew
513513e93b refactor error handling in chop_lump 2019-01-11 10:07:00 -08:00
yaoandrew
78b302eab6 refactor error handling in bite transformer 2019-01-11 10:07:00 -08:00
Edvard Hübinette
ebca338b1e VDB-302 Sleep when no missing blocks in lightSync (#129) 
* Sleep when no missing blocks in lightSync

* Fix tests and error propagation

* Correct geth.log to logrus
 2019-01-11 10:58:41 +01:00
Takayuki Goto
ac430ddf54 Merge branch 'staging' into batch-historical-headers 2019-01-08 19:06:53 -06:00
Taka Goto
aa99c861a8 batch headers by 100s 2019-01-08 16:53:26 -06:00
Ian Norden
59cdaa05e6 moving some memory allocation to init function rather than within 
execution cycles and fixing test db tear down so that it drops
checked_headers table instead of individual columns (droping columns
only hides them, they remain in memory and continue to contribute to the
1600 column limit which causes us to eventually exceed it)
 2019-01-07 13:07:31 -06:00
Edvard
2292c94e99 Merge branch 'staging' into VDB-125-aggregate-log-fetching 2019-01-07 15:25:37 +01:00
Edvard
50cc0c2459 Update tests for watcher and repository 2019-01-07 15:20:05 +01:00
Edvard
833dde62cf Remove injection of fetcher and repository to watcher 2019-01-07 15:19:31 +01:00
Ian Norden
2cbe6e7a70 method piping 2018-12-21 11:14:54 -06:00
Ian Norden
456c735087 fetch headers and logs in batches across all contracts and events 2018-12-21 10:33:31 -06:00
Ian Norden
d188329661 fetch event logs in batches- all events for a contract address at a given block height are fetched together, different contract events still fetched separately 2018-12-21 10:33:31 -06:00
Ian Norden
e390a97502 finish method polling with hash or address type argument values 
collected from watched events; 'MissingMethodsCheckedEventsIntersection'
method to find headers which have been checked for each event of
interest but methods have not yet been polled at that header; tests for
new features; travis ci go version 1.9 -> 1.11 ; consolidate omniWatcher and lightOmniWatcher into single command with light as default
 2018-12-21 10:33:31 -06:00
Ian Norden
0a59f06cac begin work on: Add checked_headers column for methods that are polled so taht we don’t duplicate; Add batching of method polling so that we arent generating a rediculously large account address list before using it to poll methods (or persist the list in pg?); User passed ABI and other ways to get ABI; Add ability to collect []byte and hashes from events and use them in method polling same manner as addresses; Event filter addrs => only those event’s addresses/hashes are used for polling; Option to persist seen address/hash/bytes lists into pg; Only generate lists of addresses, []byte, or hashes if a method will use them later 2018-12-21 10:33:31 -06:00
Edvard
e54699c039 Remove todo on chunker non-match (it's basically a filter) 2018-12-14 15:22:46 +01:00
Edvard Hübinette
a3e8633aff VDB-104 improve lightsync ui (#125) 
* Return PopulateMissingHeaders early if the sync is at the head of the chain

* Squelch logging if no blocks to sync

* Fix broken test

* Refactor repository tests
 2018-12-14 12:11:48 +01:00
Edvard
7cf694f5e7 Update transformer documentation 2018-12-14 10:30:08 +01:00
Edvard
35f7f719e6 Swap strings.Builder for bytes.Buffer for go 1.9 support 2018-12-13 16:08:57 +01:00
Edvard
129964f3bc Add tests for now shared MissingHeaders 2018-12-13 13:58:16 +01:00
Edvard
e5db740ae4 Move MockTransformer to test_data 2018-12-13 13:03:51 +01:00
Edvard
d30fcfed80 Use new config getter on shared.Transformer <3 2018-12-13 12:39:57 +01:00
Edvard
882a6dd7a2 Update tests with new adding of transformers 2018-12-12 15:41:44 +01:00
Edvard
dc43547612 Improve process of adding transformers 2018-12-12 15:41:29 +01:00
Edvard
a86a2ec5c6 Improve checked header column detection 2018-12-12 12:13:23 +01:00
Edvard
4e089c363e Fixes after merging staging 2018-12-11 15:19:27 +01:00
Edvard
ff47e2e06e Merge branch 'staging' into VDB-125-aggregate-log-fetching 2018-12-11 15:03:21 +01:00
Edvard
9000c6e9f9 Add test for watcher log delegation 2018-12-11 15:02:32 +01:00
Edvard
5d4239de05 Finish watcher and watcher tests (+fmt) 2018-12-11 13:52:02 +01:00
Edvard
d93817f346 Interface-ify shared repository 2018-12-11 11:35:13 +01:00
Edvard
83593cb5f2 Cleanup fetcher and add test todos 2018-12-10 21:12:55 +01:00
Edvard
2d81861ae5 Add chunker tests 2018-12-10 21:11:25 +01:00
Edvard
bc9fd17793 Fix remaining integration tests 2018-12-10 17:12:51 +01:00
Edvard
3cb8fe60d4 Fix persistHeader not returning proper header ID 2018-12-10 16:58:24 +01:00
Edvard
637623b752 Fix test suites! 2018-12-10 15:40:45 +01:00
Rob Mulholand
670741616e Use logrus for logging in populating blocks 
- Removes logging from test output
 2018-12-07 11:24:06 -06:00
Rob Mulholand
2a518ad518 (VDB-254) Convert numerical values for Pit.file ilk 
- convert data to ray when what == "spot" and wad when what == "line"
 2018-12-07 11:24:06 -06:00
Rob Mulholand
087af252c9 (VDB-253) Update numerical values for Pit.file debt ceiling 
- convert Line to wad
 2018-12-07 11:24:06 -06:00
Rob Mulholand
4a5765b20d (VDB-249) Update numerical values for Vat.fold 
- convert rate to ray
 2018-12-07 11:24:06 -06:00
Edvard
e1ba7ac5b4 WIP update tests for aggregate fetching 2018-12-07 18:10:36 +01:00
Edvard
47c75d055b Update repository tests for absence of MissingHeaders 2018-12-04 17:53:46 +01:00
Edvard
9789648cc2 Delegate log chunks to respective transformers 2018-12-04 17:40:39 +01:00
Edvard
8bebcdc064 Change transformer.Execute to single header 2018-12-04 17:05:34 +01:00
Edvard
38c745e8c3 Refactor fetching out from repositories to log_fetcher and watcher 2018-12-04 16:04:13 +01:00
Ian Norden
e02b33547d finishing porting omni watcher to work with light sync; split into full, light, 
and shared directories and refactor as much into shared; finish
lightSync omni watcher tests
 2018-11-30 13:33:48 -06:00
Edvard
c26736dc9e WIP continue on aggregate fetching architecture 2018-11-30 17:28:52 +01:00
Ian Norden
975f13b969 reorganizing omni directory and beginning light watcher work 2018-11-29 20:33:21 -06:00
Ian Norden
817bd76713 refactoring event converter and repository so that event logs are 
persisted as they are converted, preventing build up in memory, and finishing  method polling (for full sync)
 2018-11-29 20:33:21 -06:00
Ian Norden
390a60f7f6 port over lightSync updates from maker repo 2018-11-29 20:33:21 -06:00
Elizabeth
8ce901d1af Convert drip_file_ilk tax field to ray (#117) 2018-11-29 14:10:12 -06:00
Edvard
45a087f5d1 WIP: First steps in refactoring to aggregate fetching 2018-11-28 17:00:04 +01:00
Edvard
423fdf01b5 Add log_chunker 2018-11-28 16:59:06 +01:00
Edvard
f7ecab9dc2 Fix typo 2018-11-28 15:39:46 +01:00
Taka Goto
e8be96a4e7 use logrus for logging 2018-11-21 10:14:11 -06:00
Elizabeth
128d20c9bf Vdb 269- fetch logs by hash (#122) 
* Upgrade geth from 1.8.15 to 1.8.18

* Update vat_tune to use shared repository methods

* Query blockchain by block hash instead of block number range

* Remove hash validation from repositories

* Fix vow flog integration test

* Update README Travis build sticker

* Update constants formatting per go fmt

* Update EthPublicKeyParser.ParsePublicKey to use discv5.PubkeyID method

* Address PR comments
 2018-11-21 09:30:00 -06:00
Elizabeth
b19e77e419 Add integration test for Vow.flog (#114) 2018-11-15 16:44:49 -06:00
Ian Norden
417b18ec6a Edits to address PR issues; change license from apache to AGPL; and work 
towards generic method polling and reposito;y; config settings to
filterevents/methods by account address; refactoring some stuff out of
repo and into converter; remove fetcher and instead call
blockchain's FetchContractData directly; finishing tests
 2018-11-15 12:32:52 -06:00
Edvard
7cd512ae91 Remove PFSF label 2018-11-15 12:14:59 -06:00
Rob Mulholand
854236fd99 (VDB-267) Remove pit file stability fee 
- Method is no longer present on the Pit contract
 2018-11-15 12:14:59 -06:00
Edvard Hübinette
40408e8300 VDB-120 include tic in dent and tend (#118) 
* Add constants for TTL and tic for fakeHeader

* Add tic to tend transformer

* Update tests for tend

* Fix string conversion bug in fakes

* Fix tend integration tests after staging rebase

* Add tic to dent transformer

* Update dent tests

* Change integration tests to use hardcoded block timestamp
 2018-11-15 12:57:29 +01:00
Edvard
8f4648a9e3 Add support for never-ending transformer 2018-11-14 10:11:16 -06:00
Rob Mulholand
82fd73ba3f (VDB-68) Verify log block hash matches header block hash 
- Delete header on conflict to prompt data refresh (cascade deletes all
  data associated with that block)
- Derive header hash from rpc payload rather than computing it from data
  (prevents hash mismatch between blockchain and cache)
 2018-11-13 14:51:39 -06:00
Elizabeth
6a43cd4287 Vdb 207 the rest of the repos (#111) 
* Use shared repository test examples for frob

* Use shared repository methods for frob

* Use shared repository test examples for flop kick

* Use shared repository methods for flop kick

* Use shared repository test examples for flip kick

* Use shared repository methods for flip kick

* Use shared repository test examples for bite

* Use shared repository methods for bite

* Alphabetize check headers constants
 2018-11-12 10:50:51 -06:00
Edvard Hübinette
4f295b9aec VDB-207 refactor init move split toll tune flog (#110) 
* Use shared repo tests for vat_init

* Use shared methods in vat_init repository

* Use shared repo tests for vat_move

* Use shared methods in vat_move repository

* Use shared repo tests for vat_slip

* Use shared methods in vat_slip repository

* Use shared repo tests for vat_toll

* Use shared methods in vat_toll repository

* Use shared methods in vat_tune repository

* Use shared repo tests for vow_flog (+small namespace fix)

* Use shared methods in vow_flog repository

* Go vet + tiny oopsie
 2018-11-12 14:17:58 +01:00
Edvard Hübinette
571f300392 Add rendering of available transformer names to continuousLogSync help (#109) 2018-11-12 14:01:07 +01:00
Elizabeth
672269effa VDB 207 refactor more repo tests (#107) 
* Use shared repo tests for vat flux

* Use shared repo methods for vat flux

* Use shared repo test examples for vat fold

* Use shared repository methods for vat fold

* Use shared repo test examples for vat grab

* Use shared repo methods for vat grab

* Use shared repo test examples for vat heal

* Use shared repo methods for vat heal
 2018-11-09 14:54:22 -06:00
Elizabeth
c1e10f09fb Vdb 67 convert numbers to ray and wad (#105) 
* Add method to convert values to ray or wad units

* Convert data to ray or wad for cat_file_chop_lump

* Use shared convert functions in price feed conversion

* Pull common ray/wad values into vars

* Fix after rebase with staging
 2018-11-09 14:53:20 -06:00
Rob Mulholand
0b58efb56a Extract common repository methods 
- Inject column names to reduce duplication across common behavior
- Extract checked headers column names to constants
 2018-11-08 13:08:59 -06:00
Rob Mulholand
8c168ed332 Extract constants package 
- Reduce clutter in the shared package
 2018-11-08 13:08:59 -06:00
Rob Mulholand
8c5192f82f (VDB-210) Refactor tend repository tests 2018-11-08 13:08:59 -06:00
Rob Mulholand
1c1a9c699a (VDB-209) Refactor price feed repository tests 2018-11-08 13:08:59 -06:00
Rob Mulholand
af8bbe5ce9 (VDB-208) Refactor pit file repository tests 2018-11-08 13:08:59 -06:00
Rob Mulholand
a1c4d70a77 (VDB-99) Transform Flap deal events 
- Add flapper address to deal transformer config
 2018-11-07 10:52:55 -06:00
Rob Mulholand
9cc9d4d3b6 (VDB-95) Transform Flop dent events 
- Add flopper address to dent config
 2018-11-07 10:52:55 -06:00
Rob Mulholand
52bbedd837 (VDB-98) Transform Flap tend events 
- Add Flapper contract address to tend transformer config
 2018-11-07 10:52:55 -06:00
Rob Mulholand
b2a78f1da4 Remove unused transformer config struct 2018-11-07 10:52:55 -06:00
Rob Mulholand
cb141f7cb7 (VDB-204) Add transformer for Flap kick 2018-11-07 10:52:55 -06:00
Taka Goto
627ae288bd use self hosted parity kovan for staging 2018-11-06 12:58:45 -06:00
Takayuki Goto
7afd978cf0 Merge pull request #100 from 8thlight/update-integration-tests 
integration tests hits self hosted remote parity kovan node:
 2018-11-06 12:08:48 -06:00
Elizabeth
0ab5ea1b25 Refactor more repo tests (#101) 
* Use shared repository tests for deal

* Small updates to repo tests

* Use shared repository test examples for dent

* Use shared repository test examples for drip_drip

* Use shared repository test examples for drip_file_ilk

* Use shared repository test examples for drip_file_repo

* Use shared repository test examples for drip_file_vow

* Update formatting
 2018-11-06 10:51:26 -06:00
Elizabeth
a4abf3781e Refactor repository tests (#99) 
* Add shared test examples for repositories

- Create
- MissingHeaders
- MarkHeaderChecked

* Use shared repository tests for cat_file_chop_lump

* Use shared repository test for cat_file_flip

* Use shared repository test for cat_file_pit_vow

* Updates to shared repository tests

* Use shared repository tests for deal

* Small updates to repo tests
 2018-11-06 10:30:31 -06:00
Elizabeth
bfe4531d7e Common transformer tests (#97) 
* Move chop lump transformer test to a generic log_note_transformer_test

* Add test cases to transformer test

- cat file flip
- cat file pit vow
- deal
- dent

* Move transformer tests over to transformer factor test file

- drip drip
- drip file ilk/repo/vow
- pit file deb ceiling/ilk/stability fee
- price feeds

* Move LogNoteTransformer test into factory package and remove log specific transformer tests

* Move Transformer test into factory package and remove log event specific transformer tests

* Respond to PR comments
 2018-11-06 10:30:11 -06:00
Taka Goto
2cc9826be9 create integration toml file 2018-11-06 09:25:08 -06:00
Ian Norden
b459cf35ed beginning work on method polling; first need to generate list of token holder address in a completely generic/contratc agnostic fashion. Created address retriever that can iterate through any given contract's watched events, finding the inputs/arguments with address type, and generate a list from those values. Edit: Contract objects now cache every event emitted address as its event logs are transformed into the repo to grow a list of contract associated addresses as we go 2018-11-04 21:37:31 -06:00
Ian Norden
e9dbd771e5 tests and fixes for fetcher, parser, retriever, converter, and repository; update cmd and watcher 2018-11-04 01:49:11 -05:00
Ian Norden
8ce75fe5ad Generic watcher that takes a contract address, grabs the contract abi and starting block number, creates custom event filters, and extracts and transforms event data into postgres. Can configure to look at only a subset of events through CLI flag. Building but needs testing. 2018-11-03 14:00:25 -05:00
Ian Norden
9b41000b88 Moved fetcher to generic directory (methods have to remain public since it is in seperate package now), added FetchHash method, created ERC20 and generic getters which call the fetcher with specific contract methods (GetTotalSupply, GetBalance, GetAllowance for ERC20 getter, and GetOwner, GetStoppedStatus, GetStringName, GetHashName, GetStringSymbol, GetHashSymbol, and GetDecimals for generic getter). Getter tests cover all but GetBalance and GetAllowance, and also cover all of the Fetcher methods- but with only nil methodArgs. GetAllowance and GetBalance tests are not working against infura and these are the only contract method calls with arguments passed in so I suspect this might be where the issue lies. Have tested GetBalance using previous version of FetchContractData without the variadic input to the Pack method and it fails with the same error so I don’t think it is due to those changes. 2018-11-03 13:49:23 -05:00
Ian Norden
ccd21f73c9 adjusted FetchContractData to work with variable number of contract method arguments, changed fetcher's FetchSupply method to more general FetchBigInt method and added FetchBool and FetchAddress methods 2018-11-03 13:49:23 -05:00
Rob Mulholand
ba071ef13f Consolidate test doubles 
- Migrate various mocks of core namespaces to shared version in `fakes` pkg
- Err on the side of making test doubles less sophisticated
- Don't pull over mocks of namespaces that are only used in example code
 2018-11-03 13:49:23 -05:00
Rob Mulholand
5fe6394406 Add tests for pkg/geth/blockchain 
- inject dependencies instead of initializing them in the constructor
 2018-11-03 13:49:23 -05:00
Rob Mulholand
05186634bd Add light sync command 
- Only syncs block headers (excludes block bodies, transactions, receipts, and logs)
- Modifies validation window to include the most recent block
- Isolates validation window to the variable defined in the cmd directory (blocks
  have a separate variable defined in the block_repository for determining when
  to set a block as final)
 2018-11-03 13:49:23 -05:00
Maxim Krasilnikov
7b507667f8 Added user and password to db config 2018-11-03 13:49:23 -05:00
Takayuki Goto
0d325afbfb flop kick transformer factory (#98) 
* flop kick transformer factory

* add tests for wrong model/entity
 2018-11-02 09:13:10 -05:00
Elizabeth
981393d0a7 Use transformer factory for FlipKick (#96) 2018-11-02 09:12:51 -05:00
Taka Goto
3b722c132e integration tests hits self hosted parity kovan node: 2018-11-01 10:24:03 -05:00
Elizabeth
ab4e904713 Frob transformer factory (#95) 
* Updates to previous transformers

* Use transformer factory for Frob

* Move bite integration test to integrationt est pkg
 2018-10-29 15:08:00 -05:00
Rob Mulholand
78b8173b37 Improve test coverage for event method signature lookups 2018-10-26 15:35:18 -05:00
Rob Mulholand
602802e5f4 Don't test order of DB results 
- reduce flakiness by enabling tests to pass regardless of result order
 2018-10-26 15:35:18 -05:00
Rob Mulholand
ed624330f6 Rename Flog pack to Vow flog 
- enables differentiation if another contract implements flog
- also deletes flog-specific mocks
 2018-10-26 15:35:18 -05:00
Rob Mulholand
1b680d2a4b Use transformer factory for Drip drip 2018-10-26 15:35:18 -05:00
Edvard Hübinette
f3a82e6ba1 VDB-184 Refactor Dent (#91) 
* Factorise dent transformer

* Update dent transformer tests

* Add additional tests to dent repository test

* Update dent to use LogNoteTransformer factory
 2018-10-26 13:56:49 -05:00
Rob Mulholand
d1ee501f12 Use transformer factory for Bite (#89) 
* Use transformer factory for Bite

- introduces separate transformer factory for non-LogNote events
- converter includes `ToEntities` for events defined via ABI

* Updates after rebasing with staging
 2018-10-26 13:26:10 -05:00
Rob Mulholand
0e5b23993a Discard log output from Vat heal during tests 2018-10-26 12:02:56 -05:00
Rob Mulholand
45a52a59f9 Add repository tests for Deal 2018-10-26 12:02:56 -05:00
Edvard
0e1675354f Update tests for deal transformer 2018-10-26 12:02:56 -05:00
Edvard
35a8009410 Refactor deal transformer to use factory 2018-10-26 12:02:56 -05:00
Elizabeth
d522d33531 Update vat heal to use transformer factory (#88) 
* Update vat heal to use transformer factory
 2018-10-26 11:40:24 -05:00
Rob Mulholand
5de949f4c5 Add cat file repository tests 2018-10-26 11:36:02 -05:00
Edvard
00bb48bafb Fix whitespace errors 2018-10-26 11:36:02 -05:00
Edvard
e6e2ed3308 Update cat file transformer tests 2018-10-26 11:36:02 -05:00
Edvard
28ab5f92ab Factorise cat_file transformers 2018-10-26 11:36:02 -05:00
Rob Mulholand
47a3c35938 Use transformer factory for Price Feed events 2018-10-26 11:25:16 -05:00
Elizabeth
bd428e752a Update vat grab to use transformer factory (#85) 
* Update vat grab to use transformer factory
 2018-10-26 11:02:39 -05:00
Rob Mulholand
745b3a460e Add test for updating existing checked header row 2018-10-26 10:45:49 -05:00
Rob Mulholand
e37447ae9a Use transformer factory for Vat fold 2018-10-26 10:45:49 -05:00
Takayuki Goto
0e7f316568 flip kick integration test (#83) 
* flip kick integration test

* use timestamp instead to compare date

* fix after rebase staging
 2018-10-26 10:26:22 -05:00
Rob Mulholand
3ec5b4afab Remove unused param in integration test helper 2018-10-25 10:02:06 -05:00
Rob Mulholand
8831586fba Extract shared mock converter and repository 2018-10-25 10:02:06 -05:00
Rob Mulholand
f5b9a21a8c Use transformer factory for Vat slip 2018-10-25 10:02:06 -05:00
Rob Mulholand
1e7ee08af5 Use transformer factory for Vat toll 2018-10-25 10:02:06 -05:00
Rob Mulholand
6d662a6ce7 Add test for can't create Vat tune of wrong type 2018-10-25 10:02:06 -05:00
Rob Mulholand
a0cae99437 Use transformer factory for Vat flux 2018-10-25 10:02:06 -05:00
Edvard Hübinette
be249437c2 VDB-117: Add Vow.flog transformer (#80) 
* Add DB migrations for flog

* Add Vow addresses and constants

* Add err logging to ABI signature getter, test for flog sig

* Add flog transformer

* Add tests+data for flog transformer

* Prepend DB tables and columns with vow_

* Panic on failure to get method sig
 2018-10-25 12:19:46 +02:00
Rob Mulholand
78d746bb5b Use transformer factory for Vat tune 2018-10-24 15:49:19 -05:00
Elizabeth
5682ee988b Add log index to various events (#78) 
* Add log_index field to flip kick

* Add log index to flop kick

* Add repo tests for Cat.file

* Add log_index to vat_flux

* Add log_index to vat_slip

* Add log_index to vat_toll

* Add log_index to vat_tune

* Add test to repos for updating checked headers
 2018-10-23 16:33:04 -05:00
Takayuki Goto
28f8861f81 Merge pull request #71 from 8thlight/VDB-150-vat-flux-integration-test 
integration test for vat flux
 2018-10-23 16:01:42 -05:00
Rob Mulholand
42d60c3cbd Add log index to Vat move 2018-10-23 15:25:21 -05:00
Rob Mulholand
4ac896c246 Add log index to Vat init 2018-10-23 15:25:21 -05:00
Rob Mulholand
702f90367a Add log index to Vat heal 2018-10-23 15:25:21 -05:00
Rob Mulholand
650bbf5874 Add log index to Vat grab 2018-10-23 15:25:21 -05:00
Rob Mulholand
1e74c73591 Add log index to Vat fold 2018-10-23 15:25:21 -05:00
Rob Mulholand
db52955669 Add log index to Tend 2018-10-23 15:25:21 -05:00
Rob Mulholand
e0253cce55 Add log index to Price Feed events 2018-10-23 15:25:21 -05:00
Rob Mulholand
a0a86871b5 Add log index to Frob 2018-10-23 15:25:21 -05:00
Rob Mulholand
b3d00fe3c5 Add log index to Pit file events 2018-10-23 15:25:21 -05:00
Edvard
a0deb5eb62 Add and use transformer labels in shared constants 2018-10-23 11:23:31 +02:00
Edvard
f55449d0e5 Replace GinkgoRandomSeed with proper randomisation 2018-10-23 10:47:01 +02:00
Edvard
794a7e5f99 Nuke some leftover converter interfaces 2018-10-23 10:47:01 +02:00
Edvard
6d9547a888 Fix invalidated test for tend repository 2018-10-23 10:47:01 +02:00
Edvard
7ee3329baf Privatise DB repository field and add transformer test 2018-10-23 10:47:01 +02:00
Edvard
cb74cbe990 Add cast error checking to refactored transformers 2018-10-23 10:47:01 +02:00
Edvard
b7f8432e65 Fix git whitespace warnings 2018-10-23 10:47:00 +02:00
Edvard
bb204b6b14 Remove unnecessary type washing in factory transformer 2018-10-23 10:45:48 +02:00
Edvard
e7e511764e Implement skip to next header on empty matchingLogs 2018-10-23 10:45:48 +02:00
Edvard
33f1f62c11 Implement SingleTransformerConfig 2018-10-23 10:45:48 +02:00
Edvard
7d3bf1f0f6 Fixes after rebase 2018-10-23 10:45:48 +02:00
Edvard
70722fc9e3 Add todo for changing to single topic in shared transformer config 2018-10-23 10:45:48 +02:00
Edvard
2a9fafca08 Update drip_file tests 2018-10-23 10:45:48 +02:00
Edvard
dc4eb009c1 Refactor drip_file transformers 2018-10-23 10:45:48 +02:00
Edvard
ccca6e0229 Update old pit file shared config references 2018-10-23 10:43:03 +02:00
Edvard
d9b6603109 Update pit file debt ceiling tests 2018-10-23 10:43:03 +02:00
Edvard
24199cb67e Refactor pit file debt ceiling transformer 2018-10-23 10:43:03 +02:00
Edvard
0239c4daf6 Fix sneaky missing asterisk 2018-10-23 10:43:03 +02:00
Edvard
399b3b78a0 Update tests for pit file ilk transformer 2018-10-23 10:43:03 +02:00
Edvard
98062a7970 Refactor pit file ilk transfomer 2018-10-23 10:43:03 +02:00
Edvard
837a777c8a Update transformer name for stability fee 2018-10-23 10:43:03 +02:00
Edvard
3412866182 Update tests for stability_fee 2018-10-23 10:43:03 +02:00
Edvard
2cc303a2f5 Refactor stability_fee transfomer 2018-10-23 10:43:03 +02:00
Edvard
57b3744085 Update tend tests 2018-10-23 10:43:03 +02:00
Edvard
a4df8f348d Refactor Tend transformer using factory 2018-10-23 10:43:03 +02:00
Edvard
4d36f6200d Further refactor move and init tests 2018-10-23 10:43:03 +02:00
Edvard
c4d3783f8b Expand error logging in transformer 2018-10-23 10:43:03 +02:00
Edvard
719ffd5819 Update vat init tests for factory 2018-10-23 10:43:03 +02:00
Edvard
fedbce80f6 Refactor vat init transformer to use factory 2018-10-23 10:43:03 +02:00
Edvard
177d212cc4 Add SetDB test for vat move 2018-10-23 10:43:03 +02:00
Edvard
32654ffabc Add transformer name to config for logging in factory transformer 2018-10-23 10:43:03 +02:00
Edvard
46d8ea8d97 Snag transformer topics from config in factory 2018-10-23 10:43:03 +02:00
Edvard
155569a1d8 Update vat_move tests 2018-10-23 10:43:03 +02:00
Edvard
af1dfbc489 Implement vat_move transformer using factory 2018-10-23 10:43:03 +02:00
Edvard
15e33db33d Fix transformer factory 2018-10-23 10:43:03 +02:00
Edvard
1b9e6491da Add transformer factory 2018-10-23 10:43:03 +02:00
Elizabeth
954fcf2378 integration tests (#75) 
* Deal integration test

* Add LogIndex to CatFileChopLump to update unique constraint

* Add LogIndex to cat_file_pit_vow records

* Add integration tests for Cat.file transformers

* Add log index to Cat.file flip
 2018-10-22 13:28:42 -05:00
Rob Mulholand
d87fb14c6b Add log index to Drip file events 2018-10-22 12:14:43 -05:00
Rob Mulholand
de294ad8ea Add log index to Drip drip 2018-10-22 12:14:43 -05:00
Rob Mulholand
e5ba0496a6 Add log index to Dent 2018-10-22 12:14:43 -05:00
Rob Mulholand
1aafb7201e Add log index to Deal 2018-10-22 12:14:43 -05:00
Rob Mulholand
495f39abb8 Add log index to Bite 2018-10-22 12:14:43 -05:00
Rob Mulholand
ea3de51792 Add Flop kick integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
fa828cf041 Add Tend integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
cbb5ffa3c7 Add Pit file stability fee integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
3449a1e246 Add Pit file debt ceiling integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
a83bf5ed0d Add Pit file ilk integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
677e78891d Add Drip file vow integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
0ce3e3d829 Add Bite integration test 2018-10-19 14:21:57 -05:00
Rob Mulholand
ddc7ee5143 Add Drip drip integration test 2018-10-19 10:56:28 -05:00
Rob Mulholand
b067ba565e Add Vat fold integration test 2018-10-19 10:56:28 -05:00
Rob Mulholand
fdbe0b0b44 Add Vat heal integration test 2018-10-19 10:56:28 -05:00
Rob Mulholand
a7b9b825c8 Add Vat init integration test 2018-10-19 10:56:28 -05:00
Rob Mulholand
9b427ea1e9 Add Vat move integration test 2018-10-19 10:56:28 -05:00
Taka Goto
21088eb3a2 integration test for vat flux 2018-10-17 16:25:12 -05:00
Konstantin Zolotarev
b38c29ce43 Merge pull request #64 from 8thlight/contract_addresses 
Added configuration for contract addresses
 2018-10-17 21:34:25 +03:00
Rob Mulholand
130d11d0aa Add Vat slip transformer 2018-10-17 13:30:35 -05:00
Elizabeth
3baca64903 Update address formatting (#67) 
* Add getSignatures command for list of signatures in constants

* Update VatTune converter's formatting of addresses

and added an integration test

* Move VatTune and PriceFeeds integration tests to a common suite

* Update VatGrab converter for address format

* Update frob converter to fix urn and ilk formatting/decoding

* Update formatting

* Update make test command and add integrationtest command

* Add integration test to travis build

* Update creating headers in integration tests to include Raw and Timestamp fields
 2018-10-17 13:04:55 -05:00
Taka Goto
4721a91faa use before each for transformer test 2018-10-17 12:15:15 -05:00
Taka Goto
a3ca319e9b remove unnecessary byte slicing 2018-10-17 12:04:11 -05:00
Taka Goto
2f64269ab3 update from pr feedback; update tests and use shared fake header 2018-10-17 11:48:57 -05:00
Taka Goto
f32a4fa2a5 use kovan vat contract address 2018-10-17 10:55:30 -05:00
Taka Goto
2e72a7e940 vat flux transformer 2018-10-17 10:53:53 -05:00
Rob Mulholand
d0226dbce5 Convert Bite event field types to numeric 
- Enables arithmetic operations when constructing views
- Also removes application-level references to DB ID
 2018-10-16 15:27:13 -05:00
Rob Mulholand
d3df685c46 Add timestamp to block header 
- Enables quickly deriving timestamp from header for log events
 2018-10-12 12:00:02 -05:00
Rob Mulholand
5f67161f41 Convert raw header to JSON 
- Enables parsing values off of header via Postgres migration
 2018-10-12 12:00:02 -05:00
Edvard Hübinette
be58dd4ac8 Add vat_move transformer (#47) 
* Add vat_move transformer base

* Add vat_move migrations

* Add test data for vat_move

* Add vat_move transformer to initialisers

* Add numeric cast to psql insert of Rad

* Add new db schema

* Dependency update

* Expand abbreviation in repository

* Add test suite for vat_move

* Add header checking to transformer and mock repository

* Remove trailing zero in test data

* Fix minor mishaps

* Go fmt nitpicking

* Refactoring in tests

* Add tests covering checked headers stuff (and fix revealed bugs)

* go fmt fixes

* Implement batching behaviour of transformer

* Small fixes after review

* Go fmt
 2018-10-12 16:13:13 +02:00
Konstantin Zolotarev
0b281eab47 Added default addresses to config & rename fallback var 2018-10-12 11:11:31 +03:00
Konstantin Zolotarev
245732b7f4 Added configuration for contract addresses 2018-10-11 19:10:53 +03:00
Elizabeth
6c77f369d9 Vat heal (#56) 
* VatHeal Converter

* Add VatHeal repository

* Add VatHeal transformer

* Add VatHeal to continuousLogSync command

* Mark vat_init_checked as true when creating vat init records

* Update urn and v converting

* Return error if Repository.MarkCheckedHeader fails

* Add deleting vat heal from test cleanup method
 2018-10-10 11:56:06 -05:00
David Terry
0bb3737027 vat.fold: pull common test setup into BeforeEach blocks 2018-10-10 18:13:10 +03:00
David Terry
b1745a6403 Merge branch 'staging' into VDB-101-vat-fold-transformer 
Conflicts:
	cmd/continuousLogSync.go
	db/schema.sql
	pkg/transformers/shared/constants.go
	pkg/transformers/transformers.go
	test_config/test_config.go
 2018-10-10 14:09:14 +03:00
David Terry
89ec0a1cab documentation: document config file for transformers 2018-10-10 14:06:00 +03:00
David Terry
95f1c94bf5 vat.fold: check for errors in insert before updating checked_headers 2018-10-10 14:03:37 +03:00
David Terry
1273acb733 vat.fold: add repository tests for MarkHeaderChecked 2018-10-10 13:51:44 +03:00
David Terry
3778d7ac06 vat.fold: add transformer tests for MarkHeaderChecked 2018-10-10 13:40:25 +03:00
David Terry
7e7ad13de6 vat.fold: handle errors 2018-10-10 13:39:33 +03:00
David Terry
dde57178f1 vat.fold: break out of loop in transformer if no logs found 2018-10-10 13:38:57 +03:00
David Terry
724839ce4a documentation: document vat event types 2018-10-10 13:37:59 +03:00
Rob Mulholand
e935f2f553 Add Vat grab transformer 2018-10-09 16:48:19 -05:00
Rob Mulholand
61dca314c4 Add Vat tune transformer 2018-10-09 14:37:26 -05:00
Rob Mulholand
8ee106e14d Add Vat toll transformer 2018-10-09 11:40:35 -05:00
David Terry
ecef1cbcdb transformers: add step by step docs 2018-10-09 14:25:27 +03:00
David Terry
09770b2e50 Merge branch 'staging' into VDB-101-vat-fold-transformer 2018-10-09 14:15:07 +03:00
David Terry
81dec589fc vat.fold: rollback broken test refactoring 2018-10-09 14:11:02 +03:00
David Terry
e040383fa5 vat.fold: add test for signature generation 2018-10-09 14:10:42 +03:00
David Terry
3e9901864b vat.fold: switch to batched interface 2018-10-09 13:34:22 +03:00
Taka Goto
347a339687 update frob and bite converters and repository 2018-10-07 22:02:14 -05:00
Taka Goto
7e6302c662 update bite to take in string for urn and ilk 2018-10-04 16:35:51 -05:00
David Terry
b3e2218eb5 Merge branch 'staging' into VDB-101-vat-fold-transformer 2018-10-04 18:49:38 +03:00
David Terry
53a74c39f7 Vat.fold: correct typo 2018-10-04 17:47:03 +03:00
David Terry
dde57ea70b Vat.fold: run transformer 2018-10-04 16:04:53 +03:00
David Terry
c8cb079da5 Vat.fold: add transformer & mocks 2018-10-04 14:34:30 +03:00
David Terry
46bba30938 Vat.fold: refactor repository tests 2018-10-03 19:38:47 +02:00
David Terry
ef0e3f9e11 Vat.fold: add repository & tests 2018-10-03 18:22:05 +02:00
David Terry
0c58e0ac5b Vat.fold: verify that the event signature is correct 2018-10-03 18:21:27 +02:00
David Terry
96f8279421 Vat.fold: update test data so it matches a real deployed kovan event 2018-10-03 16:03:30 +02:00
David Terry
b3295e3a86 Vat.fold: add converter & tests 2018-10-03 14:08:01 +02:00
David Terry
ce4d928fb8 transformers: tweak docs 2018-10-03 14:07:32 +02:00
David Terry
22fb019d3f docs: add links 2018-10-03 12:50:28 +02:00
David Terry
377c57ce1b transformers: format documentation 2018-10-03 12:44:05 +02:00
Rob Mulholand
bf44ec614a Log checked headers for MCD events (#43) 
* Log checked headers for Bite events

* Log checked headers for Drip drip events

* Log checked headers for Drip file events

- mark headers checked for 3 implementations of file on the Drip contract

* Log checked headers for Frob frob events

* Log checked headers for Pit file events

- mark headers checked for 3 implementations of file on the Pit contract

* Log checked headers for Vat init events
 2018-10-02 17:25:38 -05:00
David Terry
cf437e4007 VDB-101: Add signature for vat.fold 2018-09-28 16:31:29 +02:00
David Terry
f56c882b9d increase timeout on price feeds integration tests 2018-09-28 14:19:40 +02:00
Rob Mulholand
02853014b2 Fix Pit file converters 
- Use topics 2 & 3 instead of topic 1 (call data instead of msg.sender)
- Prefer extracting fields from topics rather than data
 2018-09-27 10:32:08 -05:00
Rob Mulholand
c17ad1f826 Add missing Describe statements 2018-09-27 10:32:08 -05:00
Rob Mulholand
58f17a24ae Add transformers for Cat file events 
- transform logs from 3 implementations of file on the Cat contract
 2018-09-27 10:32:08 -05:00
Elizabeth
b0cd852aa3 Flop deal (#39) 
*Refactor deal transformer to take in several contract addresses
* Add flop address to deal transformer
 2018-09-27 10:12:29 -05:00
Elizabeth
31516ea87e Flop kick (#28) 
* FlopKick transformer
 2018-09-26 09:42:52 -05:00
Rob Mulholand
a0ba6ca6bd Return error if RPC returns empty header 
- Enables graceful failure if RPC returns empty data with no error
 2018-09-25 14:11:35 -05:00
Rob Mulholand
92525ca575 Make price feed log persistence atomic per block 
- Prevents marking header as checked if only one of several price feed
  logs is successfully persisted
 2018-09-25 14:11:35 -05:00
Rob Mulholand
ac362650db Log checked headers for tend events 2018-09-25 14:11:35 -05:00
Rob Mulholand
009cbb8aa0 Log checked headers for flip kick events 2018-09-25 14:11:35 -05:00
Rob Mulholand
d8b1a61ff5 Log checked headers for dent events 2018-09-25 14:11:35 -05:00
Rob Mulholand
c3f1fcf796 Log checked headers for deal events 
- mark header as checked if no matching logs
- mark header as checked if all matching logs persisted successfully
- batch convert and insert matching logs to enable transactional persist
- enable skipping previously checked block headers when restarting a sync
 2018-09-25 14:11:35 -05:00
Elizabeth
cee824f7ad Add transformer documentation (#32) 2018-09-24 15:39:00 -05:00
Rob Mulholand
e138e58120 Set timeout before assertions in integration tests 
- Prevent early evaluation from triggering failure
 2018-09-21 11:28:17 -05:00
Rob Mulholand
56c45b6841 Add logging for every transformer 
- Also add drip file transformer initializer
 2018-09-19 11:00:20 -05:00
Rob Mulholand
a735698f77 Update config for Kovan 2018-09-19 11:00:20 -05:00
Rob Mulholand
042f1142c0 Handle headers from POA chain 
- Fetching headers from Kovan requires custom type without mixHash/Nonce
 2018-09-19 11:00:20 -05:00
Rob Mulholand
778517f33f Add integration tests for price feed transformer 
- Verify eth/usd, mkr/usd, and rep/usd price feed logs are successfully
  fetched from Kovan and persisted to Postgres
 2018-09-19 11:00:20 -05:00
Elizabeth
cfc8773c5d Continuous sync (#23) 2018-09-19 10:14:49 -05:00
Elizabeth
60901d9095 Update tend, flipkick and dent tables for consistency (#24) 2018-09-18 09:27:14 -05:00
Rob Mulholand
06881db350 Sync unchecked headers for price feeds 
- Indicate that a header has been checked for logs if no logs returned
- Indicate that a header has been checked for logs when persisting a log
- Fetch headers that need to be checked based on absence of the above
 2018-09-17 14:14:52 -05:00
Rob Mulholand
f5595bee78 Add Drip drip transformer 2018-09-13 16:35:10 -05:00
Rob Mulholand
560305f601 Update dependencies 
- uses newer version of go-ethereum required for go1.11
 2018-09-13 16:14:35 -05:00
Elizabeth
939ead0c82 Deal transformer (#19) 2018-09-13 09:06:26 -05:00
Rob Mulholand
d69063636e Add raw log to price feeds 2018-09-12 11:35:32 -05:00
Rob Mulholand
b2ba7ee1e3 Add transaction index to flip kick 2018-09-12 11:35:32 -05:00
Rob Mulholand
a843de5eb7 Add drip file transformers 
- for all three file functions on the drip contract
 2018-09-12 11:35:32 -05:00
Elizabeth
faefd620b1 Get method/event signature from ABI (#17) 2018-09-12 08:59:43 -05:00
Rob Mulholand
4b6ef1e58b Verify pit file logs 
- assure required topics + data available before parsing
 2018-09-11 16:30:29 -05:00
Rob Mulholand
dc9bda7d68 Extract addresses to shared constants file 2018-09-11 16:30:29 -05:00
Rob Mulholand
ceeaa0f8d3 Update licenses and imports 
- Verify license added to each file
- Order imports by group and separate with line breaks
 2018-09-11 16:30:29 -05:00
Rob Mulholand
ff110592bd Add Vat init transformer 2018-09-11 16:30:29 -05:00
Elizabeth
9abe3ffa68 Dent (#10) 
Convert and persist LogNote events as dent records
 2018-09-11 14:01:45 -05:00
Rob Mulholand
d48cf38070 Add Pit file debt ceiling event 2018-09-05 13:51:18 -05:00
Rob Mulholand
e1f609528d Add Pit file stability fee event 
- file event on the pit contract is overloaded and each implementation
  logs a note
- existing Pit file transformer now specified as dealing with ilks
 2018-09-05 11:36:13 -05:00
Rob Mulholand
a30e5036fd generate log event signatures from functions 2018-09-04 16:19:03 -05:00
Rob Mulholand
8f5bc0a956 Add raw log to Frob record in DB 
- Enables parsing additional values off the log if necessary
- Also update Frob to reflect DSS changes (lad => urn)
- Move Frob constants to shared directory
 2018-09-04 16:11:54 -05:00
Rob Mulholand
db4b891398 Backfill Pit.file log events 2018-09-04 16:07:35 -05:00
Elizabeth
53635ec362 Update flip kick and tend transformers due to contract changes 
* Update FlipperAddress to new local ganache address

* Update flip_kick table

* Update flipkick transformer to handle new signature and abi

* Update tend table

* Update tend converter
 2018-09-04 15:50:29 -05:00
Takayuki Goto
807b7accd6 Merge pull request #12 from 8thlight/bite-event 
bite event transformer
 2018-09-04 09:58:51 -05:00
Taka Goto
415ec3d97c update from PR feedback 2018-08-31 10:37:02 -05:00
Taka Goto
adc7c47244 update tests and import order per PR feedback 2018-08-30 15:06:14 -05:00
Taka Goto
4ca7c10f92 fix compile error 2018-08-30 14:27:28 -05:00
Taka Goto
985fa49178 migration file to create bite; create bite repository 
add transaction index and raw log to bite table
work on converter for bite event
update bite repository, replace guy with 32byte lad; create bite converter to entity
update field type for bite event; start on bite transformer
finish bite event transformer
 2018-08-30 08:38:43 -05:00
Rob Mulholand
1916d585fa Incorporate DSS updates (#8) 
* Incorporate DSS updates
- Lad renamed to Guy
- Dink and Dart added to Frob
* update test chain data
* Remove Mom field from FlipKick
* Update Flip ABI and sample flip kick data
* Incorporate updates to Frob event
- Guy renamed to Lad
- Era and Gem removed, iArt added
- Also turn off integration tests that read from test chain while
events are actively under development
 2018-08-28 14:16:39 -05:00
Elizabeth
66ad7e3021 Transform and persist Tend log events 2018-08-16 16:36:35 -05:00
Rob Mulholand
634604d0b5 Combine price feed transformers 
- fetches logs from all three price feeds in one query
- assumes eth/usd price feed will be updated to include LogValue event
- updates transformers to run separate from header sync
 2018-08-16 11:22:16 -05:00
Rob Mulholand
9231d40369 Extract price feed contract addresses to config 2018-08-16 11:22:16 -05:00
Rob Mulholand
72a849a272 Update price feeds if header changed by validator 
- currently not validating price feeds if the underlying header already exists
  and is valid, since price feeds should have been added when initial header
  was added
 2018-08-16 11:22:16 -05:00
Rob Mulholand
6e68dc4a92 Add Rep price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand
2949996d22 Add Pip price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand
0551147453 Add Pep price feed 2018-08-16 11:22:16 -05:00
Rob Mulholand
9f7a776d70 update chain data for frob event 2018-08-14 10:47:43 -05:00
Rob Mulholand
ac63e43543 Backfill Frob log events 2018-08-14 10:47:43 -05:00
Rob Mulholand
55229cd2eb Refactorings to get tests passing with local Ganache setup 
- removes assertions against data with timestamps to facilitate running
the tests against a freshly setup local Ganache instance
- also applies a few `go vet` and `go fmt` changes
 2018-08-09 16:55:44 -05:00
Elizabeth
b37324bf3d Refactoring 
* Move flip kick files pkg/transformers/flip_kick

* Consolidate test database setup

* Pull ganache ipcPath from config

* Update README to include info about using a Ganache chain
 2018-08-07 15:17:29 -05:00
Elizabeth
c617cd9c9d Add FlipKick log events transformer 2018-08-07 10:51:34 -05:00
Rob Mulholand
a683e45855 Consolidate test doubles 
- Migrate various mocks of core namespaces to shared version in `fakes` pkg
- Err on the side of making test doubles less sophisticated
- Don't pull over mocks of namespaces that are only used in example code
 2018-07-20 11:37:46 -05:00
Rob Mulholand
63434f6bc9 Add tests for pkg/geth/blockchain 
- inject dependencies instead of initializing them in the constructor
 2018-07-18 16:34:13 -05:00