cerc-io/ipld-eth-server
18
0
Fork 1
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 37 Wiki Activity
1,271 Commits 12 Branches 52 Tags 80 MiB
79c3078c36
Commit Graph

260 Commits

Author SHA1 Message Date
Ian Norden
b5099a5051 demo commands 2019-12-02 13:24:51 -06:00
Ian Norden
2db0ce971d vulcanizedb seed node streamer and syncPublishScreenAndServe command 2019-12-02 13:24:51 -06:00
Ian Norden
34393ffb3f fixes for issues uncovered in integration 2019-12-02 13:24:51 -06:00
Ian Norden
5ebe2243d8 goimports -w 2019-12-02 13:24:50 -06:00
Ian Norden
15e044403d work on mocks and unit tests 2019-12-02 13:24:50 -06:00
Ian Norden
d702cb720c update dependencies to work with update eth-block-extractor 2019-12-02 13:24:49 -06:00
Ian Norden
31a9017c4f index all cids in a payload in a single atomic tx; misc fixes; comment additions 2019-12-02 13:24:46 -06:00
Ian Norden
79efaeb089 syncAndPublish command 2019-12-02 13:24:46 -06:00
Ian Norden
6434a7279d integrate backfill into storage watcher; documentation for storage backfill 2019-12-02 13:24:45 -06:00
Ian Norden
db0f024088 review fixes 2019-12-02 11:26:44 -06:00
Ian Norden
a834e55b9f command edits; external pkg for finding min deployment block; gofmt 2019-12-02 11:22:18 -06:00
Ian Norden
37f4a2d603 integrate backfill into storage watcher; documentation for storage backfill 2019-12-02 11:06:28 -06:00
Ian Norden
a59bd06a37 goimports -w; comments; refactor test 2019-12-02 11:03:36 -06:00
Rob Mulholand
e1236b4072 Prefer all caps for initialisms and acronyms 2019-10-31 13:42:19 -05:00
Edvard Hübinette
3fff2896aa
Rename geth to eth, signifying client independence (#161) 2019-10-28 12:30:24 +01:00
Elizabeth Engelman
f0d2741dea Allow storageDiff source to be set through an env variable 2019-10-01 10:24:53 -05:00
Rob Mulholand
d06dddbfaa Always hash storage diff contract addresses 
- Enables syncing Geth and Parity diffs with same transformer lookup
- Maybe worth always hashing the storage key so we don't need a hashed
  and not-hashed version in the key lookups?
 2019-09-25 16:36:08 -05:00
Elizabeth Engelman
639c561fdf Remove unncessary pointer receiver from NewGethRpcStorageFetcher.FetchStorageDiffs 2019-09-25 16:32:31 -05:00
Elizabeth Engelman
0cc90a1f80 Factor out an IStorageWatcher interface 2019-09-25 16:32:31 -05:00
Elizabeth Engelman
b09b8a8735 Use geth state diff source in execute command 2019-09-25 16:32:27 -05:00
Elizabeth Engelman
a577811e0a Use geth state diff source in composeAndExecute 2019-09-25 16:32:27 -05:00
Matt Griswold
20ce0ab852 Revert "option to write the plugin .go file and run the plugin migrations without building the .so file so that it can be done from the shell instead to afford complete error messages" 
This reverts commit 9bb2f27a69.
 2019-09-24 23:54:25 +00:00
Ian Norden
9bb2f27a69 option to write the plugin .go file and run the plugin migrations without building the .so file so that it can be done from the shell instead to afford complete error messages 2019-09-24 18:44:04 -05:00
Rob Mulholand
4fa19be90a Return error when no logs/headers available 
- Replaces bool and moots question of error/bool ordering
- Also make event watcher execution synchronous
 2019-09-18 20:55:15 -05:00
Rob Mulholand
13d503b851 Distinguish between missing and unchecked headers 
- Missing == not in DB
- Unchecked == logs haven't been fetched
 2019-09-10 14:39:45 -05:00
Rob Mulholand
666ea1c325 Update checked headers for new transformers 
- If a header was marked as checked before a transformer was added to
  the watcher, mark all headers since the new transformer's starting
  block number as unchecked.
 2019-08-28 09:25:14 -05:00
Rob Mulholand
63dabbb051 Extract and delegate logs concurrently 2019-08-28 09:25:13 -05:00
Elizabeth Engelman
1b3786338f Address PR feedback 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
58c23c6632 Update Copyrights 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
7e38764618 Rename ReceiptRepository -> FullSyncReceiptRepository 2019-08-23 10:11:37 -05:00
Andrew J Yao
ee77fc6521 Remove injection of SubCommand for logs 2019-07-23 13:11:20 -07:00
Andrew J Yao
a188e1dd79 Incorporate changes from Edvard PR 2019-07-23 08:10:58 -07:00
Andrew J Yao
92d153b010 Add subCommand name to log fields 2019-07-23 08:09:57 -07:00
Andrew J Yao
d7a82e353b Add calling method to debug level log messages 2019-07-23 08:08:15 -07:00
Edvard Hübinette
2c092e8d04
[VDB-751 VDB-754] Bugfix null pointer panic and improve logging (#119) 
* VDB-751 VDB-754 Bugfix null pointer panic and improve logging

* Fix typo
 2019-07-18 09:21:40 +02:00
Rob Mulholand
62cca62432 Log error on failure to read config file 
- include actual error to highlight the specific issue
 2019-06-27 13:58:58 -05:00
Andrew J Yao
11a65c525c Adds ability to set log level from toml or CLI args 2019-06-27 11:14:11 -07:00
Rob Mulholand
e2909797fc Remove handling of duplicate storage diffs in watcher 
- Can push this responsibility down to the transformers
- Update docs to reflect that transformers should handle duplicates
 2019-05-20 13:29:09 -05:00
Elizabeth Engelman
d947c8f30a Rename sync command to fullSync 2019-05-13 16:14:56 -05:00
Elizabeth Engelman
f7d520c933 Update header sync transformer alias in ContractWatcher 2019-05-08 13:42:43 -05:00
Rob Mulholand
36c4da372b (VDB-570) Handle duplicate storage diffs 
- If processing a new diff for a row that already exists in the DB,
  ignore the error without logging or queueing
- If processing a queued diff for a row that already exists, remove
  it from the queue
 2019-05-02 11:39:16 -05:00
Rob Mulholand
6716c3b92a Make queue recheck interval configurable via CLI 2019-05-01 12:32:39 -05:00
Rob Mulholand
6a86de87b4 (VDB-371) Recheck queued storage 
- Iterate through queued storage at defined interval, popping rows
  from the queue if successfully persisted
 2019-05-01 12:30:37 -05:00
Rob Mulholand
2d684c5aec Extract storage diff fetching behind an interface 
- Replaces directly reading from a CSV
- Simplifies testing
- Should hopefully make it easier to plug in other sources for storage
  diffs (e.g. differently formatted CSVs, JSON RPC, etc)
 2019-05-01 12:30:36 -05:00
Gabe Laughlin
a344432156
(VDB-560) Rename lightSync to headerSync 2019-05-01 12:12:55 -05:00
Ian Norden
1aa849bcb4 PR fixes; remove all infura token references and setup travis to use encrypted env 
variable; rest of the ethjson_rpc dependent tests extracted to integration_test
 2019-03-21 18:36:51 -05:00
Ian Norden
d3e172d9ab remove references to infura token except for in infura.toml so that tests will continue to pass on CI 2019-03-21 18:36:49 -05:00
Ian Norden
37fc038605 fixes for review comments 2019-03-21 18:36:49 -05:00
Ian Norden
24879a85aa mv pkg/omni pkg/contract_watcher 2019-03-21 18:33:56 -05:00
Ian Norden
073378de6d update © to 2019 2019-03-21 18:33:56 -05:00
Ian Norden
e4e092f542 changes to plugin and commands to accomodate changes 2019-03-21 18:33:56 -05:00
Ian Norden
8174ce4aee refactor omni code; rename omniWatcher => contractWatcher; use config instead of excessive number of CLI flags 2019-03-21 18:30:48 -05:00
Ian Norden
ff55e3ba7a TransformerInitializer => EventTransformerInitializer reorganization 2019-03-14 11:59:39 -05:00
Ian Norden
9f8c50e3ab tests for migration path ordering and errors 2019-03-13 11:42:30 -05:00
Ian Norden
05cb06c2d3 config example fixes 2019-03-13 11:42:29 -05:00
Ian Norden
b3ce1ecd78 update readme and cmd usage comments 2019-03-13 11:42:29 -05:00
Ian Norden
84aa0a7eba plugin migration order specified in config 2019-03-13 11:42:29 -05:00
Rob Mulholand
5eff2618ed Assorted cleanups 
- Remove DataDog
- Remove token_supply table
- Drop from more tables when cleaning DB for tests
 2019-03-08 11:35:10 -06:00
Ian Norden
8bd397e2eb Merge pull request #29 from vulcanize/fix 
Fix for Travis issue at https://github.com/vulcanize/mcd_transformers/pull/6
 2019-03-06 11:27:43 -06:00
Ian Norden
5fd9a59027 fix for issue with Travis building plugin from plugin repo 2019-03-05 08:57:54 -06:00
Edvard Hübinette
6ff67f26d1 Fix logrus append (#24) 2019-03-05 11:30:22 +01:00
Ian Norden
83f7daf37d compose and execute as separate commands; update README and add git checkout vendor/github.com/ethereum/go-ethereum/accounts/abi to make build 2019-02-28 18:15:16 -06:00
Ian Norden
2c5ddd03ba core repo migrations in fixed form 2019-02-26 00:52:54 -06:00
Ian Norden
17f4d3dfa5 move maker plugin generator test to mcd_transformers 2019-02-24 21:53:04 -06:00
Ian Norden
f2072561a7 removing mcd/maker references; delete test_data; update READMEs; use 
logrus
 2019-02-24 21:53:04 -06:00
Ian Norden
3d34a9e7c9 remove maker migrations and convert back to timestamps and fix bug in 
composeAndExecute command
 2019-02-24 21:52:23 -06:00
Ian Norden
b449193b16 addressing review comments; still need to reorg migrations and add helper tests 2019-02-24 21:38:48 -06:00
Ian Norden
2868cf2f73 goose changes; update plugin readme; use rel paths for transformers in config; more commenting in pkg/plugin; fix PluginWriter to work with different transformer types; reopen generator tests 2019-02-24 21:38:48 -06:00
Ian Norden
90e67d6da9 move storage mappings pkg into shared 2019-02-24 21:38:48 -06:00
Ian Norden
03a7379617 work on porting storage watcher; watcher type is defined for 
transformers in the config, storage transformers are exported from
plugin like event transformers
 2019-02-24 21:38:47 -06:00
Ian Norden
a322372713 refactoring plugin generating code 2019-02-24 21:38:47 -06:00
Ian Norden
decc2a3caf remove pkg/transformers; extract shared files needed to 
libraries/shared; work on automated db migration management
 2019-02-24 21:38:47 -06:00
Ian Norden
55fa9b8364 auto management of dependencies temporarily needed to build plugin and transformer 
migrations; new test showing it is working with external transformers runs
properly when migrations were manually performed but still need
to test automated migration management
 2019-02-24 15:23:35 -06:00
Ian Norden
6c2d895023 composeAndExecute command that loads and executes over arbitrary transformer set exported from a go plugin generated according to config file; test for plugin generation, loading, and execution; work on plugins README 2019-02-24 15:23:35 -06:00
Edvard
cd6611d2ec Add Cat storage transformer 2019-02-20 14:37:46 +01:00
Edvard Hübinette
53789c82ab Merge pull request #12 from vulcanize/VDB-392-fix-io-error-propagation 
VDB-392 Fix IO error propagation
 2019-02-20 13:58:00 +01:00
Edvard
6cd4e5ea95 Improve I/O error propagation 2019-02-14 16:03:57 +01:00
Taka Goto
c2c21325b5 vow contract storage diffing excluding sin 2019-02-11 15:28:54 -06:00
Edvard Hübinette
f73623789e Merge pull request #6 from 8thlight/VDB-337-Dockerize-vDB 
VDB-337 Dockerize vDB
 2019-02-11 12:37:56 +01:00
Edvard
0bf8bd4183 Add char replacer when searching for env variables 2019-02-11 11:27:33 +01:00
Edvard
e4968c4a75 Add back override with env variables 2019-02-11 11:26:12 +01:00
Rob Mulholand
c05d3b7836 (VDB-280) Consume vat contract storage diffs 2019-02-08 13:36:30 -06:00
Takayuki Goto
0a024d429d * update transformer to able to recheck headers (#4) 
* update transformer to able to recheck headers
* put cap on rechecking header
* integration test for recheck headers
* use enum for recheck headers param; make recheck cap configurable
* update integration tests with new test config
* update omni pkg with new recheck header column type
* update migration with new migration tool and final tweaks needed to accommodate changes in omni pkg
 2019-02-08 10:35:46 -06:00
Rob Mulholand
867f92c431 (VDB-298) Consume Pit contract storage diffs 
- Continuously parse storage diffs CSV data to read Pit contract state
- Convert ilks in database to raw bytes32 value for use in generating
  storage keys dynamically
- Persist storage diffs with block number and hash for validation
 2019-02-06 10:31:46 -06:00
Edvard Hübinette
29f2537130 Merge pull request #7 from vulcanize/VDB-347-Squelch-lightSync 
VDB-347 Squelch lightSync in stdout
 2019-02-01 11:02:48 +01:00
Edvard
8eea1b00ed Redirect validation window status to logrus instead of stdout 2019-01-31 15:55:28 +01:00
Ian Norden
8749c94994 update license to AGPL-3.0 2019-01-24 14:41:30 -06:00
Ian Norden
1f95eb6443 Merge branch 'staging' into pub_merge 2019-01-24 09:47:12 -06:00
Ian Norden
dc87f39889 change omniwatcher database test helpers so that it doesn't interfere with downstream maker tests 2019-01-23 14:55:20 -06:00
Konstantin Zolotarev
65cb6e9092 VDB-205 Fix external configuration values (#121) 
* Updated loading config from /Users/konstantinzolotarev/.vulcanizedb for addreses

* Updated contract addresses loading from env variables.

* Implement getter chain for addresses, configs and transformers

Solves the issue with declaring stuff as vars when the environment file hasn't been loaded yet. Instead, this data is fetched dynamically when the root commands are called.

* Update test data to use constant Kovan addresses

* Decouple integration tests from config file IO

* Add ABI to environment file, convert getter chains for related values

* Decouple tests suites from ABI config file IO

* Add startingBlock to environment and implement getter chain

* Exit when called without config file

* Add missing deployment blocks to staging.toml

* Remove comment
 2019-01-23 19:44:09 +01:00
Ian Norden
46b898f690 merge public vulcanizedb/master 2019-01-23 00:37:26 -06:00
Edvard Hübinette
ebca338b1e VDB-302 Sleep when no missing blocks in lightSync (#129) 
* Sleep when no missing blocks in lightSync

* Fix tests and error propagation

* Correct geth.log to logrus
 2019-01-11 10:58:41 +01:00
Ian Norden
59cdaa05e6 moving some memory allocation to init function rather than within 
execution cycles and fixing test db tear down so that it drops
checked_headers table instead of individual columns (droping columns
only hides them, they remain in memory and continue to contribute to the
1600 column limit which causes us to eventually exceed it)
 2019-01-07 13:07:31 -06:00
Edvard
833dde62cf Remove injection of fetcher and repository to watcher 2019-01-07 15:19:31 +01:00
Ian Norden
e390a97502 finish method polling with hash or address type argument values 
collected from watched events; 'MissingMethodsCheckedEventsIntersection'
method to find headers which have been checked for each event of
interest but methods have not yet been polled at that header; tests for
new features; travis ci go version 1.9 -> 1.11 ; consolidate omniWatcher and lightOmniWatcher into single command with light as default
 2018-12-21 10:33:31 -06:00
Ian Norden
0a59f06cac begin work on: Add checked_headers column for methods that are polled so taht we don’t duplicate; Add batching of method polling so that we arent generating a rediculously large account address list before using it to poll methods (or persist the list in pg?); User passed ABI and other ways to get ABI; Add ability to collect []byte and hashes from events and use them in method polling same manner as addresses; Event filter addrs => only those event’s addresses/hashes are used for polling; Option to persist seen address/hash/bytes lists into pg; Only generate lists of addresses, []byte, or hashes if a method will use them later 2018-12-21 10:33:31 -06:00
Edvard
a3737c46f6 Remove duplicate entry in initializer map 2018-12-13 13:16:49 +01:00
Edvard
d30fcfed80 Use new config getter on shared.Transformer <3 2018-12-13 12:39:57 +01:00
Edvard
dc43547612 Improve process of adding transformers 2018-12-12 15:41:29 +01:00
Edvard
4e089c363e Fixes after merging staging 2018-12-11 15:19:27 +01:00