yaoandrew
5a1f599a4d
Clean up files after rebase
2019-12-02 23:36:00 -06:00
Andrew J Yao
e26fa12af0
Adds logging for transformer start, log conversions, log persistance
2019-12-02 22:59:32 -06:00
Ian Norden
4036d9d6a0
review fixes
2019-12-02 13:49:27 -06:00
Ian Norden
5be205ffa6
super node backfill breaks batch call into smaller bins; retrieve gap
...
test
2019-12-02 13:24:58 -06:00
Ian Norden
c16ac026db
logWithCommand; rebase fixes; config for testing super node subscription
2019-12-02 13:24:58 -06:00
Ian Norden
67df8dea77
header_cids.final => header_cids.uncle
2019-12-02 13:24:58 -06:00
Ian Norden
6880611436
command for serving data without an ongoing sync process
2019-12-02 13:24:58 -06:00
Ian Norden
3a666df294
backfiller refactoring; explicity errs; golint
2019-12-02 13:24:58 -06:00
Ian Norden
40c3aff597
seed => super; port 80 => port 8080; backfill process for the super_node
2019-12-02 13:24:58 -06:00
Ian Norden
b789ab69e6
adjustments after rebase
2019-12-02 13:24:58 -06:00
Ian Norden
8fe273fb7b
getter rpc method for the seed node's geth info
2019-12-02 13:24:58 -06:00
Ian Norden
8b31d12716
fixes after rebase
2019-12-02 13:24:58 -06:00
Ian Norden
35c8f3561a
publisher unit test
2019-12-02 13:24:58 -06:00
Ian Norden
0bbb7a30d1
review fixes
2019-12-02 13:24:58 -06:00
Ian Norden
f2efbb5d01
worker pool for handling concurrent ipld putting and cid indexing without overloading Postgres connections
2019-12-02 13:24:58 -06:00
Ian Norden
3fa33fb767
index receipts by the contract address
2019-12-02 13:24:58 -06:00
Ian Norden
23a21c14f3
make subscription config rlp encodable, group subs of the
...
same type using a hash of their config, process only once for each sub
type instead of for every sub.
2019-12-02 13:24:58 -06:00
Ian Norden
b76bdc51f2
fixing dockerfile and start up script
2019-12-02 13:24:54 -06:00
Ian Norden
5356cd50bb
seed node documentation
2019-12-02 13:24:51 -06:00
Ian Norden
e7cdd6247e
update to use ReceiptForStorage; expose rpc server over ws
2019-12-02 13:24:51 -06:00
Ian Norden
723c7c6244
fix backfill operations and dependency issue; hopefully travis will work now
2019-12-02 13:24:51 -06:00
Ian Norden
4baea2923c
make db fks deferrable so that we can commit entire cid payload in single transaction; adjust buffer sizes to optimize performane and stability
2019-12-02 13:24:51 -06:00
Ian Norden
b1bb646ad5
goimports + streamSubscribe command for raw access to the seed node data
2019-12-02 13:24:51 -06:00
Ian Norden
8ccdfd4835
fix streamFilters issue
2019-12-02 13:24:51 -06:00
Ian Norden
b5099a5051
demo commands
2019-12-02 13:24:51 -06:00
Ian Norden
2db0ce971d
vulcanizedb seed node streamer and syncPublishScreenAndServe command
2019-12-02 13:24:51 -06:00
Ian Norden
34393ffb3f
fixes for issues uncovered in integration
2019-12-02 13:24:51 -06:00
Ian Norden
5ebe2243d8
goimports -w
2019-12-02 13:24:50 -06:00
Ian Norden
15e044403d
work on mocks and unit tests
2019-12-02 13:24:50 -06:00
Ian Norden
d702cb720c
update dependencies to work with update eth-block-extractor
2019-12-02 13:24:49 -06:00
Ian Norden
31a9017c4f
index all cids in a payload in a single atomic tx; misc fixes; comment additions
2019-12-02 13:24:46 -06:00
Ian Norden
79efaeb089
syncAndPublish command
2019-12-02 13:24:46 -06:00
Ian Norden
6434a7279d
integrate backfill into storage watcher; documentation for storage backfill
2019-12-02 13:24:45 -06:00
Ian Norden
db0f024088
review fixes
2019-12-02 11:26:44 -06:00
Ian Norden
a834e55b9f
command edits; external pkg for finding min deployment block; gofmt
2019-12-02 11:22:18 -06:00
Ian Norden
37f4a2d603
integrate backfill into storage watcher; documentation for storage backfill
2019-12-02 11:06:28 -06:00
Ian Norden
a59bd06a37
goimports -w; comments; refactor test
2019-12-02 11:03:36 -06:00
Rob Mulholand
e1236b4072
Prefer all caps for initialisms and acronyms
2019-10-31 13:42:19 -05:00
Edvard Hübinette
3fff2896aa
Rename geth to eth, signifying client independence ( #161 )
2019-10-28 12:30:24 +01:00
Elizabeth Engelman
f0d2741dea
Allow storageDiff source to be set through an env variable
2019-10-01 10:24:53 -05:00
Rob Mulholand
d06dddbfaa
Always hash storage diff contract addresses
...
- Enables syncing Geth and Parity diffs with same transformer lookup
- Maybe worth always hashing the storage key so we don't need a hashed
and not-hashed version in the key lookups?
2019-09-25 16:36:08 -05:00
Elizabeth Engelman
639c561fdf
Remove unncessary pointer receiver from NewGethRpcStorageFetcher.FetchStorageDiffs
2019-09-25 16:32:31 -05:00
Elizabeth Engelman
0cc90a1f80
Factor out an IStorageWatcher interface
2019-09-25 16:32:31 -05:00
Elizabeth Engelman
b09b8a8735
Use geth state diff source in execute command
2019-09-25 16:32:27 -05:00
Elizabeth Engelman
a577811e0a
Use geth state diff source in composeAndExecute
2019-09-25 16:32:27 -05:00
20ce0ab852
Revert "option to write the plugin .go file and run the plugin migrations without building the .so file so that it can be done from the shell instead to afford complete error messages"
...
This reverts commit 9bb2f27a69
.
2019-09-24 23:54:25 +00:00
Ian Norden
9bb2f27a69
option to write the plugin .go file and run the plugin migrations without building the .so file so that it can be done from the shell instead to afford complete error messages
2019-09-24 18:44:04 -05:00
Rob Mulholand
4fa19be90a
Return error when no logs/headers available
...
- Replaces bool and moots question of error/bool ordering
- Also make event watcher execution synchronous
2019-09-18 20:55:15 -05:00
Rob Mulholand
13d503b851
Distinguish between missing and unchecked headers
...
- Missing == not in DB
- Unchecked == logs haven't been fetched
2019-09-10 14:39:45 -05:00
Rob Mulholand
666ea1c325
Update checked headers for new transformers
...
- If a header was marked as checked before a transformer was added to
the watcher, mark all headers since the new transformer's starting
block number as unchecked.
2019-08-28 09:25:14 -05:00
Rob Mulholand
63dabbb051
Extract and delegate logs concurrently
2019-08-28 09:25:13 -05:00
Elizabeth Engelman
1b3786338f
Address PR feedback
2019-08-23 10:11:37 -05:00
Elizabeth Engelman
58c23c6632
Update Copyrights
2019-08-23 10:11:37 -05:00
Elizabeth Engelman
7e38764618
Rename ReceiptRepository -> FullSyncReceiptRepository
2019-08-23 10:11:37 -05:00
Andrew J Yao
ee77fc6521
Remove injection of SubCommand for logs
2019-07-23 13:11:20 -07:00
Andrew J Yao
a188e1dd79
Incorporate changes from Edvard PR
2019-07-23 08:10:58 -07:00
Andrew J Yao
92d153b010
Add subCommand name to log fields
2019-07-23 08:09:57 -07:00
Andrew J Yao
d7a82e353b
Add calling method to debug level log messages
2019-07-23 08:08:15 -07:00
Edvard Hübinette
2c092e8d04
[VDB-751 VDB-754] Bugfix null pointer panic and improve logging ( #119 )
...
* VDB-751 VDB-754 Bugfix null pointer panic and improve logging
* Fix typo
2019-07-18 09:21:40 +02:00
Rob Mulholand
62cca62432
Log error on failure to read config file
...
- include actual error to highlight the specific issue
2019-06-27 13:58:58 -05:00
Andrew J Yao
11a65c525c
Adds ability to set log level from toml or CLI args
2019-06-27 11:14:11 -07:00
Rob Mulholand
e2909797fc
Remove handling of duplicate storage diffs in watcher
...
- Can push this responsibility down to the transformers
- Update docs to reflect that transformers should handle duplicates
2019-05-20 13:29:09 -05:00
Elizabeth Engelman
d947c8f30a
Rename sync command to fullSync
2019-05-13 16:14:56 -05:00
Elizabeth Engelman
f7d520c933
Update header sync transformer alias in ContractWatcher
2019-05-08 13:42:43 -05:00
Rob Mulholand
36c4da372b
(VDB-570) Handle duplicate storage diffs
...
- If processing a new diff for a row that already exists in the DB,
ignore the error without logging or queueing
- If processing a queued diff for a row that already exists, remove
it from the queue
2019-05-02 11:39:16 -05:00
Rob Mulholand
6716c3b92a
Make queue recheck interval configurable via CLI
2019-05-01 12:32:39 -05:00
Rob Mulholand
6a86de87b4
(VDB-371) Recheck queued storage
...
- Iterate through queued storage at defined interval, popping rows
from the queue if successfully persisted
2019-05-01 12:30:37 -05:00
Rob Mulholand
2d684c5aec
Extract storage diff fetching behind an interface
...
- Replaces directly reading from a CSV
- Simplifies testing
- Should hopefully make it easier to plug in other sources for storage
diffs (e.g. differently formatted CSVs, JSON RPC, etc)
2019-05-01 12:30:36 -05:00
Gabe Laughlin
a344432156
(VDB-560) Rename lightSync to headerSync
2019-05-01 12:12:55 -05:00
Ian Norden
1aa849bcb4
PR fixes; remove all infura token references and setup travis to use encrypted env
...
variable; rest of the ethjson_rpc dependent tests extracted to integration_test
2019-03-21 18:36:51 -05:00
Ian Norden
d3e172d9ab
remove references to infura token except for in infura.toml so that tests will continue to pass on CI
2019-03-21 18:36:49 -05:00
Ian Norden
37fc038605
fixes for review comments
2019-03-21 18:36:49 -05:00
Ian Norden
24879a85aa
mv pkg/omni pkg/contract_watcher
2019-03-21 18:33:56 -05:00
Ian Norden
073378de6d
update © to 2019
2019-03-21 18:33:56 -05:00
Ian Norden
e4e092f542
changes to plugin and commands to accomodate changes
2019-03-21 18:33:56 -05:00
Ian Norden
8174ce4aee
refactor omni code; rename omniWatcher => contractWatcher; use config instead of excessive number of CLI flags
2019-03-21 18:30:48 -05:00
Ian Norden
ff55e3ba7a
TransformerInitializer => EventTransformerInitializer reorganization
2019-03-14 11:59:39 -05:00
Ian Norden
9f8c50e3ab
tests for migration path ordering and errors
2019-03-13 11:42:30 -05:00
Ian Norden
05cb06c2d3
config example fixes
2019-03-13 11:42:29 -05:00
Ian Norden
b3ce1ecd78
update readme and cmd usage comments
2019-03-13 11:42:29 -05:00
Ian Norden
84aa0a7eba
plugin migration order specified in config
2019-03-13 11:42:29 -05:00
Rob Mulholand
5eff2618ed
Assorted cleanups
...
- Remove DataDog
- Remove token_supply table
- Drop from more tables when cleaning DB for tests
2019-03-08 11:35:10 -06:00
Ian Norden
8bd397e2eb
Merge pull request #29 from vulcanize/fix
...
Fix for Travis issue at https://github.com/vulcanize/mcd_transformers/pull/6
2019-03-06 11:27:43 -06:00
Ian Norden
5fd9a59027
fix for issue with Travis building plugin from plugin repo
2019-03-05 08:57:54 -06:00
Edvard Hübinette
6ff67f26d1
Fix logrus append ( #24 )
2019-03-05 11:30:22 +01:00
Ian Norden
83f7daf37d
compose and execute as separate commands; update README and add git checkout vendor/github.com/ethereum/go-ethereum/accounts/abi to make build
2019-02-28 18:15:16 -06:00
Ian Norden
2c5ddd03ba
core repo migrations in fixed form
2019-02-26 00:52:54 -06:00
Ian Norden
17f4d3dfa5
move maker plugin generator test to mcd_transformers
2019-02-24 21:53:04 -06:00
Ian Norden
f2072561a7
removing mcd/maker references; delete test_data; update READMEs; use
...
logrus
2019-02-24 21:53:04 -06:00
Ian Norden
3d34a9e7c9
remove maker migrations and convert back to timestamps and fix bug in
...
composeAndExecute command
2019-02-24 21:52:23 -06:00
Ian Norden
b449193b16
addressing review comments; still need to reorg migrations and add helper tests
2019-02-24 21:38:48 -06:00
Ian Norden
2868cf2f73
goose changes; update plugin readme; use rel paths for transformers in config; more commenting in pkg/plugin; fix PluginWriter to work with different transformer types; reopen generator tests
2019-02-24 21:38:48 -06:00
Ian Norden
90e67d6da9
move storage mappings pkg into shared
2019-02-24 21:38:48 -06:00
Ian Norden
03a7379617
work on porting storage watcher; watcher type is defined for
...
transformers in the config, storage transformers are exported from
plugin like event transformers
2019-02-24 21:38:47 -06:00
Ian Norden
a322372713
refactoring plugin generating code
2019-02-24 21:38:47 -06:00
Ian Norden
decc2a3caf
remove pkg/transformers; extract shared files needed to
...
libraries/shared; work on automated db migration management
2019-02-24 21:38:47 -06:00
Ian Norden
55fa9b8364
auto management of dependencies temporarily needed to build plugin and transformer
...
migrations; new test showing it is working with external transformers runs
properly when migrations were manually performed but still need
to test automated migration management
2019-02-24 15:23:35 -06:00
Ian Norden
6c2d895023
composeAndExecute command that loads and executes over arbitrary transformer set exported from a go plugin generated according to config file; test for plugin generation, loading, and execution; work on plugins README
2019-02-24 15:23:35 -06:00
Edvard
cd6611d2ec
Add Cat storage transformer
2019-02-20 14:37:46 +01:00
Edvard Hübinette
53789c82ab
Merge pull request #12 from vulcanize/VDB-392-fix-io-error-propagation
...
VDB-392 Fix IO error propagation
2019-02-20 13:58:00 +01:00
Edvard
6cd4e5ea95
Improve I/O error propagation
2019-02-14 16:03:57 +01:00
Taka Goto
c2c21325b5
vow contract storage diffing excluding sin
2019-02-11 15:28:54 -06:00
Edvard Hübinette
f73623789e
Merge pull request #6 from 8thlight/VDB-337-Dockerize-vDB
...
VDB-337 Dockerize vDB
2019-02-11 12:37:56 +01:00
Edvard
0bf8bd4183
Add char replacer when searching for env variables
2019-02-11 11:27:33 +01:00
Edvard
e4968c4a75
Add back override with env variables
2019-02-11 11:26:12 +01:00
Rob Mulholand
c05d3b7836
(VDB-280) Consume vat contract storage diffs
2019-02-08 13:36:30 -06:00
Takayuki Goto
0a024d429d
* update transformer to able to recheck headers ( #4 )
...
* update transformer to able to recheck headers
* put cap on rechecking header
* integration test for recheck headers
* use enum for recheck headers param; make recheck cap configurable
* update integration tests with new test config
* update omni pkg with new recheck header column type
* update migration with new migration tool and final tweaks needed to accommodate changes in omni pkg
2019-02-08 10:35:46 -06:00
Rob Mulholand
867f92c431
(VDB-298) Consume Pit contract storage diffs
...
- Continuously parse storage diffs CSV data to read Pit contract state
- Convert ilks in database to raw bytes32 value for use in generating
storage keys dynamically
- Persist storage diffs with block number and hash for validation
2019-02-06 10:31:46 -06:00
Edvard Hübinette
29f2537130
Merge pull request #7 from vulcanize/VDB-347-Squelch-lightSync
...
VDB-347 Squelch lightSync in stdout
2019-02-01 11:02:48 +01:00
Edvard
8eea1b00ed
Redirect validation window status to logrus instead of stdout
2019-01-31 15:55:28 +01:00
Ian Norden
8749c94994
update license to AGPL-3.0
2019-01-24 14:41:30 -06:00
Ian Norden
1f95eb6443
Merge branch 'staging' into pub_merge
2019-01-24 09:47:12 -06:00
Ian Norden
dc87f39889
change omniwatcher database test helpers so that it doesn't interfere with downstream maker tests
2019-01-23 14:55:20 -06:00
Konstantin Zolotarev
65cb6e9092
VDB-205 Fix external configuration values ( #121 )
...
* Updated loading config from /Users/konstantinzolotarev/.vulcanizedb for addreses
* Updated contract addresses loading from env variables.
* Implement getter chain for addresses, configs and transformers
Solves the issue with declaring stuff as vars when the environment file hasn't been loaded yet. Instead, this data is fetched dynamically when the root commands are called.
* Update test data to use constant Kovan addresses
* Decouple integration tests from config file IO
* Add ABI to environment file, convert getter chains for related values
* Decouple tests suites from ABI config file IO
* Add startingBlock to environment and implement getter chain
* Exit when called without config file
* Add missing deployment blocks to staging.toml
* Remove comment
2019-01-23 19:44:09 +01:00
Ian Norden
46b898f690
merge public vulcanizedb/master
2019-01-23 00:37:26 -06:00
Edvard Hübinette
ebca338b1e
VDB-302 Sleep when no missing blocks in lightSync ( #129 )
...
* Sleep when no missing blocks in lightSync
* Fix tests and error propagation
* Correct geth.log to logrus
2019-01-11 10:58:41 +01:00
Ian Norden
59cdaa05e6
moving some memory allocation to init function rather than within
...
execution cycles and fixing test db tear down so that it drops
checked_headers table instead of individual columns (droping columns
only hides them, they remain in memory and continue to contribute to the
1600 column limit which causes us to eventually exceed it)
2019-01-07 13:07:31 -06:00
Edvard
833dde62cf
Remove injection of fetcher and repository to watcher
2019-01-07 15:19:31 +01:00
Ian Norden
e390a97502
finish method polling with hash or address type argument values
...
collected from watched events; 'MissingMethodsCheckedEventsIntersection'
method to find headers which have been checked for each event of
interest but methods have not yet been polled at that header; tests for
new features; travis ci go version 1.9 -> 1.11 ; consolidate omniWatcher and lightOmniWatcher into single command with light as default
2018-12-21 10:33:31 -06:00
Ian Norden
0a59f06cac
begin work on: Add checked_headers column for methods that are polled so taht we don’t duplicate; Add batching of method polling so that we arent generating a rediculously large account address list before using it to poll methods (or persist the list in pg?); User passed ABI and other ways to get ABI; Add ability to collect []byte and hashes from events and use them in method polling same manner as addresses; Event filter addrs => only those event’s addresses/hashes are used for polling; Option to persist seen address/hash/bytes lists into pg; Only generate lists of addresses, []byte, or hashes if a method will use them later
2018-12-21 10:33:31 -06:00
Edvard
a3737c46f6
Remove duplicate entry in initializer map
2018-12-13 13:16:49 +01:00
Edvard
d30fcfed80
Use new config getter on shared.Transformer <3
2018-12-13 12:39:57 +01:00
Edvard
dc43547612
Improve process of adding transformers
2018-12-12 15:41:29 +01:00
Edvard
4e089c363e
Fixes after merging staging
2018-12-11 15:19:27 +01:00
Edvard
d93817f346
Interface-ify shared repository
2018-12-11 11:35:13 +01:00
Edvard
587d2219d8
Remove bc references and cleanup pointers
2018-12-10 16:50:13 +01:00
Ian Norden
e02b33547d
finishing porting omni watcher to work with light sync; split into full, light,
...
and shared directories and refactor as much into shared; finish
lightSync omni watcher tests
2018-11-30 13:33:48 -06:00
Edvard
c26736dc9e
WIP continue on aggregate fetching architecture
2018-11-30 17:28:52 +01:00
Ian Norden
975f13b969
reorganizing omni directory and beginning light watcher work
2018-11-29 20:33:21 -06:00
Ian Norden
817bd76713
refactoring event converter and repository so that event logs are
...
persisted as they are converted, preventing build up in memory, and finishing method polling (for full sync)
2018-11-29 20:33:21 -06:00
Ian Norden
390a60f7f6
port over lightSync updates from maker repo
2018-11-29 20:33:21 -06:00
Edvard
b5dab1e83b
Nuke erc20 transformer
2018-11-29 11:48:16 +01:00
Taka Goto
e8be96a4e7
use logrus for logging
2018-11-21 10:14:11 -06:00
Ian Norden
417b18ec6a
Edits to address PR issues; change license from apache to AGPL; and work
...
towards generic method polling and reposito;y; config settings to
filterevents/methods by account address; refactoring some stuff out of
repo and into converter; remove fetcher and instead call
blockchain's FetchContractData directly; finishing tests
2018-11-15 12:32:52 -06:00
Rob Mulholand
854236fd99
(VDB-267) Remove pit file stability fee
...
- Method is no longer present on the Pit contract
2018-11-15 12:14:59 -06:00
Rob Mulholand
82fd73ba3f
(VDB-68) Verify log block hash matches header block hash
...
- Delete header on conflict to prompt data refresh (cascade deletes all
data associated with that block)
- Derive header hash from rpc payload rather than computing it from data
(prevents hash mismatch between blockchain and cache)
2018-11-13 14:51:39 -06:00
Edvard Hübinette
571f300392
Add rendering of available transformer names to continuousLogSync help ( #109 )
2018-11-12 14:01:07 +01:00
Rob Mulholand
68464d375a
Add missing signatures to getSignatures command
2018-11-08 13:08:59 -06:00
Rob Mulholand
8c168ed332
Extract constants package
...
- Reduce clutter in the shared package
2018-11-08 13:08:59 -06:00
Rob Mulholand
cb141f7cb7
(VDB-204) Add transformer for Flap kick
2018-11-07 10:52:55 -06:00
Ian Norden
b459cf35ed
beginning work on method polling; first need to generate list of token holder address in a completely generic/contratc agnostic fashion. Created address retriever that can iterate through any given contract's watched events, finding the inputs/arguments with address type, and generate a list from those values. Edit: Contract objects now cache every event emitted address as its event logs are transformed into the repo to grow a list of contract associated addresses as we go
2018-11-04 21:37:31 -06:00
Ian Norden
e9dbd771e5
tests and fixes for fetcher, parser, retriever, converter, and repository; update cmd and watcher
2018-11-04 01:49:11 -05:00
Ian Norden
8ce75fe5ad
Generic watcher that takes a contract address, grabs the contract abi and starting block number, creates custom event filters, and extracts and transforms event data into postgres. Can configure to look at only a subset of events through CLI flag. Building but needs testing.
2018-11-03 14:00:25 -05:00
Ian Norden
57820ff473
change transformer interface and watcher so that contract config is now fed into AddTransformers such that a single watcher can be loaded with transformers that use different cofigs
2018-11-03 13:49:24 -05:00
Rob Mulholand
ba071ef13f
Consolidate test doubles
...
- Migrate various mocks of core namespaces to shared version in `fakes` pkg
- Err on the side of making test doubles less sophisticated
- Don't pull over mocks of namespaces that are only used in example code
2018-11-03 13:49:23 -05:00
Rob Mulholand
5fe6394406
Add tests for pkg/geth/blockchain
...
- inject dependencies instead of initializing them in the constructor
2018-11-03 13:49:23 -05:00
Rob Mulholand
05186634bd
Add light sync command
...
- Only syncs block headers (excludes block bodies, transactions, receipts, and logs)
- Modifies validation window to include the most recent block
- Isolates validation window to the variable defined in the cmd directory (blocks
have a separate variable defined in the block_repository for determining when
to set a block as final)
2018-11-03 13:49:23 -05:00
Elizabeth Engelman
b6f93e735f
Add ERC20 token watcher example
...
- starting with the totalSupply function
- sets contract config on transformer by passing it into the transformer
initializer
- handles block records with the same number for different nodes for both creating token_supply records, and finding missing blocks
2018-11-03 13:49:23 -05:00
Maxim Krasilnikov
7b507667f8
Added user and password to db config
2018-11-03 13:49:23 -05:00
Rob Mulholand
ed624330f6
Rename Flog pack to Vow flog
...
- enables differentiation if another contract implements flog
- also deletes flog-specific mocks
2018-10-26 15:35:18 -05:00