cerc-io/ipld-eth-server
17
0
Fork 1
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 37 Wiki Activity
1,100 Commits 12 Branches 52 Tags 80 MiB
a834e55b9f
Commit Graph

44 Commits

Author SHA1 Message Date
Ian Norden
8562abd180 split backfill range up into smaller bins and process them concurrently; improve tests; review fixes 2019-12-02 11:20:49 -06:00
Ian Norden
37f4a2d603 integrate backfill into storage watcher; documentation for storage backfill 2019-12-02 11:06:28 -06:00
Ian Norden
a59bd06a37 goimports -w; comments; refactor test 2019-12-02 11:03:36 -06:00
Rob Mulholand
e1236b4072 Prefer all caps for initialisms and acronyms 2019-10-31 13:42:19 -05:00
Elizabeth Engelman
9c6182c356 Rename StorageDiff field from KeccakOfContractAddress to HashedAddress 2019-09-25 16:36:44 -05:00
Rob Mulholand
2ff88de859 Extract helper for converting hex to keccak256 hash 
- Also prefer crypto.Keccak256Hash(x) to common.BytesToHash(crypto.Keccak256(x))
 2019-09-25 16:36:08 -05:00
Rob Mulholand
d06dddbfaa Always hash storage diff contract addresses 
- Enables syncing Geth and Parity diffs with same transformer lookup
- Maybe worth always hashing the storage key so we don't need a hashed
  and not-hashed version in the key lookups?
 2019-09-25 16:36:08 -05:00
Elizabeth Engelman
66d695d93b Update keccak of address to be a common.Hash instead of a common.Address 2019-09-25 16:34:31 -05:00
Elizabeth Engelman
9226e53e0b Panic if connecting to geth subscription fails 2019-09-25 16:32:31 -05:00
Elizabeth Engelman
0cc90a1f80 Factor out an IStorageWatcher interface 2019-09-25 16:32:31 -05:00
Elizabeth Engelman
18f47b7b6a Memoizing keccak hhash of address => transformer 2019-09-25 16:32:31 -05:00
Elizabeth Engelman
045d78be25 Rename StorageDiffRow -> StorageDiff 2019-09-25 16:32:31 -05:00
Elizabeth Engelman
70fcc22a00 Apply go formatting 2019-09-25 16:32:27 -05:00
Elizabeth Engelman
8c4a4d6587 Handle different contract address format for geth vs csv 2019-09-25 16:32:27 -05:00
Rob Mulholand
b96a1ee1c6 Put secondary processes to sleep in event watcher tests 
- Prevent extract/delegate from spinning when other side is being
  simulated
 2019-09-18 22:17:02 -05:00
Rob Mulholand
4fa19be90a Return error when no logs/headers available 
- Replaces bool and moots question of error/bool ordering
- Also make event watcher execution synchronous
 2019-09-18 20:55:15 -05:00
Rob Mulholand
13d503b851 Distinguish between missing and unchecked headers 
- Missing == not in DB
- Unchecked == logs haven't been fetched
 2019-09-10 14:39:45 -05:00
Rob Mulholand
666ea1c325 Update checked headers for new transformers 
- If a header was marked as checked before a transformer was added to
  the watcher, mark all headers since the new transformer's starting
  block number as unchecked.
 2019-08-28 09:25:14 -05:00
Rob Mulholand
d76be4962b Remove unnecessary async from the event watcher 
- extract and delegate logs synchronously after initial goroutine fired
 2019-08-28 09:25:13 -05:00
Rob Mulholand
63dabbb051 Extract and delegate logs concurrently 2019-08-28 09:25:13 -05:00
Rob Mulholand
d496dad33c Decouple log extraction from transformer delegation 
- limit missing headers results set to 100 so that extraction doesn't
  excessively block delegation
- wrap checked headers functions in repository struct
- move storage repository to factory, to correspond with event
  repository path
- remove unused files
- reformat sql
- remove line breaks in imports
 2019-08-28 09:25:13 -05:00
Rob Mulholand
cb819fa9a6 Write event logs to database before transforming 
- enables decoupling event extraction/persistence from transformation
- modifies event transformer, converter, and log chunker to accept
  payload that includes internal log database ID with log data
- remove alias for transformer pkg as shared_t
- remove unused mock watcher repository
 2019-08-28 09:13:44 -05:00
Rob Mulholand
66a4e20b20 Update event transformer to take header ID 
- rather than header
- enables executing transformers without full header lookup
 2019-08-28 09:11:06 -05:00
Edvard Hübinette
2c092e8d04
[VDB-751 VDB-754] Bugfix null pointer panic and improve logging (#119) 
* VDB-751 VDB-754 Bugfix null pointer panic and improve logging

* Fix typo
 2019-07-18 09:21:40 +02:00
Elizabeth Engelman
f292888195 Refactor checked column specific SQL 2019-06-13 12:36:29 -05:00
Elizabeth Engelman
ee13d5c1df Remove unused recheckHeaders bool argument from Transformer.Execute 2019-06-13 10:33:16 -05:00
Rob Mulholand
e2909797fc Remove handling of duplicate storage diffs in watcher 
- Can push this responsibility down to the transformers
- Update docs to reflect that transformers should handle duplicates
 2019-05-20 13:29:09 -05:00
Rob Mulholand
36c4da372b (VDB-570) Handle duplicate storage diffs 
- If processing a new diff for a row that already exists in the DB,
  ignore the error without logging or queueing
- If processing a queued diff for a row that already exists, remove
  it from the queue
 2019-05-02 11:39:16 -05:00
Rob Mulholand
b036053937 Queue storage diffs if transformer execution fails 
- For any error, not just if key isn't recognized
- Means we don't lose track of diffs on random ephemeral errors
 2019-05-01 12:32:39 -05:00
Rob Mulholand
6a86de87b4 (VDB-371) Recheck queued storage 
- Iterate through queued storage at defined interval, popping rows
  from the queue if successfully persisted
 2019-05-01 12:30:37 -05:00
Rob Mulholand
2d684c5aec Extract storage diff fetching behind an interface 
- Replaces directly reading from a CSV
- Simplifies testing
- Should hopefully make it easier to plug in other sources for storage
  diffs (e.g. differently formatted CSVs, JSON RPC, etc)
 2019-05-01 12:30:36 -05:00
Rob Mulholand
54d46638a8 updates for light sync transactions 2019-03-28 14:31:17 -05:00
Ian Norden
1aa849bcb4 PR fixes; remove all infura token references and setup travis to use encrypted env 
variable; rest of the ethjson_rpc dependent tests extracted to integration_test
 2019-03-21 18:36:51 -05:00
Ian Norden
37fc038605 fixes for review comments 2019-03-21 18:36:49 -05:00
Ian Norden
073378de6d update © to 2019 2019-03-21 18:33:56 -05:00
Ian Norden
e4e092f542 changes to plugin and commands to accomodate changes 2019-03-21 18:33:56 -05:00
Ian Norden
5101ed4369 generic transformer and watcher interfaces for plugging in transformers built from omni pkging 2019-03-21 18:30:48 -05:00
Ian Norden
ff55e3ba7a TransformerInitializer => EventTransformerInitializer reorganization 2019-03-14 11:59:39 -05:00
Ian Norden
9f8c50e3ab tests for migration path ordering and errors 2019-03-13 11:42:30 -05:00
Rob Mulholand
b6bef49318 Add tests for plugin helpers 
- Prevents code analysis from showing functions as unused
- Also address other code analysis issues
 2019-03-06 12:43:39 -06:00
Ian Norden
708425c4d6 rebase; extract factories and the mocks they are dependent on to 
libraries/shared; adjust omni test_helpers to drop and recreate
checked_headers table to avoid reaching 1600 column limit after repeated
tests (dropping columns doesn't actually remove them from contributing
to that limit)
 2019-02-25 01:34:38 -06:00
Ian Norden
90e67d6da9 move storage mappings pkg into shared 2019-02-24 21:38:48 -06:00
Ian Norden
decc2a3caf remove pkg/transformers; extract shared files needed to 
libraries/shared; work on automated db migration management
 2019-02-24 21:38:47 -06:00
Ian Norden
6c2d895023 composeAndExecute command that loads and executes over arbitrary transformer set exported from a go plugin generated according to config file; test for plugin generation, loading, and execution; work on plugins README 2019-02-24 15:23:35 -06:00