cerc-io/ipld-eth-server
18
0
Fork 1
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 37 Wiki Activity
1,167 Commits 12 Branches 52 Tags 80 MiB
bf26198458
Commit Graph

169 Commits

Author SHA1 Message Date
Elizabeth Engelman
1b4a901892 Rename storage_fetcher -> csv_tail_storage_fetcher 2019-09-25 16:32:27 -05:00
Elizabeth Engelman
9de631f3a7 Add a statediff streamer 2019-09-25 16:32:27 -05:00
Rob Mulholand
b96a1ee1c6 Put secondary processes to sleep in event watcher tests 
- Prevent extract/delegate from spinning when other side is being
  simulated
 2019-09-18 22:17:02 -05:00
Rob Mulholand
4fa19be90a Return error when no logs/headers available 
- Replaces bool and moots question of error/bool ordering
- Also make event watcher execution synchronous
 2019-09-18 20:55:15 -05:00
Rob Mulholand
3f9b034c4c Rename checked_logs => watched_logs 
- We're logging that a given log has been included in any fetch calls
  for checked headers, rather than that we have already checked for
  that log
 2019-09-10 21:22:14 -05:00
Rob Mulholand
13d503b851 Distinguish between missing and unchecked headers 
- Missing == not in DB
- Unchecked == logs haven't been fetched
 2019-09-10 14:39:45 -05:00
Rob Mulholand
c568fedd89 Remove unused functions on mock event repository 2019-09-10 14:20:47 -05:00
Rob Mulholand
5ac76eee74 Make check_count a column on public.headers 
- Don't need to maintain it on public.checked_headers if we're not
  adding additional columns to that table
 2019-08-28 09:50:17 -05:00
Rob Mulholand
666ea1c325 Update checked headers for new transformers 
- If a header was marked as checked before a transformer was added to
  the watcher, mark all headers since the new transformer's starting
  block number as unchecked.
 2019-08-28 09:25:14 -05:00
Rob Mulholand
d76be4962b Remove unnecessary async from the event watcher 
- extract and delegate logs synchronously after initial goroutine fired
 2019-08-28 09:25:13 -05:00
Rob Mulholand
1883a11ab1 Update comment in log chunker 
Co-Authored-By: Edvard Hübinette <edvard@hubinette.me>
 2019-08-28 09:25:13 -05:00
Rob Mulholand
63dabbb051 Extract and delegate logs concurrently 2019-08-28 09:25:13 -05:00
Rob Mulholand
d496dad33c Decouple log extraction from transformer delegation 
- limit missing headers results set to 100 so that extraction doesn't
  excessively block delegation
- wrap checked headers functions in repository struct
- move storage repository to factory, to correspond with event
  repository path
- remove unused files
- reformat sql
- remove line breaks in imports
 2019-08-28 09:25:13 -05:00
Rob Mulholand
cb819fa9a6 Write event logs to database before transforming 
- enables decoupling event extraction/persistence from transformation
- modifies event transformer, converter, and log chunker to accept
  payload that includes internal log database ID with log data
- remove alias for transformer pkg as shared_t
- remove unused mock watcher repository
 2019-08-28 09:13:44 -05:00
Rob Mulholand
66a4e20b20 Update event transformer to take header ID 
- rather than header
- enables executing transformers without full header lookup
 2019-08-28 09:11:06 -05:00
Gabe Laughlin
4f8b057726
(VDB-768) Allow addresses in packed slots to be decoded 2019-08-01 22:40:50 -05:00
Andrew J Yao
dc8d52a58d Removes _checked from column name query 2019-07-30 18:07:25 -07:00
Elizabeth
85aba29cf2
Merge pull request #121 from vulcanize/allow-for-packed-storage-slots 
VDB-318 Allow for packed storage slots
 2019-07-23 13:48:36 -05:00
Elizabeth Engelman
2c66afbcd7 Address PR feedback 
- factor out bitsPerByte constant
- panic if the necessary args aren't passed to GetStorageValueMetadataForPackedSlot
- only have one decode integer method in the deocoder file
 2019-07-23 13:36:58 -05:00
Elizabeth Engelman
69ad521482 Apply go fmt changes and cleanup 2019-07-19 10:44:36 -05:00
Elizabeth Engelman
d30b4ea4f2 Update GetStorageValueMetadata 2019-07-19 10:13:30 -05:00
Elizabeth Engelman
51915f7506 Pass storage values to repo.create as a map 2019-07-18 18:40:38 -05:00
Elizabeth Engelman
ebfb4965c7 Add tests for decoding packed storage 2019-07-18 16:21:57 -05:00
Elizabeth Engelman
b0fff9a1dd First pass at decoded packed storage slots 2019-07-18 13:30:04 -05:00
Edvard Hübinette
2c092e8d04
[VDB-751 VDB-754] Bugfix null pointer panic and improve logging (#119) 
* VDB-751 VDB-754 Bugfix null pointer panic and improve logging

* Fix typo
 2019-07-18 09:21:40 +02:00
Andrew J Yao
11a65c525c Adds ability to set log level from toml or CLI args 2019-06-27 11:14:11 -07:00
Rob Mulholand
f0a7a7d999
Merge pull request #103 from vulcanize/remove-queued-storage-duplicates 
Don't duplicate queued storage diffs
 2019-06-17 15:55:35 -05:00
Rob Mulholand
7d6629e5b4
Merge pull request #102 from vulcanize/fix-mappings-util 
Fix mappings util
 2019-06-17 15:45:55 -05:00
Elizabeth Engelman
948246cf83 Address PR comments 2019-06-17 14:49:25 -05:00
Rob Mulholand
e11f2c8c59 Don't duplicate queued storage diffs 
- currently, if we don't recognize the same diff several times (e.g.
  if you restart the storage diff watcher pointed at the same file),
  we'll add the same row to the queue on each run.
- these changes assure we only queue an unrecognized diff once.
 2019-06-14 11:26:15 -05:00
Rob Mulholand
95fa6280c5 Fix mappings util 
- Remove hex prefix so that the function can accept keys with or
  without a hex prefix and return the same result
 2019-06-13 18:33:22 -05:00
Elizabeth Engelman
fd126c7bea Remove unused RecheckHeaders method and update Repo tests 2019-06-13 15:19:23 -05:00
Elizabeth Engelman
f292888195 Refactor checked column specific SQL 2019-06-13 12:36:29 -05:00
Elizabeth Engelman
adf91df23a Remove unused RecheckHeaders method from the events Repository interface 2019-06-13 10:33:53 -05:00
Elizabeth Engelman
ede6ad1c8f Remove unused MissingHeaders method from the events Repository interface 2019-06-13 10:33:48 -05:00
Elizabeth Engelman
ee13d5c1df Remove unused recheckHeaders bool argument from Transformer.Execute 2019-06-13 10:33:16 -05:00
Rob Mulholand
e2909797fc Remove handling of duplicate storage diffs in watcher 
- Can push this responsibility down to the transformers
- Update docs to reflect that transformers should handle duplicates
 2019-05-20 13:29:09 -05:00
Rob Mulholand
3b531950f2
Merge pull request #93 from vulcanize/vdb-570-storage-duplicates 
(VDB-570) Handle duplicate storage diffs
 2019-05-02 13:40:44 -05:00
Rob Mulholand
36c4da372b (VDB-570) Handle duplicate storage diffs 
- If processing a new diff for a row that already exists in the DB,
  ignore the error without logging or queueing
- If processing a queued diff for a row that already exists, remove
  it from the queue
 2019-05-02 11:39:16 -05:00
Elizabeth
77b7ee9e4c
Merge pull request #91 from vulcanize/vdb-558-remove-log-note-transformer 
Remove LogNote Transformer and Converter
 2019-05-02 11:26:10 -05:00
Rob Mulholand
b036053937 Queue storage diffs if transformer execution fails 
- For any error, not just if key isn't recognized
- Means we don't lose track of diffs on random ephemeral errors
 2019-05-01 12:32:39 -05:00
Rob Mulholand
76ab914bdc Add license 2019-05-01 12:32:39 -05:00
Rob Mulholand
d77f3fe012 Don't pass empty row to channel on error 2019-05-01 12:30:37 -05:00
Rob Mulholand
6a86de87b4 (VDB-371) Recheck queued storage 
- Iterate through queued storage at defined interval, popping rows
  from the queue if successfully persisted
 2019-05-01 12:30:37 -05:00
Rob Mulholand
bf4b1687a0 Add get/delete functions to storage queue 2019-05-01 12:30:37 -05:00
Rob Mulholand
2d684c5aec Extract storage diff fetching behind an interface 
- Replaces directly reading from a CSV
- Simplifies testing
- Should hopefully make it easier to plug in other sources for storage
  diffs (e.g. differently formatted CSVs, JSON RPC, etc)
 2019-05-01 12:30:36 -05:00
Gabe Laughlin
a344432156
(VDB-560) Rename lightSync to headerSync 2019-05-01 12:12:55 -05:00
Elizabeth Engelman
eb1a9cb381 Remove LogNote Transformer and Converter 2019-04-30 12:25:51 -05:00
Rob Mulholand
3fd6269b78 Don't lookup transactions if no log events 
- Prevents EOF error on transactions lookup
 2019-04-18 09:53:18 -05:00
Ian Norden
185f4c0e93 adjust block/uncle reward tests and add methods to pkg/geth blockchain and ethclient for direct fetching of eth balances 2019-04-02 10:10:17 -05:00
Rob Mulholand
39471e1d04 Remove links to Maker 2019-04-01 16:03:35 -05:00
Rob Mulholand
8b5b5b8f17 Remove links to 8th Light 2019-04-01 15:15:08 -05:00
Rob Mulholand
81dfd12665 Address PR comments 2019-03-28 14:31:17 -05:00
Rob Mulholand
54d46638a8 updates for light sync transactions 2019-03-28 14:31:17 -05:00
Elizabeth Engelman
a7022563ab Update storage mapping indexes to hex values 2019-03-25 14:28:49 -05:00
Ian Norden
1aa849bcb4 PR fixes; remove all infura token references and setup travis to use encrypted env 
variable; rest of the ethjson_rpc dependent tests extracted to integration_test
 2019-03-21 18:36:51 -05:00
Ian Norden
37fc038605 fixes for review comments 2019-03-21 18:36:49 -05:00
Ian Norden
24879a85aa mv pkg/omni pkg/contract_watcher 2019-03-21 18:33:56 -05:00
Ian Norden
073378de6d update © to 2019 2019-03-21 18:33:56 -05:00
Ian Norden
e4e092f542 changes to plugin and commands to accomodate changes 2019-03-21 18:33:56 -05:00
Ian Norden
5101ed4369 generic transformer and watcher interfaces for plugging in transformers built from omni pkging 2019-03-21 18:30:48 -05:00
Ian Norden
ff55e3ba7a TransformerInitializer => EventTransformerInitializer reorganization 2019-03-14 11:59:39 -05:00
Ian Norden
9f8c50e3ab tests for migration path ordering and errors 2019-03-13 11:42:30 -05:00
Ian Norden
35c5808a07 readme fixes 2019-03-12 12:54:36 -05:00
Ian Norden
1d6219cc83 event transformers readme and update main readme to remove last links to 
mcd_transformers
 2019-03-12 12:45:29 -05:00
Rob Mulholand
89d152e987 Add unit48 to storage decoder 2019-03-08 11:12:02 -06:00
Rob Mulholand
cdcb90c170 Merge pull request #34 from vulcanize/redsquirrel-patch-3 
Fixing some typos
 2019-03-08 10:56:38 -06:00
Rob Mulholand
c5862299e6 Merge pull request #33 from vulcanize/redsquirrel-patch-2 
Minor Solidity suggestion
 2019-03-08 10:56:20 -06:00
Dave Hoover
f03329247d Fixing some typos 2019-03-07 15:33:18 -06:00
Dave Hoover
789477bfbf Minor Solidity suggestion 
Just reading through this and figured I'd suggest a more concise snippet.
 2019-03-07 15:18:27 -06:00
Rob Mulholand
5d0467f4b8 Add variables to clarify example contracts 2019-03-06 12:43:39 -06:00
Rob Mulholand
1414779d52 Use *sqlx.Tx instead of *sql.Tx 
- requires using db.Beginx() instead of db.Begin()
- enables calling tx.Get()
 2019-03-06 12:43:39 -06:00
Rob Mulholand
60d7b34471 address PR comments 2019-03-06 12:43:39 -06:00
Rob Mulholand
7ed4dcd1fa Remove Tic logic 2019-03-06 12:43:39 -06:00
Rob Mulholand
310a52cb0c test mappings helpers against static values 2019-03-06 12:43:39 -06:00
Rob Mulholand
b6bef49318 Add tests for plugin helpers 
- Prevents code analysis from showing functions as unused
- Also address other code analysis issues
 2019-03-06 12:43:39 -06:00
Ian Norden
708425c4d6 rebase; extract factories and the mocks they are dependent on to 
libraries/shared; adjust omni test_helpers to drop and recreate
checked_headers table to avoid reaching 1600 column limit after repeated
tests (dropping columns doesn't actually remove them from contributing
to that limit)
 2019-02-25 01:34:38 -06:00
Ian Norden
f2072561a7 removing mcd/maker references; delete test_data; update READMEs; use 
logrus
 2019-02-24 21:53:04 -06:00
Ian Norden
3d34a9e7c9 remove maker migrations and convert back to timestamps and fix bug in 
composeAndExecute command
 2019-02-24 21:52:23 -06:00
Ian Norden
b449193b16 addressing review comments; still need to reorg migrations and add helper tests 2019-02-24 21:38:48 -06:00
Ian Norden
90e67d6da9 move storage mappings pkg into shared 2019-02-24 21:38:48 -06:00
Ian Norden
decc2a3caf remove pkg/transformers; extract shared files needed to 
libraries/shared; work on automated db migration management
 2019-02-24 21:38:47 -06:00
Ian Norden
6c2d895023 composeAndExecute command that loads and executes over arbitrary transformer set exported from a go plugin generated according to config file; test for plugin generation, loading, and execution; work on plugins README 2019-02-24 15:23:35 -06:00
Rob Mulholand
11dd641a84 (VDB-354) Queue unrecognized storage diffs 
- If we recognize a storage diff as coming from a watched contract but
  don't recognize the key, queue it for retrying later (after we've seen
  an event that might help us recognize the key)
- Remove unused errs and args
- Panic on unrecognized types (should not happen)
 2019-02-20 15:22:39 -06:00
Takayuki Goto
0a024d429d * update transformer to able to recheck headers (#4) 
* update transformer to able to recheck headers
* put cap on rechecking header
* integration test for recheck headers
* use enum for recheck headers param; make recheck cap configurable
* update integration tests with new test config
* update omni pkg with new recheck header column type
* update migration with new migration tool and final tweaks needed to accommodate changes in omni pkg
 2019-02-08 10:35:46 -06:00
Rob Mulholand
867f92c431 (VDB-298) Consume Pit contract storage diffs 
- Continuously parse storage diffs CSV data to read Pit contract state
- Convert ilks in database to raw bytes32 value for use in generating
  storage keys dynamically
- Persist storage diffs with block number and hash for validation
 2019-02-06 10:31:46 -06:00
Ian Norden
46b898f690 merge public vulcanizedb/master 2019-01-23 00:37:26 -06:00
Edvard Hübinette
909d23e176 VDB-284 Run watcher from earliest startingBlockNumber (#136) 
* Watcher computes earliest starting block number

* Update tests for new watcher behaviour

* Fix error oopsie

* Extract conditional to helper and suppres tests logging
 2019-01-15 11:09:13 +01:00
Edvard
50cc0c2459 Update tests for watcher and repository 2019-01-07 15:20:05 +01:00
Edvard
833dde62cf Remove injection of fetcher and repository to watcher 2019-01-07 15:19:31 +01:00
Edvard
e5db740ae4 Move MockTransformer to test_data 2018-12-13 13:03:51 +01:00
Edvard
d30fcfed80 Use new config getter on shared.Transformer <3 2018-12-13 12:39:57 +01:00
Edvard
882a6dd7a2 Update tests with new adding of transformers 2018-12-12 15:41:44 +01:00
Edvard
dc43547612 Improve process of adding transformers 2018-12-12 15:41:29 +01:00
Edvard
4e089c363e Fixes after merging staging 2018-12-11 15:19:27 +01:00
Edvard
9000c6e9f9 Add test for watcher log delegation 2018-12-11 15:02:32 +01:00
Edvard
5d4239de05 Finish watcher and watcher tests (+fmt) 2018-12-11 13:52:02 +01:00
Edvard
d93817f346 Interface-ify shared repository 2018-12-11 11:35:13 +01:00
Edvard
e03ccb094f Add error propagation and test todos for watcher 2018-12-10 21:12:31 +01:00
Edvard
587d2219d8 Remove bc references and cleanup pointers 2018-12-10 16:50:13 +01:00