cerc-io/ipld-eth-server
17
0
Fork 1
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 37 Wiki Activity
1,023 Commits 12 Branches 52 Tags 80 MiB
ee244ac6f5
Commit Graph

121 Commits

Author SHA1 Message Date
Elizabeth Engelman
ee244ac6f5 Apply go fmt 2019-09-25 16:32:27 -05:00
Elizabeth Engelman
dc06991605 Wire up the streamer with a fetcher 2019-09-25 16:32:27 -05:00
Elizabeth Engelman
1b4a901892 Rename storage_fetcher -> csv_tail_storage_fetcher 2019-09-25 16:32:27 -05:00
Elizabeth Engelman
9de631f3a7 Add a statediff streamer 2019-09-25 16:32:27 -05:00
Rob Mulholand
b96a1ee1c6 Put secondary processes to sleep in event watcher tests 
- Prevent extract/delegate from spinning when other side is being
  simulated
 2019-09-18 22:17:02 -05:00
Rob Mulholand
4fa19be90a Return error when no logs/headers available 
- Replaces bool and moots question of error/bool ordering
- Also make event watcher execution synchronous
 2019-09-18 20:55:15 -05:00
Rob Mulholand
3f9b034c4c Rename checked_logs => watched_logs 
- We're logging that a given log has been included in any fetch calls
  for checked headers, rather than that we have already checked for
  that log
 2019-09-10 21:22:14 -05:00
Rob Mulholand
13d503b851 Distinguish between missing and unchecked headers 
- Missing == not in DB
- Unchecked == logs haven't been fetched
 2019-09-10 14:39:45 -05:00
Rob Mulholand
c568fedd89 Remove unused functions on mock event repository 2019-09-10 14:20:47 -05:00
Rob Mulholand
5ac76eee74 Make check_count a column on public.headers 
- Don't need to maintain it on public.checked_headers if we're not
  adding additional columns to that table
 2019-08-28 09:50:17 -05:00
Rob Mulholand
666ea1c325 Update checked headers for new transformers 
- If a header was marked as checked before a transformer was added to
  the watcher, mark all headers since the new transformer's starting
  block number as unchecked.
 2019-08-28 09:25:14 -05:00
Rob Mulholand
d76be4962b Remove unnecessary async from the event watcher 
- extract and delegate logs synchronously after initial goroutine fired
 2019-08-28 09:25:13 -05:00
Rob Mulholand
1883a11ab1 Update comment in log chunker 
Co-Authored-By: Edvard Hübinette <edvard@hubinette.me>
 2019-08-28 09:25:13 -05:00
Rob Mulholand
63dabbb051 Extract and delegate logs concurrently 2019-08-28 09:25:13 -05:00
Rob Mulholand
d496dad33c Decouple log extraction from transformer delegation 
- limit missing headers results set to 100 so that extraction doesn't
  excessively block delegation
- wrap checked headers functions in repository struct
- move storage repository to factory, to correspond with event
  repository path
- remove unused files
- reformat sql
- remove line breaks in imports
 2019-08-28 09:25:13 -05:00
Rob Mulholand
cb819fa9a6 Write event logs to database before transforming 
- enables decoupling event extraction/persistence from transformation
- modifies event transformer, converter, and log chunker to accept
  payload that includes internal log database ID with log data
- remove alias for transformer pkg as shared_t
- remove unused mock watcher repository
 2019-08-28 09:13:44 -05:00
Rob Mulholand
66a4e20b20 Update event transformer to take header ID 
- rather than header
- enables executing transformers without full header lookup
 2019-08-28 09:11:06 -05:00
Gabe Laughlin
4f8b057726
(VDB-768) Allow addresses in packed slots to be decoded 2019-08-01 22:40:50 -05:00
Andrew J Yao
dc8d52a58d Removes _checked from column name query 2019-07-30 18:07:25 -07:00
Elizabeth
85aba29cf2
Merge pull request #121 from vulcanize/allow-for-packed-storage-slots 
VDB-318 Allow for packed storage slots
 2019-07-23 13:48:36 -05:00
Elizabeth Engelman
2c66afbcd7 Address PR feedback 
- factor out bitsPerByte constant
- panic if the necessary args aren't passed to GetStorageValueMetadataForPackedSlot
- only have one decode integer method in the deocoder file
 2019-07-23 13:36:58 -05:00
Elizabeth Engelman
69ad521482 Apply go fmt changes and cleanup 2019-07-19 10:44:36 -05:00
Elizabeth Engelman
d30b4ea4f2 Update GetStorageValueMetadata 2019-07-19 10:13:30 -05:00
Elizabeth Engelman
51915f7506 Pass storage values to repo.create as a map 2019-07-18 18:40:38 -05:00
Elizabeth Engelman
ebfb4965c7 Add tests for decoding packed storage 2019-07-18 16:21:57 -05:00
Elizabeth Engelman
b0fff9a1dd First pass at decoded packed storage slots 2019-07-18 13:30:04 -05:00
Edvard Hübinette
2c092e8d04
[VDB-751 VDB-754] Bugfix null pointer panic and improve logging (#119) 
* VDB-751 VDB-754 Bugfix null pointer panic and improve logging

* Fix typo
 2019-07-18 09:21:40 +02:00
Andrew J Yao
11a65c525c Adds ability to set log level from toml or CLI args 2019-06-27 11:14:11 -07:00
Rob Mulholand
f0a7a7d999
Merge pull request #103 from vulcanize/remove-queued-storage-duplicates 
Don't duplicate queued storage diffs
 2019-06-17 15:55:35 -05:00
Rob Mulholand
7d6629e5b4
Merge pull request #102 from vulcanize/fix-mappings-util 
Fix mappings util
 2019-06-17 15:45:55 -05:00
Elizabeth Engelman
948246cf83 Address PR comments 2019-06-17 14:49:25 -05:00
Rob Mulholand
e11f2c8c59 Don't duplicate queued storage diffs 
- currently, if we don't recognize the same diff several times (e.g.
  if you restart the storage diff watcher pointed at the same file),
  we'll add the same row to the queue on each run.
- these changes assure we only queue an unrecognized diff once.
 2019-06-14 11:26:15 -05:00
Rob Mulholand
95fa6280c5 Fix mappings util 
- Remove hex prefix so that the function can accept keys with or
  without a hex prefix and return the same result
 2019-06-13 18:33:22 -05:00
Elizabeth Engelman
fd126c7bea Remove unused RecheckHeaders method and update Repo tests 2019-06-13 15:19:23 -05:00
Elizabeth Engelman
f292888195 Refactor checked column specific SQL 2019-06-13 12:36:29 -05:00
Elizabeth Engelman
adf91df23a Remove unused RecheckHeaders method from the events Repository interface 2019-06-13 10:33:53 -05:00
Elizabeth Engelman
ede6ad1c8f Remove unused MissingHeaders method from the events Repository interface 2019-06-13 10:33:48 -05:00
Elizabeth Engelman
ee13d5c1df Remove unused recheckHeaders bool argument from Transformer.Execute 2019-06-13 10:33:16 -05:00
Rob Mulholand
e2909797fc Remove handling of duplicate storage diffs in watcher 
- Can push this responsibility down to the transformers
- Update docs to reflect that transformers should handle duplicates
 2019-05-20 13:29:09 -05:00
Rob Mulholand
3b531950f2
Merge pull request #93 from vulcanize/vdb-570-storage-duplicates 
(VDB-570) Handle duplicate storage diffs
 2019-05-02 13:40:44 -05:00
Rob Mulholand
36c4da372b (VDB-570) Handle duplicate storage diffs 
- If processing a new diff for a row that already exists in the DB,
  ignore the error without logging or queueing
- If processing a queued diff for a row that already exists, remove
  it from the queue
 2019-05-02 11:39:16 -05:00
Elizabeth
77b7ee9e4c
Merge pull request #91 from vulcanize/vdb-558-remove-log-note-transformer 
Remove LogNote Transformer and Converter
 2019-05-02 11:26:10 -05:00
Rob Mulholand
b036053937 Queue storage diffs if transformer execution fails 
- For any error, not just if key isn't recognized
- Means we don't lose track of diffs on random ephemeral errors
 2019-05-01 12:32:39 -05:00
Rob Mulholand
76ab914bdc Add license 2019-05-01 12:32:39 -05:00
Rob Mulholand
d77f3fe012 Don't pass empty row to channel on error 2019-05-01 12:30:37 -05:00
Rob Mulholand
6a86de87b4 (VDB-371) Recheck queued storage 
- Iterate through queued storage at defined interval, popping rows
  from the queue if successfully persisted
 2019-05-01 12:30:37 -05:00
Rob Mulholand
bf4b1687a0 Add get/delete functions to storage queue 2019-05-01 12:30:37 -05:00
Rob Mulholand
2d684c5aec Extract storage diff fetching behind an interface 
- Replaces directly reading from a CSV
- Simplifies testing
- Should hopefully make it easier to plug in other sources for storage
  diffs (e.g. differently formatted CSVs, JSON RPC, etc)
 2019-05-01 12:30:36 -05:00
Gabe Laughlin
a344432156
(VDB-560) Rename lightSync to headerSync 2019-05-01 12:12:55 -05:00
Elizabeth Engelman
eb1a9cb381 Remove LogNote Transformer and Converter 2019-04-30 12:25:51 -05:00