deep-stack/ipld-eth-server
4
0
Fork 0
forked from cerc-io/ipld-eth-server
Code Pull Requests Activity
1,252 Commits 12 Branches 50 Tags
a5fc994c7922722a67ec06da1facde5f27f67e05
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Rob Mulholand 56ce8bdb41 (VDB-950) Write raw diffs before transforming 
- Raw field we can reference by FK for related data
- Enables replay for unwatched or mistransformed diffs
 2019-12-03 14:51:17 -06:00
Rob Mulholand 6f837f5e88 Add indexes to columns used in plugin joins 
- Will probably want to update these depending on what's commonly used
  in join/where statements.
 2019-12-02 23:54:52 -06:00
Rob Mulholand 4a98b7942f Don't return unexported types 2019-10-31 13:42:19 -05:00
Rob Mulholand e1236b4072 Prefer all caps for initialisms and acronyms 2019-10-31 13:42:19 -05:00
Rob Mulholand eba868ff00 Drop else when if statement includes return 2019-10-31 13:33:39 -05:00
Rob Mulholand 0ded7bcd62 Prefer fmt.Errorf(s) to errors.New(fmt.Sprintf(s)) 2019-10-31 13:33:39 -05:00
Rob Mulholand eb2f3c2209 Merge pull request #164 from vulcanize/automated-migrations-with-driver 
Add driver and user to migrations in tests
 2019-10-31 13:29:21 -05:00
Rob Mulholand 0167f884b4 Enable configuring db user, host, and port in tests 2019-10-31 13:20:47 -05:00
Rob Mulholand 2bc3267cce Add driver and user to migrations in tests 
- Prevents logs that driver is missing
- Prevents goose_db_version from hanging around
 2019-10-31 13:20:47 -05:00
Rob Mulholand 59f767b9e1 Merge pull request #165 from vulcanize/use-testing-config-in-tests 
Use testing.toml and vulcanize_testing in tests
 2019-10-31 13:20:23 -05:00
Rob Mulholand 234fff83c5 Use testing.toml and vulcanize_testing in tests 
- remove legacy references to infura.toml and vulcanize_private
 2019-10-31 13:10:25 -05:00
Rob Mulholand 25f9c6c9e3 Merge pull request #150 from vulcanize/unique-headers-constraint-v2 
Add constraint to prevent duplicate headers
 2019-10-28 15:16:45 -05:00
Rob Mulholand e252229b8a Add constraint to prevent duplicate headers 
- Disallow inserts of headers with the same number, hash, and node
  fingerprint, since it will enable duplicate log fetching for the
  same header
 2019-10-28 14:57:13 -05:00
Rob Mulholand 62e1378e0c Merge pull request #163 from vulcanize/vdb-929-storage-key-lookup-cleanup 
(VDB-929) Minimize storage key lookup bespoke code
 2019-10-28 14:56:26 -05:00
Rob Mulholand b8fec5e4e3 (VDB-929) Minimize storage key lookup bespoke code 
- Extract shared namespace for looking up and hashing keys
- Require storage transformers only to implement a loader that
  associates known keys with metadata
- Move key loader/lookup utils to utils directory to avoid
  multiple "storage" packages in imports
 2019-10-28 14:29:09 -05:00
Rob Mulholand 6c055a9e12 Pin to urfave/cli version in go.mod (#154) 
* Pin to urfave/cli version in go.mod

- Attempting to fix error: github.com/vulcanize/vulcanizedb@v0.0.8
  requires gopkg.in/urfave/cli.v1@v1.0.0-00010101000000-000000000000:
  invalid version: unknown revision 000000000000
 2019-10-22 05:58:26 +09:00
Rob Mulholand 7be070fcea Merge pull request #156 from vulcanize/contract-watcher-init 
Enable contractWatcher without prior headerSync
 2019-10-10 08:37:48 +09:00
Rob Mulholand 2800e6df36 Enable contractWatcher without prior headerSync 
- Previous setup would fail if there were no headers in the db. This
  makes sense because we need headers that haven't been checked for
  logs to exist so that we can fetch logs for those headers. But it
  also prevents us from kicking off the headerSync and contractWatcher
  processes concurrently. These changes enable kicking off both
  processes at the same time with the idea that we will have unchecked
  headers upon transformer execution.
 2019-10-04 16:00:13 -05:00
Rob Mulholand 031043130e Merge pull request #145 from vulcanize/contract-watcher-logging 
Increase logging in contract watcher
 2019-10-02 16:30:17 -05:00
Rob Mulholand 4505382590 Increase logging in contract watcher 
- Focus on header mode
- Add context to errors, trace guard clauses, warn on non-returned
  errors
- Give errors distinct names so compiler will recognize if unchecked
- Remove redundant type declarations/fix typos
 2019-10-02 15:14:42 -05:00
Rob Mulholand 2ff88de859 Extract helper for converting hex to keccak256 hash 
- Also prefer crypto.Keccak256Hash(x) to common.BytesToHash(crypto.Keccak256(x))
 2019-09-25 16:36:08 -05:00
Rob Mulholand d06dddbfaa Always hash storage diff contract addresses 
- Enables syncing Geth and Parity diffs with same transformer lookup
- Maybe worth always hashing the storage key so we don't need a hashed
  and not-hashed version in the key lookups?
 2019-09-25 16:36:08 -05:00
Rob Mulholand f574407bb6 Revert "Update Mappings interface to take a diff as an argument to Lookup" 
This reverts commit cda646b78ceb6714b132891ea4489a8ff4fc95e9.
 2019-09-25 16:34:31 -05:00
Rob Mulholand 8111f4ec5e Add function for adding hashed keys to mapping 
- Tool to facilitate parsing diffs from Geth patch that emits hashed
  versions of storage keys
 2019-09-25 16:34:31 -05:00
Rob Mulholand ba4e79fc63 Merge pull request #143 from vulcanize/contract-watcher-missing-headers 
Correctly identify missing headers after contract watcher restart
 2019-09-24 08:55:25 -05:00
Rob Mulholand c21c069b69 Replace magic numbers with variables 2019-09-23 16:13:45 -05:00
Rob Mulholand eea0b0bdca Return missing headers after starting block number 
- Previously, we required that the first missing header match the
  configured starting block number. This helps guarantee that we have
  the necessary data for method polling in memory, but prevents the
  process from moving forward if restarted after the starting block
  has already been checked.
 2019-09-23 16:13:45 -05:00
Rob Mulholand a596966b7c Merge pull request #141 from vulcanize/contract-watcher-duplicate-inserts 
Do nothing on duplicate log insert in contract watcher
 2019-09-23 16:13:18 -05:00
Rob Mulholand 37e2673a8d Do nothing on duplicate log insert 
- Prevents duplicate key constraint violation from blocking the process
  from moving forward on restart.
- If header_id, log_idx, and tx_idx are the same, we can safely do
  nothing since it's definitely the same log - a reorg would cause the
  original header to be replaced with a new ID.
 2019-09-23 16:02:07 -05:00
Rob Mulholand dc30099a7c Merge pull request #139 from vulcanize/contract-watcher-uint8 
Correctly parse uint8 values in contract watcher
 2019-09-23 15:59:38 -05:00
Rob Mulholand 3466a51b4d Correctly parse uint8 values in contract watcher 
- calling string(uint8) => insert errors on numeric columns
 2019-09-20 21:37:59 -05:00
Rob Mulholand 8bbd65eabd Merge pull request #138 from vulcanize/contract-watcher-bytes32-parsing 
Parse bytes32 vals in Contract Watcher
 2019-09-20 14:23:29 -05:00
Rob Mulholand 749db40b96 Parse bytes32 vals in Contract Watcher 2019-09-20 13:48:43 -05:00
Rob Mulholand 2b151c2ba3 Merge pull request #131 from vulcanize/vdb-699-logs-table 
(VDB-699) Decouple Log extraction vs delegation to transformers
 2019-09-20 10:52:49 -05:00
Rob Mulholand 3897b288af Update Geth dep to 1.9.5 2019-09-20 10:38:16 -05:00
Rob Mulholand b96a1ee1c6 Put secondary processes to sleep in event watcher tests 
- Prevent extract/delegate from spinning when other side is being
  simulated
 2019-09-18 22:17:02 -05:00
Rob Mulholand f83e996ab8 Update comments in checked headers repository 2019-09-18 21:39:34 -05:00
Rob Mulholand 4fa19be90a Return error when no logs/headers available 
- Replaces bool and moots question of error/bool ordering
- Also make event watcher execution synchronous
 2019-09-18 20:55:15 -05:00
Rob Mulholand 2b798e00e0 Cap random fake timestamp 2019-09-13 11:20:13 -05:00
Rob Mulholand 3f9b034c4c Rename checked_logs => watched_logs 
- We're logging that a given log has been included in any fetch calls
  for checked headers, rather than that we have already checked for
  that log
 2019-09-10 21:22:14 -05:00
Rob Mulholand 13d503b851 Distinguish between missing and unchecked headers 
- Missing == not in DB
- Unchecked == logs haven't been fetched
 2019-09-10 14:39:45 -05:00
Rob Mulholand c568fedd89 Remove unused functions on mock event repository 2019-09-10 14:20:47 -05:00
Rob Mulholand 3d6c973f6e Remove extraneous db constraint 2019-09-10 13:58:58 -05:00
Rob Mulholand ce91b0d9e6 Simplify checked header repository tests 
- Use assertions instead of comments to document expectations
- Also randomize the fake timestamp in test data
 2019-09-10 13:47:43 -05:00
Rob Mulholand b9f3b9f946 Reference header sync logs address via foreign key 2019-08-28 10:41:34 -05:00
Rob Mulholand 5ac76eee74 Make check_count a column on public.headers 
- Don't need to maintain it on public.checked_headers if we're not
  adding additional columns to that table
 2019-08-28 09:50:17 -05:00
Rob Mulholand 222252f89a Remove extraneous migration formatting 2019-08-28 09:26:13 -05:00
Rob Mulholand 666ea1c325 Update checked headers for new transformers 
- If a header was marked as checked before a transformer was added to
  the watcher, mark all headers since the new transformer's starting
  block number as unchecked.
 2019-08-28 09:25:14 -05:00
Rob Mulholand d76be4962b Remove unnecessary async from the event watcher 
- extract and delegate logs synchronously after initial goroutine fired
 2019-08-28 09:25:13 -05:00
Rob MulholandandEdvard Hübinette 1883a11ab1 Update comment in log chunker 
Co-Authored-By: Edvard Hübinette <edvard@hubinette.me>
 2019-08-28 09:25:13 -05:00
Rob Mulholand 63dabbb051 Extract and delegate logs concurrently 2019-08-28 09:25:13 -05:00
Rob Mulholand d496dad33c Decouple log extraction from transformer delegation 
- limit missing headers results set to 100 so that extraction doesn't
  excessively block delegation
- wrap checked headers functions in repository struct
- move storage repository to factory, to correspond with event
  repository path
- remove unused files
- reformat sql
- remove line breaks in imports
 2019-08-28 09:25:13 -05:00
Rob Mulholand cb819fa9a6 Write event logs to database before transforming 
- enables decoupling event extraction/persistence from transformation
- modifies event transformer, converter, and log chunker to accept
  payload that includes internal log database ID with log data
- remove alias for transformer pkg as shared_t
- remove unused mock watcher repository
 2019-08-28 09:13:44 -05:00
Rob Mulholand 66a4e20b20 Update event transformer to take header ID 
- rather than header
- enables executing transformers without full header lookup
 2019-08-28 09:11:06 -05:00
Rob Mulholand 3693ed905f Rename logs to full sync logs 
- Enable creating new table for logs used in event watching based on
  header sync
 2019-08-28 09:11:06 -05:00
Rob Mulholand e093ad351f Merge pull request #130 from vulcanize/update-deps 
Update dependencies
 2019-08-21 16:04:55 -05:00
Rob Mulholand 33e7651096 Update dependencies 2019-08-20 16:17:25 -05:00
Rob Mulholand f2547b5a42 Merge pull request #117 from vulcanize/geth-1.9.0 
Update Geth to 1.9.0
 2019-07-29 15:04:37 -05:00
Rob Mulholand 62e454b7fe Update docs to reflect using go mod 2019-07-23 15:28:08 -05:00
Rob Mulholand ca4451d914 Update Travis to enable go mod 
- set GO111MODULE=on
- use go 1.12
 2019-07-23 15:28:08 -05:00
Rob Mulholand f51f5c64a0 Replace dep with go mod 2019-07-23 15:28:08 -05:00
Rob Mulholand 987abd4b2e Update Geth to 1.9.0 2019-07-23 15:26:18 -05:00
Rob Mulholand af6190ea09 Merge pull request #113 from vulcanize/plugin-updates 
Plugin updates
 2019-06-27 14:11:12 -05:00
Rob Mulholand 0f95267e84 Use goose library for running plugin migrations 
- Removes dependency on goose binary existing in plugin execution
  environment
 2019-06-27 13:58:58 -05:00
Rob Mulholand f0bcf930f4 Separate tmp migrations dir from home in join 
- enable separator between home and migrations dir
 2019-06-27 13:58:58 -05:00
Rob Mulholand 0558792282 Log error on failure to setup plugin migrations 
- include specific error
 2019-06-27 13:58:58 -05:00
Rob Mulholand 62cca62432 Log error on failure to read config file 
- include actual error to highlight the specific issue
 2019-06-27 13:58:58 -05:00
Rob Mulholand f0a7a7d999 Merge pull request #103 from vulcanize/remove-queued-storage-duplicates 
Don't duplicate queued storage diffs
 2019-06-17 15:55:35 -05:00
Rob Mulholand 7d6629e5b4 Merge pull request #102 from vulcanize/fix-mappings-util 
Fix mappings util
 2019-06-17 15:45:55 -05:00
Rob Mulholand e11f2c8c59 Don't duplicate queued storage diffs 
- currently, if we don't recognize the same diff several times (e.g.
  if you restart the storage diff watcher pointed at the same file),
  we'll add the same row to the queue on each run.
- these changes assure we only queue an unrecognized diff once.
 2019-06-14 11:26:15 -05:00
Rob Mulholand 95fa6280c5 Fix mappings util 
- Remove hex prefix so that the function can accept keys with or
  without a hex prefix and return the same result
 2019-06-13 18:33:22 -05:00
Rob Mulholand 2598704690 Merge pull request #96 from vulcanize/remove-storage-diff-duplicate-handling 
Remove handling of duplicate storage diffs in watcher
 2019-05-20 14:04:50 -05:00
Rob Mulholand e2909797fc Remove handling of duplicate storage diffs in watcher 
- Can push this responsibility down to the transformers
- Update docs to reflect that transformers should handle duplicates
 2019-05-20 13:29:09 -05:00
Rob Mulholand c5cfa85f4c Merge pull request #90 from vulcanize/vdb-563-remove-postgraphile 
(VDB-563) Remove Postgraphile bespoke code
 2019-05-08 12:51:56 -05:00
Rob Mulholand 702e839f5b Update Travis to reflect removing Postgraphile 2019-05-08 09:11:51 -05:00
Rob Mulholand ca354de229 (VDB-563) Remove Postgraphile bespoke code 
- Add documentation for running Postgraphile from their binary
 2019-05-08 09:11:51 -05:00
Rob Mulholand 3b531950f2 Merge pull request #93 from vulcanize/vdb-570-storage-duplicates 
(VDB-570) Handle duplicate storage diffs
 2019-05-02 13:40:44 -05:00
Rob Mulholand 36c4da372b (VDB-570) Handle duplicate storage diffs 
- If processing a new diff for a row that already exists in the DB,
  ignore the error without logging or queueing
- If processing a queued diff for a row that already exists, remove
  it from the queue
 2019-05-02 11:39:16 -05:00
Rob Mulholand 782e3fd095 Merge pull request #86 from vulcanize/vdb-371-recheck-queued-storage 
(VDB-371) recheck queued storage
 2019-05-01 12:49:02 -05:00
Rob Mulholand 6716c3b92a Make queue recheck interval configurable via CLI 2019-05-01 12:32:39 -05:00
Rob Mulholand b036053937 Queue storage diffs if transformer execution fails 
- For any error, not just if key isn't recognized
- Means we don't lose track of diffs on random ephemeral errors
 2019-05-01 12:32:39 -05:00
Rob Mulholand 76ab914bdc Add license 2019-05-01 12:32:39 -05:00
Rob Mulholand d77f3fe012 Don't pass empty row to channel on error 2019-05-01 12:30:37 -05:00
Rob Mulholand 6a86de87b4 (VDB-371) Recheck queued storage 
- Iterate through queued storage at defined interval, popping rows
  from the queue if successfully persisted
 2019-05-01 12:30:37 -05:00
Rob Mulholand bf4b1687a0 Add get/delete functions to storage queue 2019-05-01 12:30:37 -05:00
Rob Mulholand 2d684c5aec Extract storage diff fetching behind an interface 
- Replaces directly reading from a CSV
- Simplifies testing
- Should hopefully make it easier to plug in other sources for storage
  diffs (e.g. differently formatted CSVs, JSON RPC, etc)
 2019-05-01 12:30:36 -05:00
Rob Mulholand 9c06e9387c Merge pull request #82 from vulcanize/fixup-transactions-syncing 
Don't lookup transactions if no log events
 2019-04-18 10:11:45 -05:00
Rob Mulholand 3fd6269b78 Don't lookup transactions if no log events 
- Prevents EOF error on transactions lookup
 2019-04-18 09:53:18 -05:00
Rob Mulholand 13cd7126cc Merge pull request #84 from vulcanize/improve-postgraphile-interaction 
Add support for restricted graphql users
 2019-04-18 09:51:33 -05:00
Rob Mulholand b403e731be Merge pull request #56 from vulcanize/schema-update 
Update schema to remove uncles
 2019-04-08 16:08:22 -05:00
Rob Mulholand fb753a1468 Update schema to remove uncles 2019-04-08 15:16:43 -05:00
Rob Mulholand 4d0f7ee1bb Merge pull request #54 from vulcanize/update-schema 
Update db schema for latest changes
 2019-04-05 16:11:18 -05:00
Rob Mulholand fdd870563e Update db schema for latest changes 2019-04-05 14:51:51 -05:00
Rob Mulholand fea4ce0c7a Merge pull request #51 from vulcanize/cleanup-links 
Cleanup links
 2019-04-02 08:45:08 -05:00
Rob Mulholand a215927234 Remove links to Gitter 2019-04-01 16:04:07 -05:00
Rob Mulholand 39471e1d04 Remove links to Maker 2019-04-01 16:03:35 -05:00
Rob Mulholand 8b5b5b8f17 Remove links to 8th Light 2019-04-01 15:15:08 -05:00
Rob Mulholand 426556ceda Merge pull request #46 from vulcanize/vdb-425-transactions 
(VDB-486) transactions
 2019-03-28 18:00:23 -05:00
Rob Mulholand f5b32a11b0 Improve transaction syncing test coverage 2019-03-28 14:31:17 -05:00
Rob Mulholand 81dfd12665 Address PR comments 2019-03-28 14:31:17 -05:00