cerc-io/ipld-eth-server
17
0
Fork 1
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 37 Wiki Activity
1,447 Commits 12 Branches 52 Tags 80 MiB
75debec01a
Commit Graph

306 Commits

Author SHA1 Message Date
Ian Norden
57bdcca43c combine resync and supernode configs; load config params from env variables if available; allow config params to be passed in as cli flags 2020-03-20 13:15:50 -05:00
Ian Norden
1d4b37aca9 split running migrations into separate compose step; docker-compose files for different processes 2020-03-18 12:21:29 -05:00
Ian Norden
c72dc273ba misc fixes; config files for resyncing 2020-03-17 19:42:53 -05:00
Ian Norden
e7225317d4 split btc and eth super node processes into serpate containers; dockerfile for Postgraphile with plugins 2020-03-17 08:37:00 -05:00
Ian Norden
93e7eb35c5 resync command 2020-03-17 07:51:39 -05:00
Ian Norden
e5c5422edc adhjustments to work with statediffing geth v1.10-alpha.2 2020-03-11 13:41:59 -05:00
Ian Norden
8c2a71b16f finish wasm watcher engine 2020-02-27 15:09:20 -06:00
Ian Norden
330a083749 watcher configuration; super node versioning 2020-02-25 16:38:46 -06:00
Ian Norden
25aa4634e9 work on wasm watchers 2020-02-24 12:54:10 -06:00
Ian Norden
e3e8700d34 api test; return full ipld models to subscribers 2020-02-23 17:15:26 -06:00
Ian Norden
5f85a7ecf6 more descriptive comments/method names for super node service; send msg 
when backfill is finished
 2020-02-23 17:15:26 -06:00
Ian Norden
48fb5bcd27 separate super node instances for different chains need to be within the same process to avoid contention over ipfs repo lock 2020-02-20 16:14:17 -06:00
Ian Norden
ef3b043f97 emulate btc data streamer over http; misc fixes 2020-02-20 16:14:17 -06:00
Ian Norden
ca273a026d major refactor part 2: remove cold import, full sync, generalize node table 2020-02-20 16:14:17 -06:00
Ian Norden
48f70d4ddf btc mocks; reorg 2020-02-20 16:14:17 -06:00
Ian Norden
4dde90447e wrap generic top-level interfaces with loose types 2020-02-20 16:14:17 -06:00
Ian Norden
8099f726c3 internalized ipld types and dag_putters; adjust converters/publishers to work with them 2020-02-20 16:14:17 -06:00
Ian Norden
da844b0b83 reorg pkg/ to prepare to support chains other than ethereumm 2020-02-20 16:14:16 -06:00
Ian Norden
33ac5978f5 split uncles into their own table; add tx index to the tx index (lol); continue work on geth api endpoints 2020-01-27 14:14:03 -06:00
Ian Norden
e52284b9ba update documentation, bug/error fixes, work on eth api for drawbridge 2020-01-24 15:37:52 -06:00
Ian Norden
308ccb5d8c refactor super node commands into one 2020-01-24 15:37:52 -06:00
Ian Norden
7843312815 major refactoring of super_node to make it easier to support other chains 2020-01-24 15:37:52 -06:00
yaoandrew
5a1f599a4d Clean up files after rebase 2019-12-02 23:36:00 -06:00
Andrew J Yao
e26fa12af0 Adds logging for transformer start, log conversions, log persistance 2019-12-02 22:59:32 -06:00
Ian Norden
4036d9d6a0 review fixes 2019-12-02 13:49:27 -06:00
Ian Norden
5be205ffa6 super node backfill breaks batch call into smaller bins; retrieve gap 
test
 2019-12-02 13:24:58 -06:00
Ian Norden
c16ac026db logWithCommand; rebase fixes; config for testing super node subscription 2019-12-02 13:24:58 -06:00
Ian Norden
67df8dea77 header_cids.final => header_cids.uncle 2019-12-02 13:24:58 -06:00
Ian Norden
6880611436 command for serving data without an ongoing sync process 2019-12-02 13:24:58 -06:00
Ian Norden
3a666df294 backfiller refactoring; explicity errs; golint 2019-12-02 13:24:58 -06:00
Ian Norden
40c3aff597 seed => super; port 80 => port 8080; backfill process for the super_node 2019-12-02 13:24:58 -06:00
Ian Norden
b789ab69e6 adjustments after rebase 2019-12-02 13:24:58 -06:00
Ian Norden
8fe273fb7b getter rpc method for the seed node's geth info 2019-12-02 13:24:58 -06:00
Ian Norden
8b31d12716 fixes after rebase 2019-12-02 13:24:58 -06:00
Ian Norden
35c8f3561a publisher unit test 2019-12-02 13:24:58 -06:00
Ian Norden
0bbb7a30d1 review fixes 2019-12-02 13:24:58 -06:00
Ian Norden
f2efbb5d01 worker pool for handling concurrent ipld putting and cid indexing without overloading Postgres connections 2019-12-02 13:24:58 -06:00
Ian Norden
3fa33fb767 index receipts by the contract address 2019-12-02 13:24:58 -06:00
Ian Norden
23a21c14f3 make subscription config rlp encodable, group subs of the 
same type using a hash of their config, process only once for each sub
type instead of for every sub.
 2019-12-02 13:24:58 -06:00
Ian Norden
b76bdc51f2 fixing dockerfile and start up script 2019-12-02 13:24:54 -06:00
Ian Norden
5356cd50bb seed node documentation 2019-12-02 13:24:51 -06:00
Ian Norden
e7cdd6247e update to use ReceiptForStorage; expose rpc server over ws 2019-12-02 13:24:51 -06:00
Ian Norden
723c7c6244 fix backfill operations and dependency issue; hopefully travis will work now 2019-12-02 13:24:51 -06:00
Ian Norden
4baea2923c make db fks deferrable so that we can commit entire cid payload in single transaction; adjust buffer sizes to optimize performane and stability 2019-12-02 13:24:51 -06:00
Ian Norden
b1bb646ad5 goimports + streamSubscribe command for raw access to the seed node data 2019-12-02 13:24:51 -06:00
Ian Norden
8ccdfd4835 fix streamFilters issue 2019-12-02 13:24:51 -06:00
Ian Norden
b5099a5051 demo commands 2019-12-02 13:24:51 -06:00
Ian Norden
2db0ce971d vulcanizedb seed node streamer and syncPublishScreenAndServe command 2019-12-02 13:24:51 -06:00
Ian Norden
34393ffb3f fixes for issues uncovered in integration 2019-12-02 13:24:51 -06:00
Ian Norden
5ebe2243d8 goimports -w 2019-12-02 13:24:50 -06:00
Ian Norden
15e044403d work on mocks and unit tests 2019-12-02 13:24:50 -06:00
Ian Norden
d702cb720c update dependencies to work with update eth-block-extractor 2019-12-02 13:24:49 -06:00
Ian Norden
31a9017c4f index all cids in a payload in a single atomic tx; misc fixes; comment additions 2019-12-02 13:24:46 -06:00
Ian Norden
79efaeb089 syncAndPublish command 2019-12-02 13:24:46 -06:00
Ian Norden
6434a7279d integrate backfill into storage watcher; documentation for storage backfill 2019-12-02 13:24:45 -06:00
Ian Norden
db0f024088 review fixes 2019-12-02 11:26:44 -06:00
Ian Norden
a834e55b9f command edits; external pkg for finding min deployment block; gofmt 2019-12-02 11:22:18 -06:00
Ian Norden
37f4a2d603 integrate backfill into storage watcher; documentation for storage backfill 2019-12-02 11:06:28 -06:00
Ian Norden
a59bd06a37 goimports -w; comments; refactor test 2019-12-02 11:03:36 -06:00
Rob Mulholand
e1236b4072 Prefer all caps for initialisms and acronyms 2019-10-31 13:42:19 -05:00
Edvard Hübinette
3fff2896aa
Rename geth to eth, signifying client independence (#161) 2019-10-28 12:30:24 +01:00
Elizabeth Engelman
f0d2741dea Allow storageDiff source to be set through an env variable 2019-10-01 10:24:53 -05:00
Rob Mulholand
d06dddbfaa Always hash storage diff contract addresses 
- Enables syncing Geth and Parity diffs with same transformer lookup
- Maybe worth always hashing the storage key so we don't need a hashed
  and not-hashed version in the key lookups?
 2019-09-25 16:36:08 -05:00
Elizabeth Engelman
639c561fdf Remove unncessary pointer receiver from NewGethRpcStorageFetcher.FetchStorageDiffs 2019-09-25 16:32:31 -05:00
Elizabeth Engelman
0cc90a1f80 Factor out an IStorageWatcher interface 2019-09-25 16:32:31 -05:00
Elizabeth Engelman
b09b8a8735 Use geth state diff source in execute command 2019-09-25 16:32:27 -05:00
Elizabeth Engelman
a577811e0a Use geth state diff source in composeAndExecute 2019-09-25 16:32:27 -05:00
Matt Griswold
20ce0ab852 Revert "option to write the plugin .go file and run the plugin migrations without building the .so file so that it can be done from the shell instead to afford complete error messages" 
This reverts commit 9bb2f27a69.
 2019-09-24 23:54:25 +00:00
Ian Norden
9bb2f27a69 option to write the plugin .go file and run the plugin migrations without building the .so file so that it can be done from the shell instead to afford complete error messages 2019-09-24 18:44:04 -05:00
Rob Mulholand
4fa19be90a Return error when no logs/headers available 
- Replaces bool and moots question of error/bool ordering
- Also make event watcher execution synchronous
 2019-09-18 20:55:15 -05:00
Rob Mulholand
13d503b851 Distinguish between missing and unchecked headers 
- Missing == not in DB
- Unchecked == logs haven't been fetched
 2019-09-10 14:39:45 -05:00
Rob Mulholand
666ea1c325 Update checked headers for new transformers 
- If a header was marked as checked before a transformer was added to
  the watcher, mark all headers since the new transformer's starting
  block number as unchecked.
 2019-08-28 09:25:14 -05:00
Rob Mulholand
63dabbb051 Extract and delegate logs concurrently 2019-08-28 09:25:13 -05:00
Elizabeth Engelman
1b3786338f Address PR feedback 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
58c23c6632 Update Copyrights 2019-08-23 10:11:37 -05:00
Elizabeth Engelman
7e38764618 Rename ReceiptRepository -> FullSyncReceiptRepository 2019-08-23 10:11:37 -05:00
Andrew J Yao
ee77fc6521 Remove injection of SubCommand for logs 2019-07-23 13:11:20 -07:00
Andrew J Yao
a188e1dd79 Incorporate changes from Edvard PR 2019-07-23 08:10:58 -07:00
Andrew J Yao
92d153b010 Add subCommand name to log fields 2019-07-23 08:09:57 -07:00
Andrew J Yao
d7a82e353b Add calling method to debug level log messages 2019-07-23 08:08:15 -07:00
Edvard Hübinette
2c092e8d04
[VDB-751 VDB-754] Bugfix null pointer panic and improve logging (#119) 
* VDB-751 VDB-754 Bugfix null pointer panic and improve logging

* Fix typo
 2019-07-18 09:21:40 +02:00
Rob Mulholand
62cca62432 Log error on failure to read config file 
- include actual error to highlight the specific issue
 2019-06-27 13:58:58 -05:00
Andrew J Yao
11a65c525c Adds ability to set log level from toml or CLI args 2019-06-27 11:14:11 -07:00
Rob Mulholand
e2909797fc Remove handling of duplicate storage diffs in watcher 
- Can push this responsibility down to the transformers
- Update docs to reflect that transformers should handle duplicates
 2019-05-20 13:29:09 -05:00
Elizabeth Engelman
d947c8f30a Rename sync command to fullSync 2019-05-13 16:14:56 -05:00
Elizabeth Engelman
f7d520c933 Update header sync transformer alias in ContractWatcher 2019-05-08 13:42:43 -05:00
Rob Mulholand
36c4da372b (VDB-570) Handle duplicate storage diffs 
- If processing a new diff for a row that already exists in the DB,
  ignore the error without logging or queueing
- If processing a queued diff for a row that already exists, remove
  it from the queue
 2019-05-02 11:39:16 -05:00
Rob Mulholand
6716c3b92a Make queue recheck interval configurable via CLI 2019-05-01 12:32:39 -05:00
Rob Mulholand
6a86de87b4 (VDB-371) Recheck queued storage 
- Iterate through queued storage at defined interval, popping rows
  from the queue if successfully persisted
 2019-05-01 12:30:37 -05:00
Rob Mulholand
2d684c5aec Extract storage diff fetching behind an interface 
- Replaces directly reading from a CSV
- Simplifies testing
- Should hopefully make it easier to plug in other sources for storage
  diffs (e.g. differently formatted CSVs, JSON RPC, etc)
 2019-05-01 12:30:36 -05:00
Gabe Laughlin
a344432156
(VDB-560) Rename lightSync to headerSync 2019-05-01 12:12:55 -05:00
Ian Norden
1aa849bcb4 PR fixes; remove all infura token references and setup travis to use encrypted env 
variable; rest of the ethjson_rpc dependent tests extracted to integration_test
 2019-03-21 18:36:51 -05:00
Ian Norden
d3e172d9ab remove references to infura token except for in infura.toml so that tests will continue to pass on CI 2019-03-21 18:36:49 -05:00
Ian Norden
37fc038605 fixes for review comments 2019-03-21 18:36:49 -05:00
Ian Norden
24879a85aa mv pkg/omni pkg/contract_watcher 2019-03-21 18:33:56 -05:00
Ian Norden
073378de6d update © to 2019 2019-03-21 18:33:56 -05:00
Ian Norden
e4e092f542 changes to plugin and commands to accomodate changes 2019-03-21 18:33:56 -05:00
Ian Norden
8174ce4aee refactor omni code; rename omniWatcher => contractWatcher; use config instead of excessive number of CLI flags 2019-03-21 18:30:48 -05:00
Ian Norden
ff55e3ba7a TransformerInitializer => EventTransformerInitializer reorganization 2019-03-14 11:59:39 -05:00
Ian Norden
9f8c50e3ab tests for migration path ordering and errors 2019-03-13 11:42:30 -05:00