cerc-io/ipld-eth-server
18
0
Fork 1
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 37 Wiki Activity
1,675 Commits 12 Branches 52 Tags 80 MiB
0c44882cb2
Commit Graph

860 Commits

Author SHA1 Message Date
Ian Norden
9f81ffa8e0 minor clenaup/adjustments 2021-02-24 16:20:06 -06:00
Ian Norden
211ec12009 cache misses trigger call out to statediffing geth to fill in the gap in Postgres 2021-02-24 10:50:26 -06:00
Ramil Amerzyanov
e92d35b084
update geth statediff to version v1.9.25-statediff-0.0.14 (#27) 
* update geth statediff to version v1.9.25-statediff-0.0.14
run integration tests in github actions

* fix goose install issue

* fix unit test bug.
Added sorting by tx index for transactions and receipts queries
 2021-02-19 23:23:45 +03:00
Ian Norden
b11fb949f5 final touches, test canonicty functions and get tests passing again 2020-11-13 07:40:33 -06:00
Ian Norden
c0a91b9d9f updated canonical header finder functions 2020-11-09 10:29:04 -06:00
Ian Norden
b128f894c4 postgres functions to check if node was removed (e.g. a la EIP158) in a range; update backend to use these to retrieve state and storage leafs in single (albeit complex) SELECT query 2020-10-31 15:00:03 -05:00
Ian Norden
7c06d4b3a1 additional unit tests 2020-10-30 18:21:44 -05:00
Ian Norden
b208281ad6 optimize GetStorageAt; GetStorageAt unit tests 2020-10-30 12:06:41 -05:00
Ian Norden
b664aee621 unit tests for GetTransactionCount, GetTransactionReceipt, GetBalance, and GetCode 2020-10-30 00:44:41 -05:00
Ian Norden
cffceb53db optimize GetTransactionCount, GetBalance, and GetCode to use secondary indexes instead of operating through ethdb where we have to iterate down trie from root to leaf (multiple db lookups) to access account info 2020-10-29 22:07:39 -05:00
Ian Norden
dc25ea7f87 new transaction endpoint unit tests (except for GetTransactionCount which is still TODO because testing its current implementation requires complete state (e.g. like eth_call test) so- ideally- will first refactor it to use state diff 2ndary indexes rather than proceeding through regular ethdb interface) 2020-10-29 15:20:58 -05:00
Ian Norden
e1026d5261 remaining block endpoint unit test; uncle endpoints unit tests 2020-10-29 14:59:09 -05:00
Ian Norden
1d4abcb69b finish cache miss forwarding for new endpoints 2020-10-28 08:54:06 -05:00
Ian Norden
b5d57b6afc the remaining, currently supportable, eth_* endpoints 2020-10-28 08:54:06 -05:00
Ian Norden
7a2ccaa8a7 fixes for test; update readme and gomodules 2020-10-28 08:40:00 -05:00
Ian Norden
16aa9652a5 integrate into serve command 2020-10-28 08:23:17 -05:00
Ian Norden
a8dd77294a graphql service on top of rpc endpoints 2020-10-28 08:23:17 -05:00
Ian Norden
4f4ab1dd4f update eth backend 2020-10-28 08:23:17 -05:00
Ian Norden
e561fd3178 minor cleanup to prom files 2020-10-27 14:18:59 -05:00
Ian Norden
2d0367fe6c update env 2020-10-27 12:57:26 -05:00
Ian Norden
e1cab4fadc forward cache misses (err and/or empty results) to remote node 2020-10-27 12:29:28 -05:00
ramil
909c85b547 monitoring documentation, prometheus config, grafana dashboard 2020-10-21 15:38:48 +03:00
ramil
c5eae5fb75 Merge remote-tracking branch 'remotes/vulc/master' into metrics 
# Conflicts:
#	go.sum
#	pkg/serve/config.go
 2020-10-21 13:53:38 +03:00
Ian Norden
20af343efb fixes after dirty rebase; canonical hash/header finder function is in this commit now 2020-10-20 15:36:51 -05:00
Ian Norden
5b177e716c test chain maker for eth_call testing 2020-10-20 15:29:00 -05:00
Ian Norden
33a0c8e0e7 begin eth_Call backend integration 2020-10-20 15:28:57 -05:00
Ian Norden
71bc3f8e7b ipld retriever 2020-10-20 15:28:57 -05:00
ramil
bf46200c07 revert local changes for testing 2020-10-20 18:54:06 +03:00
ramil
797450dd2a fix tests 2020-10-20 18:48:35 +03:00
ramil
6e127acbf3 upgrade ipld-eth-indexer from v0.2.0-alpha to v0.5.0-alpha and test fixes 2020-10-20 17:42:09 +03:00
Ilnur Galiev
4c18554fbe add websocket and unixsocket counters 2020-10-19 23:00:09 +03:00
Ilnur Galiev
f627c2edfa add prometheus-middlewares for http and ws endpoint 2020-10-19 18:00:55 +03:00
Ilnur Galiev
a0d70b544b add prometheus metrics 2020-10-19 16:07:29 +03:00
ramil
1d57f75dcf docker compose for dapptools, postgraphile, eth-indexer and eth-server 2020-09-06 12:32:11 +03:00
Ian Norden
add6afadd7 fix subscribe command and client (remove rlp encoding of subscription params) 2020-09-02 14:13:51 -05:00
Ian Norden
dbcb2c0cf7 updat dep and serve cmd 2020-09-02 10:19:25 -05:00
Ian Norden
0d28234804 continue refactor/purge 2020-09-01 16:15:31 -05:00
Ian Norden
d645f52e87 watch => serve 2020-08-31 10:59:41 -05:00
Ian Norden
5830df44a4 decouple from sync 2020-08-31 10:47:06 -05:00
Ian Norden
558599dd32 remove btc stuff 2020-08-31 10:42:01 -05:00
Ian Norden
611fc2c11e eth json rpc optimizations; begin ipfs-ethdb integration 2020-08-14 13:14:19 -05:00
Ian Norden
a532e17e21 remove option for publishing through ipfs node interface 2020-08-12 09:26:18 -05:00
Ian Norden
39354b2114 write code to pg-ipfs table, for codehash lookups 2020-08-12 09:26:18 -05:00
Ian Norden
52b5c99760 fix support for eth testnet chains (use correct tx signer) 2020-08-12 09:26:18 -05:00
Ian Norden
7b0b613b92 misc adjustments 2020-08-07 23:17:16 -05:00
Ian Norden
74752d15aa remove diff unique connstraint, overwrite as diff 2020-08-07 23:17:11 -05:00
Ian Norden
2b8e5ad05a fix flaky test 2020-08-05 11:55:36 -05:00
Ian Norden
b9fbd2da53 update docs 2020-08-05 00:23:06 -05:00
Ian Norden
77b7bcc94c adjust everything to work with mh fks 2020-08-04 22:35:05 -05:00
Ian Norden
0ab55ef9d8 distinguish between differential state/storage nodes and eventual ones 2020-07-01 13:44:04 -05:00
Ian Norden
e2bcc06f8a major refactor pt 3 2020-06-29 19:16:52 -05:00
Ian Norden
449d23757e pkg rename 2020-06-29 16:56:25 -05:00
Ian Norden
3dc1bf6955 update ipfs stack, specifically dshelp.CidToDsKey => dshelp.MultihashToDsKey 2020-06-22 13:15:29 -05:00
Ian Norden
9f3e4f1102 finish updating documentation after refactor 2020-06-20 10:59:32 -05:00
Ian Norden
fe083c12f9 trimming down to ipfs watchers 2020-05-29 22:02:47 -05:00
Ian Norden
421987ff72 update version in go.mod 2020-05-22 19:22:52 -05:00
Ian Norden
0101c4791a pair with new statediffing geth version; travis tests will fail til release is up 2020-05-21 15:34:43 -05:00
Ian Norden
5fb1cc0696 refactor- focus on simplifying concurrent fetching; graceful shutdown for superNode command 2020-05-12 15:34:25 -05:00
Ian Norden
3320df264f avoid decode error on storage branch nodes when element won't decode into byte array 2020-05-12 09:32:33 -05:00
Ian Norden
e2ccd3ffdb update configs 2020-05-11 20:50:19 -05:00
Ian Norden
8dc31e4ca5 make connection pool params configurable 2020-05-11 10:50:05 -05:00
Ian Norden
fb6fdfcc13 separate db conneciton pools for each process 2020-05-06 16:16:56 -05:00
Ian Norden
dfb66eb67b update version; minor refactoring 2020-05-04 12:14:57 -05:00
Ian Norden
9b0ded692b clean up db transaction rollback/commit handling 2020-05-01 17:00:00 -05:00
Ian Norden
1541c62ddc eth api tests needed to be updated too 2020-05-01 11:12:01 -05:00
Ian Norden
ff5472ccf5 direct fetching of iplds from pg-ipfs for btc and eth 2020-04-30 17:22:22 -05:00
Ian Norden
059664ffc9 combined/directp publish and indexing for btc; tests 2020-04-30 17:22:22 -05:00
Ian Norden
65da8cafe7 update higher level interfaces and constructs 2020-04-30 17:22:22 -05:00
Ian Norden
475ca3e6d1 publish and index test for eth 2020-04-30 17:22:22 -05:00
Ian Norden
680a86aee2 combine publishing and indexing for eth; write directly to pg and avoid need for ipfs node 2020-04-30 17:22:22 -05:00
Ian Norden
d345e6b954 reduce chan buffer size, make circular buffer 2020-04-30 17:22:22 -05:00
Ian Norden
9db0fb219a ipfs mode enum; configuration 2020-04-30 17:22:18 -05:00
Ian Norden
ef77688145 additional logging at initialization 2020-04-28 22:08:46 -05:00
Ian Norden
43c254b5f6 fixes after tests 2020-04-20 11:16:08 -05:00
Ian Norden
eceaa0aecb use configurable timeout for geth batch http requests; additional error log info in payload fetchers 2020-04-20 11:05:12 -05:00
Ian Norden
35e625695e updates to documentation for super node 2020-04-19 18:28:36 -05:00
Ian Norden
79c3078c36 version command; log resync and super node configs; fix logfile cli flag 2020-04-14 12:36:07 -05:00
Ian Norden
f2e853e9c9 add back raw contract address 2020-04-04 15:45:55 -05:00
Ian Norden
8960cde4f7 backfill based on validation level 2020-04-04 15:45:54 -05:00
Ian Norden
649fd54a9f store receipt contract hash instead of contract 2020-04-04 15:45:54 -05:00
Ian Norden
00031e2b83 resync reset validation level 2020-04-04 15:45:54 -05:00
Ian Norden
060e3430c1 times_validated field in header 2020-04-04 15:45:54 -05:00
Ian Norden
aa9f78a028 new fields in eth.receipt_cids to index contract addresses seen in 
logs; handle null dst/contract addrs in trxs/rcts
 2020-04-04 15:45:54 -05:00
Ian Norden
657abbb7ec vacuum analyze public.blocks after cleanup 2020-03-26 04:25:00 -05:00
Ian Norden
834e2bcff4 additional logging to resync service; rename state/storage key to leaf key and allow insert of null values for intermediate nodes 2020-03-25 19:16:26 -05:00
Ian Norden
e570a2c9d4 VACUUM ANALYZE after cleaning out old data 2020-03-25 19:16:26 -05:00
Ian Norden
716cc3663f don't throw error on duplicate key insert 2020-03-25 19:16:26 -05:00
Ian Norden
1aded75645 additional log.info for data streamed at head 2020-03-22 09:45:49 -05:00
Ian Norden
f81c440418 for some reason every now and then a branch node would throw a panic here; still need to figure out why; quick fix 2020-03-22 09:05:26 -05:00
Ian Norden
b2d5345c02 add new fields to test data to get tests passing again 2020-03-21 21:10:28 -05:00
Ian Norden
ba5d6f9b9f state account support and add roots and additional metadata to header 
index
 2020-03-21 20:57:02 -05:00
Ian Norden
57bdcca43c combine resync and supernode configs; load config params from env variables if available; allow config params to be passed in as cli flags 2020-03-20 13:15:50 -05:00
Ian Norden
c72dc273ba misc fixes; config files for resyncing 2020-03-17 19:42:53 -05:00
Ian Norden
d47ba24373 adjust btc and eth publishers and publisher tests 2020-03-17 13:05:19 -05:00
Ian Norden
1e2b09045a fix state and storage iplds/dag_putters; add support for eth rct trie and eth and btc tx tries 2020-03-17 11:18:20 -05:00
Ian Norden
e7225317d4 split btc and eth super node processes into serpate containers; dockerfile for Postgraphile with plugins 2020-03-17 08:37:00 -05:00
Ian Norden
36b714b6df resync config initializer 2020-03-17 07:51:00 -05:00
Ian Norden
b934f6c963 btc and eth unit tests for db cleaning 2020-03-17 07:50:38 -05:00
Ian Norden
f814b64543 bug fix 2020-03-12 13:59:23 -05:00
Ian Norden
77b1c25bc1 resync service 2020-03-12 13:52:23 -05:00
Ian Norden
b4005eff39 db cleaners to eth and btc 2020-03-12 13:51:58 -05:00
Ian Norden
e5c5422edc adhjustments to work with statediffing geth v1.10-alpha.2 2020-03-11 13:41:59 -05:00
Ian Norden
be875c0100 db comments to resolve postgraphile type conflicts 2020-03-10 09:38:07 -05:00
Ian Norden
aad318c67b improved logging in super node service 2020-03-08 11:47:49 -05:00
Ian Norden
8c2a71b16f finish wasm watcher engine 2020-02-27 15:09:20 -06:00
Ian Norden
330a083749 watcher configuration; super node versioning 2020-02-25 16:38:46 -06:00
Ian Norden
25aa4634e9 work on wasm watchers 2020-02-24 12:54:10 -06:00
Ian Norden
e3e8700d34 api test; return full ipld models to subscribers 2020-02-23 17:15:26 -06:00
Ian Norden
fb360d8562 changes to super node to improve compatibility with watcher 2020-02-23 17:15:26 -06:00
Ian Norden
f0c5ff8077 begin wasm watcher engine 2020-02-23 17:15:26 -06:00
Ian Norden
94aefafd7c more explicity payload err and msg 2020-02-23 17:15:26 -06:00
Ian Norden
2fbf97da9d wasm instantiator 2020-02-23 17:15:26 -06:00
Ian Norden
5f85a7ecf6 more descriptive comments/method names for super node service; send msg 
when backfill is finished
 2020-02-23 17:15:26 -06:00
Ian Norden
48fb5bcd27 separate super node instances for different chains need to be within the same process to avoid contention over ipfs repo lock 2020-02-20 16:14:17 -06:00
Ian Norden
ef3b043f97 emulate btc data streamer over http; misc fixes 2020-02-20 16:14:17 -06:00
Ian Norden
5173edf563 fix broken go-ipld-eth trie node dag putters 2020-02-20 16:14:17 -06:00
Ian Norden
ca273a026d major refactor part 2: remove cold import, full sync, generalize node table 2020-02-20 16:14:17 -06:00
Ian Norden
0f765df12c finish constructors 2020-02-20 16:14:17 -06:00
Ian Norden
5d46e02509 tx inputs directly reference tx ouputs 2020-02-20 16:14:17 -06:00
Ian Norden
642e08a04b btc converter, publisher, and indexer unit tests 2020-02-20 16:14:17 -06:00
Ian Norden
8643a7f3b6 btc ipld fetcher, paylaod fetcher, and ipld resolver 2020-02-20 16:14:17 -06:00
Ian Norden
aa2f8bd0a5 btc retriever and filterer 2020-02-20 16:14:17 -06:00
Ian Norden
f33cc3f34b TxOutputs: extract and index pkscript metadata (script type, addresses, #required sigs); TxInputs: outpoint_hash => outpoint_tx_id that references transaction_cids.id 2020-02-20 16:14:17 -06:00
Ian Norden
48f70d4ddf btc mocks; reorg 2020-02-20 16:14:17 -06:00
Ian Norden
808f1b5662 btc cid indexer 2020-02-20 16:14:17 -06:00
Ian Norden
5094b975fc postgres schema and tables for btc 2020-02-20 16:14:17 -06:00
Ian Norden
9018e551ba move eth tables under eth schema 2020-02-20 16:14:17 -06:00
Ian Norden
4dde90447e wrap generic top-level interfaces with loose types 2020-02-20 16:14:17 -06:00
Ian Norden
68d8e68144 btc ipld publisher 2020-02-20 16:14:17 -06:00
Ian Norden
8099f726c3 internalized ipld types and dag_putters; adjust converters/publishers to work with them 2020-02-20 16:14:17 -06:00
Ian Norden
2e81f8d31a btc payload converter 2020-02-20 16:14:17 -06:00
Ian Norden
076903b174 btc payload streamer 2020-02-20 16:14:17 -06:00
Ian Norden
da844b0b83 reorg pkg/ to prepare to support chains other than ethereumm 2020-02-20 16:14:16 -06:00
Ian Norden
33ac5978f5 split uncles into their own table; add tx index to the tx index (lol); continue work on geth api endpoints 2020-01-27 14:14:03 -06:00
Ian Norden
e52284b9ba update documentation, bug/error fixes, work on eth api for drawbridge 2020-01-24 15:37:52 -06:00
Ian Norden
7843312815 major refactoring of super_node to make it easier to support other chains 2020-01-24 15:37:52 -06:00
Ian Norden
5ff05225c2 begin eth api recapitulation 2020-01-24 15:37:52 -06:00
Ian Norden
0785507a7d support for total difficulty (needed to support some eth endpoints) 2020-01-24 15:37:52 -06:00
Ian Norden
358575335b expand retriever, fetcher, and resolver interfaces in prep for eth api mimick 2020-01-24 15:37:52 -06:00
Rob Mulholand
56ce8bdb41 (VDB-950) Write raw diffs before transforming 
- Raw field we can reference by FK for related data
- Enables replay for unwatched or mistransformed diffs
 2019-12-03 14:51:17 -06:00
Ian Norden
74a90fd124 use Debugf instead of strconv 2019-12-02 23:37:01 -06:00
yaoandrew
5a1f599a4d Clean up files after rebase 2019-12-02 23:36:00 -06:00
yaoandrew
614ea3bba1 Adds logging for storage and RPC 2019-12-02 23:10:32 -06:00
yaoandrew
0b9f29c300 Removes import alias for logrus 2019-12-02 23:08:48 -06:00
Ian Norden
4036d9d6a0 review fixes 2019-12-02 13:49:27 -06:00
Ian Norden
5be205ffa6 super node backfill breaks batch call into smaller bins; retrieve gap 
test
 2019-12-02 13:24:58 -06:00
Ian Norden
c16ac026db logWithCommand; rebase fixes; config for testing super node subscription 2019-12-02 13:24:58 -06:00
Ian Norden
67df8dea77 header_cids.final => header_cids.uncle 2019-12-02 13:24:58 -06:00
Ian Norden
3a666df294 backfiller refactoring; explicity errs; golint 2019-12-02 13:24:58 -06:00
Ian Norden
40c3aff597 seed => super; port 80 => port 8080; backfill process for the super_node 2019-12-02 13:24:58 -06:00