cerc-io/go-ethereum
12
0
Fork 0
Code Issues 9 Pull Requests Packages Projects Releases 66 Wiki Activity

Separate out logs into own table #79

New Issue
Closed
opened 2021-06-01 09:59:35 +00:00 by ashwinphatak · 7 comments
ashwinphatak commented 2021-06-01 09:59:35 +00:00 (Migrated from github.com)
Copy Link
Copy Source

Currently, logs don't have their own table and are stored in additional columns in the receipts table. It's desirable to create a separate table and IPLD schema for them, allowing for more efficient query and distribution of individual logs.

DEPENDENCY: https://github.com/vulcanize/go-codec-dageth/issues/25.

After finishing https://github.com/vulcanize/go-codec-dageth/issues/25 it will be known precisely how we want to represent logs as IPLD objects.

Currently, logs don't have their own table and are stored in additional columns in the receipts table. It's desirable to create a separate table and IPLD schema for them, allowing for more efficient query and distribution of individual logs. ---- DEPENDENCY: https://github.com/vulcanize/go-codec-dageth/issues/25. After finishing https://github.com/vulcanize/go-codec-dageth/issues/25 it will be known precisely how we want to represent logs as IPLD objects.
i-norden commented 2021-06-11 13:54:37 +00:00
Member
Copy Link
Copy Source

When we create the new logs table, we need to also create a process to backfill the existing data.

When we create the new logs table, we need to also create a process to backfill the existing data.
👍 1
AlexBinesh commented 2021-06-11 16:34:42 +00:00 (Migrated from github.com)
Copy Link
Copy Source

Had a meeting with Ian, Rick and Ashwin. Agreed on the following structure for a new table to hold the Log data as described below:
Auto Index,

Receipt_cids.id
log.address,
(Name log.address as Address ),
log.Topics 0 (Varchar 66),
log.Topics 1 (Varchar 66),
log.Topics 2 (Varchar 66),
log.Topics 3 (Varchar 66),
Data

Additionally: Add B Tree indexes for the Topic columns and Data columns

Also : When we create the new logs table, we need to also create a process to backfill the existing data.

Had a meeting with Ian, Rick and Ashwin. Agreed on the following structure for a new table to hold the Log data as described below: Auto Index, <Position Of the log in the reciept> Receipt_cids.id log.address, (Name log.address as Address ), log.Topics 0 (Varchar 66), log.Topics 1 (Varchar 66), log.Topics 2 (Varchar 66), log.Topics 3 (Varchar 66), Data Additionally: Add B Tree indexes for the Topic columns and Data columns Also : When we create the new logs table, we need to also create a process to backfill the existing data.
AlexBinesh commented 2021-06-14 20:06:59 +00:00 (Migrated from github.com)
Copy Link
Copy Source

Worked on the coding. Have to parse the Logs in the receipt next

Worked on the coding. Have to parse the Logs in the receipt next
AlexBinesh commented 2021-06-15 19:38:31 +00:00 (Migrated from github.com)
Copy Link
Copy Source

Created a local Logs table with the structure below using the following SQL statements:

CREATE TABLE Logs (
id SERIAL,
CONSTRAINT receipt_id
FOREIGN KEY(id)
REFERENCES eth.receipt_cids(id),

address char(42),
Topics0 Varchar (66),
Topics1 Varchar (66),
Topics2 Varchar (66),
Topics3 Varchar (66),
Data bytea);

CREATE UNIQUE INDEX topic0_idx ON Logs (Topics0);
CREATE UNIQUE INDEX topic1_idx ON Logs (Topics1);
CREATE UNIQUE INDEX topic2_idx ON Logs (Topics2);
CREATE UNIQUE INDEX topic3_idx ON Logs (Topics3);
CREATE UNIQUE INDEX address_idx ON Logs (address);

Table "public.logs"
Column | Type | Collation | Nullable | Default
---------+-----------------------+-----------+----------+----------------------------------
id | integer | | not null | nextval('logs_id_seq'::regclass)
address | character(42) | | |
topics0 | character varying(66) | | |
topics1 | character varying(66) | | |
topics2 | character varying(66) | | |
topics3 | character varying(66) | | |
data | bytea | | |
Indexes:
"address_idx" UNIQUE, btree (address)
"topic0_idx" UNIQUE, btree (topics0)
"topic1_idx" UNIQUE, btree (topics1)
"topic2_idx" UNIQUE, btree (topics2)
"topic3_idx" UNIQUE, btree (topics3)
Foreign-key constraints:
"receipt_id" FOREIGN KEY (id) REFERENCES eth.receipt_cids(id)

Created a local Logs table with the structure below using the following SQL statements: CREATE TABLE Logs ( id SERIAL, CONSTRAINT receipt_id FOREIGN KEY(id) REFERENCES eth.receipt_cids(id), address char(42), Topics0 Varchar (66), Topics1 Varchar (66), Topics2 Varchar (66), Topics3 Varchar (66), Data bytea); CREATE UNIQUE INDEX topic0_idx ON Logs (Topics0); CREATE UNIQUE INDEX topic1_idx ON Logs (Topics1); CREATE UNIQUE INDEX topic2_idx ON Logs (Topics2); CREATE UNIQUE INDEX topic3_idx ON Logs (Topics3); CREATE UNIQUE INDEX address_idx ON Logs (address); Table "public.logs" Column | Type | Collation | Nullable | Default ---------+-----------------------+-----------+----------+---------------------------------- id | integer | | not null | nextval('logs_id_seq'::regclass) address | character(42) | | | topics0 | character varying(66) | | | topics1 | character varying(66) | | | topics2 | character varying(66) | | | topics3 | character varying(66) | | | data | bytea | | | Indexes: "address_idx" UNIQUE, btree (address) "topic0_idx" UNIQUE, btree (topics0) "topic1_idx" UNIQUE, btree (topics1) "topic2_idx" UNIQUE, btree (topics2) "topic3_idx" UNIQUE, btree (topics3) Foreign-key constraints: "receipt_id" FOREIGN KEY (id) REFERENCES eth.receipt_cids(id)
i-norden commented 2021-07-26 19:05:13 +00:00
Member
Copy Link
Copy Source

Related issue: https://github.com/vulcanize/go-codec-dageth/issues/25

Related issue: https://github.com/vulcanize/go-codec-dageth/issues/25
i-norden commented 2021-08-04 03:19:07 +00:00
Member
Copy Link
Copy Source

We need to

  1. Remove the current log rows in the receipt_cids table and add a new VARCHAR(66) row name log_root (should eventually be NOT NULL, but we need to backfill the old missing values first).
  2. Create a new table, log_cids, for indexing log IPLDs, rows in this table reference the parent receipt by fk.
  3. When processing receipts, iterate over their logs in the same manner that we iterate over txs and rcts to materialize their tries.
  4. Write the root hash of the materialized logTrie to the new log_root row in the receipt_cids table.
  5. Publish all the logTrie nodes to the IPLD blockstore like we do with the txTrie and rctTrie nodes, we don't index each individual node for these tries. Use 0x99 as the multicodec type for the CID, but it's not official yet.
  6. RLP encode and publish each log as an IPLD, in analogous fashion to how we encode and publish each individual tx and rct. Use 0x9a as the multicodec type for the CID, but it's not official yet.
  7. Index these log IPLDs with the new log_cids table by referencing them by multihash fk.
We need to 1. Remove the current log rows in the receipt_cids table and add a new `VARCHAR(66)` row name `log_root` (should eventually be `NOT NULL`, but we need to backfill the old missing values first). 2. Create a new table, `log_cids`, for indexing log IPLDs, rows in this table reference the parent receipt by fk. 3. When processing receipts, iterate over their logs in the same manner that we iterate over txs and rcts to materialize their tries. 4. Write the root hash of the materialized logTrie to the new `log_root` row in the `receipt_cids` table. 5. Publish all the logTrie nodes to the IPLD blockstore like we do with the txTrie and rctTrie nodes, we don't index each individual node for these tries. Use `0x99` as the multicodec type for the CID, but it's not official yet. 6. RLP encode and publish each log as an IPLD, in analogous fashion to how we encode and publish each individual tx and rct. Use `0x9a` as the multicodec type for the CID, but it's not official yet. 7. Index these log IPLDs with the new `log_cids` table by referencing them by multihash fk.
i-norden commented 2021-08-20 15:35:18 +00:00
Member
Copy Link
Copy Source

Linking to https://github.com/vulcanize/statediff-migrations/pull/7
Linking to https://github.com/vulcanize/go-ethereum/pull/102

Linking to https://github.com/vulcanize/statediff-migrations/pull/7 Linking to https://github.com/vulcanize/go-ethereum/pull/102
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
critical
duplicate
enhancement
epic
help wanted
in progress
invalid
low priority
question
rebase
v1
v5
wontfix
Copied from Github
An issue or PR manually copied from GitHub.
 Kind/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
Reviewed
Confirmed
Issue has been confirmed
Reviewed
Duplicate
This issue or pull request already exists
Reviewed
Invalid
Invalid issue
Reviewed
Won't Fix
This issue won't be fixed
Status
Abandoned
Somebody has started to work on this but abandoned work
Status
Blocked
Something is blocking this issue or pull request
Status
Need More Info
Feedback is required to reproduce issue or to continue work
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
AFDudley (A. F. Dudley) Alex ashwin cerccicd ericlewis i-norden (Ian Norden) prathamesh (Prathamesh Musale) rachmaninovquar rjk-laconic roysc (Roy Crihfield) srw (srw) zramsay
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cerc-io/go-ethereum#79