Separate out logs into own table #79

New Issue

i-norden · 2021-06-01T09:59:35Z

ashwinphatak commented

2021-06-01 09:59:35 +00:00

(Migrated from github.com)

Currently, logs don't have their own table and are stored in additional columns in the receipts table. It's desirable to create a separate table and IPLD schema for them, allowing for more efficient query and distribution of individual logs.

DEPENDENCY: https://github.com/vulcanize/go-codec-dageth/issues/25.

After finishing https://github.com/vulcanize/go-codec-dageth/issues/25 it will be known precisely how we want to represent logs as IPLD objects.

Currently, logs don't have their own table and are stored in additional columns in the receipts table. It's desirable to create a separate table and IPLD schema for them, allowing for more efficient query and distribution of individual logs. ---- DEPENDENCY: https://github.com/vulcanize/go-codec-dageth/issues/25. After finishing https://github.com/vulcanize/go-codec-dageth/issues/25 it will be known precisely how we want to represent logs as IPLD objects.

i-norden commented

2021-06-11 13:54:37 +00:00

When we create the new logs table, we need to also create a process to backfill the existing data.

👍 1

AlexBinesh commented

2021-06-11 16:34:42 +00:00

(Migrated from github.com)

Had a meeting with Ian, Rick and Ashwin. Agreed on the following structure for a new table to hold the Log data as described below:
Auto Index,

Receipt_cids.id
log.address,
(Name log.address as Address ),
log.Topics 0 (Varchar 66),
log.Topics 1 (Varchar 66),
log.Topics 2 (Varchar 66),
log.Topics 3 (Varchar 66),
Data

Additionally: Add B Tree indexes for the Topic columns and Data columns

Also : When we create the new logs table, we need to also create a process to backfill the existing data.

Had a meeting with Ian, Rick and Ashwin. Agreed on the following structure for a new table to hold the Log data as described below: Auto Index, <Position Of the log in the reciept> Receipt_cids.id log.address, (Name log.address as Address ), log.Topics 0 (Varchar 66), log.Topics 1 (Varchar 66), log.Topics 2 (Varchar 66), log.Topics 3 (Varchar 66), Data Additionally: Add B Tree indexes for the Topic columns and Data columns Also : When we create the new logs table, we need to also create a process to backfill the existing data.

AlexBinesh commented

2021-06-14 20:06:59 +00:00

(Migrated from github.com)

Worked on the coding. Have to parse the Logs in the receipt next

AlexBinesh commented

2021-06-15 19:38:31 +00:00

(Migrated from github.com)

Created a local Logs table with the structure below using the following SQL statements:

CREATE TABLE Logs (
id SERIAL,
CONSTRAINT receipt_id
FOREIGN KEY(id)
REFERENCES eth.receipt_cids(id),

address char(42),
Topics0 Varchar (66),
Topics1 Varchar (66),
Topics2 Varchar (66),
Topics3 Varchar (66),
Data bytea);

CREATE UNIQUE INDEX topic0_idx ON Logs (Topics0);
CREATE UNIQUE INDEX topic1_idx ON Logs (Topics1);
CREATE UNIQUE INDEX topic2_idx ON Logs (Topics2);
CREATE UNIQUE INDEX topic3_idx ON Logs (Topics3);
CREATE UNIQUE INDEX address_idx ON Logs (address);

Created a local Logs table with the structure below using the following SQL statements: CREATE TABLE Logs ( id SERIAL, CONSTRAINT receipt_id FOREIGN KEY(id) REFERENCES eth.receipt_cids(id), address char(42), Topics0 Varchar (66), Topics1 Varchar (66), Topics2 Varchar (66), Topics3 Varchar (66), Data bytea); CREATE UNIQUE INDEX topic0_idx ON Logs (Topics0); CREATE UNIQUE INDEX topic1_idx ON Logs (Topics1); CREATE UNIQUE INDEX topic2_idx ON Logs (Topics2); CREATE UNIQUE INDEX topic3_idx ON Logs (Topics3); CREATE UNIQUE INDEX address_idx ON Logs (address); Table "public.logs" Column | Type | Collation | Nullable | Default ---------+-----------------------+-----------+----------+---------------------------------- id | integer | | not null | nextval('logs_id_seq'::regclass) address | character(42) | | | topics0 | character varying(66) | | | topics1 | character varying(66) | | | topics2 | character varying(66) | | | topics3 | character varying(66) | | | data | bytea | | | Indexes: "address_idx" UNIQUE, btree (address) "topic0_idx" UNIQUE, btree (topics0) "topic1_idx" UNIQUE, btree (topics1) "topic2_idx" UNIQUE, btree (topics2) "topic3_idx" UNIQUE, btree (topics3) Foreign-key constraints: "receipt_id" FOREIGN KEY (id) REFERENCES eth.receipt_cids(id)

i-norden commented

2021-07-26 19:05:13 +00:00

Related issue: https://github.com/vulcanize/go-codec-dageth/issues/25

i-norden commented

2021-08-04 03:19:07 +00:00

We need to

Remove the current log rows in the receipt_cids table and add a new VARCHAR(66) row name log_root (should eventually be NOT NULL, but we need to backfill the old missing values first).
Create a new table, log_cids, for indexing log IPLDs, rows in this table reference the parent receipt by fk.
When processing receipts, iterate over their logs in the same manner that we iterate over txs and rcts to materialize their tries.
Write the root hash of the materialized logTrie to the new log_root row in the receipt_cids table.
Publish all the logTrie nodes to the IPLD blockstore like we do with the txTrie and rctTrie nodes, we don't index each individual node for these tries. Use 0x99 as the multicodec type for the CID, but it's not official yet.
RLP encode and publish each log as an IPLD, in analogous fashion to how we encode and publish each individual tx and rct. Use 0x9a as the multicodec type for the CID, but it's not official yet.
Index these log IPLDs with the new log_cids table by referencing them by multihash fk.

We need to 1. Remove the current log rows in the receipt_cids table and add a new `VARCHAR(66)` row name `log_root` (should eventually be `NOT NULL`, but we need to backfill the old missing values first). 2. Create a new table, `log_cids`, for indexing log IPLDs, rows in this table reference the parent receipt by fk. 3. When processing receipts, iterate over their logs in the same manner that we iterate over txs and rcts to materialize their tries. 4. Write the root hash of the materialized logTrie to the new `log_root` row in the `receipt_cids` table. 5. Publish all the logTrie nodes to the IPLD blockstore like we do with the txTrie and rctTrie nodes, we don't index each individual node for these tries. Use `0x99` as the multicodec type for the CID, but it's not official yet. 6. RLP encode and publish each log as an IPLD, in analogous fashion to how we encode and publish each individual tx and rct. Use `0x9a` as the multicodec type for the CID, but it's not official yet. 7. Index these log IPLDs with the new `log_cids` table by referencing them by multihash fk.

i-norden commented

2021-08-20 15:35:18 +00:00

Linking to https://github.com/vulcanize/statediff-migrations/pull/7
Linking to https://github.com/vulcanize/go-ethereum/pull/102

Linking to https://github.com/vulcanize/statediff-migrations/pull/7 Linking to https://github.com/vulcanize/go-ethereum/pull/102

Sign in to join this conversation.

No Milestone

No project

No Assignees

2 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/go-ethereum#79