Chain congestion prevents setting names, publishing records, etc. #55

New Issue

zramsay · 2024-08-20T19:41:24Z

zramsay commented

2024-08-20 19:41:24 +00:00

for example:

2024-08-18 14:07:53.652126 - bafyreif4kgenticfrbyf4zmbaup2xqavvoaigns4xmllsdzghugsiyvxfa-1723990011-34db9321 - 0:00:00.001761 (step): Publishing DnsRecord.
record:
  meta:
    so: 5f79f7368e2841718c83d7d369101be3
  name: pwa-test-613-03bea2fad1.pwa.realitynetwork.store
  request: bafyreif4kgenticfrbyf4zmbaup2xqavvoaigns4xmllsdzghugsiyvxfa
  resource_type: A
  type: DnsRecord
  version: 0.0.1

laconic -c /etc/config/laconic.yml registry record publish --filename /tmp/tmp1npf8erv/record.yml
laconic -c /etc/config/laconic.yml registry name set lrn://lcn-example/dns/pwa-test-613-03bea2fad1.pwa.realitynetwork.store bafyreics5ayn7jyuhdp5ovkhymtkc7tix2orvhfkmnkm5b5mg7rf43ilaq
laconic registry name set [name] [id]

Set name (create name to record ID mapping).

Options:
      --version   Show version number                                  [boolean]
  -v, --verbose   Verbose output                      [boolean] [default: false]
  -c, --config    Config file path.             [string] [default: "config.yml"]
  -o, --output    Gives output in json format when specified.           [string]
      --user-key                                                        [string]
      --tx-key                                                          [string]
      --bond-id                                                         [string]
      --chain-id                                                        [string]
  -f, --filename
      --id                                                              [string]
      --address                                                         [string]
      --gas                                                             [string]
      --fees                                                            [string]
      --help      Show help                                            [boolean]

TimeoutError: Transaction with ID 49BEA6FE54B16B7914BFDD28EDE632BE859AB0212CEC1985F744C7160E76CAAF was submitted but was not yet found on the chain. You might want to check later. There was a wait of 60 seconds.
    at pollForTx (/usr/lib/node_modules/@cerc-io/laconic-registry-cli/node_modules/@cosmjs/stargate/build/stargateclient.js:249:23)
    at pollForTx (/usr/lib/node_modules/@cerc-io/laconic-registry-cli/node_modules/@cosmjs/stargate/build/stargateclient.js:265:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  txId: '49BEA6FE54B16B7914BFDD28EDE632BE859AB0212CEC1985F744C7160E76CAAF'
}

rm -rf /tmp/tmp1npf8erv
2024-08-18 14:09:19.980237 - bafyreif4kgenticfrbyf4zmbaup2xqavvoaigns4xmllsdzghugsiyvxfa-1723990011-34db9321 - 0:01:26.328098 (step): ERROR: Command '('laconic', '-c', '/etc/config/laconic.yml', 'registry', 'name', 'set', 'lrn://lcn-example/dns/pwa-test-613-03bea2fad1.pwa.realitynetwork.store', 'bafyreics5ayn7jyuhdp5ovkhymtkc7tix2orvhfkmnkm5b5mg7rf43ilaq')' returned non-zero exit status 1.
2024-08-18 14:09:19.980317 - bafyreif4kgenticfrbyf4zmbaup2xqavvoaigns4xmllsdzghugsiyvxfa-1723990011-34db9321 - 0:02:27.981609 (total): DONE with status ERROR

from cerc-io/webapp-deployment-status-api#13 (comment)

for example: ``` 2024-08-18 14:07:53.652126 - bafyreif4kgenticfrbyf4zmbaup2xqavvoaigns4xmllsdzghugsiyvxfa-1723990011-34db9321 - 0:00:00.001761 (step): Publishing DnsRecord. record: meta: so: 5f79f7368e2841718c83d7d369101be3 name: pwa-test-613-03bea2fad1.pwa.realitynetwork.store request: bafyreif4kgenticfrbyf4zmbaup2xqavvoaigns4xmllsdzghugsiyvxfa resource_type: A type: DnsRecord version: 0.0.1 laconic -c /etc/config/laconic.yml registry record publish --filename /tmp/tmp1npf8erv/record.yml laconic -c /etc/config/laconic.yml registry name set lrn://lcn-example/dns/pwa-test-613-03bea2fad1.pwa.realitynetwork.store bafyreics5ayn7jyuhdp5ovkhymtkc7tix2orvhfkmnkm5b5mg7rf43ilaq laconic registry name set [name] [id] Set name (create name to record ID mapping). Options: --version Show version number [boolean] -v, --verbose Verbose output [boolean] [default: false] -c, --config Config file path. [string] [default: "config.yml"] -o, --output Gives output in json format when specified. [string] --user-key [string] --tx-key [string] --bond-id [string] --chain-id [string] -f, --filename --id [string] --address [string] --gas [string] --fees [string] --help Show help [boolean] TimeoutError: Transaction with ID 49BEA6FE54B16B7914BFDD28EDE632BE859AB0212CEC1985F744C7160E76CAAF was submitted but was not yet found on the chain. You might want to check later. There was a wait of 60 seconds. at pollForTx (/usr/lib/node_modules/@cerc-io/laconic-registry-cli/node_modules/@cosmjs/stargate/build/stargateclient.js:249:23) at pollForTx (/usr/lib/node_modules/@cerc-io/laconic-registry-cli/node_modules/@cosmjs/stargate/build/stargateclient.js:265:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { txId: '49BEA6FE54B16B7914BFDD28EDE632BE859AB0212CEC1985F744C7160E76CAAF' } rm -rf /tmp/tmp1npf8erv 2024-08-18 14:09:19.980237 - bafyreif4kgenticfrbyf4zmbaup2xqavvoaigns4xmllsdzghugsiyvxfa-1723990011-34db9321 - 0:01:26.328098 (step): ERROR: Command '('laconic', '-c', '/etc/config/laconic.yml', 'registry', 'name', 'set', 'lrn://lcn-example/dns/pwa-test-613-03bea2fad1.pwa.realitynetwork.store', 'bafyreics5ayn7jyuhdp5ovkhymtkc7tix2orvhfkmnkm5b5mg7rf43ilaq')' returned non-zero exit status 1. 2024-08-18 14:09:19.980317 - bafyreif4kgenticfrbyf4zmbaup2xqavvoaigns4xmllsdzghugsiyvxfa-1723990011-34db9321 - 0:02:27.981609 (total): DONE with status ERROR ``` from https://git.vdb.to/cerc-io/webapp-deployment-status-api/issues/13#issuecomment-9189

zramsay referenced this issue from cerc-io/webapp-deployment-status-api

2024-08-20 19:42:43 +00:00

The deployer can deploy the app but fail to publish DnsRecord and ApplicationDeploymentRecord #13

zramsay changed title from ~~Chain congestion prevents setting names~~ to Chain congestion prevents setting names, publishing records, etc.

2024-08-21 18:37:21 +00:00

zramsay commented

2024-08-21 18:42:38 +00:00

This issue has been reported by multiple users for multiple transaction types, and may be related to high fee/gas settings, as well as blocks only including 1 txn

AFDudley commented

2024-08-21 18:46:56 +00:00

What do you mean blocks only include 1 tx?

ashwin was assigned by zramsay

2024-08-21 21:02:42 +00:00

prathamesh commented

2024-08-22 09:59:10 +00:00

With registry config (gas: 9950000 and fees: 500000alnt) from SP docs, txs might run into the follow error:

BroadcastTxError: Broadcasting transaction failed with code 13 (codespace: sdk). Log: insufficient fees; got: 500000alnt required: 9950000alnt: insufficient fee

This happens if the fees provided is not sufficient according to min gas price set by the node the txs are being sent to (1alnt on stage1 deployment node)

However, looking at the recent txs on stage1 chain where gas_wanted (provided in tx by user) is 9,950,000 (MsgSetRecord, MsgSetName, etc.), the actual gas_used is < 100,000:

# Against stage1 deployment node
✗ laconicd --node https://laconicd.laconic.com:443 query txs --query "tx.height > 67196 AND tx.height < 67200" --output json | jq '{total_count, txs: [.txs[] | {gas_used: .gas_used, gas_wanted: .gas_wanted, height: .height, timestamp: .timestamp, messages: [.tx.body.messages[] | {"@type": .["@type"]}]}]}'
{
  "total_count": "4",
  "txs": [
    {
      "gas_used": "47412",
      "gas_wanted": "200000",
      "height": "67197",
      "timestamp": "2024-08-22T07:40:24Z",
      "messages": [
        {
          "@type": "/cerc.auction.v1.MsgCommitBid"
        }
      ]
    },
    {
      "gas_used": "51359",
      "gas_wanted": "9550000",
      "height": "67197",
      "timestamp": "2024-08-22T07:40:24Z",
      "messages": [
        {
          "@type": "/cerc.registry.v1.MsgSetRecord"
        }
      ]
    },
    {
      "gas_used": "51229",
      "gas_wanted": "9950000",
      "height": "67198",
      "timestamp": "2024-08-22T07:40:28Z",
      "messages": [
        {
          "@type": "/cerc.registry.v1.MsgSetRecord"
        }
      ]
    },
    {
      "gas_used": "51128",
      "gas_wanted": "9950000",
      "height": "67199",
      "timestamp": "2024-08-22T07:40:32Z",
      "messages": [
        {
          "@type": "/cerc.registry.v1.MsgSetRecord"
        }
      ]
    }
  ]
}

So setting the gas as 100,000 - 200,000 should suffice for the registry txs

Regarding chain congestion, the gas value 9,950,000 is very close to the max_gas limit for a block (10,000,000)

The default consensus params are as follows:

# Against stage1 deployment node
✗ laconicd query consensus params --node https://laconicd.laconic.com:443
params:
  abci: {}
  block:
    max_bytes: "22020096"
    max_gas: "10000000"
  ...

So with users sending txs with gas set to 9,950,000 results in only one tx being included in the block and other users getting the error like:

TimeoutError: Transaction with ID 49BEA6FE54B16B7914BFDD28EDE632BE859AB0212CEC1985F744C7160E76CAAF was submitted but was not yet found on the chain. You might want to check later. There was a wait of 60 seconds.

Refernce: https://github.com/cosmos/cosmos-sdk/issues/2150

To handle both insufficient fees and TimeoutError errors, we can ask users to set gas to a lower value in the SP docs

- With registry config (`gas: 9950000` and `fees: 500000alnt`) from SP [docs](https://github.com/hyphacoop/loro-testnet/blob/main/docs/service-provider-setup.md#deploy-backend), txs might run into the follow error: ```bash BroadcastTxError: Broadcasting transaction failed with code 13 (codespace: sdk). Log: insufficient fees; got: 500000alnt required: 9950000alnt: insufficient fee ``` - This happens if the fees provided is not sufficient according to min gas price set by the node the txs are being sent to (`1alnt` on stage1 deployment node) - However, looking at the recent txs on stage1 chain where `gas_wanted` (provided in tx by user) is `9,950,000` (`MsgSetRecord`, `MsgSetName`, etc.), the actual `gas_used` is < 100,000: ```bash # Against stage1 deployment node ✗ laconicd --node https://laconicd.laconic.com:443 query txs --query "tx.height > 67196 AND tx.height < 67200" --output json | jq '{total_count, txs: [.txs[] | {gas_used: .gas_used, gas_wanted: .gas_wanted, height: .height, timestamp: .timestamp, messages: [.tx.body.messages[] | {"@type": .["@type"]}]}]}' { "total_count": "4", "txs": [ { "gas_used": "47412", "gas_wanted": "200000", "height": "67197", "timestamp": "2024-08-22T07:40:24Z", "messages": [ { "@type": "/cerc.auction.v1.MsgCommitBid" } ] }, { "gas_used": "51359", "gas_wanted": "9550000", "height": "67197", "timestamp": "2024-08-22T07:40:24Z", "messages": [ { "@type": "/cerc.registry.v1.MsgSetRecord" } ] }, { "gas_used": "51229", "gas_wanted": "9950000", "height": "67198", "timestamp": "2024-08-22T07:40:28Z", "messages": [ { "@type": "/cerc.registry.v1.MsgSetRecord" } ] }, { "gas_used": "51128", "gas_wanted": "9950000", "height": "67199", "timestamp": "2024-08-22T07:40:32Z", "messages": [ { "@type": "/cerc.registry.v1.MsgSetRecord" } ] } ] } ``` So setting the gas as 100,000 - 200,000 should suffice for the registry txs - Regarding chain congestion, the gas value 9,950,000 is very close to the `max_gas` limit for a block (10,000,000) - The default consensus params are as follows: ```bash # Against stage1 deployment node ✗ laconicd query consensus params --node https://laconicd.laconic.com:443 params: abci: {} block: max_bytes: "22020096" max_gas: "10000000" ... ``` - So with users sending txs with gas set to 9,950,000 results in only one tx being included in the block and other users getting the error like: ```bash TimeoutError: Transaction with ID 49BEA6FE54B16B7914BFDD28EDE632BE859AB0212CEC1985F744C7160E76CAAF was submitted but was not yet found on the chain. You might want to check later. There was a wait of 60 seconds. ``` - Refernce: https://github.com/cosmos/cosmos-sdk/issues/2150 - To handle both `insufficient fees` and `TimeoutError` errors, we can ask users to set gas to a lower value in the SP [docs](https://github.com/hyphacoop/loro-testnet/blob/main/docs/service-provider-setup.md#deploy-backend)

AFDudley commented

2024-08-22 11:20:21 +00:00

Wonderful. Thanks for the quick solution. What commands do existing testnet validators run to fix their configurations?

prathamesh commented

2024-08-22 11:30:11 +00:00

Wonderful. Thanks for the quick solution. What commands do existing testnet validators run to fix their configurations?

No changes are required to the validator node's config, just need to set lower gas amount in the web-deployer config:
https://github.com/hyphacoop/loro-testnet/blob/main/docs/service-provider-setup.md#deploy-backend

services:
  registry:
    rpcEndpoint: 'https://lcn-daemon.laconic.com:26657'
    gqlEndpoint: 'https://lcn-daemon.laconic.com:9473/api'
    userKey: e64ae9d07b21c62081b3d6d48e78bf44275ffe0575f788ea7b36f71ea559724b
    bondId: ad9c977f4a641c2cf26ce37dcc9d9eb95325e9f317aee6c9f33388cdd8f2abb8
    chainId: lorotestnet-1
    gas: 9950000
    fees: 500000alnt

> Wonderful. Thanks for the quick solution. What commands do existing testnet validators run to fix their configurations? No changes are required to the validator node's config, just need to set lower gas amount in the web-deployer config: https://github.com/hyphacoop/loro-testnet/blob/main/docs/service-provider-setup.md#deploy-backend ```bash services: registry: rpcEndpoint: 'https://lcn-daemon.laconic.com:26657' gqlEndpoint: 'https://lcn-daemon.laconic.com:9473/api' userKey: e64ae9d07b21c62081b3d6d48e78bf44275ffe0575f788ea7b36f71ea559724b bondId: ad9c977f4a641c2cf26ce37dcc9d9eb95325e9f317aee6c9f33388cdd8f2abb8 chainId: lorotestnet-1 gas: 9950000 fees: 500000alnt ```

AFDudley commented

2024-08-22 11:31:28 +00:00

Thanks for clarifying.

dboreham commented

2024-08-22 14:29:13 +00:00

I assume the very high gas limit came about due to a previous problem where the nodes were charging more and more gas as the size of the registry grew, leading to txns failing from lack of gas. I doubt anyone was aware at the time that there was the potential to have this second problem.

👀 1

AFDudley commented

2024-09-25 12:20:45 +00:00

@zramsay wasn't this issue resolved with the auto fee calculation patches?

👍 1

zramsay closed this issue

2024-09-25 15:36:51 +00:00

Sign in to join this conversation.