laconicd-stack/docs/run-first-validator.md

# Run First Validator Node

## Prerequisites

- Machine 1: Where the TMKMS service is to be setup

- Machine 2: Where the mainnet first validator node is to be setup

  - Check [domain port mappings](./domain-port-mappings.md) to ensure that required domains are pointing to correct ports

- Machine 3: Where the genesis file is to be generated

  - Install `zstd` using `sudo apt install zstd` (Linux) or `brew install zstd` (macOS)

  - [LSP distribution JSON](./export-testnet.md#prepare-lps-distribution-json)

  - [Exported testnet state](./export-testnet.md#export-testnet-state)

- Following tools are required in all machines:

  - [ansible](../playbooks/README.md#ansible-installation)

  - [laconic-so](https://github.com/cerc-io/stack-orchestrator/?tab=readme-ov-file#install)

  - laconicd-stack

    ```bash
    laconic-so fetch-stack git.vdb.to/cerc-io/laconicd-stack --pull
    ```

## Build laconicd to generate genesis file

- Run the following steps in a secure machine (machine 3) separate from the one where the node is to be setup (machine 2)

- Run playbook to build laconicd container:

  ```bash
  ansible-playbook -i localhost, -c local ~/cerc/laconicd-stack/playbooks/validator/build-laconicd.yml
  ```

## Setup bootstrap node deployment

- Run the following steps in the machine where the mainnet node is to be setup (machine 2)

- Copy the example variables file if not already done:

  ```bash
  cp ~/cerc/laconicd-stack/playbooks/first-validator/first-validator-vars.example.yml ~/cerc/laconicd-stack/playbooks/first-validator/first-validator-vars.yml
  ```

- Update `~/cerc/laconicd-stack/playbooks/first-validator/first-validator-vars.yml` with required values:

  ```bash
  # Set custom moniker for the node
  cerc_moniker: "LaconicMainnetNode"

  # Set desired key name
  key_name: "laconic-validator"

  # Enable TMKMS
  tmkms_enabled: true
  ```

- Export the data directory and mainnet deployment directory as environment variables:

  ```bash
  # Parent directory where the deployment directory will live
  export DATA_DIRECTORY=

  # Set mainnet deployment directory
  export MAINNET_DEPLOYMENT_DIR=mainnet-laconicd-deployment
  ```

- Run ansible playbook to setup the node:

  ```bash
  ansible-playbook -i localhost, -c local ~/cerc/laconicd-stack/playbooks/first-validator/setup-first-validator.yml
  ```

- Get the public key of your node:

  ```bash
  docker run -it \
    -v $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR/data/laconicd-data:/root/.laconicd \
    cerc/laconicd:local bash -c "laconicd tendermint show-validator"
  ```

  NOTE: This public key is required in [next step to generate the genesis file](#generate-mainnet-genesis-file)

- Copy over the `priv_validator_key.json` located at `$DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR/data/laconicd-data/config/priv_validator_key.json` to the machine where the TMKMS service is to be setup (machine 1)

  ```bash
  # Example command to transfer file from machine 2 to machine 1 (run on machine 2)
  scp -C $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR/data/laconicd-data/config/priv_validator_key.json <user>@<machine-ip-address>:<absolute-path-to-desired-destination-directory>
  ```

## Setup TMKMS deployment

- For integrating TMKMS with laconicd, follow steps below in the machine where the TMKMS service is to be setup (machine 1)

- Export the data directory and TMKMS deployment directory as environment variables:

  ```bash
  # Parent directory where the deployment directory will live
  export DATA_DIRECTORY=
  ```

- Run ansible playbook to setup the TMKMS service:

  ```bash
  ansible-playbook -i localhost, -c local ~/cerc/laconicd-stack/playbooks/tmkms/setup-tmkms.yml
  ```


## Generate mainnet genesis file

- Run the following steps in machine where [the genesis file is to be generated (machine 3)](#build-laconicd-to-generate-genesis-file)

- Set envs:

  ```bash
  # File path where exported testnet state JSON file will be created
  export EXPORTED_STATE_PATH=<absolute/path/to/testnet-state.json>

  # File path where LPS distribution JSON file will be created
  export LPS_DISTRIBUTION_PATH=<absolute/path/to/distribution.json>

  # Parent directory where the deployment directory will be setup (required for generating genesis)
  export DATA_DIRECTORY=
  ```

  NOTE: Steps for creating LPS distribution JSON are in [export-testnet.md](./export-testnet.md#prepare-lps-distribution-json)

- Copy over the compressed `testnet-state.zst` file (from [export-testnet.md](./export-testnet.md#export-testnet-state)):

  ```bash
  # Example command to transfer testnet state file
  scp -C <user>@<machine-ip-address>:<absolute-path-to-testnet-deployment>/export/testnet-state.zst <absolute-path-to-compressed-file>
  ```

- Extract the testnet-state JSON file:

  ```bash
  zstd -dc <absolute-path-to-compressed-file>/testnet-state.zst > $EXPORTED_STATE_PATH

  # Remove zst folder
  rm -rf <absolute-path-to-compressed-file>/testnet-state.zst
  ```

- Copy the example variables file:

  ```bash
  cp ~/cerc/laconicd-stack/playbooks/first-validator/first-validator-vars.example.yml ~/cerc/laconicd-stack/playbooks/first-validator/first-validator-vars.yml
  ```

- Edit `~/cerc/laconicd-stack/playbooks/first-validator/first-validator-vars.yml` with required values:

  NOTE: Use the public key exported in [previous steps](#setup-node)

  ```bash
  # Make sure to wrap it with single quotes ('')
  validator_pub_key: '<public-key-of-your-node>'
  ```

- Change to the deployments directory

  ```bash
  cd $DATA_DIRECTORY
  ```

- Run playbook for generating mainnet genesis file with gentx:

  ```bash
  ansible-playbook -i localhost, -c local ~/cerc/laconicd-stack/playbooks/first-validator/generate-genesis.yml -e "exported_state_path=$EXPORTED_STATE_PATH" -e "lps_distribution_path=$LPS_DISTRIBUTION_PATH"
  ```

  - When prompted for private key, use key of the existing account that was exported in [export-testnet.md](./export-testnet.md#export-testnet-state)

  - Private key should be of the account which will create the first validator bootstrap node

- Genesis file will be generated in output directory along with a file specifying the staking amount

  ```bash
  # List files in output directory - genesis.json and staking-amount.json
  ls -l output
  ```

  NOTE: Staking amount will be used by the playbooks to send create validator txs


## Start Mainnet

### Start TMKMS

- Run these steps in the machine where [the TMKMS service is setup (machine 1)](#setup-tmkms-deployment)

- Copy the example variables file:

  ```bash
  cp ~/cerc/laconicd-stack/playbooks/tmkms/tmkms-vars.example.yml ~/cerc/laconicd-stack/playbooks/tmkms/tmkms-vars.yml
  ```

- Update `~/cerc/laconicd-stack/playbooks/tmkms/tmkms-vars.yml` with required values:

  NOTE: Use the `priv_validator_key.json` file copied from the node setup machine (Machine 2) in [previous step](#setup-bootstrap-node-deployment)

  ```yaml
  # Absolute path to the node's private validator key file
  priv_validator_key_file_path: "</path/to/priv_validator_key.json>"

  # Set the IP address of the machine where the laconicd node is setup
  node_ip: "laconicd-mainnet.laconic.com"

  # Set the port of the laconicd node
  node_port: "26659"
  ```

- Run ansible playbook to run the TMKMS:

  ```bash
  ansible-playbook -i localhost, -c local ~/cerc/laconicd-stack/playbooks/tmkms/run-tmkms.yml
  ```

- Check logs to ensure that TMKMS is running:

  ```bash
  laconic-so deployment --dir $DATA_DIRECTORY/tmkms-deployment logs tmkms -f
  ```

  - Expected example output:

    ```bash
     INFO tmkms::commands::start: tmkms 0.14.0 starting up...
     INFO tmkms::keyring: [keyring:softsign] added consensus Ed25519 key: {"@type":"/cosmos.crypto.ed25519.PubKey","key":"T24No1A1FmetNRVCOSg2G2XAKWh97oBXuELdAD6DFgw="}
     INFO tmkms::connection::tcp: KMS node ID: 7f5fd8dae8953e964e7e56edd4700f597ea0d45c
    ERROR tmkms::client: [laconic-mainnet@tcp://<node-ip>:26659] I/O error: Connection refused (os error 111)
    ```

    NOTE: The errors dissapear once the laconicd node starts

- Note the pubkey logged at start for comparing later with validator pubkey on chain

### Start laconicd node

- Run the following steps in the machine where [the mainnet node is setup (machine 2)](#setup-bootstrap-node-deployment)

- Remove the validator key from node deployment as it is no longer required:

  NOTE: Store it safely offline in case of an emergency

  ```bash
  rm -rf $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR/data/laconicd-data/config/priv_validator_key.json
  ```

- Copy the genesis file generated in [Generate mainnet genesis file section](#generate-mainnet-genesis-file) from machine 3 to the machine 2 mainnet deployment tmp directory:

  ```bash
  # Example command to transfer file from machine 3 to machine 2 (run on machine 3)
  scp -C $DATA_DIRECTORY/output/genesis.json <user>@<machine-ip-address>:<absolute-path-to-deployments-directory>/mainnet-laconicd-deployment/data/laconicd-data/tmp/
  ```

- Command to run node:

  ```bash
  laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR start
  ```

- Check logs to ensure that node is running:

  ```bash
  laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR logs laconicd -f
  ```

  NOTE: The node takes a long time to start generating blocks `~30 seconds`

- Verify that validator and TMKMS pubkeys match:

  - Get validator pubkey on chain

    ```bash
    # Check consensus_pubkey in output
    laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR exec laconicd 'laconicd query staking validators -o json | jq .validators'
    ```

  - Compare it with the pubkey noted from logs in TMKMS

- Check bonds list to confirm that testnet state was transferred properly:

  ```bash
  laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR exec laconicd 'laconicd query bond list'
  ```

- Check `alps` and `alnt` tokens total supply:

  ```bash
  laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR exec laconicd 'laconicd query bank total-supply'
  ```

- Query the `lps_lockup` account and view distribution:

  ```bash
  laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR exec laconicd 'laconicd query auth module-account lps_lockup'
  ```

- Query the `lps_lockup` account balance:

  ```bash
  lockup_account_address=$(laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR exec laconicd 'laconicd query auth module-account lps_lockup -o json | jq -r .account.value.base_account.address')
  laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR exec laconicd "laconicd query bank balances $lockup_account_address"
  ```

## Publish required artifacts

- Run the following steps in machine where the genesis file and staking amount files were generated (machine 3)

  - Copy the genesis file to [config](./config) folder:

    ```bash
    cp $DATA_DIRECTORY/output/genesis.json ~/cerc/laconicd-stack/config/mainnet-genesis.json
    ```

  - Copy the staking amount file to [config](./config) folder:

    ```bash
    cp $DATA_DIRECTORY/output/staking-amount.json ~/cerc/laconicd-stack/config/staking-amount.json
    ```
  
  - Check git status of the stack repo

    ```bash
    cd ~/cerc/laconicd-stack
    git status
    ```

    The following files should show up with changes:
    - config/mainnet-genesis.json
    - config/staking-amount.json
  
  - Create a PR (to this repo and to https://github.com/LaconicNetwork/mainnet/) with the genesis file and staking amount file so that it is available to other validators

- Run the following steps in machine where the mainnet node is running (machine 2)

  - Get your node's address by running the following command:

    ```bash
    laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR exec laconicd 'echo $(laconicd cometbft show-node-id)@laconicd-mainnet.laconic.com:26656'
    ```

  - Add your node's address to [node-addresses.yml](../node-addresses.yml)

    - Update the file `~/cerc/laconicd-stack/node-addresses.yml` in machine where genesis file is generated (machine 3)

  - Check git status of the stack repo

    ```bash
    cd ~/cerc/laconicd-stack
    git status
    ```

    The `node-addresses.yml` file should show up with changes

  - Submit a PR with the node address so that it is available to other validators

## Update config

- Run following steps to update the config for TMKMS and node

### TMKMS

- Run these steps in the machine where the TMKMS service is setup (machine 1)

- Stop the TMKMS deployment:

  ```bash
  laconic-so deployment --dir $DATA_DIRECTORY/$TMKMS_DEPLOYMENT_DIR stop
  ```

- Update `~/cerc/laconicd-stack/playbooks/tmkms/tmkms-vars.yml` with required values

- Run ansible playbook to run the TMKMS:

  ```bash
  ansible-playbook -i localhost, -c local ~/cerc/laconicd-stack/playbooks/tmkms/run-tmkms.yml
  ```

### Node

- Run these steps in the machine where the mainnet node is setup (machine 2)

- Stop the node deployment:

  ```bash
  laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR stop
  ```

- Update `$DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR/config.env` with required values

- Start the node deployment:

  ```bash
  laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR start
  ```

## Rebuild Images

- Follow these steps to rebuild the images for TMKMS and node in case of any code changes

### TMKMS

- Run these steps in the machine where the TMKMS service is setup (machine 1)

- Stop the TMKMS deployment:

  ```bash
  laconic-so deployment --dir $DATA_DIRECTORY/tmkms-deployment stop
  ```

- Run ansible playbook to rebuild the TMKMS image:

  ```bash
  BUILD_ONLY=true ansible-playbook -i localhost, -c local ~/cerc/laconicd-stack/playbooks/tmkms/setup-tmkms.yml
  ```

- Start the TMKMS deployment:

  ```bash
  ansible-playbook -i localhost, -c local ~/cerc/laconicd-stack/playbooks/tmkms/run-tmkms.yml
  ```

### Node

- Run these steps in the machine where the mainnet node is setup (machine 2)

- Stop the node deployment:

  ```bash
  laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR stop
  ```

- Run ansible playbook to rebuild the node image:

  ```bash
  BUILD_ONLY=true ansible-playbook -i localhost, -c local ~/cerc/laconicd-stack/playbooks/first-validator/setup-first-validator.yml
  ```

- Start the node deployment:

  ```bash
  laconic-so deployment --dir $DATA_DIRECTORY/$MAINNET_DEPLOYMENT_DIR start
  ```