testnet-laconicd-stack/ops/sp-deployment-from-scratch.md

# Service Provider deployments from scratch

## container-registry

* Reference: <https://github.com/LaconicNetwork/loro-testnet/blob/main/docs/service-provider-setup.md#deploy-docker-image-container-registry>

* Target dir: `/srv/service-provider/container-registry`

* Cleanup an existing deployment if required:
  ```bash
  cd /srv/service-provider/container-registry

  # Stop the deployment
  laconic-so deployment --dir container-registry stop --delete-volumes

  # Remove the deployment dir
  sudo rm -rf container-registrty

  # Remove the existing spec file
  rm container-registry.spec
  ```

### Setup

- Generate the spec file for the container-registry stack
  ```bash
  laconic-so --stack container-registry deploy init --output container-registry.spec
  ```

- Modify the `container-registry.spec` as shown below
  ```
  stack: container-registry
  deploy-to: k8s
  kube-config: /home/dev/.kube/config-vs-narwhal.yaml
  network:
    ports:
      registry:
      - '5000'
    http-proxy:
      - host-name: container-registry.apps.vaasl.io
        routes:
          - path: '/'
            proxy-to: registry:5000
  volumes:
    registry-data:
  configmaps:
    config: ./configmaps/config
  ```

- Create the deployment directory for the `container-registry` stack
  ```bash
  laconic-so --stack container-registry deploy create --deployment-dir container-registry --spec-file container-registry.spec
  ```

- Modify file `container-registry/kubeconfig.yml` if required
  ```
  apiVersion: v1
  ...
  contexts:
    - context:
        cluster: ***
        user: ***
      name: default
  ...
  ```
  NOTE: `context.name` must be default to use with SO

- Base64 encode the container registry credentials  
  NOTE: Use actual credentials for container registry (credentials set in `container-registry/credentials.txt`)
  ```bash
  echo -n "so-reg-user:pXDwO5zLU7M88x3aA" | base64 -w0
  
  # Output: c28tcmVnLXVzZXI6cFhEd081ekxVN004OHgzYUE=
  ```

- Install `apache2-utils` for next step
  ```bash
  sudo apt install apache2-utils
  ```

- Encrypt the container registry credentials to create an `htpasswd` file
  ```bash
  htpasswd -bB -c container-registry/configmaps/config/htpasswd so-reg-user pXDwO5zLU7M88x3aA
  ```

  Resulting file should look like this
  ```
  cat container-registry/configmaps/config/htpasswd
  # so-reg-user:$2y$05$6EdxIwwDNlJfNhhQxZRr4eNd.aYrdmbBjAdw422w0u2j3TihQXgd2
  ```

- Using the credentials from the previous steps, create a `container-registry/my_password.json` file
  ```json
  {
    "auths": {
      "container-registry.apps.vaasl.io": {
        "username": "so-reg-user",
        "password": "$2y$05$6EdxIwwDNlJfNhhQxZRr4eNd.aYrdmbBjAdw422w0u2j3TihQXgd2",
        "auth": "c28tcmVnLXVzZXI6cFhEd081ekxVN004OHgzYUE="
      }
    }
  }
  ```

- Configure the file `container-registry/config.env` as follows
  ```env
  REGISTRY_AUTH=htpasswd
  REGISTRY_AUTH_HTPASSWD_REALM="VSL Service Provider Image Registry"
  REGISTRY_AUTH_HTPASSWD_PATH="/config/htpasswd"
  REGISTRY_HTTP_SECRET='$2y$05$6EdxIwwDNlJfNhhQxZRr4eNd.aYrdmbBjAdw422w0u2j3TihQXgd2'
  ```

- Load context for k8s
  ```bash
  kubie ctx vs-narwhal
  ```

- Add the container registry credentials as a secret available to the cluster
  ```bash
  kubectl create secret generic laconic-registry --from-file=.dockerconfigjson=container-registry/my_password.json --type=kubernetes.io/dockerconfigjson
  ```

### Run

- Deploy the container registry
  ```bash
  laconic-so deployment --dir container-registry start
  ```

- Check the logs
  ```bash
  laconic-so deployment --dir container-registry logs
  ```

- Check status and await succesful deployment:
  ```bash
  laconic-so deployment --dir container-registry status
  ```

- Confirm deployment by logging in:
  ```
  docker login container-registry.apps.vaasl.io --username so-reg-user --password pXDwO5zLU7M88x3aA
  ```

- Set ingress annotations

  - Set the `cluster-id` found in `container-registry/deployment.yml` and then run the following commands:
    ```
    export CLUSTER_ID=<cluster-id>
    # Example
    # export CLUSTER_ID=laconic-26cc70be8a3db3f4
    
    kubectl annotate ingress $CLUSTER_ID-ingress nginx.ingress.kubernetes.io/proxy-body-size=0
    kubectl annotate ingress $CLUSTER_ID-ingress nginx.ingress.kubernetes.io/proxy-read-timeout=600
    kubectl annotate ingress $CLUSTER_ID-ingress nginx.ingress.kubernetes.io/proxy-send-timeout=600
    ```

## webapp-deployer

### Backend

* Reference: <https://github.com/LaconicNetwork/loro-testnet/blob/main/docs/service-provider-setup.md#deploy-backend>

* Target dir: `/srv/service-provider/webapp-deployer`

* Cleanup an existing deployment if required:
  ```bash
  cd /srv/service-provider/webapp-deployer

  # Stop the deployment
  laconic-so deployment --dir webapp-deployer stop

  # Remove the deployment dir
  sudo rm -rf webapp-deployer

  # Remove the existing spec file
  rm webapp-deployer.spec
  ```

#### Setup

- Initialize a spec file for the deployer backend.
  ```bash
  laconic-so --stack webapp-deployer-backend setup-repositories
  laconic-so --stack webapp-deployer-backend build-containers
  laconic-so --stack webapp-deployer-backend deploy init --output webapp-deployer.spec
  ```

- Modify the contents of `webapp-deployer.spec`:
  ```
  stack: webapp-deployer-backend
  deploy-to: k8s
  kube-config: /home/dev/.kube/config-vs-narwhal.yaml
  image-registry: container-registry.apps.vaasl.io/laconic-registry
  network:
    ports:
      server:
        - '9555'
    http-proxy:
      - host-name: webapp-deployer-api.apps.vaasl.io
        routes:
          - path: '/'
            proxy-to: server:9555
  volumes:
    srv:
  configmaps:
    config: ./data/config
  annotations:
    container.apparmor.security.beta.kubernetes.io/{name}: unconfined
  labels:
    container.kubeaudit.io/{name}.allow-disabled-apparmor: "podman"
  security:
    privileged: true

  resources:
    containers:
      reservations:
        cpus: 3
        memory: 8G
      limits:
        cpus: 7
        memory: 16G
    volumes:
      reservations:
        storage: 200G
  ```

- Create the deployment directory from the spec file.
  ```
  laconic-so --stack webapp-deployer-backend deploy create --deployment-dir webapp-deployer --spec-file webapp-deployer.spec
  ```

- Modify file `webapp-deployer/kubeconfig.yml` if required
  ```
  apiVersion: v1
  ...
  contexts:
    - context:
        cluster: ***
        user: ***
      name: default
  ...
  ```
  NOTE: `context.name` must be default to use with SO

- Copy `webapp-deployer/kubeconfig.yml` from the k8s cluster creation step to `webapp-deployer/data/config/kube.yml`
  ```bash
  cp webapp-deployer/kubeconfig.yml webapp-deployer/data/config/kube.yml
  ```

- Create `webapp-deployer/data/config/laconic.yml`, it should look like this:
  ```
  services:
    registry:
      # Using public endpoint does not work inside machine where laconicd chain is deployed
      rpcEndpoint: 'http://host.docker.internal:36657'
      gqlEndpoint: 'http://host.docker.internal:3473/api'

      # Set user key of account with balance and bond owned by the user
      userKey:
      bondId:

      chainId: laconic-testnet-2
      gasPrice: 1alnt
  ```
  NOTE: Modify the user key and bond ID according to your configuration

* Publish a `WebappDeployer` record for the deployer backend by following the steps below:

  * Setup GPG keys by following [these steps to create and export a key](https://git.vdb.to/cerc-io/webapp-deployment-status-api#keys)
    ```
    cd webapp-deployer

    # Create a key
    gpg --batch --passphrase "SECRET" --quick-generate-key webapp-deployer-api.apps.vaasl.io default default never

    # Export the public key
    gpg --export webapp-deployer-api.apps.vaasl.io > webapp-deployer-api.apps.vaasl.io.pgp.pub

    # Export the private key
    gpg --export-secret-keys webapp-deployer-api.apps.vaasl.io > webapp-deployer-api.apps.vaasl.io.pgp.key

    cd -
    ```
    NOTE: Use "SECRET" for passphrase prompt

  * Copy the GPG pub key file generated above to `webapp-deployer/data/config` directory. This ensures the Docker container has access to the key during the publish process
    ```bash
    cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.pub webapp-deployer/data/config
    ```


  * Publish the webapp deployer record using the `publish-deployer-to-registry` command

    ```
    docker run -i -t \
      --add-host=host.docker.internal:host-gateway \
      -v /srv/service-provider/webapp-deployer/data/config:/config \
      cerc/webapp-deployer-backend:local laconic-so publish-deployer-to-registry \
        --laconic-config /config/laconic.yml \
        --api-url https://webapp-deployer-api.apps.vaasl.io \
        --public-key-file /config/webapp-deployer-api.apps.vaasl.io.pgp.pub \
        --lrn lrn://vaasl-provider/deployers/webapp-deployer-api.apps.vaasl.io \
        --min-required-payment 10000
    ```

- Modify the contents of `webapp-deployer/config.env`:

  ```
  DEPLOYMENT_DNS_SUFFIX="apps.vaasl.io"

  # this should match the name authority reserved above
  DEPLOYMENT_RECORD_NAMESPACE="vaasl-provider"

  # url of the deployed docker image registry
  IMAGE_REGISTRY="container-registry.apps.vaasl.io"

  # credentials from the htpasswd section above in container-registry setup
  IMAGE_REGISTRY_USER=
  IMAGE_REGISTRY_CREDS=

  # configs
  CLEAN_DEPLOYMENTS=false
  CLEAN_LOGS=false
  CLEAN_CONTAINERS=false
  SYSTEM_PRUNE=false
  WEBAPP_IMAGE_PRUNE=true
  CHECK_INTERVAL=10
  FQDN_POLICY="allow"

  # lrn of the webapp deployer
  LRN="lrn://vaasl-provider/deployers/webapp-deployer-api.apps.vaasl.io"

  # Path to the GPG key file inside the webapp-deployer container
  OPENPGP_PRIVATE_KEY_FILE="webapp-deployer-api.apps.vaasl.io.pgp.key"
  # Passphrase used when creating the GPG key
  OPENPGP_PASSPHRASE="SECRET"
  
  DEPLOYER_STATE="srv-test/deployments/autodeploy.state"
  UNDEPLOYER_STATE="srv-test/deployments/autoundeploy.state"
  UPLOAD_DIRECTORY="srv-test/uploads"
  HANDLE_AUCTION_REQUESTS=true
  AUCTION_BID_AMOUNT=10000

  # Minimum payment amount required for single webapp deployment
  MIN_REQUIRED_PAYMENT=10000
  ```

- Push the image to the container registry
  ```
  laconic-so deployment --dir webapp-deployer push-images
  ```

- Modify `webapp-deployer/data/config/laconic.yml`:
  ```
  services:
    registry:
      rpcEndpoint: 'https://laconicd-sapo.laconic.com/'
      gqlEndpoint: 'https://laconicd-sapo.laconic.com/api'

      # Set user key of account with balance and bond owned by the user
      userKey:
      bondId:

      chainId: laconic-testnet-2
      gasPrice: 1alnt
  ```

#### Run

- Start the deployer.
  ```
  laconic-so deployment --dir webapp-deployer start
  ```

- Load context for k8s
  ```bash
  kubie ctx vs-narwhal
  ```

- Copy the GPG key file to the webapp-deployer container

  ```bash
  # Get the webapp-deployer pod id
  laconic-so deployment --dir webapp-deployer ps
  
  # Expected output
  # Running containers:
  # id: default/laconic-096fed46af974a47-deployment-644db859c7-snbq6, name: laconic-096fed46af974a47-deployment-644db859c7-snbq6, ports: 10.42.2.11:9555->9555

  # Set pod id
  export POD_ID=
  # Example:
  # export POD_ID=laconic-096fed46af974a47-deployment-644db859c7-snbq6

  # Copy GPG key files to the pod
  kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.key $POD_ID:/app
  kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.pub $POD_ID:/app
  ```

- Publishing records to the registry will trigger deployments in backend now

### Frontend

* Target dir: `/srv/service-provider/webapp-ui`

* Cleanup an existing deployment if required:
  ```bash
  cd /srv/service-provider/webapp-ui

  # Stop the deployment
  laconic-so deployment --dir webapp-ui stop

  # Remove the deployment dir
  sudo rm -rf webapp-ui

  # Remove the existing spec file
  rm webapp-ui.spec
  ```

#### Setup

* Clone and build the deployer UI
  ```
  git clone https://git.vdb.to/cerc-io/webapp-deployment-status-ui.git ~/cerc/webapp-deployment-status-ui

  laconic-so build-webapp --source-repo ~/cerc/webapp-deployment-status-ui
  ```

* Create a deployment
  ```bash
  export KUBECONFIG_PATH=/home/dev/.kube/config-vs-narwhal.yaml
  # NOTE: Use actual kubeconfig path

  laconic-so deploy-webapp create --kube-config $KUBECONFIG_PATH --image-registry container-registry.apps.vaasl.io --deployment-dir webapp-ui --image cerc/webapp-deployment-status-ui:local --url https://webapp-deployer-ui.apps.vaasl.io --env-file ~/cerc/webapp-deployment-status-ui/.env
  ```

* Modify file `webapp-ui/kubeconfig.yml` if required
  ```yml
  apiVersion: v1
  ...
  contexts:
    - context:
        cluster: ***
        user: ***
      name: default
  ...
  ```
  NOTE: `context.name` must be default to use with SO

- Push the image to the container registry.
  ```
  laconic-so deployment --dir webapp-ui push-images
  ```

- Modify `webapp-ui/config.env` like [this Pull Request](https://git.vdb.to/cerc-io/webapp-deployment-status-ui/pulls/6) but with your host details.

#### Run

- Start the deployer UI
  ```bash
  laconic-so deployment --dir webapp-ui start
  ```

- Wait a moment, then go to https://webapp-deployer-ui.apps.vaasl.io for the status and logs of each deployment