ops/deployments-from-scratch.md

@@ -976,6 +976,10 @@
 
 </details>
 
+## Service Provider
+
+Follow [sp-deployment-from-scratch.md](./sp-deployment-from-scratch.md)
+
 <details open>
   <summary>deploy-backend</summary>
 
@@ -1755,6 +1759,11 @@ https://console-sapo.laconic.com          -> 4002
 Open p2p ports:
 36656
 
+# Service provider (deployed on K8s)
+https://container-registry.apps.vaasl.io
+https://webapp-deployer-api.apps.vaasl.io
+https://webapp-deployer-ui.apps.vaasl.io
+
 # Machine 2
 https://sepolia.laconic.com           -> 8545
 wss://sepolia.laconic.com             -> 8546

ops/sp-deployment-from-scratch.md

@@ -0,0 +1,465 @@
+# Service Provider deployments from scratch
+
+## container-registry
+
+* Reference: <https://github.com/LaconicNetwork/loro-testnet/blob/main/docs/service-provider-setup.md#deploy-docker-image-container-registry>
+
+* Target dir: `/srv/service-provider/container-registry`
+
+* Cleanup an existing deployment if required:
+  ```bash
+  cd /srv/service-provider/container-registry
+
+  # Stop the deployment
+  laconic-so deployment --dir container-registry stop --delete-volumes
+
+  # Remove the deployment dir
+  sudo rm -rf container-registrty
+
+  # Remove the existing spec file
+  rm container-registry.spec
+  ```
+
+### Setup
+
+- Generate the spec file for the container-registry stack
+  ```bash
+  laconic-so --stack container-registry deploy init --output container-registry.spec
+  ```
+
+- Modify the `container-registry.spec` as shown below
+  ```
+  stack: container-registry
+  deploy-to: k8s
+  kube-config: /home/dev/.kube/config-vs-narwhal.yaml
+  network:
+    ports:
+      registry:
+      - '5000'
+    http-proxy:
+      - host-name: container-registry.apps.vaasl.io
+        routes:
+          - path: '/'
+            proxy-to: registry:5000
+  volumes:
+    registry-data:
+  configmaps:
+    config: ./configmaps/config
+  ```
+
+- Create the deployment directory for the `container-registry` stack
+  ```bash
+  laconic-so --stack container-registry deploy create --deployment-dir container-registry --spec-file container-registry.spec
+  ```
+
+- Modify file `container-registry/kubeconfig.yml` if required
+  ```
+  apiVersion: v1
+  ...
+  contexts:
+    - context:
+        cluster: ***
+        user: ***
+      name: default
+  ...
+  ```
+  NOTE: `context.name` must be default to use with SO
+
+- Base64 encode the container registry credentials  
+  NOTE: Use actual credentials for container registry (credentials set in `container-registry/credentials.txt`)
+  ```bash
+  echo -n "so-reg-user:pXDwO5zLU7M88x3aA" | base64 -w0
+  
+  # Output: c28tcmVnLXVzZXI6cFhEd081ekxVN004OHgzYUE=
+  ```
+
+- Install `apache2-utils` for next step
+  ```bash
+  sudo apt install apache2-utils
+  ```
+
+- Encrypt the container registry credentials to create an `htpasswd` file
+  ```bash
+  htpasswd -bB -c container-registry/configmaps/config/htpasswd so-reg-user pXDwO5zLU7M88x3aA
+  ```
+
+  Resulting file should look like this
+  ```
+  cat container-registry/configmaps/config/htpasswd
+  # so-reg-user:$2y$05$6EdxIwwDNlJfNhhQxZRr4eNd.aYrdmbBjAdw422w0u2j3TihQXgd2
+  ```
+
+- Using the credentials from the previous steps, create a `container-registry/my_password.json` file
+  ```json
+  {
+    "auths": {
+      "container-registry.apps.vaasl.io": {
+        "username": "so-reg-user",
+        "password": "$2y$05$6EdxIwwDNlJfNhhQxZRr4eNd.aYrdmbBjAdw422w0u2j3TihQXgd2",
+        "auth": "c28tcmVnLXVzZXI6cFhEd081ekxVN004OHgzYUE="
+      }
+    }
+  }
+  ```
+
+- Configure the file `container-registry/config.env` as follows
+  ```env
+  REGISTRY_AUTH=htpasswd
+  REGISTRY_AUTH_HTPASSWD_REALM="VSL Service Provider Image Registry"
+  REGISTRY_AUTH_HTPASSWD_PATH="/config/htpasswd"
+  REGISTRY_HTTP_SECRET='$2y$05$6EdxIwwDNlJfNhhQxZRr4eNd.aYrdmbBjAdw422w0u2j3TihQXgd2'
+  ```
+
+- Load context for k8s
+  ```bash
+  kubie ctx vs-narwhal
+  ```
+
+- Add the container registry credentials as a secret available to the cluster
+  ```bash
+  kubectl create secret generic laconic-registry --from-file=.dockerconfigjson=container-registry/my_password.json --type=kubernetes.io/dockerconfigjson
+  ```
+
+### Run
+
+- Deploy the container registry
+  ```bash
+  laconic-so deployment --dir container-registry start
+  ```
+
+- Check the logs
+  ```bash
+  laconic-so deployment --dir container-registry logs
+  ```
+
+- Check status and await succesful deployment:
+  ```bash
+  laconic-so deployment --dir container-registry status
+  ```
+
+- Confirm deployment by logging in:
+  ```
+  docker login container-registry.apps.vaasl.io --username so-reg-user --password pXDwO5zLU7M88x3aA
+  ```
+
+- Set ingress annotations
+
+  - Set the `cluster-id` found in `container-registry/deployment.yml` and then run the following commands:
+    ```
+    export CLUSTER_ID=<cluster-id>
+    # Example
+    # export CLUSTER_ID=laconic-26cc70be8a3db3f4
+    
+    kubectl annotate ingress $CLUSTER_ID-ingress nginx.ingress.kubernetes.io/proxy-body-size=0
+    kubectl annotate ingress $CLUSTER_ID-ingress nginx.ingress.kubernetes.io/proxy-read-timeout=600
+    kubectl annotate ingress $CLUSTER_ID-ingress nginx.ingress.kubernetes.io/proxy-send-timeout=600
+    ```
+
+## webapp-deployer
+
+### Backend
+
+* Reference: <https://github.com/LaconicNetwork/loro-testnet/blob/main/docs/service-provider-setup.md#deploy-backend>
+
+* Target dir: `/srv/service-provider/webapp-deployer`
+
+* Cleanup an existing deployment if required:
+  ```bash
+  cd /srv/service-provider/webapp-deployer
+
+  # Stop the deployment
+  laconic-so deployment --dir webapp-deployer stop
+
+  # Remove the deployment dir
+  sudo rm -rf webapp-deployer
+
+  # Remove the existing spec file
+  rm webapp-deployer.spec
+  ```
+
+#### Setup
+
+- Initialize a spec file for the deployer backend.
+  ```bash
+  laconic-so --stack webapp-deployer-backend setup-repositories
+  laconic-so --stack webapp-deployer-backend build-containers
+  laconic-so --stack webapp-deployer-backend deploy init --output webapp-deployer.spec
+  ```
+
+- Modify the contents of `webapp-deployer.spec`:
+  ```
+  stack: webapp-deployer-backend
+  deploy-to: k8s
+  kube-config: /home/dev/.kube/config-vs-narwhal.yaml
+  image-registry: container-registry.apps.vaasl.io/laconic-registry
+  network:
+    ports:
+      server:
+        - '9555'
+    http-proxy:
+      - host-name: webapp-deployer-api.apps.vaasl.io
+        routes:
+          - path: '/'
+            proxy-to: server:9555
+  volumes:
+    srv:
+  configmaps:
+    config: ./data/config
+  annotations:
+    container.apparmor.security.beta.kubernetes.io/{name}: unconfined
+  labels:
+    container.kubeaudit.io/{name}.allow-disabled-apparmor: "podman"
+  security:
+    privileged: true
+
+  resources:
+    containers:
+      reservations:
+        cpus: 3
+        memory: 8G
+      limits:
+        cpus: 7
+        memory: 16G
+    volumes:
+      reservations:
+        storage: 200G
+  ```
+
+- Create the deployment directory from the spec file.
+  ```
+  laconic-so --stack webapp-deployer-backend deploy create --deployment-dir webapp-deployer --spec-file webapp-deployer.spec
+  ```
+
+- Modify file `webapp-deployer/kubeconfig.yml` if required
+  ```
+  apiVersion: v1
+  ...
+  contexts:
+    - context:
+        cluster: ***
+        user: ***
+      name: default
+  ...
+  ```
+  NOTE: `context.name` must be default to use with SO
+
+- Copy `webapp-deployer/kubeconfig.yml` from the k8s cluster creation step to `webapp-deployer/data/config/kube.yml`
+  ```bash
+  cp webapp-deployer/kubeconfig.yml webapp-deployer/data/config/kube.yml
+  ```
+
+- Create `webapp-deployer/data/config/laconic.yml`, it should look like this:
+  ```
+  services:
+    registry:
+      # Using public endpoint does not work inside machine where laconicd chain is deployed
+      rpcEndpoint: 'http://host.docker.internal:36657'
+      gqlEndpoint: 'http://host.docker.internal:3473/api'
+
+      # Set user key of account with balance and bond owned by the user
+      userKey:
+      bondId:
+
+      chainId: laconic-testnet-2
+      gasPrice: 1alnt
+  ```
+  NOTE: Modify the user key and bond ID according to your configuration
+
+* Publish a `WebappDeployer` record for the deployer backend by following the steps below:
+
+  * Setup GPG keys by following [these steps to create and export a key](https://git.vdb.to/cerc-io/webapp-deployment-status-api#keys)
+    ```
+    cd webapp-deployer
+
+    # Create a key
+    gpg --batch --passphrase "SECRET" --quick-generate-key webapp-deployer-api.apps.vaasl.io default default never
+
+    # Export the public key
+    gpg --export webapp-deployer-api.apps.vaasl.io > webapp-deployer-api.apps.vaasl.io.pgp.pub
+
+    # Export the private key
+    gpg --export-secret-keys webapp-deployer-api.apps.vaasl.io > webapp-deployer-api.apps.vaasl.io.pgp.key
+
+    cd -
+    ```
+    NOTE: Use "SECRET" for passphrase prompt
+
+  * Copy the GPG pub key file generated above to `webapp-deployer/data/config` directory. This ensures the Docker container has access to the key during the publish process
+    ```bash
+    cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.pub webapp-deployer/data/config
+    ```
+
+
+  * Publish the webapp deployer record using the `publish-deployer-to-registry` command
+
+    ```
+    docker run -i -t \
+      --add-host=host.docker.internal:host-gateway \
+      -v /srv/service-provider/webapp-deployer/data/config:/config \
+      cerc/webapp-deployer-backend:local laconic-so publish-deployer-to-registry \
+        --laconic-config /config/laconic.yml \
+        --api-url https://webapp-deployer-api.apps.vaasl.io \
+        --public-key-file /config/webapp-deployer-api.apps.vaasl.io.pgp.pub \
+        --lrn lrn://vaasl-provider/deployers/webapp-deployer-api.apps.vaasl.io \
+        --min-required-payment 10000
+    ```
+
+- Modify the contents of `webapp-deployer/config.env`:
+
+  ```
+  DEPLOYMENT_DNS_SUFFIX="apps.vaasl.io"
+
+  # this should match the name authority reserved above
+  DEPLOYMENT_RECORD_NAMESPACE="vaasl-provider"
+
+  # url of the deployed docker image registry
+  IMAGE_REGISTRY="container-registry.apps.vaasl.io"
+
+  # credentials from the htpasswd section above in container-registry setup
+  IMAGE_REGISTRY_USER=
+  IMAGE_REGISTRY_CREDS=
+
+  # configs
+  CLEAN_DEPLOYMENTS=false
+  CLEAN_LOGS=false
+  CLEAN_CONTAINERS=false
+  SYSTEM_PRUNE=false
+  WEBAPP_IMAGE_PRUNE=true
+  CHECK_INTERVAL=10
+  FQDN_POLICY="allow"
+
+  # lrn of the webapp deployer
+  LRN="lrn://vaasl-provider/deployers/webapp-deployer-api.apps.vaasl.io"
+
+  # Path to the GPG key file inside the webapp-deployer container
+  OPENPGP_PRIVATE_KEY_FILE="webapp-deployer-api.apps.vaasl.io.pgp.key"
+  # Passphrase used when creating the GPG key
+  OPENPGP_PASSPHRASE="SECRET"
+  
+  DEPLOYER_STATE="srv-test/deployments/autodeploy.state"
+  UNDEPLOYER_STATE="srv-test/deployments/autoundeploy.state"
+  UPLOAD_DIRECTORY="srv-test/uploads"
+  HANDLE_AUCTION_REQUESTS=true
+  AUCTION_BID_AMOUNT=10000
+
+  # Minimum payment amount required for single webapp deployment
+  MIN_REQUIRED_PAYMENT=10000
+  ```
+
+- Push the image to the container registry
+  ```
+  laconic-so deployment --dir webapp-deployer push-images
+  ```
+
+- Modify `webapp-deployer/data/config/laconic.yml`:
+  ```
+  services:
+    registry:
+      rpcEndpoint: 'https://laconicd-sapo.laconic.com/'
+      gqlEndpoint: 'https://laconicd-sapo.laconic.com/api'
+
+      # Set user key of account with balance and bond owned by the user
+      userKey:
+      bondId:
+
+      chainId: laconic-testnet-2
+      gasPrice: 1alnt
+  ```
+
+#### Run
+
+- Start the deployer.
+  ```
+  laconic-so deployment --dir webapp-deployer start
+  ```
+
+- Load context for k8s
+  ```bash
+  kubie ctx vs-narwhal
+  ```
+
+- Copy the GPG key file to the webapp-deployer container
+
+  ```bash
+  # Get the webapp-deployer pod id
+  laconic-so deployment --dir webapp-deployer ps
+  
+  # Expected output
+  # Running containers:
+  # id: default/laconic-096fed46af974a47-deployment-644db859c7-snbq6, name: laconic-096fed46af974a47-deployment-644db859c7-snbq6, ports: 10.42.2.11:9555->9555
+
+  # Set pod id
+  export POD_ID=
+  # Example:
+  # export POD_ID=laconic-096fed46af974a47-deployment-644db859c7-snbq6
+
+  # Copy GPG key files to the pod
+  kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.key $POD_ID:/app
+  kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.pub $POD_ID:/app
+  ```
+
+- Publishing records to the registry will trigger deployments in backend now
+
+### Frontend
+
+* Target dir: `/srv/service-provider/webapp-ui`
+
+* Cleanup an existing deployment if required:
+  ```bash
+  cd /srv/service-provider/webapp-ui
+
+  # Stop the deployment
+  laconic-so deployment --dir webapp-ui stop
+
+  # Remove the deployment dir
+  sudo rm -rf webapp-ui
+
+  # Remove the existing spec file
+  rm webapp-ui.spec
+  ```
+
+#### Setup
+
+* Clone and build the deployer UI
+  ```
+  git clone https://git.vdb.to/cerc-io/webapp-deployment-status-ui.git ~/cerc/webapp-deployment-status-ui
+
+  laconic-so build-webapp --source-repo ~/cerc/webapp-deployment-status-ui
+  ```
+
+* Create a deployment
+  ```bash
+  export KUBECONFIG_PATH=/home/dev/.kube/config-vs-narwhal.yaml
+  # NOTE: Use actual kubeconfig path
+
+  laconic-so deploy-webapp create --kube-config $KUBECONFIG_PATH --image-registry container-registry.apps.vaasl.io --deployment-dir webapp-ui --image cerc/webapp-deployment-status-ui:local --url https://webapp-deployer-ui.apps.vaasl.io --env-file ~/cerc/webapp-deployment-status-ui/.env
+  ```
+
+* Modify file `webapp-ui/kubeconfig.yml` if required
+  ```yml
+  apiVersion: v1
+  ...
+  contexts:
+    - context:
+        cluster: ***
+        user: ***
+      name: default
+  ...
+  ```
+  NOTE: `context.name` must be default to use with SO
+
+- Push the image to the container registry.
+  ```
+  laconic-so deployment --dir webapp-ui push-images
+  ```
+
+- Modify `webapp-ui/config.env` like [this Pull Request](https://git.vdb.to/cerc-io/webapp-deployment-status-ui/pulls/6) but with your host details.
+
+#### Run
+
+- Start the deployer UI
+  ```bash
+  laconic-so deployment --dir webapp-ui start
+  ```
+
+- Wait a moment, then go to https://webapp-deployer-ui.apps.vaasl.io for the status and logs of each deployment

ops/update-deployments.md

@@ -325,6 +325,128 @@ Instructions to reset / update the deployments
   laconic-so deployment --dir laconic-shopify-deployment start
   ```
 
+## webapp-deployer
+
+### Backend
+
+* Deployment dir: `/srv/service-provider/webapp-deployer`
+
+* If code has changed, fetch and build with updated source code:
+
+  ```bash
+  laconic-so --stack webapp-deployer-backend setup-repositories --git-ssh --pull
+
+  laconic-so --stack webapp-deployer-backend build-containers --force-rebuild
+  ```
+
+* Update the configuration, if required in :
+  * `/srv/service-provider/webapp-deployer/data/config/laconic.yml`
+  * `/srv/service-provider/webapp-deployer/config.env`
+
+* Restart the deployment:
+
+  ```bash
+  laconic-so deployment --dir webapp-deployer stop
+
+  laconic-so deployment --dir webapp-deployer start
+  ```
+
+- Load context for k8s
+  ```bash
+  kubie ctx vs-narwhal
+  ```
+
+- Copy the GPG key file to the webapp-deployer container
+
+  ```bash
+  # Get the webapp-deployer pod id
+  laconic-so deployment --dir webapp-deployer ps
+  
+  # Expected output
+  # Running containers:
+  # id: default/laconic-096fed46af974a47-deployment-644db859c7-snbq6, name: laconic-096fed46af974a47-deployment-644db859c7-snbq6, ports: 10.42.2.11:9555->9555
+
+  # Set pod id
+  export POD_ID=
+  # Example:
+  # export POD_ID=laconic-096fed46af974a47-deployment-644db859c7-snbq6
+
+  # Copy GPG key files to the pod
+  kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.key $POD_ID:/app
+  kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.pub $POD_ID:/app
+  ```
+
+### Frontend
+
+* Deployment dir: `/srv/service-provider/webapp-ui`
+
+* If code has changed, fetch and build with updated source code:
+
+  ```bash
+  cd ~/cerc/webapp-deployment-status-ui
+
+  # Pull latest changes, or checkout to the required branch
+  git pull
+
+  # Confirm the latest commit hash
+  git log
+
+  laconic-so build-webapp --source-repo ~/cerc/webapp-deployment-status-ui
+  ```
+
+- Modify `/srv/service-provider/webapp-ui/config.env` like [this Pull Request](https://git.vdb.to/cerc-io/webapp-deployment-status-ui/pulls/6) but with your host details.
+
+* Restart the deployment:
+
+  ```bash
+  laconic-so deployment --dir webapp-ui stop
+
+  laconic-so deployment --dir webapp-ui start
+  ```
+
+## Deploy Backend
+
+* Deployment dir: `/srv/deploy-backend/backend-deployment`
+
+* If code has changed, fetch and build with updated source code:
+
+  ```bash
+  laconic-so --stack ~/cerc/snowballtools-base-api-stack/stack-orchestrator/stacks/snowballtools-base-backend setup-repositories --git-ssh --pull
+
+  # rebuild containers
+  laconic-so --stack ~/cerc/snowballtools-base-api-stack/stack-orchestrator/stacks/snowballtools-base-backend build-containers --force-rebuild
+  ```
+
+* Push updated images to the container registry:
+
+  ```bash
+  cd /srv/deploy-backend
+
+  # login to container registry
+  CONTAINER_REGISTRY_URL=container-registry.apps.vaasl.io
+  CONTAINER_REGISTRY_USERNAME=
+  CONTAINER_REGISTRY_PASSWORD=
+
+  docker login $CONTAINER_REGISTRY_URL --username $CONTAINER_REGISTRY_USERNAME --password $CONTAINER_REGISTRY_PASSWORD
+
+  # Push backend images
+  laconic-so deployment --dir backend-deployment push-images
+  ```
+
+* Update the configuration if required in `backend-deployment/configmaps/config/prod.toml`
+
+* Restart the deployment:
+
+  ```bash
+  laconic-so deployment --dir backend-deployment stop
+
+  laconic-so deployment --dir backend-deployment start
+  ```
+
+## Deply Frontend
+
+* Follow steps from [deployments-from-scratch.md](./deployments-from-scratch.md#deploy-frontend) to deploy the snowball frontend
+
 ## Fixturenet Eth
 
 * Deployment dir: `/srv/fixturenet-eth/fixturenet-eth-deployment`
@@ -376,46 +498,3 @@ Instructions to reset / update the deployments
 
   laconic-so deployment --dir bridge-deployment start
   ```
-
-## Backend Deployment
-
-* Deployment dir: `/srv/deploy-backend/backend-deployment`
-
-* If code has changed, fetch and build with updated source code:
-
-  ```bash
-  laconic-so --stack ~/cerc/snowballtools-base-api-stack/stack-orchestrator/stacks/snowballtools-base-backend setup-repositories --git-ssh --pull
-
-  # rebuild containers
-  laconic-so --stack ~/cerc/snowballtools-base-api-stack/stack-orchestrator/stacks/snowballtools-base-backend build-containers --force-rebuild
-  ```
-
-* Push updated images to the container registry:
-
-  ```bash
-  cd /srv/deploy-backend
-
-  # login to container registry
-  CONTAINER_REGISTRY_URL=container-registry.apps.vaasl.io
-  CONTAINER_REGISTRY_USERNAME=
-  CONTAINER_REGISTRY_PASSWORD=
-
-  docker login $CONTAINER_REGISTRY_URL --username $CONTAINER_REGISTRY_USERNAME --password $CONTAINER_REGISTRY_PASSWORD
-
-  # Push backend images
-  laconic-so deployment --dir backend-deployment push-images
-  ```
-
-* Update the configuration if required in `backend-deployment/configmaps/config/prod.toml`
-
-* Restart the deployment:
-
-  ```bash
-  laconic-so deployment --dir backend-deployment stop
-
-  laconic-so deployment --dir backend-deployment start
-  ```
-
-## Frontend Deployment
-
-* Follow steps from [deployments-from-scratch.md](./deployments-from-scratch.md#deploy-frontend) to deploy the snowball frontend