diff --git a/ops/deployments-from-scratch.md b/ops/deployments-from-scratch.md index 4d71ea4..0d6cd29 100644 --- a/ops/deployments-from-scratch.md +++ b/ops/deployments-from-scratch.md @@ -976,6 +976,10 @@ +## Service Provider + +Follow [sp-deployment-from-scratch.md](./sp-deployment-from-scratch.md) +
deploy-backend @@ -1755,6 +1759,11 @@ https://console-sapo.laconic.com -> 4002 Open p2p ports: 36656 +# Service provider (deployed on K8s) +https://container-registry.apps.vaasl.io +https://webapp-deployer-api.apps.vaasl.io +https://webapp-deployer-ui.apps.vaasl.io + # Machine 2 https://sepolia.laconic.com -> 8545 wss://sepolia.laconic.com -> 8546 diff --git a/ops/sp-deployment-from-scratch.md b/ops/sp-deployment-from-scratch.md new file mode 100644 index 0000000..23e103b --- /dev/null +++ b/ops/sp-deployment-from-scratch.md @@ -0,0 +1,465 @@ +# Service Provider deployments from scratch + +## container-registry + +* Reference: + +* Target dir: `/srv/service-provider/container-registry` + +* Cleanup an existing deployment if required: + ```bash + cd /srv/service-provider/container-registry + + # Stop the deployment + laconic-so deployment --dir container-registry stop --delete-volumes + + # Remove the deployment dir + sudo rm -rf container-registrty + + # Remove the existing spec file + rm container-registry.spec + ``` + +### Setup + +- Generate the spec file for the container-registry stack + ```bash + laconic-so --stack container-registry deploy init --output container-registry.spec + ``` + +- Modify the `container-registry.spec` as shown below + ``` + stack: container-registry + deploy-to: k8s + kube-config: /home/dev/.kube/config-vs-narwhal.yaml + network: + ports: + registry: + - '5000' + http-proxy: + - host-name: container-registry.apps.vaasl.io + routes: + - path: '/' + proxy-to: registry:5000 + volumes: + registry-data: + configmaps: + config: ./configmaps/config + ``` + +- Create the deployment directory for the `container-registry` stack + ```bash + laconic-so --stack container-registry deploy create --deployment-dir container-registry --spec-file container-registry.spec + ``` + +- Modify file `container-registry/kubeconfig.yml` if required + ``` + apiVersion: v1 + ... + contexts: + - context: + cluster: *** + user: *** + name: default + ... + ``` + NOTE: `context.name` must be default to use with SO + +- Base64 encode the container registry credentials + NOTE: Use actual credentials for container registry (credentials set in `container-registry/credentials.txt`) + ```bash + echo -n "so-reg-user:pXDwO5zLU7M88x3aA" | base64 -w0 + + # Output: c28tcmVnLXVzZXI6cFhEd081ekxVN004OHgzYUE= + ``` + +- Install `apache2-utils` for next step + ```bash + sudo apt install apache2-utils + ``` + +- Encrypt the container registry credentials to create an `htpasswd` file + ```bash + htpasswd -bB -c container-registry/configmaps/config/htpasswd so-reg-user pXDwO5zLU7M88x3aA + ``` + + Resulting file should look like this + ``` + cat container-registry/configmaps/config/htpasswd + # so-reg-user:$2y$05$6EdxIwwDNlJfNhhQxZRr4eNd.aYrdmbBjAdw422w0u2j3TihQXgd2 + ``` + +- Using the credentials from the previous steps, create a `container-registry/my_password.json` file + ```json + { + "auths": { + "container-registry.apps.vaasl.io": { + "username": "so-reg-user", + "password": "$2y$05$6EdxIwwDNlJfNhhQxZRr4eNd.aYrdmbBjAdw422w0u2j3TihQXgd2", + "auth": "c28tcmVnLXVzZXI6cFhEd081ekxVN004OHgzYUE=" + } + } + } + ``` + +- Configure the file `container-registry/config.env` as follows + ```env + REGISTRY_AUTH=htpasswd + REGISTRY_AUTH_HTPASSWD_REALM="VSL Service Provider Image Registry" + REGISTRY_AUTH_HTPASSWD_PATH="/config/htpasswd" + REGISTRY_HTTP_SECRET='$2y$05$6EdxIwwDNlJfNhhQxZRr4eNd.aYrdmbBjAdw422w0u2j3TihQXgd2' + ``` + +- Load context for k8s + ```bash + kubie ctx vs-narwhal + ``` + +- Add the container registry credentials as a secret available to the cluster + ```bash + kubectl create secret generic laconic-registry --from-file=.dockerconfigjson=container-registry/my_password.json --type=kubernetes.io/dockerconfigjson + ``` + +### Run + +- Deploy the container registry + ```bash + laconic-so deployment --dir container-registry start + ``` + +- Check the logs + ```bash + laconic-so deployment --dir container-registry logs + ``` + +- Check status and await succesful deployment: + ```bash + laconic-so deployment --dir container-registry status + ``` + +- Confirm deployment by logging in: + ``` + docker login container-registry.apps.vaasl.io --username so-reg-user --password pXDwO5zLU7M88x3aA + ``` + +- Set ingress annotations + + - Set the `cluster-id` found in `container-registry/deployment.yml` and then run the following commands: + ``` + export CLUSTER_ID= + # Example + # export CLUSTER_ID=laconic-26cc70be8a3db3f4 + + kubectl annotate ingress $CLUSTER_ID-ingress nginx.ingress.kubernetes.io/proxy-body-size=0 + kubectl annotate ingress $CLUSTER_ID-ingress nginx.ingress.kubernetes.io/proxy-read-timeout=600 + kubectl annotate ingress $CLUSTER_ID-ingress nginx.ingress.kubernetes.io/proxy-send-timeout=600 + ``` + +## webapp-deployer + +### Backend + +* Reference: + +* Target dir: `/srv/service-provider/webapp-deployer` + +* Cleanup an existing deployment if required: + ```bash + cd /srv/service-provider/webapp-deployer + + # Stop the deployment + laconic-so deployment --dir webapp-deployer stop + + # Remove the deployment dir + sudo rm -rf webapp-deployer + + # Remove the existing spec file + rm webapp-deployer.spec + ``` + +#### Setup + +- Initialize a spec file for the deployer backend. + ```bash + laconic-so --stack webapp-deployer-backend setup-repositories + laconic-so --stack webapp-deployer-backend build-containers + laconic-so --stack webapp-deployer-backend deploy init --output webapp-deployer.spec + ``` + +- Modify the contents of `webapp-deployer.spec`: + ``` + stack: webapp-deployer-backend + deploy-to: k8s + kube-config: /home/dev/.kube/config-vs-narwhal.yaml + image-registry: container-registry.apps.vaasl.io/laconic-registry + network: + ports: + server: + - '9555' + http-proxy: + - host-name: webapp-deployer-api.apps.vaasl.io + routes: + - path: '/' + proxy-to: server:9555 + volumes: + srv: + configmaps: + config: ./data/config + annotations: + container.apparmor.security.beta.kubernetes.io/{name}: unconfined + labels: + container.kubeaudit.io/{name}.allow-disabled-apparmor: "podman" + security: + privileged: true + + resources: + containers: + reservations: + cpus: 3 + memory: 8G + limits: + cpus: 7 + memory: 16G + volumes: + reservations: + storage: 200G + ``` + +- Create the deployment directory from the spec file. + ``` + laconic-so --stack webapp-deployer-backend deploy create --deployment-dir webapp-deployer --spec-file webapp-deployer.spec + ``` + +- Modify file `webapp-deployer/kubeconfig.yml` if required + ``` + apiVersion: v1 + ... + contexts: + - context: + cluster: *** + user: *** + name: default + ... + ``` + NOTE: `context.name` must be default to use with SO + +- Copy `webapp-deployer/kubeconfig.yml` from the k8s cluster creation step to `webapp-deployer/data/config/kube.yml` + ```bash + cp webapp-deployer/kubeconfig.yml webapp-deployer/data/config/kube.yml + ``` + +- Create `webapp-deployer/data/config/laconic.yml`, it should look like this: + ``` + services: + registry: + # Using public endpoint does not work inside machine where laconicd chain is deployed + rpcEndpoint: 'http://host.docker.internal:36657' + gqlEndpoint: 'http://host.docker.internal:3473/api' + + # Set user key of account with balance and bond owned by the user + userKey: + bondId: + + chainId: laconic-testnet-2 + gasPrice: 1alnt + ``` + NOTE: Modify the user key and bond ID according to your configuration + +* Publish a `WebappDeployer` record for the deployer backend by following the steps below: + + * Setup GPG keys by following [these steps to create and export a key](https://git.vdb.to/cerc-io/webapp-deployment-status-api#keys) + ``` + cd webapp-deployer + + # Create a key + gpg --batch --passphrase "SECRET" --quick-generate-key webapp-deployer-api.apps.vaasl.io default default never + + # Export the public key + gpg --export webapp-deployer-api.apps.vaasl.io > webapp-deployer-api.apps.vaasl.io.pgp.pub + + # Export the private key + gpg --export-secret-keys webapp-deployer-api.apps.vaasl.io > webapp-deployer-api.apps.vaasl.io.pgp.key + + cd - + ``` + NOTE: Use "SECRET" for passphrase prompt + + * Copy the GPG pub key file generated above to `webapp-deployer/data/config` directory. This ensures the Docker container has access to the key during the publish process + ```bash + cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.pub webapp-deployer/data/config + ``` + + + * Publish the webapp deployer record using the `publish-deployer-to-registry` command + + ``` + docker run -i -t \ + --add-host=host.docker.internal:host-gateway \ + -v /srv/service-provider/webapp-deployer/data/config:/config \ + cerc/webapp-deployer-backend:local laconic-so publish-deployer-to-registry \ + --laconic-config /config/laconic.yml \ + --api-url https://webapp-deployer-api.apps.vaasl.io \ + --public-key-file /config/webapp-deployer-api.apps.vaasl.io.pgp.pub \ + --lrn lrn://vaasl-provider/deployers/webapp-deployer-api.apps.vaasl.io \ + --min-required-payment 10000 + ``` + +- Modify the contents of `webapp-deployer/config.env`: + + ``` + DEPLOYMENT_DNS_SUFFIX="apps.vaasl.io" + + # this should match the name authority reserved above + DEPLOYMENT_RECORD_NAMESPACE="vaasl-provider" + + # url of the deployed docker image registry + IMAGE_REGISTRY="container-registry.apps.vaasl.io" + + # credentials from the htpasswd section above in container-registry setup + IMAGE_REGISTRY_USER= + IMAGE_REGISTRY_CREDS= + + # configs + CLEAN_DEPLOYMENTS=false + CLEAN_LOGS=false + CLEAN_CONTAINERS=false + SYSTEM_PRUNE=false + WEBAPP_IMAGE_PRUNE=true + CHECK_INTERVAL=10 + FQDN_POLICY="allow" + + # lrn of the webapp deployer + LRN="lrn://vaasl-provider/deployers/webapp-deployer-api.apps.vaasl.io" + + # Path to the GPG key file inside the webapp-deployer container + OPENPGP_PRIVATE_KEY_FILE="webapp-deployer-api.apps.vaasl.io.pgp.key" + # Passphrase used when creating the GPG key + OPENPGP_PASSPHRASE="SECRET" + + DEPLOYER_STATE="srv-test/deployments/autodeploy.state" + UNDEPLOYER_STATE="srv-test/deployments/autoundeploy.state" + UPLOAD_DIRECTORY="srv-test/uploads" + HANDLE_AUCTION_REQUESTS=true + AUCTION_BID_AMOUNT=10000 + + # Minimum payment amount required for single webapp deployment + MIN_REQUIRED_PAYMENT=10000 + ``` + +- Push the image to the container registry + ``` + laconic-so deployment --dir webapp-deployer push-images + ``` + +- Modify `webapp-deployer/data/config/laconic.yml`: + ``` + services: + registry: + rpcEndpoint: 'https://laconicd-sapo.laconic.com/' + gqlEndpoint: 'https://laconicd-sapo.laconic.com/api' + + # Set user key of account with balance and bond owned by the user + userKey: + bondId: + + chainId: laconic-testnet-2 + gasPrice: 1alnt + ``` + +#### Run + +- Start the deployer. + ``` + laconic-so deployment --dir webapp-deployer start + ``` + +- Load context for k8s + ```bash + kubie ctx vs-narwhal + ``` + +- Copy the GPG key file to the webapp-deployer container + + ```bash + # Get the webapp-deployer pod id + laconic-so deployment --dir webapp-deployer ps + + # Expected output + # Running containers: + # id: default/laconic-096fed46af974a47-deployment-644db859c7-snbq6, name: laconic-096fed46af974a47-deployment-644db859c7-snbq6, ports: 10.42.2.11:9555->9555 + + # Set pod id + export POD_ID= + # Example: + # export POD_ID=laconic-096fed46af974a47-deployment-644db859c7-snbq6 + + # Copy GPG key files to the pod + kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.key $POD_ID:/app + kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.pub $POD_ID:/app + ``` + +- Publishing records to the registry will trigger deployments in backend now + +### Frontend + +* Target dir: `/srv/service-provider/webapp-ui` + +* Cleanup an existing deployment if required: + ```bash + cd /srv/service-provider/webapp-ui + + # Stop the deployment + laconic-so deployment --dir webapp-ui stop + + # Remove the deployment dir + sudo rm -rf webapp-ui + + # Remove the existing spec file + rm webapp-ui.spec + ``` + +#### Setup + +* Clone and build the deployer UI + ``` + git clone https://git.vdb.to/cerc-io/webapp-deployment-status-ui.git ~/cerc/webapp-deployment-status-ui + + laconic-so build-webapp --source-repo ~/cerc/webapp-deployment-status-ui + ``` + +* Create a deployment + ```bash + export KUBECONFIG_PATH=/home/dev/.kube/config-vs-narwhal.yaml + # NOTE: Use actual kubeconfig path + + laconic-so deploy-webapp create --kube-config $KUBECONFIG_PATH --image-registry container-registry.apps.vaasl.io --deployment-dir webapp-ui --image cerc/webapp-deployment-status-ui:local --url https://webapp-deployer-ui.apps.vaasl.io --env-file ~/cerc/webapp-deployment-status-ui/.env + ``` + +* Modify file `webapp-ui/kubeconfig.yml` if required + ```yml + apiVersion: v1 + ... + contexts: + - context: + cluster: *** + user: *** + name: default + ... + ``` + NOTE: `context.name` must be default to use with SO + +- Push the image to the container registry. + ``` + laconic-so deployment --dir webapp-ui push-images + ``` + +- Modify `webapp-ui/config.env` like [this Pull Request](https://git.vdb.to/cerc-io/webapp-deployment-status-ui/pulls/6) but with your host details. + +#### Run + +- Start the deployer UI + ```bash + laconic-so deployment --dir webapp-ui start + ``` + +- Wait a moment, then go to https://webapp-deployer-ui.apps.vaasl.io for the status and logs of each deployment diff --git a/ops/update-deployments.md b/ops/update-deployments.md index 4db6067..220a7ae 100644 --- a/ops/update-deployments.md +++ b/ops/update-deployments.md @@ -325,6 +325,128 @@ Instructions to reset / update the deployments laconic-so deployment --dir laconic-shopify-deployment start ``` +## webapp-deployer + +### Backend + +* Deployment dir: `/srv/service-provider/webapp-deployer` + +* If code has changed, fetch and build with updated source code: + + ```bash + laconic-so --stack webapp-deployer-backend setup-repositories --git-ssh --pull + + laconic-so --stack webapp-deployer-backend build-containers --force-rebuild + ``` + +* Update the configuration, if required in : + * `/srv/service-provider/webapp-deployer/data/config/laconic.yml` + * `/srv/service-provider/webapp-deployer/config.env` + +* Restart the deployment: + + ```bash + laconic-so deployment --dir webapp-deployer stop + + laconic-so deployment --dir webapp-deployer start + ``` + +- Load context for k8s + ```bash + kubie ctx vs-narwhal + ``` + +- Copy the GPG key file to the webapp-deployer container + + ```bash + # Get the webapp-deployer pod id + laconic-so deployment --dir webapp-deployer ps + + # Expected output + # Running containers: + # id: default/laconic-096fed46af974a47-deployment-644db859c7-snbq6, name: laconic-096fed46af974a47-deployment-644db859c7-snbq6, ports: 10.42.2.11:9555->9555 + + # Set pod id + export POD_ID= + # Example: + # export POD_ID=laconic-096fed46af974a47-deployment-644db859c7-snbq6 + + # Copy GPG key files to the pod + kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.key $POD_ID:/app + kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.pub $POD_ID:/app + ``` + +### Frontend + +* Deployment dir: `/srv/service-provider/webapp-ui` + +* If code has changed, fetch and build with updated source code: + + ```bash + cd ~/cerc/webapp-deployment-status-ui + + # Pull latest changes, or checkout to the required branch + git pull + + # Confirm the latest commit hash + git log + + laconic-so build-webapp --source-repo ~/cerc/webapp-deployment-status-ui + ``` + +- Modify `/srv/service-provider/webapp-ui/config.env` like [this Pull Request](https://git.vdb.to/cerc-io/webapp-deployment-status-ui/pulls/6) but with your host details. + +* Restart the deployment: + + ```bash + laconic-so deployment --dir webapp-ui stop + + laconic-so deployment --dir webapp-ui start + ``` + +## Deploy Backend + +* Deployment dir: `/srv/deploy-backend/backend-deployment` + +* If code has changed, fetch and build with updated source code: + + ```bash + laconic-so --stack ~/cerc/snowballtools-base-api-stack/stack-orchestrator/stacks/snowballtools-base-backend setup-repositories --git-ssh --pull + + # rebuild containers + laconic-so --stack ~/cerc/snowballtools-base-api-stack/stack-orchestrator/stacks/snowballtools-base-backend build-containers --force-rebuild + ``` + +* Push updated images to the container registry: + + ```bash + cd /srv/deploy-backend + + # login to container registry + CONTAINER_REGISTRY_URL=container-registry.apps.vaasl.io + CONTAINER_REGISTRY_USERNAME= + CONTAINER_REGISTRY_PASSWORD= + + docker login $CONTAINER_REGISTRY_URL --username $CONTAINER_REGISTRY_USERNAME --password $CONTAINER_REGISTRY_PASSWORD + + # Push backend images + laconic-so deployment --dir backend-deployment push-images + ``` + +* Update the configuration if required in `backend-deployment/configmaps/config/prod.toml` + +* Restart the deployment: + + ```bash + laconic-so deployment --dir backend-deployment stop + + laconic-so deployment --dir backend-deployment start + ``` + +## Deply Frontend + +* Follow steps from [deployments-from-scratch.md](./deployments-from-scratch.md#deploy-frontend) to deploy the snowball frontend + ## Fixturenet Eth * Deployment dir: `/srv/fixturenet-eth/fixturenet-eth-deployment` @@ -376,46 +498,3 @@ Instructions to reset / update the deployments laconic-so deployment --dir bridge-deployment start ``` - -## Backend Deployment - -* Deployment dir: `/srv/deploy-backend/backend-deployment` - -* If code has changed, fetch and build with updated source code: - - ```bash - laconic-so --stack ~/cerc/snowballtools-base-api-stack/stack-orchestrator/stacks/snowballtools-base-backend setup-repositories --git-ssh --pull - - # rebuild containers - laconic-so --stack ~/cerc/snowballtools-base-api-stack/stack-orchestrator/stacks/snowballtools-base-backend build-containers --force-rebuild - ``` - -* Push updated images to the container registry: - - ```bash - cd /srv/deploy-backend - - # login to container registry - CONTAINER_REGISTRY_URL=container-registry.apps.vaasl.io - CONTAINER_REGISTRY_USERNAME= - CONTAINER_REGISTRY_PASSWORD= - - docker login $CONTAINER_REGISTRY_URL --username $CONTAINER_REGISTRY_USERNAME --password $CONTAINER_REGISTRY_PASSWORD - - # Push backend images - laconic-so deployment --dir backend-deployment push-images - ``` - -* Update the configuration if required in `backend-deployment/configmaps/config/prod.toml` - -* Restart the deployment: - - ```bash - laconic-so deployment --dir backend-deployment stop - - laconic-so deployment --dir backend-deployment start - ``` - -## Frontend Deployment - -* Follow steps from [deployments-from-scratch.md](./deployments-from-scratch.md#deploy-frontend) to deploy the snowball frontend