Add deployment steps from scratch for vaasl SP #39

Merged
nabarun merged 1 commits from deep-stack/testnet-laconicd-stack:ng-sp-setup into main 2024-11-04 11:33:11 +00:00
3 changed files with 596 additions and 43 deletions

View File

@ -976,6 +976,10 @@
</details>
## Service Provider
Follow [sp-deployment-from-scratch.md](./sp-deployment-from-scratch.md)
<details open>
<summary>deploy-backend</summary>
@ -1755,6 +1759,11 @@ https://console-sapo.laconic.com -> 4002
Open p2p ports:
36656
# Service provider (deployed on K8s)
https://container-registry.apps.vaasl.io
https://webapp-deployer-api.apps.vaasl.io
https://webapp-deployer-ui.apps.vaasl.io
# Machine 2
https://sepolia.laconic.com -> 8545
wss://sepolia.laconic.com -> 8546

View File

@ -0,0 +1,465 @@
# Service Provider deployments from scratch
## container-registry
* Reference: <https://github.com/LaconicNetwork/loro-testnet/blob/main/docs/service-provider-setup.md#deploy-docker-image-container-registry>
* Target dir: `/srv/service-provider/container-registry`
* Cleanup an existing deployment if required:
```bash
cd /srv/service-provider/container-registry
# Stop the deployment
laconic-so deployment --dir container-registry stop --delete-volumes
# Remove the deployment dir
sudo rm -rf container-registrty
# Remove the existing spec file
rm container-registry.spec
```
### Setup
- Generate the spec file for the container-registry stack
```bash
laconic-so --stack container-registry deploy init --output container-registry.spec
```
- Modify the `container-registry.spec` as shown below
```
stack: container-registry
deploy-to: k8s
kube-config: /home/dev/.kube/config-vs-narwhal.yaml
network:
ports:
registry:
- '5000'
http-proxy:
- host-name: container-registry.apps.vaasl.io
routes:
- path: '/'
proxy-to: registry:5000
volumes:
registry-data:
configmaps:
config: ./configmaps/config
```
- Create the deployment directory for the `container-registry` stack
```bash
laconic-so --stack container-registry deploy create --deployment-dir container-registry --spec-file container-registry.spec
```
- Modify file `container-registry/kubeconfig.yml` if required
```
apiVersion: v1
...
contexts:
- context:
cluster: ***
user: ***
name: default
...
```
NOTE: `context.name` must be default to use with SO
- Base64 encode the container registry credentials
NOTE: Use actual credentials for container registry (credentials set in `container-registry/credentials.txt`)
```bash
echo -n "so-reg-user:pXDwO5zLU7M88x3aA" | base64 -w0
# Output: c28tcmVnLXVzZXI6cFhEd081ekxVN004OHgzYUE=
```
- Install `apache2-utils` for next step
```bash
sudo apt install apache2-utils
```
- Encrypt the container registry credentials to create an `htpasswd` file
```bash
htpasswd -bB -c container-registry/configmaps/config/htpasswd so-reg-user pXDwO5zLU7M88x3aA
```
Resulting file should look like this
```
cat container-registry/configmaps/config/htpasswd
# so-reg-user:$2y$05$6EdxIwwDNlJfNhhQxZRr4eNd.aYrdmbBjAdw422w0u2j3TihQXgd2
```
- Using the credentials from the previous steps, create a `container-registry/my_password.json` file
```json
{
"auths": {
"container-registry.apps.vaasl.io": {
"username": "so-reg-user",
"password": "$2y$05$6EdxIwwDNlJfNhhQxZRr4eNd.aYrdmbBjAdw422w0u2j3TihQXgd2",
"auth": "c28tcmVnLXVzZXI6cFhEd081ekxVN004OHgzYUE="
}
}
}
```
- Configure the file `container-registry/config.env` as follows
```env
REGISTRY_AUTH=htpasswd
REGISTRY_AUTH_HTPASSWD_REALM="VSL Service Provider Image Registry"
REGISTRY_AUTH_HTPASSWD_PATH="/config/htpasswd"
REGISTRY_HTTP_SECRET='$2y$05$6EdxIwwDNlJfNhhQxZRr4eNd.aYrdmbBjAdw422w0u2j3TihQXgd2'
```
- Load context for k8s
```bash
kubie ctx vs-narwhal
```
- Add the container registry credentials as a secret available to the cluster
```bash
kubectl create secret generic laconic-registry --from-file=.dockerconfigjson=container-registry/my_password.json --type=kubernetes.io/dockerconfigjson
```
### Run
- Deploy the container registry
```bash
laconic-so deployment --dir container-registry start
```
- Check the logs
```bash
laconic-so deployment --dir container-registry logs
```
- Check status and await succesful deployment:
```bash
laconic-so deployment --dir container-registry status
```
- Confirm deployment by logging in:
```
docker login container-registry.apps.vaasl.io --username so-reg-user --password pXDwO5zLU7M88x3aA
```
- Set ingress annotations
- Set the `cluster-id` found in `container-registry/deployment.yml` and then run the following commands:
```
export CLUSTER_ID=<cluster-id>
# Example
# export CLUSTER_ID=laconic-26cc70be8a3db3f4
kubectl annotate ingress $CLUSTER_ID-ingress nginx.ingress.kubernetes.io/proxy-body-size=0
kubectl annotate ingress $CLUSTER_ID-ingress nginx.ingress.kubernetes.io/proxy-read-timeout=600
kubectl annotate ingress $CLUSTER_ID-ingress nginx.ingress.kubernetes.io/proxy-send-timeout=600
```
## webapp-deployer
### Backend
* Reference: <https://github.com/LaconicNetwork/loro-testnet/blob/main/docs/service-provider-setup.md#deploy-backend>
* Target dir: `/srv/service-provider/webapp-deployer`
* Cleanup an existing deployment if required:
```bash
cd /srv/service-provider/webapp-deployer
# Stop the deployment
laconic-so deployment --dir webapp-deployer stop
# Remove the deployment dir
sudo rm -rf webapp-deployer
# Remove the existing spec file
rm webapp-deployer.spec
```
#### Setup
- Initialize a spec file for the deployer backend.
```bash
laconic-so --stack webapp-deployer-backend setup-repositories
laconic-so --stack webapp-deployer-backend build-containers
laconic-so --stack webapp-deployer-backend deploy init --output webapp-deployer.spec
```
- Modify the contents of `webapp-deployer.spec`:
```
stack: webapp-deployer-backend
deploy-to: k8s
kube-config: /home/dev/.kube/config-vs-narwhal.yaml
image-registry: container-registry.apps.vaasl.io/laconic-registry
network:
ports:
server:
- '9555'
http-proxy:
- host-name: webapp-deployer-api.apps.vaasl.io
routes:
- path: '/'
proxy-to: server:9555
volumes:
srv:
configmaps:
config: ./data/config
annotations:
container.apparmor.security.beta.kubernetes.io/{name}: unconfined
labels:
container.kubeaudit.io/{name}.allow-disabled-apparmor: "podman"
security:
privileged: true
resources:
containers:
reservations:
cpus: 3
memory: 8G
limits:
cpus: 7
memory: 16G
volumes:
reservations:
storage: 200G
```
- Create the deployment directory from the spec file.
```
laconic-so --stack webapp-deployer-backend deploy create --deployment-dir webapp-deployer --spec-file webapp-deployer.spec
```
- Modify file `webapp-deployer/kubeconfig.yml` if required
```
apiVersion: v1
...
contexts:
- context:
cluster: ***
user: ***
name: default
...
```
NOTE: `context.name` must be default to use with SO
- Copy `webapp-deployer/kubeconfig.yml` from the k8s cluster creation step to `webapp-deployer/data/config/kube.yml`
```bash
cp webapp-deployer/kubeconfig.yml webapp-deployer/data/config/kube.yml
```
- Create `webapp-deployer/data/config/laconic.yml`, it should look like this:
```
services:
registry:
# Using public endpoint does not work inside machine where laconicd chain is deployed
rpcEndpoint: 'http://host.docker.internal:36657'
gqlEndpoint: 'http://host.docker.internal:3473/api'
# Set user key of account with balance and bond owned by the user
userKey:
bondId:
chainId: laconic-testnet-2
gasPrice: 1alnt
```
NOTE: Modify the user key and bond ID according to your configuration
* Publish a `WebappDeployer` record for the deployer backend by following the steps below:
* Setup GPG keys by following [these steps to create and export a key](https://git.vdb.to/cerc-io/webapp-deployment-status-api#keys)
```
cd webapp-deployer
# Create a key
gpg --batch --passphrase "SECRET" --quick-generate-key webapp-deployer-api.apps.vaasl.io default default never
# Export the public key
gpg --export webapp-deployer-api.apps.vaasl.io > webapp-deployer-api.apps.vaasl.io.pgp.pub
# Export the private key
gpg --export-secret-keys webapp-deployer-api.apps.vaasl.io > webapp-deployer-api.apps.vaasl.io.pgp.key
cd -
```
NOTE: Use "SECRET" for passphrase prompt
* Copy the GPG pub key file generated above to `webapp-deployer/data/config` directory. This ensures the Docker container has access to the key during the publish process
```bash
cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.pub webapp-deployer/data/config
```
* Publish the webapp deployer record using the `publish-deployer-to-registry` command
```
docker run -i -t \
--add-host=host.docker.internal:host-gateway \
-v /srv/service-provider/webapp-deployer/data/config:/config \
cerc/webapp-deployer-backend:local laconic-so publish-deployer-to-registry \
--laconic-config /config/laconic.yml \
--api-url https://webapp-deployer-api.apps.vaasl.io \
--public-key-file /config/webapp-deployer-api.apps.vaasl.io.pgp.pub \
--lrn lrn://vaasl-provider/deployers/webapp-deployer-api.apps.vaasl.io \
--min-required-payment 10000
```
- Modify the contents of `webapp-deployer/config.env`:
```
DEPLOYMENT_DNS_SUFFIX="apps.vaasl.io"
# this should match the name authority reserved above
DEPLOYMENT_RECORD_NAMESPACE="vaasl-provider"
# url of the deployed docker image registry
IMAGE_REGISTRY="container-registry.apps.vaasl.io"
# credentials from the htpasswd section above in container-registry setup
IMAGE_REGISTRY_USER=
IMAGE_REGISTRY_CREDS=
# configs
CLEAN_DEPLOYMENTS=false
CLEAN_LOGS=false
CLEAN_CONTAINERS=false
SYSTEM_PRUNE=false
WEBAPP_IMAGE_PRUNE=true
CHECK_INTERVAL=10
FQDN_POLICY="allow"
# lrn of the webapp deployer
LRN="lrn://vaasl-provider/deployers/webapp-deployer-api.apps.vaasl.io"
# Path to the GPG key file inside the webapp-deployer container
OPENPGP_PRIVATE_KEY_FILE="webapp-deployer-api.apps.vaasl.io.pgp.key"
# Passphrase used when creating the GPG key
OPENPGP_PASSPHRASE="SECRET"
DEPLOYER_STATE="srv-test/deployments/autodeploy.state"
UNDEPLOYER_STATE="srv-test/deployments/autoundeploy.state"
UPLOAD_DIRECTORY="srv-test/uploads"
HANDLE_AUCTION_REQUESTS=true
AUCTION_BID_AMOUNT=10000
# Minimum payment amount required for single webapp deployment
MIN_REQUIRED_PAYMENT=10000
```
- Push the image to the container registry
```
laconic-so deployment --dir webapp-deployer push-images
```
- Modify `webapp-deployer/data/config/laconic.yml`:
```
services:
registry:
rpcEndpoint: 'https://laconicd-sapo.laconic.com/'
gqlEndpoint: 'https://laconicd-sapo.laconic.com/api'
# Set user key of account with balance and bond owned by the user
userKey:
bondId:
chainId: laconic-testnet-2
gasPrice: 1alnt
```
#### Run
- Start the deployer.
```
laconic-so deployment --dir webapp-deployer start
```
- Load context for k8s
```bash
kubie ctx vs-narwhal
```
- Copy the GPG key file to the webapp-deployer container
```bash
# Get the webapp-deployer pod id
laconic-so deployment --dir webapp-deployer ps
# Expected output
# Running containers:
# id: default/laconic-096fed46af974a47-deployment-644db859c7-snbq6, name: laconic-096fed46af974a47-deployment-644db859c7-snbq6, ports: 10.42.2.11:9555->9555
# Set pod id
export POD_ID=
# Example:
# export POD_ID=laconic-096fed46af974a47-deployment-644db859c7-snbq6
# Copy GPG key files to the pod
kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.key $POD_ID:/app
kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.pub $POD_ID:/app
```
- Publishing records to the registry will trigger deployments in backend now
### Frontend
* Target dir: `/srv/service-provider/webapp-ui`
* Cleanup an existing deployment if required:
```bash
cd /srv/service-provider/webapp-ui
# Stop the deployment
laconic-so deployment --dir webapp-ui stop
# Remove the deployment dir
sudo rm -rf webapp-ui
# Remove the existing spec file
rm webapp-ui.spec
```
#### Setup
* Clone and build the deployer UI
```
git clone https://git.vdb.to/cerc-io/webapp-deployment-status-ui.git ~/cerc/webapp-deployment-status-ui
laconic-so build-webapp --source-repo ~/cerc/webapp-deployment-status-ui
```
* Create a deployment
```bash
export KUBECONFIG_PATH=/home/dev/.kube/config-vs-narwhal.yaml
# NOTE: Use actual kubeconfig path
laconic-so deploy-webapp create --kube-config $KUBECONFIG_PATH --image-registry container-registry.apps.vaasl.io --deployment-dir webapp-ui --image cerc/webapp-deployment-status-ui:local --url https://webapp-deployer-ui.apps.vaasl.io --env-file ~/cerc/webapp-deployment-status-ui/.env
```
* Modify file `webapp-ui/kubeconfig.yml` if required
```yml
apiVersion: v1
...
contexts:
- context:
cluster: ***
user: ***
name: default
...
```
NOTE: `context.name` must be default to use with SO
- Push the image to the container registry.
```
laconic-so deployment --dir webapp-ui push-images
```
- Modify `webapp-ui/config.env` like [this Pull Request](https://git.vdb.to/cerc-io/webapp-deployment-status-ui/pulls/6) but with your host details.
#### Run
- Start the deployer UI
```bash
laconic-so deployment --dir webapp-ui start
```
- Wait a moment, then go to https://webapp-deployer-ui.apps.vaasl.io for the status and logs of each deployment

View File

@ -325,6 +325,128 @@ Instructions to reset / update the deployments
laconic-so deployment --dir laconic-shopify-deployment start
```
## webapp-deployer
### Backend
* Deployment dir: `/srv/service-provider/webapp-deployer`
* If code has changed, fetch and build with updated source code:
```bash
laconic-so --stack webapp-deployer-backend setup-repositories --git-ssh --pull
laconic-so --stack webapp-deployer-backend build-containers --force-rebuild
```
* Update the configuration, if required in :
* `/srv/service-provider/webapp-deployer/data/config/laconic.yml`
* `/srv/service-provider/webapp-deployer/config.env`
* Restart the deployment:
```bash
laconic-so deployment --dir webapp-deployer stop
laconic-so deployment --dir webapp-deployer start
```
- Load context for k8s
```bash
kubie ctx vs-narwhal
```
- Copy the GPG key file to the webapp-deployer container
```bash
# Get the webapp-deployer pod id
laconic-so deployment --dir webapp-deployer ps
# Expected output
# Running containers:
# id: default/laconic-096fed46af974a47-deployment-644db859c7-snbq6, name: laconic-096fed46af974a47-deployment-644db859c7-snbq6, ports: 10.42.2.11:9555->9555
# Set pod id
export POD_ID=
# Example:
# export POD_ID=laconic-096fed46af974a47-deployment-644db859c7-snbq6
# Copy GPG key files to the pod
kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.key $POD_ID:/app
kubectl cp webapp-deployer/webapp-deployer-api.apps.vaasl.io.pgp.pub $POD_ID:/app
```
### Frontend
* Deployment dir: `/srv/service-provider/webapp-ui`
* If code has changed, fetch and build with updated source code:
```bash
cd ~/cerc/webapp-deployment-status-ui
# Pull latest changes, or checkout to the required branch
git pull
# Confirm the latest commit hash
git log
laconic-so build-webapp --source-repo ~/cerc/webapp-deployment-status-ui
```
- Modify `/srv/service-provider/webapp-ui/config.env` like [this Pull Request](https://git.vdb.to/cerc-io/webapp-deployment-status-ui/pulls/6) but with your host details.
* Restart the deployment:
```bash
laconic-so deployment --dir webapp-ui stop
laconic-so deployment --dir webapp-ui start
```
## Deploy Backend
* Deployment dir: `/srv/deploy-backend/backend-deployment`
* If code has changed, fetch and build with updated source code:
```bash
laconic-so --stack ~/cerc/snowballtools-base-api-stack/stack-orchestrator/stacks/snowballtools-base-backend setup-repositories --git-ssh --pull
# rebuild containers
laconic-so --stack ~/cerc/snowballtools-base-api-stack/stack-orchestrator/stacks/snowballtools-base-backend build-containers --force-rebuild
```
* Push updated images to the container registry:
```bash
cd /srv/deploy-backend
# login to container registry
CONTAINER_REGISTRY_URL=container-registry.apps.vaasl.io
CONTAINER_REGISTRY_USERNAME=
CONTAINER_REGISTRY_PASSWORD=
docker login $CONTAINER_REGISTRY_URL --username $CONTAINER_REGISTRY_USERNAME --password $CONTAINER_REGISTRY_PASSWORD
# Push backend images
laconic-so deployment --dir backend-deployment push-images
```
* Update the configuration if required in `backend-deployment/configmaps/config/prod.toml`
* Restart the deployment:
```bash
laconic-so deployment --dir backend-deployment stop
laconic-so deployment --dir backend-deployment start
```
## Deply Frontend
* Follow steps from [deployments-from-scratch.md](./deployments-from-scratch.md#deploy-frontend) to deploy the snowball frontend
## Fixturenet Eth
* Deployment dir: `/srv/fixturenet-eth/fixturenet-eth-deployment`
@ -376,46 +498,3 @@ Instructions to reset / update the deployments
laconic-so deployment --dir bridge-deployment start
```
## Backend Deployment
* Deployment dir: `/srv/deploy-backend/backend-deployment`
* If code has changed, fetch and build with updated source code:
```bash
laconic-so --stack ~/cerc/snowballtools-base-api-stack/stack-orchestrator/stacks/snowballtools-base-backend setup-repositories --git-ssh --pull
# rebuild containers
laconic-so --stack ~/cerc/snowballtools-base-api-stack/stack-orchestrator/stacks/snowballtools-base-backend build-containers --force-rebuild
```
* Push updated images to the container registry:
```bash
cd /srv/deploy-backend
# login to container registry
CONTAINER_REGISTRY_URL=container-registry.apps.vaasl.io
CONTAINER_REGISTRY_USERNAME=
CONTAINER_REGISTRY_PASSWORD=
docker login $CONTAINER_REGISTRY_URL --username $CONTAINER_REGISTRY_USERNAME --password $CONTAINER_REGISTRY_PASSWORD
# Push backend images
laconic-so deployment --dir backend-deployment push-images
```
* Update the configuration if required in `backend-deployment/configmaps/config/prod.toml`
* Restart the deployment:
```bash
laconic-so deployment --dir backend-deployment stop
laconic-so deployment --dir backend-deployment start
```
## Frontend Deployment
* Follow steps from [deployments-from-scratch.md](./deployments-from-scratch.md#deploy-frontend) to deploy the snowball frontend