Add ansible playbook to automate service provider setup (#10)

Part of [Service Provider setup](https://www.notion.so/Service-provider-setup-a09e2207e1f34f3a847f7ce9713b7ac5) - Added ansible playbooks for: - Adding a new user with passwordless sudo - Configuring DNS records - Setting up the system with required packages and gpg key - Deploying k8s - Setting up container registry - Setting up laconicd and laconic-console - Setting up and starting webapp-deployer-api and webapp-deployer-ui - TODOs: - Mount gpg keys in webapp-deployer-api container Co-authored-by: Adw8 <adwaitgharpure@gmail.com> Reviewed-on: cerc-io/testnet-ops#10
2024-10-01 12:17:10 +00:00 · 2024-10-01 12:17:10 +00:00 · 18df60a291
commit 18df60a291
parent 631a859080
37 changed files with 1370 additions and 0 deletions
--- a/README.md
+++ b/README.md
@ -43,3 +43,4 @@
 - [nitro-node-setup](./nitro-nodes-setup/README.md)
 - [nitro-bridge-setup](./nitro-bridge-setup/README.md)
 - [nitro-contracts-setup](./nitro-contracts-setup/README.md)
+- [service-provider-setup](./service-provider-setup/README.md)
--- a/service-provider-setup/.gitignore
+++ b/service-provider-setup/.gitignore
@ -0,0 +1,2 @@
+vars/*.yml
+!vars/*.example.yml
--- a/service-provider-setup/README.md
+++ b/service-provider-setup/README.md
@ -0,0 +1,163 @@
+# service-provider-setup
+
+## Setup Ansible
+
+To get started, follow the [installation](../README.md#installation) guide to setup ansible on your machine
+
+## Prerequisites
+
+- Set up a DigitalOcean Droplet with passwordless SSH access
+
+- Buy a domain and configure [nameservers pointing to DigitalOcean](https://docs.digitalocean.com/products/networking/dns/getting-started/dns-registrars/)
+
+- Generate a DigitalOcean access token, used for API authentication and managing cloud resources
+
+## Setup a new User
+
+- Create a new `hosts.ini` file:
+
+   ```bash
+   cp ../hosts.example.ini hosts.ini
+   ```
+
+- Edit the [`hosts.ini`](./hosts.ini) file to run the playbook on a remote machine:
+
+  ```ini
+  [root_host]
+  <host_name> ansible_host=<target_ip> ansible_user=<ssh_user> ansible_ssh_common_args='-o ForwardAgent=yes'
+  ```
+
+  - Replace `<host_name>` with the desired `hostname` of the remote machine
+  - Replace `<target_ip>` with the IP address or hostname of the target machine
+  - Replace `<ssh_user>` with `root`
+
+- Verify that you are able to connect to the host using the following command:
+
+  ```bash
+  ansible all -m ping -i hosts.ini
+
+  # Expected output:
+
+  # <host_name> | SUCCESS => {
+  #  "ansible_facts": {
+  #      "discovered_interpreter_python": "/usr/bin/python3.10"
+  #  },
+  #  "changed": false,
+  #  "ping": "pong"
+  # }
+  ```
+
+- Setup `user-vars.yml`  using the example file
+
+  ```bash
+  cp vars/user-vars.example.yml vars/user-vars.yml
+  ```
+
+- Edit the `user-vars.yml` file:
+
+  ```bash
+  # name of the user you want to setup on the target host
+  username: ""
+
+  # password of the user you want to setup on the target host
+  password: ""
+
+  # path to the ssh key on your machine, eg: "/home/dev/.ssh/id_rsa.pub"
+  path_to_ssh_key: ""
+  ```
+
+- Execute the `setup-user.yml` Ansible playbook to create a user with passwordless sudo permissions:
+
+  ```bash
+  cd ../
+  LANG=en_US.utf8 ansible-playbook setup-user.yml -i hosts.ini --extra-vars='{ "target_host": "deployment_host" }'
+  ```
+
+## Become a Service Provider
+
+### Setup
+
+- Copy the vars files:
+
+  ```bash
+  cd vars
+  cp dns-vars.example.yml dns-vars.yml
+  cp gpg-vars.example.yml gpg-vars.yml
+  cp k8s-vars.example.yml k8s-vars.yml
+  cp container-vars.example.yml container-vars.yml
+  cp webapp-vars.example.yml webapp-vars.yml
+  cd -
+  ```
+
+- Update the following values in the respective variable files:
+
+  ```bash
+  # vars/dns-vars.yml
+  full_domain: ""               # eg: laconic.com
+  subdomain_prefix: ""          # eg: lcn-cad
+  service_provider_ip: ""        # eg: 23.111.78.179
+  do_api_token: ""              # Digital Ocean access token that you generated, eg: dop_v1...
+
+  # vars/gpg-vars.yml
+  gpg_user_name: ""         # Full name of the user for the GPG key
+  gpg_user_email: ""        # Email address associated with the GPG key
+  gpg_passphrase: ""        # Passphrase for securing the GPG key
+
+  # vars/k8s-vars.yml
+  target_host: "deployment_host"
+  org_id: ""                    # eg: lcn
+  location_id: ""               # eg: cad
+  base_domain: ""               # eg: laconic
+  support_email: ""             # eg: support@laconic.com
+
+  # vars/container-vars.yml
+  container_registry_username: "" # username to login to the container registry
+  container_registry_password: "" # password to login to the container registry
+
+  # vars/webapp-vars.yml
+  authority_name: ""      # eg: my-org-name
+  cpu_reservation: ""     # Minimum number of cpu cores to be used, eg: 2
+  memory_reservation: ""  # Minimum amount of memory in GB to be used, eg: 4G
+  deployer_gpg_passphrase: "" # passphrase for creating GPG key used by webapp-deployer, eg: SECRET
+  ```
+
+- Update the [`hosts.ini`](./hosts.ini) file:
+
+  ```ini
+  [root_host]
+  <host_name> ansible_host=<target_ip> ansible_user=root ansible_ssh_common_args='-o ForwardAgent=yes'
+
+  [deployment_host]
+  <host_name> ansible_host=<target_ip> ansible_user=<new_username> ansible_ssh_common_args='-o ForwardAgent=yes'
+  ```
+
+  - Replace `<host_name>` with the desired `hostname` of the remote machine
+  - Replace `<target_ip>` with the IP address or hostname of the target machine
+  - Under `deployment_host`, Replace `<ansible_user>` with the name of the user you have created
+
+- Verify that you are able to connect to the host using the following command:
+
+  ```bash
+  ansible all -m ping -i hosts.ini
+
+  # Expected output:
+
+  # <host_name> | SUCCESS => {
+  #  "ansible_facts": {
+  #      "discovered_interpreter_python": "/usr/bin/python3.10"
+  #  },
+  #  "changed": false,
+  #  "ping": "pong"
+  # }
+  ```
+
+- Run the `service-provider-setup.yml` ansible-playbook to:
+  - Create DNS records
+  - Deploy k8s
+  - Setup laconicd and laconic console
+  - Setup container registry
+  - Deploy the webapp-deployer API and webapp-deployer UI
+
+  ```bash
+  LANG=en_US.utf8 ansible-playbook service-provider-setup.yml -i hosts.ini --extra-vars='{ target_host: "deployment_host" }' --user $USER
+  ```
--- a/service-provider-setup/deploy-backend.yml
+++ b/service-provider-setup/deploy-backend.yml
@ -0,0 +1,127 @@
+- name: Deploy webapp-deployer backend
+  hosts: "{{ target_host }}"
+
+  environment:
+    PATH: "{{ ansible_env.PATH }}:/home/{{ansible_user}}/bin"
+    KUBECONFIG: "{{ ansible_env.HOME }}/.kube/config-default.yaml"
+
+  vars_files:
+    - vars/webapp-vars.yml
+    - vars/container-vars.yml
+    - vars/k8s-vars.yml
+    - vars/dns-vars.yml
+
+  tasks:
+    - name: Ensure gpg-keys directory exists
+      file:
+        path: ~/gpg-keys
+        state: directory
+        mode: '0700'
+
+    - name: Create a GPG key
+      shell: gpg --batch --passphrase "{{ deployer_gpg_passphrase }}" --quick-generate-key webapp-deployer-api.{{ full_domain }} default default never
+
+    - name: Export the public key
+      shell: gpg --export webapp-deployer-api.{{ full_domain }} > ~/gpg-keys/webapp-deployer-api.{{ full_domain }}.pgp.pub
+      args:
+        creates: ~/gpg-keys/webapp-deployer-api.{{ full_domain }}.pgp.pub
+
+    - name: Export the GPG private key with passphrase
+      shell: gpg --pinentry-mode=loopback --passphrase "{{ deployer_gpg_passphrase }}" --export-secret-keys webapp-deployer-api.{{ full_domain }} > ~/gpg-keys/webapp-deployer-api.{{ full_domain }}.pgp.key
+
+    - name: Setup repositories for webapp-deployer-backend
+      command: laconic-so --stack webapp-deployer-backend setup-repositories
+
+    - name: Build containers for webapp-deployer-backend
+      command: laconic-so --stack webapp-deployer-backend build-containers
+
+    - name: Ensure the config directory exists
+      file:
+        path: "{{ ansible_env.HOME }}/config"
+        state: directory
+
+    - name: Create laconic config file
+      template:
+        src: "./templates/laconic.yml.j2"
+        dest: "config/laconic.yml"
+
+    - name: Copy the gpg private key file to config dir
+      copy:
+        src: "gpg-keys/webapp-deployer-api.{{ full_domain }}.pgp.key"
+        dest: "config"
+        remote_src: true
+
+    - name: Copy the gpg public key file to config dir
+      copy:
+        src: "gpg-keys/webapp-deployer-api.{{ full_domain }}.pgp.pub"
+        dest: "config"
+        remote_src: true
+
+    - name: Publish the webapp-deployer record using laconic-so
+      shell: |
+        docker run -i -t \
+          -v /home/{{ ansible_user }}/config:/home/root/config \
+          cerc/webapp-deployer-backend:local laconic-so publish-deployer-to-registry \
+            --laconic-config /home/root/config/laconic.yml \
+            --api-url https://webapp-deployer-api.pwa.{{ full_domain }} \
+            --public-key-file /home/root/config/webapp-deployer-api.{{ full_domain }}.pgp.pub \
+            --lrn lrn://{{ authority_name }}/deployers/webapp-deployer-api.{{ full_domain }} \
+            --min-required-payment 0
+      register: publish_output
+
+    - name: Display publish output
+      debug:
+        var: publish_output.stdout
+
+    - name: Generate spec file for webapp-deployer-backend
+      template:
+        src: "./templates/specs/webapp-deployer.spec.j2"
+        dest: "webapp-deployer.spec"
+
+    - name: Create the deployment directory from the spec file
+      command: >
+        laconic-so --stack webapp-deployer-backend deploy create
+        --deployment-dir webapp-deployer --spec-file webapp-deployer.spec
+
+    - name: Update config for webapp-deployer-backend
+      template:
+        src: "./templates/configs/webapp-deployer-config.env.j2"
+        dest: "webapp-deployer/config.env"
+
+    - name: Copy the kube config file to webapp-deployer directory
+      copy:
+        src: "{{ansible_env.HOME}}/.kube/config-default.yaml"
+        dest: "webapp-deployer/data/config/kube.yml"
+        remote_src: true
+
+    - name: Create laconic config file
+      template:
+        src: "./templates/laconic.yml.j2"
+        dest: "webapp-deployer/data/config/laconic.yml"
+
+    - name: login to the container registry
+      command: "docker login container-registry.pwa.{{ full_domain }} --username  {{ container_registry_username }} --password {{ container_registry_password}}"
+
+    - name: Push images to container registry
+      command: laconic-so deployment --dir webapp-deployer push-images
+
+    - name: Start the webapp deployer
+      command: laconic-so deployment --dir webapp-deployer start
+
+    - name: Get the most recent pod for the deployment
+      shell: kubectl get pods --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}'
+      register: webapp_deployer_pod
+
+    - name: Set pod ID to a variable
+      set_fact:
+        pod_id: "{{ webapp_deployer_pod.stdout }}"
+
+    - name: Wait for the recent pod to be ready
+      command: kubectl wait --for=condition=Ready pod/{{ pod_id }} --timeout=300s
+      register: wait_result
+
+    - name: Copy gpg private key file to webapp deployer pod
+      shell: kubectl cp gpg-keys/webapp-deployer-api.{{ full_domain }}.pgp.key {{ pod_id }}:/app
+
+    - name: Copy gpg public key file to webapp deployer pod
+      shell: kubectl cp gpg-keys/webapp-deployer-api.{{ full_domain }}.pgp.pub {{ pod_id }}:/app
--- a/service-provider-setup/deploy-frontend.yml
+++ b/service-provider-setup/deploy-frontend.yml
@ -0,0 +1,43 @@
+- name: Deploy webapp-deployer ui
+  hosts: "{{ target_host }}"
+
+  environment:
+    PATH: "{{ ansible_env.PATH }}:/home/{{ansible_user}}/bin"
+
+  vars_files:
+    - vars/webapp-vars.yml
+    - vars/dns-vars.yml
+    - vars/k8s-vars.yml
+
+  tasks:
+    - name: Clone webapp-deployment-status-ui repository
+      git:
+        repo: "https://git.vdb.to/cerc-io/webapp-deployment-status-ui.git"
+        dest: "{{ ansible_env.HOME }}/cerc/webapp-deployment-status-ui"
+        update: yes
+
+    - name: Build webapp-deployer-status-ui
+      command: laconic-so build-webapp --source-repo {{ ansible_env.HOME }}/cerc/webapp-deployment-status-ui
+
+    - name: Create a deployment for webapp-ui
+      command: |
+        laconic-so deploy-webapp create --kube-config {{ ansible_env.HOME }}/.kube/config-default.yaml
+        --image-registry container-registry.pwa.{{ full_domain }} --deployment-dir webapp-ui
+        --image cerc/webapp-deployment-status-ui:local --url https://webapp-deployer-ui.pwa.{{ full_domain }}
+        --env-file ~/cerc/webapp-deployment-status-ui/.env
+
+    - name: Push image to container registry
+      command: laconic-so deployment --dir webapp-ui push-images
+
+    - name: Update config file for webapp ui
+      template:
+        src: "./templates/configs/webapp-ui-config.env.j2"
+        dest: "webapp-ui/config.env"
+
+    - name: Start the deployer ui
+      command: laconic-so deployment --dir webapp-ui start
+
+    - name: Create .out file
+      file:
+        path: "{{ ansible_env.HOME }}/.out"
+        state: touch
--- a/service-provider-setup/run-laconic-console.yml
+++ b/service-provider-setup/run-laconic-console.yml
@ -0,0 +1,77 @@
+- name: Setup and run laconic console
+  hosts: "{{target_host}}"
+
+  environment:
+    PATH: "{{ ansible_env.PATH }}:/home/{{ansible_user}}/bin"
+
+  vars_files:
+    - vars/webapp-vars.yml
+    - vars/dns-vars.yml
+    - vars/k8s-vars.yml
+
+  tasks:
+    - name: Clone the stack repo
+      command: laconic-so fetch-stack git.vdb.to/cerc-io/testnet-laconicd-stack --pull
+      ignore_errors: yes
+
+    - name: Clone required repositories for laconic-console
+      command: laconic-so --stack ~/cerc/testnet-laconicd-stack/stack-orchestrator/stacks/laconic-console setup-repositories --pull
+
+    - name: Build container images
+      command: laconic-so --stack ~/cerc/testnet-laconicd-stack/stack-orchestrator/stacks/laconic-console build-containers --force-rebuild
+
+    - name: Generate spec file for laconic console deployment
+      template:
+        src: "./templates/specs/laconic-console-spec.yml.j2"
+        dest: "laconic-console-spec.yml"
+
+    - name: Check if the deployment directory exists
+      stat:
+        path: laconic-console-deployment
+      register: deployment_dir
+
+    - name: Create a deployment from the spec file
+      command: laconic-so --stack ~/cerc/testnet-laconicd-stack/stack-orchestrator/stacks/laconic-console deploy create --spec-file laconic-console-spec.yml --deployment-dir laconic-console-deployment
+      when: not deployment_dir.stat.exists
+
+    - name: Place deployment in the same namespace as fixturenet-laconicd
+      copy:
+        src: "fixturenet-laconicd-deployment/deployment.yml"
+        dest: "laconic-console-deployment/deployment.yml"
+        remote_src: yes
+
+    - name: Fetch user key from laconicd
+      command: laconic-so deployment --dir fixturenet-laconicd-deployment exec laconicd "echo y | laconicd keys export alice --unarmored-hex --unsafe"
+      register: alice_pk
+
+    - name: Set Private key for console deployment
+      set_fact:
+        ALICE_PK: "{{ alice_pk.stdout }}"
+
+    - name: Start the laconic console deployment
+      command: laconic-so deployment --dir laconic-console-deployment start
+
+    - name: Create a bond using cli
+      shell: laconic-so deployment --dir laconic-console-deployment exec cli "laconic registry bond create --type alnt --quantity 1000000000000 --user-key {{ALICE_PK}}" | jq -r '.bondId'
+      register: bond_id
+
+    - name: Set Bond ID for console deployment
+      set_fact:
+        BOND_ID: "{{ bond_id.stdout }}"
+
+    - name: Stop the console deployment
+      command: laconic-so deployment --dir laconic-console-deployment stop
+
+    - name: Modify the console config with alice_pk and bond_id
+      template:
+        src: "./templates/configs/console-config.env.j2"
+        dest: "laconic-console-deployment/config.env"
+
+    - name: Start the laconic console deployment with updated config
+      command: laconic-so deployment --dir laconic-console-deployment start
+
+    - name: Reserve an authority
+      command: laconic-so deployment --dir laconic-console-deployment exec cli "laconic registry authority reserve {{authority_name}}"
+
+    - name: Set authority using bond id
+      command: laconic-so deployment --dir laconic-console-deployment exec cli "laconic registry authority bond set {{authority_name}} {{BOND_ID}}"
--- a/service-provider-setup/run-laconicd.yml
+++ b/service-provider-setup/run-laconicd.yml
@ -0,0 +1,33 @@
+- name: Setup and run fixturnet-laconicd-stack
+  hosts: "{{ target_host }}"
+
+  environment:
+    PATH: "{{ ansible_env.PATH }}:/home/{{ansible_user}}/bin"
+
+  tasks:
+    - name: Clone the fixturenet-laconicd-stack repo
+      command: laconic-so fetch-stack git.vdb.to/cerc-io/fixturenet-laconicd-stack --pull
+      ignore_errors: yes
+
+    - name: Setup repos for fixturenet-laconicd
+      command: laconic-so --stack ~/cerc/fixturenet-laconicd-stack/stack-orchestrator/stacks/fixturenet-laconicd setup-repositories
+
+    - name: Build container images
+      command: laconic-so --stack ~/cerc/fixturenet-laconicd-stack/stack-orchestrator/stacks/fixturenet-laconicd build-containers --force-rebuild
+
+    - name: Generate over spec file for laconicd deployment
+      template:
+        src: "./templates/specs/fixturenet-laconicd-spec.yml.j2"
+        dest: "fixturenet-laconicd-spec.yml"
+
+    - name: Check if the deployment directory exists
+      stat:
+        path: "fixturenet-laconicd-deployment"
+      register: deployment_dir
+
+    - name: Create the deployment from the spec file
+      command: laconic-so --stack ~/cerc/fixturenet-laconicd-stack/stack-orchestrator/stacks/fixturenet-laconicd deploy create --spec-file fixturenet-laconicd-spec.yml --deployment-dir fixturenet-laconicd-deployment
+      when: not deployment_dir.stat.exists
+
+    - name: Start the deployment
+      command: laconic-so deployment --dir fixturenet-laconicd-deployment start
--- a/service-provider-setup/service-provider-setup.yml
+++ b/service-provider-setup/service-provider-setup.yml
@ -0,0 +1,20 @@
+- hosts: "{{ target_host }}"
+  tasks:
+    - name: Check if .out file exists
+      stat:
+        path: "{{ ansible_env.HOME }}/.out"
+      register: out_file
+
+    - name: Exit playbook if .out file exists
+      fail:
+        msg: ".out file exists, exiting playbook."
+      when: out_file.stat.exists
+
+- import_playbook: setup-dns.yml
+- import_playbook: setup-system.yml
+- import_playbook: setup-k8s.yml
+- import_playbook: setup-container-registry.yml
+- import_playbook: run-laconicd.yml
+- import_playbook: run-laconic-console.yml
+- import_playbook: deploy-backend.yml
+- import_playbook: deploy-frontend.yml
--- a/service-provider-setup/setup-container-registry.yml
+++ b/service-provider-setup/setup-container-registry.yml
@ -0,0 +1,161 @@
+- name: Setup container registry
+  hosts: "{{ target_host }}"
+
+  environment:
+    PATH: "{{ ansible_env.PATH }}:/home/{{ansible_user}}/bin"
+
+  vars_files:
+    - vars/k8s-vars.yml
+    - vars/container-vars.yml
+    - vars/dns-vars.yml
+
+  tasks:
+    - name: Generate spec file for the container-registry stack
+      template:
+        src: "./templates/specs/container-registry.spec.j2"
+        dest: "{{ansible_env.HOME}}/container-registry.spec"
+
+    - name: Create a deployment for the container-registry stack
+      command: laconic-so --stack container-registry deploy create --deployment-dir container-registry --spec-file container-registry.spec
+
+    - name: Base64 encode the container registry credentials
+      set_fact:
+        b64_encoded_cred: "{{ (container_registry_username + ':' + container_registry_password) | b64encode }}"
+
+    - name: Encrypt the container registry credentials to create an htpasswd file
+      command: >
+        htpasswd -bB -c container-registry/configmaps/config/htpasswd
+        {{ container_registry_username }} {{ container_registry_password }}
+      register: htpasswd_file
+
+    - name: Read the htpasswd file
+      slurp:
+        src: "container-registry/configmaps/config/htpasswd"
+      register: htpasswd_file_content
+
+    - name: Extract the hashed password (after the colon)
+      set_fact:
+        hashed_password: "{{ (htpasswd_file_content.content | b64decode).split(':')[1] | trim }}"
+
+    - name: Create container-registry/my_password.json file
+      template:
+        src: "./templates/my_password.json.j2"
+        dest: "container-registry/my_password.json"
+
+    - name: Configure the file container-registry/config.env
+      copy:
+        dest: "container-registry/config.env"
+        content: |
+          REGISTRY_AUTH=htpasswd
+          REGISTRY_AUTH_HTPASSWD_REALM="{{org_id}} Service Provider Image Registry"
+          REGISTRY_AUTH_HTPASSWD_PATH="/config/htpasswd"
+          REGISTRY_HTTP_SECRET='{{ hashed_password }}'
+
+    - name: Set KUBECONFIG environment variable
+      set_fact:
+        kubeconfig_path: "{{ ansible_env.HOME }}/.kube/config-default.yaml"
+
+    - name: Add the container registry credentials as a secret available to the cluster
+      command: >
+        kubectl create secret generic laconic-registry
+        --from-file=.dockerconfigjson=container-registry/my_password.json
+        --type=kubernetes.io/dockerconfigjson
+      environment:
+        KUBECONFIG: "{{ kubeconfig_path }}"
+
+    # TODO: Investigate why container registry throws error if started immediately
+    - name: Wait for 90 seconds
+      pause:
+        seconds: 90
+
+    - block:
+        - name: Get Kubernetes nodes with wide output
+          command: kubectl get nodes -o wide
+          environment:
+            KUBECONFIG: "{{ kubeconfig_path }}"
+          register: nodes_output
+
+        - name: Print output of 'kubectl get nodes -o wide'
+          debug:
+            var: nodes_output.stdout
+
+        - name: Get all secrets from all namespaces
+          command: kubectl get secrets --all-namespaces
+          environment:
+            KUBECONFIG: "{{ kubeconfig_path }}"
+          register: secrets_output
+
+        - name: Print output of 'kubectl get secrets --all-namespaces'
+          debug:
+            var: secrets_output.stdout
+
+        - name: Get cluster issuers
+          command: kubectl get clusterissuer
+          environment:
+            KUBECONFIG: "{{ kubeconfig_path }}"
+          register: clusterissuer_output
+
+        - name: Print output of 'kubectl get clusterissuer'
+          debug:
+            var: clusterissuer_output.stdout
+
+        - name: Get certificates
+          command: kubectl get certificates
+          environment:
+            KUBECONFIG: "{{ kubeconfig_path }}"
+          register: certificates_output
+
+        - name: Print output of 'kubectl get certificates'
+          debug:
+            var: certificates_output.stdout
+
+        - name: Get DaemonSets in all namespaces
+          command: kubectl get ds --all-namespaces
+          environment:
+            KUBECONFIG: "{{ kubeconfig_path }}"
+          register: daemonsets_output
+
+        - name: Print output of 'kubectl get ds --all-namespaces'
+          debug:
+            var: daemonsets_output.stdout
+
+      ignore_errors: yes
+      
+    - name: Deploy the container registry
+      command: >
+        laconic-so deployment --dir container-registry start
+
+    - name: Get cluster_id from container-registry-deployment
+      slurp:
+        src: container-registry/deployment.yml
+      register: deployment_file
+
+    - name: Decode and extract cluster-id
+      set_fact:
+        extracted_cluster_id: "{{ deployment_file.content | b64decode | regex_search('cluster-id: (.+)', '\\1') }}"
+
+    - name: Set modified cluster-id
+      set_fact:
+        formatted_cluster_id: "{{ extracted_cluster_id | replace('[', '') | replace(']', '') | replace(\"'\", '') }}"
+
+    - name: Display the cluster ID
+      debug:
+        msg: "The cluster ID is: {{ formatted_cluster_id }}"
+
+    - name: Annotate ingress for proxy body size
+      command: >
+        kubectl annotate ingress {{ formatted_cluster_id }}-ingress nginx.ingress.kubernetes.io/proxy-body-size=0
+      environment:
+        KUBECONFIG: "{{ ansible_env.HOME }}/.kube/config-default.yaml"
+
+    - name: Annotate ingress for proxy read timeout
+      command: >
+        kubectl annotate ingress {{ formatted_cluster_id }}-ingress nginx.ingress.kubernetes.io/proxy-read-timeout=600
+      environment:
+        KUBECONFIG: "{{ ansible_env.HOME }}/.kube/config-default.yaml"
+
+    - name: Annotate ingress for proxy send timeout
+      command: >
+        kubectl annotate ingress {{ formatted_cluster_id }}-ingress nginx.ingress.kubernetes.io/proxy-send-timeout=600
+      environment:
+        KUBECONFIG: "{{ ansible_env.HOME }}/.kube/config-default.yaml"
--- a/service-provider-setup/setup-dns.yml
+++ b/service-provider-setup/setup-dns.yml
@ -0,0 +1,72 @@
+- name: Configure DNS records
+  hosts: localhost
+
+  vars_files:
+    - vars/dns-vars.yml
+
+  tasks:
+    - name: Create a domain
+      community.digitalocean.digital_ocean_domain:
+        state: present
+        oauth_token: "{{ do_api_token }}"
+        name: "{{ full_domain }}"
+        ip: "{{ service_provider_ip }}"
+
+    - name: Create record for cluster control machine
+      community.digitalocean.digital_ocean_domain_record:
+        state: present
+        oauth_token: "{{ do_api_token }}"
+        domain: "{{ full_domain }}"
+        type: A
+        name: "{{ subdomain_prefix }}-cluster-control"
+        data: "{{ service_provider_ip }}"
+
+    - name: Create CNAME record for www
+      community.digitalocean.digital_ocean_domain_record:
+        state: present
+        oauth_token: "{{ do_api_token }}"
+        data: "{{ full_domain }}"
+        domain: "{{ full_domain }}"
+        type: CNAME
+        name: www
+        ttl: 43200
+
+    - name: Create CNAME record for subdomain
+      community.digitalocean.digital_ocean_domain_record:
+        state: present
+        oauth_token: "{{ do_api_token }}"
+        data: "{{ subdomain_cluster_control }}.{{ full_domain }}"
+        domain: "{{ full_domain }}"
+        type: CNAME
+        name: "{{ subdomain_prefix }}"
+        ttl: 43200
+
+    - name: Create wildcard CNAME record for subdomain
+      community.digitalocean.digital_ocean_domain_record:
+        state: present
+        oauth_token: "{{ do_api_token }}"
+        data: "{{ subdomain_cluster_control }}.{{ full_domain }}"
+        domain: "{{ full_domain }}"
+        type: CNAME
+        name: "*.{{ subdomain_prefix }}"
+        ttl: 43200
+
+    - name: Create CNAME record for pwa
+      community.digitalocean.digital_ocean_domain_record:
+        state: present
+        oauth_token: "{{ do_api_token }}"
+        data: "{{ subdomain_cluster_control }}.{{ full_domain }}"
+        domain: "{{ full_domain }}"
+        type: CNAME
+        name: "pwa"
+        ttl: 43200
+
+    - name: Create wildcard CNAME record for pwa
+      community.digitalocean.digital_ocean_domain_record:
+        state: present
+        oauth_token: "{{ do_api_token }}"
+        data: "{{ subdomain_cluster_control }}.{{ full_domain }}"
+        domain: "{{ full_domain }}"
+        type: CNAME
+        name: "*.pwa"
+        ttl: 43200
--- a/service-provider-setup/setup-k8s.yml
+++ b/service-provider-setup/setup-k8s.yml
@ -0,0 +1,183 @@
+- name: Install Stack Orchestrator if it isn't present
+  import_playbook: ../stack-orchestrator-setup/setup-laconic-so.yml
+
+- name: Setup k8s
+  hosts: "{{ target_host }}"
+
+  environment:
+    PATH: "{{ ansible_env.PATH }}:/home/{{ansible_user}}/.local/bin"
+    VAULT_KEY: "{{ vault_passphrase }}"
+
+  vars_files:
+    - vars/dns-vars.yml
+    - vars/gpg-vars.yml
+    - vars/k8s-vars.yml
+
+  tasks:
+    - name: Install Python and pip
+      apt:
+        name: "{{ item }}"
+        state: present
+      become: yes
+      loop:
+        - python3
+        - python3-pip
+
+    - name: Add user to docker group
+      user:
+        name: "{{ ansible_user }}"
+        groups: docker
+        append: true
+
+    - name: Install Ansible on remote host
+      pip:
+        name: ansible
+        extra_args: --user
+      when: target_host != "localhost"
+
+    - name: Ensure ~/.local/bin is in PATH in .bashrc
+      lineinfile:
+        path: ~/.bashrc
+        line: 'export PATH="$HOME/.local/bin:$PATH"'
+        state: present
+        create: yes
+
+    - name: Ensure ~/.local/bin is in PATH in .zshrc
+      lineinfile:
+        path: ~/.zshrc
+        line: 'export PATH="$HOME/.local/bin:$PATH"'
+        state: present
+        create: yes
+
+    - name: Clone the service provider template repo
+      git:
+        repo: "https://git.vdb.to/cerc-io/service-provider-template.git"
+        dest: "{{ ansible_env.HOME }}/service-provider-template"
+
+    - name: Update .vault/vault-keys file
+      lineinfile:
+        path: "service-provider-template/.vault/vault-keys"
+        regexp: '^.*$'
+        line: "{{ gpg_key_id }}"
+        create: yes
+
+    - name: Start GPG agent
+      command: gpg-agent --daemon
+      ignore_errors: yes
+
+    - name: Sign a dummy string using gpg-key
+      shell: echo "This is a dummy string." | gpg --batch --yes --local-user "{{ gpg_key_id }}" --passphrase "{{ vault_passphrase }}" --pinentry-mode loopback --sign -
+
+    - name: Run vault-rekey.sh
+      shell: bash .vault/vault-rekey.sh
+      args:
+        chdir: "service-provider-template"
+      register: rekey_result
+      until: rekey_result.stderr == ""
+      retries: 5
+      delay: 5
+
+    - name: Ensure the target directory exists
+      file:
+        path: "{{ ansible_env.HOME }}/service-provider-template"
+        state: directory
+        mode: '0755'
+
+    - name: Change directory name in group_vars
+      command: mv lcn_cad {{ org_id }}_{{ location_id }}
+      args:
+        chdir: "{{ ansible_env.HOME }}/service-provider-template/group_vars"
+
+    - name: Change control directory name in host_vars
+      command: mv lcn-cad-cluster-control {{ org_id }}-{{ location_id }}-cluster-control
+      args:
+        chdir: "{{ ansible_env.HOME }}/service-provider-template/host_vars"
+
+    - name: Change daemon directory name in host_vars
+      command: mv lcn-daemon {{ org_id }}-daemon
+      args:
+        chdir: "{{ ansible_env.HOME }}/service-provider-template/host_vars"
+
+    - name: Copy control-firewalld.yml to the remote VM
+      template:
+        src: ./templates/control-firewalld.yml.j2
+        dest: "{{ ansible_env.HOME }}/service-provider-template/host_vars/{{ org_id }}-{{ location_id }}-cluster-control/firewalld.yml"
+
+    - name: Copy daemon-firewalld.yml to the remote VM
+      template:
+        src: ./templates/daemon-firewalld.yml.j2
+        dest: "{{ ansible_env.HOME }}/service-provider-template/host_vars/{{ org_id }}-daemon/firewalld.yml"
+
+    - name: Copy nginx.yml to the remote VM
+      template:
+        src: ./templates/nginx.yml.j2
+        dest: "{{ ansible_env.HOME }}/service-provider-template/host_vars/{{ org_id }}-daemon/nginx.yml"
+
+    - name: Copy hosts file to the remote VM
+      template:
+        src: ./templates/hosts.j2
+        dest: "{{ ansible_env.HOME }}/service-provider-template/hosts"
+
+    - name: Copy k8s.yml to the remote VM
+      template:
+        src: ./templates/k8s.yml.j2
+        dest: "{{ ansible_env.HOME }}/service-provider-template/group_vars/{{ org_id }}_{{ location_id }}/k8s.yml"
+
+    - name: Copy wildcard-pwa-{{ base_domain }}.yaml to the remote VM
+      template:
+        src: ./templates/wildcard-pwa-example.yml.j2
+        dest: "{{ ansible_env.HOME }}/service-provider-template/files/manifests/wildcard-pwa-{{ base_domain }}.yaml"
+
+    - name: Delete old wildcard-pwa file
+      file:
+        path: "{{ ansible_env.HOME }}/service-provider-template/files/manifests/wildcard-pwa-laconic.yaml"
+        state: absent
+
+    - name: Install required ansible roles
+      shell: ansible-galaxy install -f -p roles -r roles/requirements.yml
+      args:
+        chdir: "{{ ansible_env.HOME }}/service-provider-template"
+
+    - name: Install Kubernetes helper tools
+      shell: ./roles/k8s/files/scripts/get-kube-tools.sh
+      args:
+        chdir: "{{ ansible_env.HOME }}/service-provider-template"
+      become: yes
+
+    - name: Update group_vars/all/vault.yml with support email using template
+      template:
+        src: ./templates/vault.yml.j2
+        dest: "{{ ansible_env.HOME }}/service-provider-template/group_vars/all/vault.yml"
+
+    - name: Base64 encode DigitalOcean token
+      set_fact:
+        b64_encoded_token: "{{ do_api_token | b64encode }}"
+
+    - name: Update secret-digitalocean-dns.yaml with encoded token
+      template:
+        src: ./templates/secret-digitalocean-dns.yml.j2
+        dest: "{{ ansible_env.HOME }}/service-provider-template/files/manifests/secret-digitalocean-dns.yaml"
+      vars:
+        b64_encoded_token: "{{ b64_encoded_token }}"
+
+    - name: Remove k8s-vault.yml file
+      file:
+        path: "{{ ansible_env.HOME }}/service-provider-template/group_vars/{{ org_id }}_{{ location_id }}/k8s-vault.yml"
+        state: absent
+
+    - name: Generate token for the cluster
+      command: ./roles/k8s/files/scripts/token-vault.sh ./group_vars/{{ org_id }}_{{ location_id }}/k8s-vault.yml
+      args:
+        chdir: "{{ ansible_env.HOME }}/service-provider-template"
+
+    - name: Configure firewalld and nginx
+      command: ansible-playbook -i hosts site.yml --tags=firewalld,nginx
+      args:
+        chdir: "{{ ansible_env.HOME }}/service-provider-template"
+      environment:
+        ANSIBLE_HOST_KEY_CHECKING: "False"
+
+    - name: Deploy Kubernetes
+      command: ansible-playbook -i hosts site.yml --tags=k8s --limit={{ org_id }}_{{ location_id }} --user {{ ansible_user }}
+      args:
+        chdir: "{{ ansible_env.HOME }}/service-provider-template"
--- a/service-provider-setup/setup-system.yml
+++ b/service-provider-setup/setup-system.yml
@ -0,0 +1,134 @@
+- name: Setup system for the service provider setup
+  hosts: "{{ target_host }}"
+
+  environment:
+    GNUPGHOME: /home/{{ ansible_user }}/.gnupg
+
+  vars_files:
+    - vars/k8s-vars.yml
+    - vars/dns-vars.yml
+    - vars/gpg-vars.yml
+
+  tasks:
+    - name: Install required packages
+      apt:
+        name:
+          - doas
+          - zsh
+          - tmux
+          - git
+          - jq
+          - acl
+          - curl
+          - wget
+          - netcat-traditional
+          - fping
+          - rsync
+          - htop
+          - iotop
+          - iftop
+          - tar
+          - less
+          - firewalld
+          - sshguard
+          - wireguard
+          - iproute2
+          - iperf3
+          - zfsutils-linux
+          - net-tools
+          - ca-certificates
+          - gnupg
+          - sshpass
+          - apache2-utils
+        state: latest
+        update_cache: true
+      become: yes
+
+    - name: Set unique hostname
+      hostname:
+        name: "{{ inventory_hostname }}"
+      when: ansible_hostname != inventory_hostname
+
+    - name: Verify status of firewalld and enable sshguard
+      systemd:
+        name: "{{ item }}"
+        enabled: yes
+        state: started
+      loop:
+        - firewalld
+        - sshguard
+      ignore_errors: yes
+
+    - name: Disable and remove snapd
+      block:
+        - name: Disable snapd services
+          systemd:
+            name: "{{ item }}"
+            enabled: no
+            state: stopped
+          loop:
+            - snapd.service
+            - snapd.socket
+            - snapd.seeded
+            - snapd.snap-repair.timer
+          ignore_errors: yes
+
+        - name: Purge snapd
+          apt:
+            name: snapd
+            state: absent
+
+        - name: Remove snap directories
+          file:
+            path: "{{ item }}"
+            state: absent
+          loop:
+            - "{{ ansible_env.HOME }}/snap"
+            - /snap
+            - /var/snap
+            - /var/lib/snapd
+          become: yes
+          ignore_errors: yes
+
+    - name: Ensure GPG directory exists
+      file:
+        path: "{{ ansible_env.HOME }}/.gnupg"
+        state: directory
+        mode: '0700'
+
+    - name: Create GPG key parameters file
+      copy:
+        dest: /tmp/gpg_key_params.txt
+        content: |
+          Key-Type: RSA
+          Key-Length: 4096
+          Subkey-Type: RSA
+          Name-Real: {{ gpg_user_name }}
+          Name-Email: {{ gpg_user_email }}
+          Expire-Date: 0
+          Passphrase: {{ gpg_passphrase }}
+          %no-protection
+          %commit
+        mode: '0600'
+
+    - name: Generate GPG key using the parameter file
+      command: gpg --batch --gen-key /tmp/gpg_key_params.txt
+      become_user: "{{ ansible_user }}"
+      register: gpg_keygen_output
+      ignore_errors: yes
+
+    - name: Show GPG key generation output
+      debug:
+        var: gpg_keygen_output.stdout
+
+    - name: Fetch the Key ID of the most recently created GPG key
+      shell: gpg --list-secret-keys --keyid-format=long | grep 'sec' | tail -n 1 | awk -F'/' '{print $2}' | awk '{print $1}'
+      register: gpg_key_output
+
+    - name: Set the GPG key ID to a variable
+      set_fact:
+        sec_key_id: "{{ gpg_key_output.stdout }}"
+
+    - name: Show GPG Key ID
+      debug:
+        msg: "GPG Key ID: {{ sec_key_id }}"
--- a/service-provider-setup/setup-user.yml
+++ b/service-provider-setup/setup-user.yml
@ -0,0 +1,46 @@
+- name: Configure system
+  hosts: root_host
+  become: yes
+
+  vars_files:
+    - vars/user-vars.yml
+
+  tasks:
+    - name: Create a user
+      user:
+        name: "{{ username }}"
+        password: "{{ '{{ password }}' | password_hash('sha512') }}"
+        shell: /bin/bash
+        state: present
+
+    - name: Add user to sudoers group
+      user:
+        name: "{{ username }}"
+        groups: sudo
+        append: yes
+
+    - name: Ensure .ssh directory exists for user
+      file:
+        path: /home/{{ username }}/.ssh
+        state: directory
+        owner: "{{ username }}"
+        group: "{{ username }}"
+        mode: '0700'
+
+    - name: Append SSH public key to authorized_keys
+      lineinfile:
+        path: /home/{{ username }}/.ssh/authorized_keys
+        line: "{{ lookup('file', path_to_ssh_key) }}"
+        create: yes
+        owner: "{{ username }}"
+        group: "{{ username }}"
+        mode: '0600'
+        state: present
+
+    - name: Add user to sudoers for passwordless sudo
+      lineinfile:
+        path: /etc/sudoers
+        state: present
+        regexp: '^{{ username }} ALL=\(ALL\) NOPASSWD:ALL'
+        line: '{{ username }} ALL=(ALL) NOPASSWD:ALL'
+        validate: 'visudo -cf %s'
--- a/service-provider-setup/templates/configs/console-config.env.j2
+++ b/service-provider-setup/templates/configs/console-config.env.j2
@ -0,0 +1,5 @@
+CERC_LACONICD_USER_KEY={{ALICE_PK}}
+CERC_LACONICD_BOND_ID={{BOND_ID}}
+CERC_LACONICD_RPC_ENDPOINT=http://{{ org_id }}-{{ location_id }}-cluster-control.{{ full_domain }}:26657
+CERC_LACONICD_GQL_ENDPOINT=http://{{ org_id }}-{{ location_id }}-cluster-control.{{ full_domain }}:9473/api
+LACONIC_HOSTED_ENDPOINT=http://{{ org_id }}-{{ location_id }}-cluster-control.{{ full_domain }}:9473
--- a/service-provider-setup/templates/configs/webapp-deployer-config.env.j2
+++ b/service-provider-setup/templates/configs/webapp-deployer-config.env.j2
@ -0,0 +1,28 @@
+DEPLOYMENT_DNS_SUFFIX="pwa.{{ full_domain }}"
+
+# Name of reserved authority
+DEPLOYMENT_RECORD_NAMESPACE="{{ authority_name }}"
+
+# url of the deployed docker image registry
+IMAGE_REGISTRY="container-registry.pwa.{{ full_domain }}"
+
+# htpasswd credentials
+IMAGE_REGISTRY_USER="{{ container_registry_username }}"
+IMAGE_REGISTRY_CREDS="{{ container_registry_password }}"
+
+# configs
+CLEAN_DEPLOYMENTS=false
+CLEAN_LOGS=false
+CLEAN_CONTAINERS=false
+SYSTEM_PRUNE=false
+WEBAPP_IMAGE_PRUNE=true
+CHECK_INTERVAL=5
+FQDN_POLICY="allow"
+
+# lrn of the webapp deployer
+LRN="lrn://{{ authority_name }}/deployers/webapp-deployer-api.{{ full_domain }}"
+export OPENPGP_PRIVATE_KEY_FILE="webapp-deployer-api.{{ full_domain }}.pgp.key"
+export OPENPGP_PASSPHRASE="{{ deployer_gpg_passphrase }}"
+export DEPLOYER_STATE="srv-test/deployments/autodeploy.state"
+export UNDEPLOYER_STATE="srv-test/deployments/autoundeploy.state"
+export UPLOAD_DIRECTORY="srv-test/uploads"
--- a/service-provider-setup/templates/configs/webapp-ui-config.env.j2
+++ b/service-provider-setup/templates/configs/webapp-ui-config.env.j2
@ -0,0 +1,3 @@
+CERC_WEBAPP_DEBUG=0.1.0
+LACONIC_HOSTED_CONFIG_app_api_url=https://webapp-deployer-api.pwa.{{ full_domain }}
+LACONIC_HOSTED_CONFIG_app_console_link=http://{{ org_id }}-{{ location_id }}-cluster-control.{{ full_domain }}:9473/console?query=%0A%20%20fragment%20ValueParts%20on%20Value%20%7B%0A%20%20%20%20...%20on%20BooleanValue%20%7B%0A%20%20%20%20%20%20bool%3A%20value%0A%20%20%20%20%7D%0A%20%20%20%20...%20on%20IntValue%20%7B%0A%20%20%20%20%20%20int%3A%20value%0A%20%20%20%20%7D%0A%20%20%20%20...%20on%20FloatValue%20%7B%0A%20%20%20%20%20%20float%3A%20value%0A%20%20%20%20%7D%0A%20%20%20%20...%20on%20StringValue%20%7B%0A%20%20%20%20%20%20string%3A%20value%0A%20%20%20%20%7D%0A%20%20%20%20...%20on%20BytesValue%20%7B%0A%20%20%20%20%20%20bytes%3A%20value%0A%20%20%20%20%7D%0A%20%20%20%20...%20on%20LinkValue%20%7B%0A%20%20%20%20%20%20link%3A%20value%0A%20%20%20%20%7D%0A%20%20%7D%0A%0A%20%20fragment%20AttrParts%20on%20Attribute%20%7B%0A%20%20%20%20key%0A%20%20%20%20value%20%7B%0A%20%20%20%20%20%20...ValueParts%0A%20%20%20%20%20%20...%20on%20ArrayValue%20%7B%0A%20%20%20%20%20%20%20%20value%20%7B%0A%20%20%20%20%20%20%20%20%20%20...ValueParts%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%0A%20%20%7B%0A%20%20%20%20getRecordsByIds(ids%3A%20%5B%22#RQID#%22%5D)%20%7B%0A%20%20%20%20%20%20id%0A%20%20%20%20%20%20names%0A%20%20%20%20%20%20bondId%0A%20%20%20%20%20%20createTime%0A%20%20%20%20%20%20expiryTime%0A%20%20%20%20%20%20owners%0A%20%20%20%20%20%20attributes%20%7B%0A%20%20%20%20%20%20%20%20...AttrParts%0A%20%20%20%20%20%20%20%20value%20%7B%0A%20%20%20%20%20%20%20%20%20%20...%20on%20MapValue%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20map%3A%20value%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20...AttrParts%0A%20%20%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A
--- a/service-provider-setup/templates/control-firewalld.yml.j2
+++ b/service-provider-setup/templates/control-firewalld.yml.j2
@ -0,0 +1,16 @@
+---
+firewalld_add:
+  - name: public
+    interfaces:
+      - enp9s0
+    services:
+      - http
+      - https
+    ports:
+      - 6443/tcp
+
+  - name: trusted
+    sources:
+      - 10.42.0.0/16
+      - 10.43.0.0/16
+      - {{ service_provider_ip }}
--- a/service-provider-setup/templates/daemon-firewalld.yml.j2
+++ b/service-provider-setup/templates/daemon-firewalld.yml.j2
@ -0,0 +1,16 @@
+---
+firewalld_add:
+  - name: public
+    interfaces:
+      - ens3
+    services:
+      - http
+      - https
+    ports:
+      - 26657/tcp
+      - 26656/tcp
+      - 1317/tcp
+
+  - name: trusted
+    sources:
+      - {{ service_provider_ip }}
--- a/service-provider-setup/templates/hosts.j2
+++ b/service-provider-setup/templates/hosts.j2
@ -0,0 +1,12 @@
+[all]
+{{ org_id }}-daemon ansible_host={{ service_provider_ip }}
+{{ org_id }}-{{ location_id }}-cluster-control ansible_host={{ service_provider_ip }}
+
+[so]
+{{ org_id }}-daemon
+
+[{{ org_id }}_{{ location_id }}]
+{{ org_id }}-{{ location_id }}-cluster-control k8s_node_type=bootstrap k8s_pod_limit=1024 k8s_external_ip={{ service_provider_ip }}
+
+[k8s:children]
+{{ org_id }}_{{ location_id }}
--- a/service-provider-setup/templates/k8s.yml.j2
+++ b/service-provider-setup/templates/k8s.yml.j2
@ -0,0 +1,55 @@
+---
+# default context is used for stack orchestrator deployments, for testing a custom context name can be usefull
+#k8s_cluster_name: {{ org_id }}-{{ location_id }}-cluster
+k8s_cluster_name: default
+k8s_cluster_url: {{ org_id }}-{{ location_id }}-cluster-control.{{ full_domain }}
+k8s_taint_servers: false
+
+k8s_acme_email: "{{ support_email }}"
+
+# k3s bundles traefik as the default ingress controller, we will disable it and use nginx instead
+k8s_disable:
+  - traefik
+
+# secrets can be stored in a file or as a template, the template secrets gets dynamically base64 encoded while file based secrets must be encoded by hand
+k8s_secrets:
+  - name: digitalocean-dns
+    type: file
+    source: secret-digitalocean-dns.yaml
+
+k8s_manifests:
+  # ingress controller, replaces traefik which is explicitly disabled
+  - name: ingress-nginx
+    type: url
+    source: https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.1/deploy/static/provider/cloud/deploy.yaml
+
+  # cert-manager, required for letsencrypt
+  - name: cert-manager
+    type: url
+    source: https://github.com/cert-manager/cert-manager/releases/download/v1.15.1/cert-manager.yaml
+
+  # issuer for basic http certs
+  - name: letsencrypt-prod
+    type: template
+    source: shared/clusterissuer-acme.yaml
+    server: https://acme-v02.api.letsencrypt.org/directory
+    solvers:
+      - type: http
+        ingress: nginx
+
+  # issuer for wildcard dns certs
+  - name: letsencrypt-prod-wild
+    type: template
+    source: shared/clusterissuer-acme.yaml
+    server: https://acme-v02.api.letsencrypt.org/directory
+    solvers:
+      - type: dns
+        provider: digitalocean
+        tokenref: tokenSecretRef
+        secret_name: digitalocean-dns
+        secret_key: access-token
+
+  # initiate wildcard cert
+  - name: pwa.{{ full_domain }}
+    type: file
+    source: wildcard-pwa-{{ base_domain }}.yaml
--- a/service-provider-setup/templates/laconic.yml.j2
+++ b/service-provider-setup/templates/laconic.yml.j2
@ -0,0 +1,9 @@
+services:
+  registry:
+    rpcEndpoint: 'http://{{ subdomain_cluster_control }}.{{ full_domain }}:26657'
+    gqlEndpoint: 'http://{{ subdomain_cluster_control}}.{{ full_domain }}:9473/api'
+    userKey: "{{ ALICE_PK }}"
+    bondId: "{{ BOND_ID }}"
+    chainId: lorotestnet-1
+    gas: 200000
+    fees: 200000alnt
--- a/service-provider-setup/templates/my_password.json.j2
+++ b/service-provider-setup/templates/my_password.json.j2
@ -0,0 +1,9 @@
+{
+  "auths": {
+    "{{container_registry_domain}}": {
+      "username": "{{ container_registry_username }}",
+      "password": "{{ hashed_password }}",
+      "auth": "{{ b64_encoded_cred }}"
+    }
+  }
+}
--- a/service-provider-setup/templates/nginx.yml.j2
+++ b/service-provider-setup/templates/nginx.yml.j2
@ -0,0 +1,21 @@
+---
+nginx_packages_intall: false
+nginx_server_name_hash: 64
+nginx_proxy_read_timeout: 1200
+nginx_proxy_send_timeout: 1200
+nginx_proxy_connection_timeout: 75
+
+nginx_sites:
+  - name: {{ org_id }}-console
+    url: {{ org_id }}-console.{{ full_domain }}
+    upstream: http://localhost:8080
+    template: basic-proxy
+    ssl: true
+
+  - name: {{ org_id }}-daemon
+    url: {{ org_id }}-daemon.{{ full_domain }}
+    upstream: http://localhost:9473
+    configs:
+      - rewrite ^/deployer(/.*)? https://webapp-deployer.pwa.{{full_domain}} permanent
+    template: websocket-proxy
+    ssl: true
--- a/service-provider-setup/templates/secret-digitalocean-dns.yml.j2
+++ b/service-provider-setup/templates/secret-digitalocean-dns.yml.j2
@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: cert-manager
+---
+apiVersion: v1
+data:
+  access-token: {{ b64_encoded_token }}
+kind: Secret
+metadata:
+  name: digitalocean-dns
+  namespace: cert-manager
--- a/service-provider-setup/templates/specs/container-registry.spec.j2
+++ b/service-provider-setup/templates/specs/container-registry.spec.j2
@ -0,0 +1,16 @@
+stack: container-registry
+deploy-to: k8s
+kube-config: /home/{{ ansible_user }}/.kube/config-default.yaml
+network:
+  ports:
+    registry:
+     - '5000'
+  http-proxy:
+    - host-name: container-registry.pwa.{{full_domain}}
+      routes:
+        - path: '/'
+          proxy-to: registry:5000
+volumes:
+  registry-data:
+configmaps:
+  config: ./configmaps/config
--- a/service-provider-setup/templates/specs/fixturenet-laconicd-spec.yml.j2
+++ b/service-provider-setup/templates/specs/fixturenet-laconicd-spec.yml.j2
@ -0,0 +1,15 @@
+stack:
+  /home/{{ansible_user}}/cerc/fixturenet-laconicd-stack/stack-orchestrator/stacks/fixturenet-laconicd
+deploy-to: compose
+network:
+  ports:
+    laconicd:
+     - '6060:6060'
+     - '26657:26657'
+     - '26656:26656'
+     - '9473:9473'
+     - '9090:9090'
+     - '1317:1317'
+volumes:
+  laconicd-data: ./data/laconicd-data
+  genesis-config: ./data/genesis-config
--- a/service-provider-setup/templates/specs/laconic-console-spec.yml.j2
+++ b/service-provider-setup/templates/specs/laconic-console-spec.yml.j2
@ -0,0 +1,9 @@
+stack:
+  /home/{{ansible_user}}/cerc/testnet-laconicd-stack/stack-orchestrator/stacks/laconic-console
+deploy-to: compose
+network:
+  ports:
+    console:
+     - '8080:80'
+volumes:
+  laconic-registry-data: ./data/laconic-registry-data
--- a/service-provider-setup/templates/specs/webapp-deployer.spec.j2
+++ b/service-provider-setup/templates/specs/webapp-deployer.spec.j2
@ -0,0 +1,35 @@
+stack: webapp-deployer-backend
+deploy-to: k8s
+kube-config: {{ansible_env.HOME}}/.kube/config-default.yaml
+image-registry: container-registry.pwa.{{full_domain}}/laconic-registry
+network:
+  ports:
+    server:
+      - '9555'
+  http-proxy:
+    - host-name: webapp-deployer-api.pwa.{{ full_domain }}
+      routes:
+        - path: '/'
+          proxy-to: server:9555
+volumes:
+  srv:
+configmaps:
+  config: ./data/config
+annotations:
+  container.apparmor.security.beta.kubernetes.io/{name}: unconfined
+labels:
+  container.kubeaudit.io/{name}.allow-disabled-apparmor: "podman"
+security:
+  privileged: true
+
+resources:
+  containers:
+    reservations:
+      cpus: "{{ cpu_reservation }}"
+      memory: "{{ memory_reservation }}"
+    limits:
+      cpus: 6
+      memory: 16G
+  volumes:
+    reservations:
+      storage: 200G
--- a/service-provider-setup/templates/vault.yml.j2
+++ b/service-provider-setup/templates/vault.yml.j2
@ -0,0 +1,2 @@
+---
+support_email: {{ support_email }}
--- a/service-provider-setup/templates/wildcard-pwa-example.yml.j2
+++ b/service-provider-setup/templates/wildcard-pwa-example.yml.j2
@ -0,0 +1,15 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: pwa.{{ full_domain }}
+  namespace: default
+spec:
+  secretName: pwa.{{ full_domain }}
+  issuerRef:
+    name: letsencrypt-prod-wild
+    kind: ClusterIssuer
+    group: cert-manager.io
+  commonName: *.pwa.{{ full_domain }}
+  dnsNames:
+    - pwa.{{ full_domain }}
+    - *.pwa.{{ full_domain }}
--- a/service-provider-setup/vars/container-vars.example.yml
+++ b/service-provider-setup/vars/container-vars.example.yml
@ -0,0 +1,3 @@
+container_registry_username: ""
+container_registry_password: ""
+container_registry_domain: "container-registry.pwa.{{ full_domain }}"
--- a/service-provider-setup/vars/dns-vars.example.yml
+++ b/service-provider-setup/vars/dns-vars.example.yml
@ -0,0 +1,5 @@
+full_domain: ""
+subdomain_prefix: ""
+subdomain_cluster_control: "{{ subdomain_prefix }}-cluster-control"
+service_provider_ip: ""
+do_api_token: ""
--- a/service-provider-setup/vars/gpg-vars.example.yml
+++ b/service-provider-setup/vars/gpg-vars.example.yml
@ -0,0 +1,3 @@
+gpg_user_name: ""
+gpg_user_email: ""
+gpg_passphrase: ""
--- a/service-provider-setup/vars/k8s-vars.example.yml
+++ b/service-provider-setup/vars/k8s-vars.example.yml
@ -0,0 +1,8 @@
+target_host: "deployment_host"
+gpg_key_id: "{{ sec_key_id }}"
+vault_passphrase: "{{ gpg_passphrase }}"
+org_id: ""
+location_id: ""
+base_domain: ""
+support_email: ""
+ansible_ssh_extra_args: '-o StrictHostKeyChecking=no'
--- a/service-provider-setup/vars/user-vars.example.yml
+++ b/service-provider-setup/vars/user-vars.example.yml
@ -0,0 +1,3 @@
+username: ""
+password: ""
+path_to_ssh_key: ""
--- a/service-provider-setup/vars/webapp-vars.example.yml
+++ b/service-provider-setup/vars/webapp-vars.example.yml
@ -0,0 +1,6 @@
+ALICE_PK: "{{ ALICE_PK }}"
+BOND_ID: "{{ BOND_ID }}"
+authority_name: ""
+cpu_reservation: ""
+memory_reservation: ""
+deployer_gpg_passphrase: ""
--- a/stack-orchestrator-setup/README.md
+++ b/stack-orchestrator-setup/README.md
@ -81,9 +81,11 @@ To run the playbook on a remote host:
  ```bash
  # For bash users
  echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bashrc
+  source ~/.bashrc

  # For zsh users
  echo 'export PATH="$HOME/bin:$PATH"' >> ~/.zshrc
+  source ~/.zshrc
  ```

 - Once the PATH is set, verify the installation by running the following commands: