Add logs to debug why api is not working

Handle deployment auction requests (#19 )
Part of [Service provider auctions for web deployments](https://www.notion.so/Service-provider-auctions-for-web-deployments-104a6b22d47280dbad51d28aa3a91d75) and cerc-io/stack-orchestrator#948 Requires cerc-io/stack-orchestrator#950 Reviewed-on: cerc-io/webapp-deployment-status-api#19 Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com> Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2024-10-24 18:36:39 +05:30 · 2024-10-21 07:07:06 +00:00
3 changed files with 185 additions and 86 deletions
--- a/README.md
+++ b/README.md
@ -5,10 +5,10 @@ files used by those requests.

 ## Build and Run

-```
-$ yarn
-$ yarn build
-$ yarn start
+```bash
+yarn
+yarn build
+yarn start
 ```

 ## Configuration
@ -20,22 +20,22 @@ On upload, the configuration is temporarily decrypted for validation, but stored

 To create and export a key in the necessary format use:

-```
+```bash
 # Create a key
-$ gpg --batch --passphrase "SECRET" --quick-generate-key webapp-deployer-api.my.domain.com default default never
+gpg --batch --passphrase "SECRET" --quick-generate-key webapp-deployer-api.my.domain.com default default never

 # Export the public key
-$ gpg --export webapp-deployer-api.my.domain.com > webapp-deployer-api.my.domain.com.pgp.pub
+gpg --export webapp-deployer-api.my.domain.com > webapp-deployer-api.my.domain.com.pgp.pub

 # Export the private key
-$ gpg --export-secret-keys webapp-deployer-api.my.domain.com > webapp-deployer-api.my.domain.com.pgp.key
+gpg --export-secret-keys webapp-deployer-api.my.domain.com > webapp-deployer-api.my.domain.com.pgp.key
 ```

 ### Create the Deployer Record

 Every webapp deployer should have `WebappDeployer` record in the registry which looks something like:

-```
+```yml
 record:
  type: WebappDeployer
  version: 1.0.0
@ -48,16 +48,34 @@ record:

 This record can most easily be created using `laconic-so publish-deployer-to-registry`.

-```
-$ laconic-so publish-deployer-to-registry \
+```bash
+laconic-so publish-deployer-to-registry \
  --laconic-config ~/.laconic/registry.yml \
-  --api-url https://webapp-deployer-api.my.domain.com 
+  --api-url https://webapp-deployer-api.my.domain.com
  --public-key-file webapp-deployer-api.my.domain.com.pgp.pub \
  --lrn lrn://laconic/deployers/webapp-deployer-api.my.domain.com  \
  --min-required-payment 100
 ```

-This will create the record in the proper format and assign its LRN. 
+This will create the record in the proper format and assign its LRN.
+
+### Publish Deployment Auction
+
+Users can optionally create an auction for app deployment with desired number of providers and max price they are willing to pay for a deployment:
+
+```bash
+laconic-so publish-deployment-auction \
+  --laconic-config ./config.yml \
+  --app lrn://cerc-io/applications/webapp-hello-world@0.1.3 \
+  --commits-duration 3600 \
+  --reveals-duration 3600 \
+  --commit-fee 10000 \
+  --reveal-fee 10000 \
+  --max-price 5000000 \
+  --num-providers 3
+```
+
+This will create a `provider` auction with given params and publish a deployment auction record.

 ### Request Deployment

@ -67,11 +85,11 @@ Users can now request deployment using the LRN of the deployer.  This will allow
 1. Obtain the public key for encrypting config.
 1. See the minimum required payment.

-The request can be made using `laconic-so request-webapp-deployment`.  This will handle encrypting and uploading the
+The request can be made using `laconic-so request-webapp-deployment`. This will handle encrypting and uploading the
 config automatically, as well as making a payment (if necessary).

-```
-$ laconic-so request-webapp-deployment \
+```bash
+laconic-so request-webapp-deployment \
  --laconic-config ~/.laconic/registry.yml \
  --deployer lrn://laconic/deployers/webapp-deployer-api.my.domain.com \
  --app lrn://cerc-io/applications/webapp-hello-world@0.1.3 \
@ -79,9 +97,33 @@ $ laconic-so request-webapp-deployment \
  --make-payment auto
 ```

+Alternatively, users can also use a deployment auction they created instead of making the payment to any specific deployer directly:
+
+```bash
+laconic-so request-webapp-deployment \
+  --laconic-config ~/.laconic/registry.yml \
+  --app lrn://cerc-io/applications/webapp-hello-world@0.1.3 \
+  --env-file hello.env \
+  --auction-id 4c9701c22651e143202e991056b6e7649853acc5bc0e97e3a98e09c9f3355909
+```
+
+This creates deployment requests targeted towards all the deployers who have won the auction. Similar to requests with payments, the config is automatically encrypted and uploaded to all the deployers.
+
+### Request Undeployment
+
+Users can also request removal of an existing deployment using the deployment record id:
+
+```bash
+laconic-so request-webapp-undeployment \
+  --laconic-config ~/.laconic/registry.yml \
+  --deployer lrn://laconic/deployers/webapp-deployer-api.my.domain.com \
+  --deployment bafyreigeopr72dmp6rhvnomgdz3cljbqzhh75epcrigit7ue6i6vjullme \
+  --make-payment auto
+```
+
 ### Example Config

-```
+```bash
 UPLOAD_DIRECTORY="/srv/uploads/config"
 UPLOAD_MAX_SIZE="1MB"
 DEPLOYER_STATE="/srv/deployments/autodeploy.state"
@ -90,4 +132,7 @@ BUILD_LOGS="/srv/logs"
 OPENPGP_PASSPHRASE="SECRET"
 OPENPGP_PRIVATE_KEY_FILE="/etc/config/webapp-deployer-api.my.domain.com.pgp.key"
 LACONIC_CONFIG="/etc/config/registry.yml"
+LRN=lrn://laconic/deployers/webapp-deployer-api.my.domain.com
+HANDLE_AUCTION_REQUESTS=true
+AUCTION_BID_AMOUNT=50000
 ```
--- a/run.sh
+++ b/run.sh
@ -35,6 +35,16 @@ if [ ! -f "/etc/config/kube.yml" ]; then
  exit 2
 fi

+AUCTION_OPTS=""
+if [ "$HANDLE_AUCTION_REQUESTS" = "true" ]; then
+  if [ -z "$AUCTION_BID_AMOUNT" ]; then
+    echo "AUCTION_BID_AMOUNT is required when handling auction requsts."
+    exit 2
+  fi
+
+  AUCTION_OPTS="--auction-requests"
+fi
+
 STORAGE_ROOT="${STORAGE_ROOT:-/srv}"
 DEPLOYMENTS_DIR="${DEPLOYMENTS_DIR:-$STORAGE_ROOT/deployments}"
 LOG_DIR="${LOG_DIR:-$STORAGE_ROOT/logs}"
@ -113,67 +123,82 @@ while true; do
    yarn start &
  fi

-  echo "########### UNDEPLOY ############"
-  laconic-so undeploy-webapp-from-registry \
-    --laconic-config /etc/config/laconic.yml \
-    --deployment-parent-dir "${DEPLOYMENTS_DIR}" \
-    --delete-names \
-    --delete-volumes \
-    --state-file "${DEPLOYMENTS_DIR}/autoremove.state" \
-    --include-tags "$INCLUDE_TAGS" \
-    --exclude-tags "$EXCLUDE_TAGS" \
-    --lrn "$LRN" \
-    --min-required-payment ${MIN_REQUIRED_PAYMENT:-0} \
-    $EXTRA_UNDEPLOY_OPTS \
-    $UPDATE_OPTS \
-    --discover
-  rc=$?
-  if [ $rc -eq 0 ]; then
-    echo "############ UNDEPLOY SUCCESS #############"
-  else
-    echo "############ UNDEPLOY FAILURE STATUS $rc #############"
-  fi
+#   echo "########### UNDEPLOY ############"
+#   laconic-so undeploy-webapp-from-registry \
+#     --laconic-config /etc/config/laconic.yml \
+#     --deployment-parent-dir "${DEPLOYMENTS_DIR}" \
+#     --delete-names \
+#     --delete-volumes \
+#     --state-file "${DEPLOYMENTS_DIR}/autoremove.state" \
+#     --include-tags "$INCLUDE_TAGS" \
+#     --exclude-tags "$EXCLUDE_TAGS" \
+#     --lrn "$LRN" \
+#     --min-required-payment ${MIN_REQUIRED_PAYMENT:-0} \
+#     $EXTRA_UNDEPLOY_OPTS \
+#     $UPDATE_OPTS \
+#     --discover
+#   rc=$?
+#   if [ $rc -eq 0 ]; then
+#     echo "############ UNDEPLOY SUCCESS #############"
+#   else
+#     echo "############ UNDEPLOY FAILURE STATUS $rc #############"
+#   fi

-  echo "############ DEPLOY #############"
-  laconic-so deploy-webapp-from-registry \
-    --kube-config /etc/config/kube.yml \
-    --laconic-config /etc/config/laconic.yml \
-    --image-registry ${IMAGE_REGISTRY} \
-    --deployment-parent-dir "${DEPLOYMENTS_DIR}" \
-    --dns-suffix ${DEPLOYMENT_DNS_SUFFIX} \
-    --record-namespace-dns lrn://${DEPLOYMENT_RECORD_NAMESPACE}/dns \
-    --record-namespace-deployments lrn://${DEPLOYMENT_RECORD_NAMESPACE}/deployments \
-    --state-file "${DEPLOYMENTS_DIR}/autodeploy.state" \
-    --include-tags "$INCLUDE_TAGS" \
-    --exclude-tags "$EXCLUDE_TAGS" \
-    --fqdn-policy "${FQDN_POLICY:-prohibit}" \
-    --lrn "$LRN" \
-    --min-required-payment ${MIN_REQUIRED_PAYMENT:-0} \
-    --config-upload-dir "$UPLOAD_DIRECTORY" \
-    --private-key-file "$OPENPGP_PRIVATE_KEY_FILE" \
-    --private-key-passphrase "$OPENPGP_PASSPHRASE" \
-    $LOG_OPTS \
-    $EXTRA_DEPLOY_OPTS \
-    $UPDATE_OPTS \
-    --discover
-  rc=$?
-  if [ $rc -eq 0 ]; then
-    echo "############ DEPLOY SUCCESS #############"
-  else
-    echo "############ DEPLOY FAILURE STATUS $rc #############"
-  fi
+#   echo "############ DEPLOY #############"
+#   laconic-so deploy-webapp-from-registry \
+#     --kube-config /etc/config/kube.yml \
+#     --laconic-config /etc/config/laconic.yml \
+#     --image-registry ${IMAGE_REGISTRY} \
+#     --deployment-parent-dir "${DEPLOYMENTS_DIR}" \
+#     --dns-suffix ${DEPLOYMENT_DNS_SUFFIX} \
+#     --record-namespace-dns lrn://${DEPLOYMENT_RECORD_NAMESPACE}/dns \
+#     --record-namespace-deployments lrn://${DEPLOYMENT_RECORD_NAMESPACE}/deployments \
+#     --state-file "${DEPLOYMENTS_DIR}/autodeploy.state" \
+#     --include-tags "$INCLUDE_TAGS" \
+#     --exclude-tags "$EXCLUDE_TAGS" \
+#     --fqdn-policy "${FQDN_POLICY:-prohibit}" \
+#     --lrn "$LRN" \
+#     --min-required-payment ${MIN_REQUIRED_PAYMENT:-0} \
+#     --config-upload-dir "$UPLOAD_DIRECTORY" \
+#     --private-key-file "$OPENPGP_PRIVATE_KEY_FILE" \
+#     --private-key-passphrase "$OPENPGP_PASSPHRASE" \
+#     $AUCTION_OPTS \
+#     $LOG_OPTS \
+#     $EXTRA_DEPLOY_OPTS \
+#     $UPDATE_OPTS \
+#     --discover
+#   rc=$?
+#   if [ $rc -eq 0 ]; then
+#     echo "############ DEPLOY SUCCESS #############"
+#   else
+#     echo "############ DEPLOY FAILURE STATUS $rc #############"
+#   fi

-  # Cleanup any build leftovers
-  if [[ "${SYSTEM_PRUNE:-false}" == "true" ]]; then
-    docker system prune --all --force
-  fi
-  if [[ "${WEBAPP_IMAGE_PRUNE:-true}" == "true" ]]; then
-    APP_IMAGES="$(docker image ls --quiet --filter 'reference=laconic-webapp')"
-    DANGLING_IMAGES="$(docker image ls --quiet --filter 'dangling=true')"
-    if [[ -n "$APP_IMAGES" ]] || [[ -n "$DANGLING_IMAGES" ]]; then
-      echo "Pruning images: $APP_IMAGES $DANGLING_IMAGES"
-      docker image rm -f $APP_IMAGES $DANGLING_IMAGES
-    fi
-  fi
+#   if [ "$HANDLE_AUCTION_REQUESTS" = "true" ]; then
+#     echo "############ DEPLOYMENT AUCTION #############"
+#     laconic-so handle-deployment-auction \
+#       --laconic-config /etc/config/laconic.yml \
+#       --state-file "${DEPLOYMENTS_DIR}/autoauction.state" \
+#       --bid-amount ${AUCTION_BID_AMOUNT}
+#     rc=$?
+#     if [ $rc -eq 0 ]; then
+#       echo "############ DEPLOYMENT AUCTION SUCCESS #############"
+#     else
+#       echo "############ DEPLOYMENT AUCTION FAILURE STATUS $rc #############"
+#     fi
+#   fi
+
+#   # Cleanup any build leftovers
+#   if [[ "${SYSTEM_PRUNE:-false}" == "true" ]]; then
+#     docker system prune --all --force
+#   fi
+#   if [[ "${WEBAPP_IMAGE_PRUNE:-true}" == "true" ]]; then
+#     APP_IMAGES="$(docker image ls --quiet --filter 'reference=laconic-webapp')"
+#     DANGLING_IMAGES="$(docker image ls --quiet --filter 'dangling=true')"
+#     if [[ -n "$APP_IMAGES" ]] || [[ -n "$DANGLING_IMAGES" ]]; then
+#       echo "Pruning images: $APP_IMAGES $DANGLING_IMAGES"
+#       docker image rm -f $APP_IMAGES $DANGLING_IMAGES
+#     fi
+#   fi
  sleep ${CHECK_INTERVAL:-15}
 done
--- a/src/deployments.ts
+++ b/src/deployments.ts
@ -108,8 +108,12 @@ export class RegHelper {
  }

  async deploymentRequestStatus(requestId?: string) {
+    console.log('Starting deploymentRequestStatus with requestId:', requestId);
+
    const requests: any[] = [];
    const deployments: any[] = [];
+
+    // Querying for removal requests
    const removalRequests = await this.queryRecords({
      type: 'ApplicationDeploymentRemovalRequest',
    });
@ -118,53 +122,75 @@ export class RegHelper {
      const request = await this.getRecordById(requestId);
      if (request) {
        requests.push(request);
+        console.log('Found request:', request);
+      } else {
+        console.log('Request not found for requestId:', requestId);
      }
-      deployments.push(...await this.queryRecords({
-        type: 'ApplicationDeploymentRecord', request: requestId
-      }));
-    } else {
-      requests.push(...await this.queryRecords({
-        type: 'ApplicationDeploymentRequest',
-      }));
-      deployments.push(...await this.queryRecords({
+
+      const foundDeployments = await this.queryRecords({
        type: 'ApplicationDeploymentRecord',
-      }));
+        request: requestId
+      });
+      deployments.push(...foundDeployments);
+    } else {
+      console.log('Fetching all ApplicationDeploymentRequests');
+      const allRequests = await this.queryRecords({
+        type: 'ApplicationDeploymentRequest',
+      });
+      requests.push(...allRequests);
+      console.log('All requests:', allRequests);
+
+      console.log('Fetching all ApplicationDeploymentRecords');
+      const allDeployments = await this.queryRecords({
+        type: 'ApplicationDeploymentRecord',
+      });
+      deployments.push(...allDeployments);
+      console.log('All deployments:', allDeployments);
    }

+    console.log('Sorting requests by createTime');
    requests.sort((a, b) => a.createTime === b.createTime ? 0 : a.createTime > b.createTime ? 1 : -1,);
    requests.reverse();
+    console.log('Sorted requests:', requests);

    const deploymentsByRequest = new Map<string, any>();
    for (const d of deployments) {
      deploymentsByRequest.set(d.attributes.request, d);
    }
+    console.log('Deployments by request:', deploymentsByRequest);
+
    const removalsByRequest = new Map<string, any>();
    for (const rr of removalRequests) {
      if (rr.attributes.request) {
        removalsByRequest.set(rr.attributes.request, rr);
      }
    }
+    console.log('Removals by request:', removalsByRequest);

    const latestByHostname = new Map<string, any>();

    const ret = [];
    for (const r of requests) {
+      console.log('Processing request:', r.id);
      const status = new RequestStatus(r.id, r.createTime);
      ret.push(status);

      const app = await this.getRecord(r.attributes.application);
      if (!app) {
+        console.log('Error: Application not found for request:', r.id);
        status.lastState = 'ERROR';
        continue;
      }

      status.app = r.attributes.application;
      const hostname = r.attributes.dns ?? generateHostnameForApp(app);
+      console.log('Hostname for app:', hostname);

      if (deploymentsByRequest.has(r.id)) {
        const deployment = deploymentsByRequest.get(r.id);
        status.url = deployment.attributes.url;
        status.lastUpdate = deployment.createTime;
+        console.log('Deployment found for request:', r.id, 'with deployment:', deployment);

        if (!latestByHostname.has(hostname)) {
          latestByHostname.set(hostname, status);
@ -179,11 +205,13 @@ export class RegHelper {
      }

      if (removalsByRequest.has(r.id)) {
+        console.log('Removal request found for request:', r.id);
        status.lastState = 'CANCELLED';
        continue;
      }

      if (latestByHostname.has(hostname)) {
+        console.log('Cancellation found for hostname:', hostname);
        status.lastState = 'CANCELLED';
        continue;
      }
@ -191,6 +219,7 @@ export class RegHelper {
      latestByHostname.set(hostname, status);
    }

+    console.log('Final status array:', ret);
    return ret;
  }