Merge pull request #213 from cerc-io/dboreham/fix-non-stack-deploy

Fix for #212 - exception on non-stack deploy

Former-commit-id: ee778356c0

app/data/stacks/build-support/README.md

@@ -11,20 +11,25 @@ To use a user-supplied registry set these environment variables:
 
 Leave `CERC_NPM_REGISTRY_URL` un-set to use the local gitea registry.
 
-### Build support containers
+### 1. Build support containers
 ```
 $ laconic-so --stack build-support build-containers
 ```
-### Deploy Gitea Package Registry
+Note that the scheme/gerbil builder container can take a while to build so if you aren't going to build scheme projects it can be skipped with:
+```
+$ laconic-so --stack build-support build-containers --exclude cerc/builder-gerbil
+```
+### 2. Deploy Gitea Package Registry
 
 ```
 $ laconic-so --stack package-registry setup-repositories
-$ laconic-so --stack package-registry deploy-system up
+$ laconic-so --stack package-registry deploy up
 This is your gitea access token: 84fe66a73698bf11edbdccd0a338236b7d1d5c45. Keep it safe and secure, it can not be fetched again from gitea.
-$ export CERC_NPM_AUTH_TOKEN=84fe66a73698bf11edbdccd0a338236b7d1d5c45
 ```
 Now npm packages can be built:
 ### Build npm Packages
+Ensure that `CERC_NPM_AUTH_TOKEN` is set with the token printed above when the package-registry stack was deployed (the actual token value will be different than shown in this example):
 ```
+$ export CERC_NPM_AUTH_TOKEN=84fe66a73698bf11edbdccd0a338236b7d1d5c45
 $ laconic-so build-npms --include laconic-sdk
 ```

app/data/stacks/fixturenet-laconicd/README.md

@@ -1,18 +1,50 @@
 # Laconicd Fixturenet
 
-Instructions for deploying a local Laconic blockchain "fixturenet" for development and testing purposes using laconic-stack-orchestrator (the installation of which is covered [here](https://github.com/cerc-io/stack-orchestrator#user-mode)):
+Instructions for deploying a local Laconic blockchain "fixturenet" for development and testing purposes using laconic-stack-orchestrator.
 
-## Clone required repositories
+## 1. Install Laconic Stack Orchestrator
+Installation is covered in detail [here](https://github.com/cerc-io/stack-orchestrator#user-mode) but if you're on Linux and already have docker installed it should be as simple as:
 ```
-$ laconic-so setup-repositories --include cerc-io/laconicd,cerc-io/laconic-sdk,cerc-io/laconic-registry-cli
+$ mkdir my-working-dir
+$ cd my-working-dir
+$ curl -L -o ./laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
+$ chmod +x ./laconic-so
+$ export PATH=$PATH:$(pwd)  # Or move laconic-so to ~/bin or your favorite on-path directory
 ```
-## Build the laconicd container
+## 2. Prepare the local build environment
+Note that this step needs only to be done once on a new machine. 
+Detailed instructions can be found [here](../build-support/README.md). For the impatient run these commands:
 ```
-$ laconic-so build-containers --include cerc/laconicd
+$ laconic-so --stack build-support build-containers --exclude cerc/builder-gerbil
+$ laconic-so --stack package-registry setup-repositories
+$ laconic-so --stack package-registry deploy-system up
 ```
-This should create a container with tag `cerc/laconicd` in the local image registry.
+Then add the localhost alias `gitea.local` and set `CERC_NPM_AUTH_TOKEN` to the token printed when the package-registry stack was deployed above:
-## Deploy the stack
 ```
-$ laconic-so deploy-system --include fixturenet-laconicd up
+$ sudo vi /etc/hosts
+$ export CERC_NPM_AUTH_TOKEN=<my-token>
+```
+
+## 3. Clone required repositories
+```
+$ laconic-so --stack fixturenet-laconicd setup-repositories
+```
+## 4. Build the stack's packages and containers
+```
+$ laconic-so --stack fixturenet-laconicd build-npms
+$ laconic-so --stack fixturenet-laconicd build-containers
+```
+## 5. Deploy the stack
+```
+$ laconic-so --stack fixturenet-laconicd deploy up
+```
+Correct operation should be verified by checking the laconicd container's logs with:
+```
+$ laconic-so --stack fixturenet-laconicd deploy logs
+```
+## 6. Test with the Registry CLI
+```
+$ laconic-so --stack fixturenet-laconicd deploy exec laconicd "sh /docker-entrypoint-scripts.d/export-mykey.sh"
+$ laconic-so --stack fixturenet-laconicd deploy exec cli "sh /scripts/import-key.sh xxxxxxxxxxxxxxxxxxxxxxx"
+$ laconic-so --stack fixturenet-laconicd deploy exec cli "laconic cns status"
 ```
-Correct operation should be verified by checking the laconicd container's log.

app/data/stacks/fixturenet-laconicd/stack.yml

@@ -13,3 +13,6 @@ containers:
   - cerc/laconic-registry-cli
 pods:
   - fixturenet-laconicd
+config:
+  cli:
+    key: laconicd.mykey

app/data/stacks/package-registry/README.md

@@ -0,0 +1,5 @@
+# Package Registry Stack
+
+The Package Registry Stack supports a build environment that requires a package registry (initially for NPM packages only).
+
+Setup instructions can be found [here](../build-support/README.md).

app/data/version.txt

@@ -1,2 +1,2 @@
 # This file should be re-generated running: scripts/update-version-file.sh script
-v1.0.22-4fdb442
+v1.0.25-75376d7

app/deploy_system.py

@@ -19,6 +19,7 @@ import hashlib
 import copy
 import os
 import sys
+from dataclasses import dataclass
 from decouple import config
 import subprocess
 from python_on_whales import DockerClient
@@ -66,6 +67,9 @@ def command(ctx, include, exclude, cluster, command, extra_args):
             docker.compose.up(detach=True, services=extra_args_list)
             for post_start_command in cluster_context.post_start_commands:
                 _run_command(ctx.obj, cluster_context.cluster, post_start_command)
+
+            _orchestrate_cluster_config(ctx.obj, cluster_context.config, docker, container_exec_env)
+
         elif command == "down":
             if verbose:
                 print("Running compose down")
@@ -114,7 +118,8 @@ def command(ctx, include, exclude, cluster, command, extra_args):
         elif command == "logs":
             if verbose:
                 print("Running compose logs")
-            docker.compose.logs()
+            logs_output = docker.compose.logs(services=extra_args_list if extra_args_list is not None else [])
+            print(logs_output)
 
 
 def get_stack_status(ctx, stack):
@@ -180,8 +185,10 @@ def _make_cluster_context(ctx, include, exclude, cluster):
         stack_config = get_parsed_stack_config(ctx.stack)
         # TODO: syntax check the input here
         pods_in_scope = stack_config['pods']
+        cluster_config = stack_config['config'] if 'config' in stack_config else None
     else:
         pods_in_scope = all_pods
+        cluster_config = None
 
     # Convert all pod definitions to v1.1 format
     pods_in_scope = _convert_to_new_format(pods_in_scope)
@@ -218,15 +225,16 @@ def _make_cluster_context(ctx, include, exclude, cluster):
     if ctx.verbose:
         print(f"files: {compose_files}")
 
-    return cluster_context(cluster, compose_files, pre_start_commands, post_start_commands)
+    return cluster_context(cluster, compose_files, pre_start_commands, post_start_commands, cluster_config)
 
 
 class cluster_context:
-    def __init__(self, cluster, compose_files, pre_start_commands, post_start_commands) -> None:
+    def __init__(self, cluster, compose_files, pre_start_commands, post_start_commands, config) -> None:
         self.cluster = cluster
         self.compose_files = compose_files
         self.pre_start_commands = pre_start_commands
         self.post_start_commands = post_start_commands
+        self.config = config
 
 
 def _convert_to_new_format(old_pod_array):
@@ -257,3 +265,45 @@ def _run_command(ctx, cluster_name, command):
     if command_result.returncode != 0:
         print(f"FATAL Error running command: {command}")
         sys.exit(1)
+
+
+def _orchestrate_cluster_config(ctx, cluster_config, docker, container_exec_env):
+
+    @dataclass
+    class ConfigDirective:
+        source_container: str
+        source_variable: str
+        destination_container: str
+        destination_variable: str
+
+    if cluster_config is not None:
+        for container in cluster_config:
+            container_config = cluster_config[container]
+            if ctx.verbose:
+                print(f"{container} config: {container_config}")
+            for directive in container_config:
+                pd = ConfigDirective(
+                    container_config[directive].split(".")[0],
+                    container_config[directive].split(".")[1],
+                    container,
+                    directive
+                )
+                if ctx.verbose:
+                    print(f"Setting {pd.destination_container}.{pd.destination_variable} = {pd.source_container}.{pd.source_variable}")
+                # TODO: fix the script paths so they're consistent between containers
+                source_value = docker.compose.execute(pd.source_container,
+                                                      ["sh", "-c",
+                                                          f"sh /docker-entrypoint-scripts.d/export-{pd.source_variable}.sh"],
+                                                      tty=False,
+                                                      envs=container_exec_env)
+                # TODO: handle the case that the value is not yet available
+                if ctx.debug:
+                    print(f"fetched source value: {source_value}")
+                destination_output = docker.compose.execute(pd.destination_container,
+                                                            ["sh", "-c",
+                                                             f"sh /scripts/import-{pd.destination_variable}.sh {pd.source_variable}"],
+                                                            tty=False,
+                                                            envs=container_exec_env)
+                if ctx.debug:
+                    print(f"destination output: {destination_output}")
+                # TODO: detect errors here

tests/smoke-test/run-smoke-test.sh

@@ -24,4 +24,11 @@ $TEST_TARGET_SO --stack test deploy-system up
 # TODO: test that we can use the deployed container somehow
 # Clean up
 $TEST_TARGET_SO --stack test deploy-system down
+# Run same test but not using the stack definition
+# Test building the a stack container
+$TEST_TARGET_SO build-containers --include cerc/test-container
+# Deploy the test container
+$TEST_TARGET_SO deploy-system --include test up
+# Clean up
+$TEST_TARGET_SO deploy-system --include test down
 echo "Test passed"