Merge pull request #50 from cerc-io/dboreham/fix-shiv-package

Fix shiv package

README.md

@@ -3,29 +3,18 @@
 Stack Orchestrator allows building and deployment of a Laconic stack on a single machine with minimial prerequisites.
 
 ## Setup
-### Developer Mode
-Developer mode runs the orchestrator from a cloned git repository.
+### User Mode
+User mode runs the orchestrator from a "binary" single-file release and does not require special Python environment setup. Use this mode unless you plan to make changes to the orchestrator source code.
 #### Prerequisites
 Stack Orchestrator is a Python3 CLI tool that runs on any OS with Python3 and Docker. Tested on: Ubuntu 20/22.
 
 Ensure that the following are already installed:
 
-1. Python3 (the version 3.8 available in Ubuntu 20/22 works)
+1. Python3 (the stock Python3 version available in Ubuntu 20 and 22 is suitable)
    ```
    $ python3 --version
    Python 3.8.10
    ```
-1. Python venv package
-   This may or may not be already installed depending on the host OS and version. Check by running:
-   ```
-   $ python3 -m venv
-   usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear] [--upgrade] [--without-pip] [--prompt PROMPT] ENV_DIR [ENV_DIR ...]
-   venv: error: the following arguments are required: ENV_DIR
-   ```
-   If the venv package is missing you should see a message indicating how to install it, for example with:
-   ```
-   $ apt install python3.8-venv
-   ```
 2. Docker (Install a current version from dockerco, don't use the version from any Linux distro)
    ```
    $ docker --version
@@ -42,6 +31,47 @@ Ensure that the following are already installed:
    # or to install for all users.
    ```
 #### Install
+1. Download the latest release from [this page](https://github.com/cerc-io/stack-orchestrator/tags), into a suitable directory (e.g. `~/bin`):
+   ```
+   $ cd ~/bin
+   $ curl https://github.com/cerc-io/stack-orchestrator/releases/download/v1.0.1-alpha/laconic-so
+   ```
+1. Ensure `laconic-so` is on the `PATH`
+1. Verify operation:
+   ```
+   $ ~/bin/laconic-so --help
+   Usage: python -m laconic-so [OPTIONS] COMMAND [ARGS]...
+
+     Laconic Stack Orchestrator
+
+   Options:
+     --quiet
+     --verbose
+     --dry-run
+     --local-stack
+     -h, --help     Show this message and exit.
+
+   Commands:
+     build-containers    build the set of containers required for a complete...
+     deploy-system       deploy a stack
+     setup-repositories  git clone the set of repositories required to build...
+   ```
+### Developer mode
+Suitable for developers either modifying or debugging the orchestrator Python code:
+#### Prerequisites
+In addition to the binary install prerequisites listed above, the following are required:
+1. Python venv package
+   This may or may not be already installed depending on the host OS and version. Check by running:
+   ```
+   $ python3 -m venv
+   usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear] [--upgrade] [--without-pip] [--prompt PROMPT] ENV_DIR [ENV_DIR ...]
+   venv: error: the following arguments are required: ENV_DIR
+   ```
+   If the venv package is missing you should see a message indicating how to install it, for example with:
+   ```
+   $ apt install python3.8-venv
+   ```
+#### Install
 1. Clone this repository:
    ```
    $ git clone (https://github.com/cerc-io/stack-orchestrator.git
@@ -78,11 +108,13 @@ Ensure that the following are already installed:
     deploy-system       deploy a stack
     setup-repositories  git clone the set of repositories required to build...
    ```
+
 #### Build a zipapp (single file distributable script)
 Use shiv to build a single file Python executable zip archive of laconic-so:
 1. Install [shiv](https://github.com/linkedin/shiv):
    ```
    $ (venv) pip install shiv
+   $ (venv) pip install wheel
    ```
 1. Run shiv to create a zipapp file:
    ```

app/build_containers.py

@@ -25,6 +25,7 @@ import sys
 from decouple import config
 import subprocess
 import click
+import pkg_resources
 from .util import include_exclude_check
 
 # TODO: find a place for this
@@ -55,8 +56,8 @@ def command(ctx, include, exclude):
     if not os.path.isdir(dev_root_path):
         print('Dev root directory doesn\'t exist, creating')
 
-    with open("container-image-list.txt") as container_list_file:
-        containers = container_list_file.read().splitlines()
+    with pkg_resources.resource_stream(__name__, "data/container-image-list.txt") as container_list_file:
+        containers = container_list_file.read().decode().splitlines()
 
     if verbose:
         print(f'Containers: {containers}')

app/deploy_system.py

@@ -20,6 +20,7 @@ import os
 import sys
 from python_on_whales import DockerClient
 import click
+import pkg_resources
 from .util import include_exclude_check
 
 
@@ -48,11 +49,11 @@ def command(ctx, include, exclude, cluster, command, services):
         if verbose:
             print(f"Using cluster name: {cluster}")
 
-    with open("pod-list.txt") as pod_list_file:
-        pods = pod_list_file.read().splitlines()
+    with pkg_resources.resource_stream(__name__, "data/pod-list.txt") as pod_list_file:
+        pods = pod_list_file.read().decode().splitlines()
 
     if verbose:
-        print(f'Pods: {pods}')
+        print(f"Pods: {pods}")
 
     # Construct a docker compose command suitable for our purpose
 

app/setup_repositories.py

@@ -22,6 +22,8 @@ from decouple import config
 import git
 from tqdm import tqdm
 import click
+import pkg_resources
+from .util import include_exclude_check
 
 
 class GitProgress(git.RemoteProgress):
@@ -49,11 +51,13 @@ def is_git_repo(path):
 
 
 @click.command()
+@click.option("--include", help="only clone these repositories")
+@click.option("--exclude", help="don\'t clone these repositories")
 @click.option('--check-only', is_flag=True, default=False)
 @click.option('--pull', is_flag=True, default=False)
 @click.option('--branches-file', help="checkout branches specified in this file")
 @click.pass_context
-def command(ctx, check_only, pull, branches_file):
+def command(ctx, include, exclude, check_only, pull, branches_file):
     '''git clone the set of repositories required to build the complete system from source'''
 
     quiet = ctx.obj.quiet
@@ -74,42 +78,50 @@ def command(ctx, check_only, pull, branches_file):
 
     if local_stack:
         dev_root_path = os.getcwd()[0:os.getcwd().rindex("stack-orchestrator")]
-        print(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
+        print(f"Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}")
     else:
         dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc"))
 
     if not quiet:
-        print(f'Dev Root is: {dev_root_path}')
+        print(f"Dev Root is: {dev_root_path}")
 
     if not os.path.isdir(dev_root_path):
         if not quiet:
             print('Dev root directory doesn\'t exist, creating')
         os.makedirs(dev_root_path)
 
-    with open("repository-list.txt") as repository_list_file:
-        repos = repository_list_file.read().splitlines()
+    with pkg_resources.resource_stream(__name__, "data/repository-list.txt") as repository_list_file: 
+        all_repos = repository_list_file.read().decode().splitlines()
 
     if verbose:
-        print(f'Repos: {repos}')
+        print(f"Repos: {all_repos}")
+
+    repos = []
+    for repo in all_repos:
+        if include_exclude_check(repo, include, exclude):
+            repos.append(repo)
+        else:
+            if not quiet:
+                print(f"Excluding: {repo}")
 
     def process_repo(repo):
-        full_github_repo_path = f'git@github.com:{repo}'
+        full_github_repo_path = f"git@github.com:{repo}"
         repoName = repo.split("/")[-1]
         full_filesystem_repo_path = os.path.join(dev_root_path, repoName)
         is_present = os.path.isdir(full_filesystem_repo_path)
         if not quiet:
-            present_text = f'already exists active branch: {git.Repo(full_filesystem_repo_path).active_branch}' if is_present \
+            present_text = f"already exists active branch: {git.Repo(full_filesystem_repo_path).active_branch}" if is_present \
                 else 'Needs to be fetched'
-            print(f'Checking: {full_filesystem_repo_path}: {present_text}')
+            print(f"Checking: {full_filesystem_repo_path}: {present_text}")
         # Quick check that it's actually a repo
         if is_present:
             if not is_git_repo(full_filesystem_repo_path):
-                print(f'Error: {full_filesystem_repo_path} does not contain a valid git repository')
+                print(f"Error: {full_filesystem_repo_path} does not contain a valid git repository")
                 sys.exit(1)
             else:
                 if pull:
                     if verbose:
-                        print(f'Running git pull for {full_filesystem_repo_path}')
+                        print(f"Running git pull for {full_filesystem_repo_path}")
                     if not check_only:
                         git_repo = git.Repo(full_filesystem_repo_path)
                         origin = git_repo.remotes.origin

setup.py

@@ -18,6 +18,8 @@ setup(
     packages=find_packages(),
     install_requires=[requirements],
     python_requires='>=3.7',
+    include_package_data=True,
+    package_data={'': ['data/*.txt']},
     classifiers=[
         "Programming Language :: Python :: 3.8",
         "Operating System :: OS Independent",

stacks/laconicd/README.md

@@ -0,0 +1,19 @@
+# Laconicd Fixturenet
+
+Instructions for deploying a local Laconic blockchain "fixturenet" for development and testing purposes using laconic-stack-orchestrator (the installation of which is covered [here](https://github.com/cerc-io/stack-orchestrator#user-mode)):
+
+## Clone required repositories
+```
+$ laconic-so setup-repositories --include cerc-io/laconicd,cerc-io/laconic-sdk,cerc-io/laconic-cns-cli
+```
+## Build the laconicd container
+```
+$ laconic-sh build-containers --include cerc/laconicd
+```
+This should create a container with tag `cerc/watcher-mobymask` in the local image registry.
+## Deploy the stack
+First the watcher database has to be initialized. Start only the watcher-db service:
+```
+$ laconic-so deploy-system --include fixturenet-laconicd
+```
+Correct operation should be verified by checking the laconicd container's log.

stacks/laconicd/stack.yml

@@ -0,0 +1,11 @@
+version: "1.0"
+name: laconicd-fixturenet
+repos:
+  - cerc-io/laconicd
+  - cerc-io/laconic-sdk
+  - cerc-io/laconic-cns-cli
+containers:
+  - cerc/laconicd
+  - cerc/laconic-cns-cli
+pods:
+  - fixturenet-laconicd

stacks/mobymask/README.md

@@ -3,12 +3,12 @@
 The MobyMask watcher is a Laconic Network component that provides efficient access to MobyMask contract data from Ethereum, along with evidence allowing users to verify the correctness of that data. The watcher source code is available in [this repository](https://github.com/cerc-io/watcher-ts/tree/main/packages/mobymask-watcher) and a developer-oriented Docker Compose setup for the watcher can be found [here](https://github.com/cerc-io/mobymask-watcher). The watcher can be deployed automatically using the Laconic Stack Orchestrator tool as detailed below:
 
 ## Deploy the MobyMask Watcher
-The instructions below show how to deploy a MobyMask watcher using laconic-stack-orchestrator (the installation of which is covered [here](https://github.com/cerc-io/stack-orchestrator#install)).
+The instructions below show how to deploy a MobyMask watcher using laconic-stack-orchestrator (the installation of which is covered [here](https://github.com/cerc-io/stack-orchestrator#user-mode)).
 
 This deployment expects that ipld-eth-server's endpoints are available on the local machine at http://ipld-eth-server.example.com:8083/graphql and http://ipld-eth-server.example.com:8082. More advanced configurations are supported by modifying the watcher's [config file](../../config/watcher-mobymask/mobymask-watcher.toml).
 ## Clone required repositories
 ```
-$ laconic-so setup-repositories
+$ laconic-so setup-repositories --include vulcanize/assemblyscript,cerc-io/watcher-ts
 ```
 Checkout required branches for the current release:
 ```