Compare commits

..

14 Commits

Author SHA1 Message Date
75bed6f10b fix external stack path resolution for deployments
All checks were successful
Lint Checks / Run linter (pull_request) Successful in 33s
Webapp Test / Run webapp test suite (pull_request) Successful in 4m37s
Smoke Test / Run basic test suite (pull_request) Successful in 4m23s
Deploy Test / Run deploy test suite (pull_request) Successful in 5m2s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 8m0s
2024-07-04 11:17:08 +08:00
8dfca4b227 fix stack path resolution
This was broken for "build-containers" with external stacks. Now setup and build use the same functions.
2024-07-03 19:42:40 +08:00
8fb0e4a328 lint fix 2024-07-03 19:42:40 +08:00
426357bbdb fall back to SHA if not on branch or tag 2024-07-03 19:42:40 +08:00
e0a9046685 support version in source/dist install 2024-07-03 19:42:40 +08:00
9b27ba1619 fix deployment ports command 2024-07-03 19:42:40 +08:00
45704f8bff fix extra config detection 2024-07-03 19:42:40 +08:00
df776c1b4c Install git for apps that want to get their commit sha (#854)
Some checks failed
Lint Checks / Run linter (push) Successful in 35s
Publish / Build and publish (push) Successful in 1m9s
Deploy Test / Run deploy test suite (push) Successful in 4m40s
Webapp Test / Run webapp test suite (push) Successful in 4m13s
Smoke Test / Run basic test suite (push) Successful in 3m51s
Fixturenet-Laconicd-Test / Run Laconicd fixturenet and Laconic CLI tests (push) Successful in 13m15s
K8s Deploy Test / Run deploy test suite on kind/k8s (push) Successful in 7m10s
Fixturenet-Eth-Plugeth-Test / Run an Ethereum plugeth fixturenet test (push) Failing after 3h10m0s
Fixturenet-Eth-Plugeth-Arm-Test / Run an Ethereum plugeth fixturenet test (push) Failing after 3h9m59s
Database Test / Run database hosting test on kind/k8s (push) Successful in 9m16s
Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Successful in 3m33s
External Stack Test / Run external stack test suite (push) Successful in 4m31s
Reviewed-on: #854
Reviewed-by: Thomas E Lackey <telackey@noreply.git.vdb.to>
Co-authored-by: David Boreham <dboreham@noreply.git.vdb.to>
Co-committed-by: David Boreham <dboreham@noreply.git.vdb.to>
2024-06-25 05:03:49 +00:00
48a3e79e6a Merge pull request 'Fix argument errors in command code' (#853) from dboreham/fix-laconic-mainnet-command into main
All checks were successful
Lint Checks / Run linter (push) Successful in 34s
Publish / Build and publish (push) Successful in 1m20s
Webapp Test / Run webapp test suite (push) Successful in 4m31s
Smoke Test / Run basic test suite (push) Successful in 3m54s
Deploy Test / Run deploy test suite (push) Successful in 4m58s
Reviewed-on: #853
2024-06-24 20:21:08 +00:00
0eaa5b8f09 Fix argument errors in command code
All checks were successful
Lint Checks / Run linter (pull_request) Successful in 35s
Webapp Test / Run webapp test suite (pull_request) Successful in 4m24s
Smoke Test / Run basic test suite (pull_request) Successful in 4m0s
Deploy Test / Run deploy test suite (pull_request) Successful in 4m54s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 7m38s
2024-06-24 14:15:46 -06:00
8980ac2581 Merge pull request 'Fixes for current SO objects' (#852) from dboreham/fix-mainnet-laconic into main
All checks were successful
Lint Checks / Run linter (push) Successful in 41s
Publish / Build and publish (push) Successful in 1m18s
Webapp Test / Run webapp test suite (push) Successful in 4m17s
Smoke Test / Run basic test suite (push) Successful in 3m48s
Deploy Test / Run deploy test suite (push) Successful in 4m44s
Reviewed-on: #852
2024-06-24 19:54:04 +00:00
fd15252c3f Fixes for current SO objects
All checks were successful
Lint Checks / Run linter (pull_request) Successful in 33s
Webapp Test / Run webapp test suite (pull_request) Successful in 4m23s
Deploy Test / Run deploy test suite (pull_request) Successful in 4m56s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 7m27s
Smoke Test / Run basic test suite (pull_request) Successful in 3m56s
2024-06-24 13:41:15 -06:00
b4e82ebc19 Merge pull request 'Fix mainnet laconic deploy setup' (#850) from dboreham/fix-mainnet-laconic-init into main
Some checks failed
Lint Checks / Run linter (push) Successful in 48s
Publish / Build and publish (push) Successful in 1m15s
Webapp Test / Run webapp test suite (push) Successful in 4m19s
Smoke Test / Run basic test suite (push) Successful in 3m52s
Deploy Test / Run deploy test suite (push) Successful in 4m47s
Fixturenet-Laconicd-Test / Run Laconicd fixturenet and Laconic CLI tests (push) Successful in 13m19s
Fixturenet-Eth-Plugeth-Test / Run an Ethereum plugeth fixturenet test (push) Failing after 56m43s
K8s Deploy Test / Run deploy test suite on kind/k8s (push) Successful in 7m23s
Fixturenet-Eth-Plugeth-Arm-Test / Run an Ethereum plugeth fixturenet test (push) Failing after 1h12m31s
Database Test / Run database hosting test on kind/k8s (push) Successful in 10m46s
Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Successful in 3m36s
External Stack Test / Run external stack test suite (push) Successful in 4m30s
Reviewed-on: #850
2024-06-22 01:25:07 +00:00
2364924a59 Fix mainnet laconic deploy setup
All checks were successful
Lint Checks / Run linter (pull_request) Successful in 31s
Deploy Test / Run deploy test suite (pull_request) Successful in 5m6s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 8m9s
Webapp Test / Run webapp test suite (pull_request) Successful in 4m33s
Smoke Test / Run basic test suite (pull_request) Successful in 4m9s
2024-06-21 19:24:33 -06:00
7 changed files with 54 additions and 47 deletions

View File

@ -32,7 +32,7 @@ RUN \
# [Optional] Uncomment this section to install additional OS packages.
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
&& apt-get -y install --no-install-recommends jq gettext-base
&& apt-get -y install --no-install-recommends jq gettext-base git
# [Optional] Uncomment if you want to install an additional version of node using nvm
# ARG EXTRA_NODE_VERSION=10

View File

@ -14,10 +14,11 @@
# along with this program. If not, see <http:#www.gnu.org/licenses/>.
from stack_orchestrator.util import get_yaml
from stack_orchestrator.deploy.deploy_types import DeployCommandContext, LaconicStackSetupCommand, DeploymentContext
from stack_orchestrator.deploy.deploy_types import DeployCommandContext, LaconicStackSetupCommand
from stack_orchestrator.deploy.deployment_context import DeploymentContext
from stack_orchestrator.deploy.stack_state import State
from stack_orchestrator.deploy.deploy_util import VolumeMapping, run_container_command
from stack_orchestrator.command_types import CommandOptions
from stack_orchestrator.opts import opts
from enum import Enum
from pathlib import Path
from shutil import copyfile, copytree
@ -61,7 +62,7 @@ def _get_node_moniker_from_config(network_dir: Path):
return moniker
def _get_node_key_from_gentx(options: CommandOptions, gentx_file_name: str):
def _get_node_key_from_gentx(gentx_file_name: str):
gentx_file_path = Path(gentx_file_name)
if gentx_file_path.exists():
with open(Path(gentx_file_name), "rb") as f:
@ -76,24 +77,24 @@ def _comma_delimited_to_list(list_str: str):
return list_str.split(",") if list_str else []
def _get_node_keys_from_gentx_files(options: CommandOptions, gentx_file_list: str):
def _get_node_keys_from_gentx_files(gentx_file_list: str):
node_keys = []
gentx_files = _comma_delimited_to_list(gentx_file_list)
for gentx_file in gentx_files:
node_key = _get_node_key_from_gentx(options, gentx_file)
node_key = _get_node_key_from_gentx(gentx_file)
if node_key:
node_keys.append(node_key)
return node_keys
def _copy_gentx_files(options: CommandOptions, network_dir: Path, gentx_file_list: str):
def _copy_gentx_files(network_dir: Path, gentx_file_list: str):
gentx_files = _comma_delimited_to_list(gentx_file_list)
for gentx_file in gentx_files:
gentx_file_path = Path(gentx_file)
copyfile(gentx_file_path, os.path.join(network_dir, "config", "gentx", os.path.basename(gentx_file_path)))
def _remove_persistent_peers(options: CommandOptions, network_dir: Path):
def _remove_persistent_peers(network_dir: Path):
config_file_path = _config_toml_path(network_dir)
if not config_file_path.exists():
print("Error: config.toml not found")
@ -107,7 +108,7 @@ def _remove_persistent_peers(options: CommandOptions, network_dir: Path):
output_file.write(config_file_content)
def _insert_persistent_peers(options: CommandOptions, config_dir: Path, new_persistent_peers: str):
def _insert_persistent_peers(config_dir: Path, new_persistent_peers: str):
config_file_path = config_dir.joinpath("config.toml")
if not config_file_path.exists():
print("Error: config.toml not found")
@ -150,7 +151,7 @@ def _phase_from_params(parameters):
def setup(command_context: DeployCommandContext, parameters: LaconicStackSetupCommand, extra_args):
options = command_context.cluster_context.options
options = opts.o
currency = "stake" # Does this need to be a parameter?
@ -237,7 +238,7 @@ def setup(command_context: DeployCommandContext, parameters: LaconicStackSetupCo
print("Error: --gentx-files must be supplied")
sys.exit(1)
# First look in the supplied gentx files for the other nodes' keys
other_node_keys = _get_node_keys_from_gentx_files(options, parameters.gentx_file_list)
other_node_keys = _get_node_keys_from_gentx_files(parameters.gentx_file_list)
# Add those keys to our genesis, with balances we determine here (why?)
for other_node_key in other_node_keys:
outputk, statusk = run_container_command(
@ -246,7 +247,7 @@ def setup(command_context: DeployCommandContext, parameters: LaconicStackSetupCo
if options.debug:
print(f"Command output: {outputk}")
# Copy the gentx json files into our network dir
_copy_gentx_files(options, network_dir, parameters.gentx_file_list)
_copy_gentx_files(network_dir, parameters.gentx_file_list)
# Now we can run collect-gentxs
output1, status1 = run_container_command(
command_context, "laconicd", f"laconicd collect-gentxs --home {laconicd_home_path_in_container}", mounts)
@ -255,7 +256,7 @@ def setup(command_context: DeployCommandContext, parameters: LaconicStackSetupCo
print(f"Generated genesis file, please copy to other nodes as required: \
{os.path.join(network_dir, 'config', 'genesis.json')}")
# Last thing, collect-gentxs puts a likely bogus set of persistent_peers in config.toml so we remove that now
_remove_persistent_peers(options, network_dir)
_remove_persistent_peers(network_dir)
# In both cases we validate the genesis file now
output2, status1 = run_container_command(
command_context, "laconicd", f"laconicd validate-genesis --home {laconicd_home_path_in_container}", mounts)
@ -266,7 +267,7 @@ def setup(command_context: DeployCommandContext, parameters: LaconicStackSetupCo
sys.exit(1)
def create(context: DeploymentContext, extra_args):
def create(deployment_context: DeploymentContext, extra_args):
network_dir = extra_args[0]
if network_dir is None:
print("Error: --network-dir must be supplied")
@ -285,15 +286,15 @@ def create(context: DeploymentContext, extra_args):
sys.exit(1)
# Copy the network directory contents into our deployment
# TODO: change this to work with non local paths
deployment_config_dir = context.deployment_dir.joinpath("data", "laconicd-config")
deployment_config_dir = deployment_context.deployment_dir.joinpath("data", "laconicd-config")
copytree(config_dir_path, deployment_config_dir, dirs_exist_ok=True)
# If supplied, add the initial persistent peers to the config file
if extra_args[1]:
initial_persistent_peers = extra_args[1]
_insert_persistent_peers(context.command_context.cluster_context.options, deployment_config_dir, initial_persistent_peers)
_insert_persistent_peers(deployment_config_dir, initial_persistent_peers)
# Copy the data directory contents into our deployment
# TODO: change this to work with non local paths
deployment_data_dir = context.deployment_dir.joinpath("data", "laconicd-data")
deployment_data_dir = deployment_context.deployment_dir.joinpath("data", "laconicd-data")
copytree(data_dir_path, deployment_data_dir, dirs_exist_ok=True)

View File

@ -26,8 +26,15 @@ import click
from pathlib import Path
from stack_orchestrator import constants
from stack_orchestrator.opts import opts
from stack_orchestrator.util import include_exclude_check, get_parsed_stack_config, global_options2, get_dev_root_path
from stack_orchestrator.util import resolve_compose_file
from stack_orchestrator.util import (
get_stack_path,
include_exclude_check,
get_parsed_stack_config,
global_options2,
get_dev_root_path,
stack_is_in_deployment,
resolve_compose_file,
)
from stack_orchestrator.deploy.deployer import Deployer, DeployerException
from stack_orchestrator.deploy.deployer_factory import getDeployer
from stack_orchestrator.deploy.deploy_types import ClusterContext, DeployCommandContext
@ -60,6 +67,7 @@ def command(ctx, include, exclude, env_file, cluster, deploy_to):
if deploy_to is None:
deploy_to = "compose"
stack = get_stack_path(stack)
ctx.obj = create_deploy_context(global_options2(ctx), None, stack, include, exclude, cluster, env_file, deploy_to)
# Subcommand is executed now, by the magic of click
@ -77,6 +85,7 @@ def create_deploy_context(
if deployment_context and cluster is None:
cluster = deployment_context.get_cluster_id()
cluster_context = _make_cluster_context(global_context, stack, include, exclude, cluster, env_file)
# breakpoint()
deployer = getDeployer(deploy_to, deployment_context, compose_files=cluster_context.compose_files,
compose_project_name=cluster_context.cluster,
compose_env_file=cluster_context.env_file)
@ -274,16 +283,12 @@ def _make_default_cluster_name(deployment, compose_dir, stack, include, exclude)
# stack has to be either PathLike pointing to a stack yml file, or a string with the name of a known stack
def _make_cluster_context(ctx, stack, include, exclude, cluster, env_file):
dev_root_path = get_dev_root_path(ctx)
# TODO: huge hack, fix this
# If the caller passed a path for the stack file, then we know that we can get the compose files
# from the same directory
deployment = False
if isinstance(stack, os.PathLike):
compose_dir = stack.parent.joinpath("compose")
deployment = True
# TODO: hack, this should be encapsulated by the deployment context.
deployment = stack_is_in_deployment(stack)
if deployment:
compose_dir = stack.joinpath("compose")
else:
# See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
compose_dir = Path(__file__).absolute().parent.parent.joinpath("data", "compose")

View File

@ -50,15 +50,15 @@ def command(ctx, dir):
def make_deploy_context(ctx) -> DeployCommandContext:
context: DeploymentContext = ctx.obj
stack_file_path = context.get_stack_file()
env_file = context.get_env_file()
cluster_name = context.get_cluster_id()
if constants.deploy_to_key in context.spec.obj:
deployment_type = context.spec.obj[constants.deploy_to_key]
else:
deployment_type = constants.compose_deploy_type
return create_deploy_context(ctx.parent.parent.obj, context, stack_file_path, None, None, cluster_name, env_file,
deployment_type)
stack = context.deployment_dir
return create_deploy_context(ctx.parent.parent.obj, context, stack, None, None,
cluster_name, env_file, deployment_type)
@command.command()

View File

@ -24,7 +24,7 @@ from secrets import token_hex
import sys
from stack_orchestrator import constants
from stack_orchestrator.opts import opts
from stack_orchestrator.util import (get_internal_stack_file_path, get_parsed_deployment_spec, get_parsed_stack_config,
from stack_orchestrator.util import (get_stack_path, get_parsed_deployment_spec, get_parsed_stack_config,
global_options, get_yaml, get_pod_list, get_pod_file_path, pod_has_scripts,
get_pod_script_paths, get_plugin_code_paths, error_exit, env_var_map_from_file,
resolve_config_dir)
@ -459,7 +459,7 @@ def create_operation(deployment_command_context, spec_file, deployment_dir, netw
_check_volume_definitions(parsed_spec)
stack_name = parsed_spec["stack"]
deployment_type = parsed_spec[constants.deploy_to_key]
stack_file = get_internal_stack_file_path(stack_name)
stack_file = get_stack_path(stack_name).joinpath(constants.stack_file_name)
parsed_stack = get_parsed_stack_config(stack_name)
if opts.o.debug:
print(f"parsed spec: {parsed_spec}")
@ -472,7 +472,7 @@ def create_operation(deployment_command_context, spec_file, deployment_dir, netw
os.mkdir(deployment_dir_path)
# Copy spec file and the stack file into the deployment dir
copyfile(spec_file, deployment_dir_path.joinpath(constants.spec_file_name))
copyfile(stack_file, deployment_dir_path.joinpath(os.path.basename(stack_file)))
copyfile(stack_file, deployment_dir_path.joinpath(constants.stack_file_name))
_create_deployment_file(deployment_dir_path)
# Copy any config varibles from the spec file into an env file suitable for compose
_write_config_file(spec_file, deployment_dir_path.joinpath(constants.config_file_name))

View File

@ -24,9 +24,6 @@ from git.exc import GitCommandError
from tqdm import tqdm
import click
import importlib.resources
from pathlib import Path
import yaml
from stack_orchestrator.constants import stack_file_name
from stack_orchestrator.opts import opts
from stack_orchestrator.util import get_parsed_stack_config, include_exclude_check, error_exit, warn_exit

View File

@ -20,7 +20,7 @@ import ruamel.yaml
from pathlib import Path
from dotenv import dotenv_values
from typing import Mapping, Set, List
from stack_orchestrator.constants import stack_file_name
from stack_orchestrator.constants import stack_file_name, deployment_file_name
def include_exclude_check(s, include, exclude):
@ -34,11 +34,14 @@ def include_exclude_check(s, include, exclude):
return s not in exclude_list
def get_internal_stack_file_path(stack):
# In order to be compatible with Python 3.8 we need to use this hack to get the path:
# See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
stack_file_path = Path(__file__).absolute().parent.joinpath("data", "stacks", stack, stack_file_name)
return stack_file_path
def get_stack_path(stack):
if stack_is_external(stack):
stack_path = Path(stack)
else:
# In order to be compatible with Python 3.8 we need to use this hack to get the path:
# See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
stack_path = Path(__file__).absolute().parent.joinpath("data", "stacks", stack)
return stack_path
def get_dev_root_path(ctx):
@ -53,10 +56,7 @@ def get_dev_root_path(ctx):
# Caller can pass either the name of a stack, or a path to a stack file
def get_parsed_stack_config(stack):
if stack_is_external(stack):
stack_file_path = Path(stack).joinpath(stack_file_name)
else:
stack_file_path = get_internal_stack_file_path(stack)
stack_file_path = get_stack_path(stack).joinpath(stack_file_name)
if stack_file_path.exists():
return get_yaml().load(open(stack_file_path, "r"))
# We try here to generate a useful diagnostic error
@ -84,7 +84,7 @@ def get_plugin_code_paths(stack) -> List[Path]:
result: Set[Path] = set()
for pod in pods:
if type(pod) is str:
result.add(get_internal_stack_file_path(stack).parent)
result.add(get_stack_path(stack))
else:
pod_root_dir = os.path.join(get_dev_root_path(None), pod["repository"].split("/")[-1], pod["path"])
result.add(Path(os.path.join(pod_root_dir, "stack")))
@ -196,6 +196,10 @@ def stack_is_external(stack: str):
return Path(stack).exists() if stack is not None else False
def stack_is_in_deployment(stack: Path):
return stack.joinpath(deployment_file_name).exists()
def get_yaml():
# See: https://stackoverflow.com/a/45701840/1701505
yaml = ruamel.yaml.YAML()