deep-stack/stack-orchestrator
3
0
Fork 0
forked from cerc-io/stack-orchestrator
Code Pull Requests Activity

More logging for webapp deployment (#923)

Browse Source 
```
cerc-webapp-deployer: ############ DEPLOY #############
cerc-webapp-deployer: 2024-08-15 02:13:08.321991 -  - 0:00:00.000031 (step): Discovering deployment requests...
cerc-webapp-deployer: laconic -c /etc/config/laconic.yml registry record list --all --type ApplicationDeploymentRequest
cerc-webapp-deployer: 2024-08-15 02:13:08.815428 -  - 0:00:00.493420 (step): Loading known requests from /srv/deployments/autodeploy.state...
cerc-webapp-deployer: 2024-08-15 02:13:08.815626 -  - 0:00:00.000158 (step): BEGIN: Examining request bafyreigiltcdscwt7rqldnilo4ohrhgoulrlfceixde5ycewsym64sefgi
cerc-webapp-deployer: 2024-08-15 02:13:08.815645 -  - 0:00:00.000008 (step): Skipping request bafyreigiltcdscwt7rqldnilo4ohrhgoulrlfceixde5ycewsym64sefgi, we've already seen it.
cerc-webapp-deployer: 2024-08-15 02:13:08.815653 -  - 0:00:00.000005 (step): DONE Examining request bafyreigiltcdscwt7rqldnilo4ohrhgoulrlfceixde5ycewsym64sefgi with result SKIP.
cerc-webapp-deployer: 2024-08-15 02:13:08.815664 -  - 0:00:00.000005 (step): BEGIN: Examining request bafyreicoxippgdwab6cz72py4rgv63rvvbsea73y62hashlhqpcsxyfkue
cerc-webapp-deployer: 2024-08-15 02:13:08.815674 -  - 0:00:00.000006 (step): Skipping request bafyreicoxippgdwab6cz72py4rgv63rvvbsea73y62hashlhqpcsxyfkue, we've already seen it.
cerc-webapp-deployer: 2024-08-15 02:13:08.815684 -  - 0:00:00.000004 (step): DONE Examining request bafyreicoxippgdwab6cz72py4rgv63rvvbsea73y62hashlhqpcsxyfkue with result SKIP.
cerc-webapp-deployer: 2024-08-15 02:13:08.815692 -  - 0:00:00.000005 (step): BEGIN: Examining request bafyreih3gt44pvahnbg7ag26mlk3iie4s5m5znhygajja5dcovheti72ne
cerc-webapp-deployer: 2024-08-15 02:13:08.815705 -  - 0:00:00.000007 (step): Skipping request bafyreih3gt44pvahnbg7ag26mlk3iie4s5m5znhygajja5dcovheti72ne, we've already seen it.
cerc-webapp-deployer: 2024-08-15 02:13:08.815714 -  - 0:00:00.000005 (step): DONE Examining request bafyreih3gt44pvahnbg7ag26mlk3iie4s5m5znhygajja5dcovheti72ne with result SKIP.
cerc-webapp-deployer: 2024-08-15 02:13:08.815724 -  - 0:00:00.000004 (step): BEGIN: Examining request bafyreigjnbio47rug6x5tufzc6cwfcqpl3ck3xldzotrlz5bt663dh2pua
cerc-webapp-deployer: 2024-08-15 02:13:08.815733 -  - 0:00:00.000005 (step): Skipping request bafyreigjnbio47rug6x5tufzc6cwfcqpl3ck3xldzotrlz5bt663dh2pua, we've already seen it.
cerc-webapp-deployer: 2024-08-15 02:13:08.815743 -  - 0:00:00.000005 (step): DONE Examining request bafyreigjnbio47rug6x5tufzc6cwfcqpl3ck3xldzotrlz5bt663dh2pua with result SKIP.
cerc-webapp-deployer: 2024-08-15 02:13:08.815751 -  - 0:00:00.000004 (step): BEGIN: Examining request bafyreihsfno4s6lkxcp5a7g7pjj7kklrp3xaqo57mr2pz76nk3h4jukayy
cerc-webapp-deployer: 2024-08-15 02:13:08.815761 -  - 0:00:00.000006 (step): Skipping request bafyreihsfno4s6lkxcp5a7g7pjj7kklrp3xaqo57mr2pz76nk3h4jukayy, we've already seen it.
cerc-webapp-deployer: 2024-08-15 02:13:08.815770 -  - 0:00:00.000005 (step): DONE Examining request bafyreihsfno4s6lkxcp5a7g7pjj7kklrp3xaqo57mr2pz76nk3h4jukayy with result SKIP.
cerc-webapp-deployer: 2024-08-15 02:13:08.815779 -  - 0:00:00.000005 (step): BEGIN: Examining request bafyreicyfyj4ncmtuy5pain2rvc67v645cg2bbsiakizvhdiwvkx7asvdy
cerc-webapp-deployer: 2024-08-15 02:13:08.815791 -  - 0:00:00.000007 (step): Skipping request bafyreicyfyj4ncmtuy5pain2rvc67v645cg2bbsiakizvhdiwvkx7asvdy, we've already seen it.
cerc-webapp-deployer: 2024-08-15 02:13:08.815800 -  - 0:00:00.000004 (step): DONE Examining request bafyreicyfyj4ncmtuy5pain2rvc67v645cg2bbsiakizvhdiwvkx7asvdy with result SKIP.
cerc-webapp-deployer: 2024-08-15 02:13:08.815808 -  - 0:00:00.000004 (step): Discovering existing app deployments...
cerc-webapp-deployer: laconic -c /etc/config/laconic.yml registry record list --all --type ApplicationDeploymentRecord
cerc-webapp-deployer: 2024-08-15 02:13:09.330655 -  - 0:00:00.514858 (step): Discovering deployment removal and cancellation requests...
cerc-webapp-deployer: laconic -c /etc/config/laconic.yml registry record list --all --type ApplicationDeploymentRemovalRequest
cerc-webapp-deployer: 2024-08-15 02:13:09.825145 -  - 0:00:00.494460 (step): Found 0 unsatisfied request(s) to process.
cerc-webapp-deployer: ############ DEPLOY SUCCESS #############
```

Reviewed-on: cerc-io/stack-orchestrator#923
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
This commit is contained in:
Thomas E Lackey 2024-08-15 02:57:47 +00:00 committed by Thomas E Lackey
parent 952389abb0
commit 60d34217f8
1 changed files with 346 additions and 184 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

342
stack_orchestrator/deploy/webapp/deploy_webapp_from_registry.py
View File

@ -26,11 +26,19 @@ import click
from stack_orchestrator.deploy.images import remote_image_exists from stack_orchestrator.deploy.images import remote_image_exists
from stack_orchestrator.deploy.webapp import deploy_webapp from stack_orchestrator.deploy.webapp import deploy_webapp
from stack_orchestrator.deploy.webapp.util import (LaconicRegistryClient, TimedLogger, from stack_orchestrator.deploy.webapp.util import (
build_container_image, push_container_image, LaconicRegistryClient,
file_hash, deploy_to_k8s, publish_deployment, TimedLogger,
hostname_for_deployment_request, generate_hostname_for_app, build_container_image,
match_owner, skip_by_tag) push_container_image,
file_hash,
deploy_to_k8s,
publish_deployment,
hostname_for_deployment_request,
generate_hostname_for_app,
match_owner,
skip_by_tag,
)
def process_app_deployment_request( def process_app_deployment_request(
@ -46,12 +54,14 @@ def process_app_deployment_request(
force_rebuild, force_rebuild,
fqdn_policy, fqdn_policy,
recreate_on_deploy, recreate_on_deploy,
logger logger,
): ):
logger.log("BEGIN - process_app_deployment_request") logger.log("BEGIN - process_app_deployment_request")
# 1. look up application # 1. look up application
app = laconic.get_record(app_deployment_request.attributes.application, require=True) app = laconic.get_record(
app_deployment_request.attributes.application, require=True
)
logger.log(f"Retrieved app record {app_deployment_request.attributes.application}") logger.log(f"Retrieved app record {app_deployment_request.attributes.application}")
# 2. determine dns # 2. determine dns
@ -62,7 +72,9 @@ def process_app_deployment_request(
if "allow" == fqdn_policy or "preexisting" == fqdn_policy: if "allow" == fqdn_policy or "preexisting" == fqdn_policy:
fqdn = requested_name fqdn = requested_name
else: else:
raise Exception(f"{requested_name} is invalid: only unqualified hostnames are allowed.") raise Exception(
f"{requested_name} is invalid: only unqualified hostnames are allowed."
)
else: else:
fqdn = f"{requested_name}.{default_dns_suffix}" fqdn = f"{requested_name}.{default_dns_suffix}"
@ -72,21 +84,31 @@ def process_app_deployment_request(
if dns_record: if dns_record:
matched_owner = match_owner(app_deployment_request, dns_record) matched_owner = match_owner(app_deployment_request, dns_record)
if not matched_owner and dns_record.attributes.request: if not matched_owner and dns_record.attributes.request:
matched_owner = match_owner(app_deployment_request, laconic.get_record(dns_record.attributes.request, require=True)) matched_owner = match_owner(
app_deployment_request,
laconic.get_record(dns_record.attributes.request, require=True),
)
if matched_owner: if matched_owner:
logger.log(f"Matched DnsRecord ownership: {matched_owner}") logger.log(f"Matched DnsRecord ownership: {matched_owner}")
else: else:
raise Exception("Unable to confirm ownership of DnsRecord %s for request %s" % raise Exception(
(dns_lrn, app_deployment_request.id)) "Unable to confirm ownership of DnsRecord %s for request %s"
% (dns_lrn, app_deployment_request.id)
)
elif "preexisting" == fqdn_policy: elif "preexisting" == fqdn_policy:
raise Exception(f"No pre-existing DnsRecord {dns_lrn} could be found for request {app_deployment_request.id}.") raise Exception(
f"No pre-existing DnsRecord {dns_lrn} could be found for request {app_deployment_request.id}."
)
# 4. get build and runtime config from request # 4. get build and runtime config from request
env_filename = None env_filename = None
if app_deployment_request.attributes.config and "env" in app_deployment_request.attributes.config: if (
app_deployment_request.attributes.config
and "env" in app_deployment_request.attributes.config
):
env_filename = tempfile.mktemp() env_filename = tempfile.mktemp()
with open(env_filename, 'w') as file: with open(env_filename, "w") as file:
for k, v in app_deployment_request.attributes.config["env"].items(): for k, v in app_deployment_request.attributes.config["env"].items():
file.write("%s=%s\n" % (k, shlex.quote(str(v)))) file.write("%s=%s\n" % (k, shlex.quote(str(v))))
@ -96,7 +118,10 @@ def process_app_deployment_request(
if app_deployment_request.attributes.deployment: if app_deployment_request.attributes.deployment:
app_deployment_lrn = app_deployment_request.attributes.deployment app_deployment_lrn = app_deployment_request.attributes.deployment
if not app_deployment_lrn.startswith(deployment_record_namespace): if not app_deployment_lrn.startswith(deployment_record_namespace):
raise Exception("Deployment CRN %s is not in a supported namespace" % app_deployment_request.attributes.deployment) raise Exception(
"Deployment CRN %s is not in a supported namespace"
% app_deployment_request.attributes.deployment
)
deployment_record = laconic.get_record(app_deployment_lrn) deployment_record = laconic.get_record(app_deployment_lrn)
deployment_dir = os.path.join(deployment_parent_dir, fqdn) deployment_dir = os.path.join(deployment_parent_dir, fqdn)
@ -109,20 +134,37 @@ def process_app_deployment_request(
# b. check for deployment directory (create if necessary) # b. check for deployment directory (create if necessary)
if not os.path.exists(deployment_dir): if not os.path.exists(deployment_dir):
if deployment_record: if deployment_record:
raise Exception("Deployment record %s exists, but not deployment dir %s. Please remove name." % raise Exception(
(app_deployment_lrn, deployment_dir)) "Deployment record %s exists, but not deployment dir %s. Please remove name."
logger.log(f"Creating webapp deployment in: {deployment_dir} with container id: {deployment_container_tag}") % (app_deployment_lrn, deployment_dir)
deploy_webapp.create_deployment(ctx, deployment_dir, deployment_container_tag, )
f"https://{fqdn}", kube_config, image_registry, env_filename) logger.log(
f"Creating webapp deployment in: {deployment_dir} with container id: {deployment_container_tag}"
)
deploy_webapp.create_deployment(
ctx,
deployment_dir,
deployment_container_tag,
f"https://{fqdn}",
kube_config,
image_registry,
env_filename,
)
elif env_filename: elif env_filename:
shutil.copyfile(env_filename, deployment_config_file) shutil.copyfile(env_filename, deployment_config_file)
needs_k8s_deploy = False needs_k8s_deploy = False
if force_rebuild: if force_rebuild:
logger.log("--force-rebuild is enabled so the container will always be built now, even if nothing has changed in the app") logger.log(
"--force-rebuild is enabled so the container will always be built now, even if nothing has changed in the app"
)
# 6. build container (if needed) # 6. build container (if needed)
# TODO: add a comment that explains what this code is doing (not clear to me) # TODO: add a comment that explains what this code is doing (not clear to me)
if not deployment_record or deployment_record.attributes.application != app.id or force_rebuild: if (
not deployment_record
or deployment_record.attributes.application != app.id
or force_rebuild
):
needs_k8s_deploy = True needs_k8s_deploy = True
# check if the image already exists # check if the image already exists
shared_tag_exists = remote_image_exists(image_registry, app_image_shared_tag) shared_tag_exists = remote_image_exists(image_registry, app_image_shared_tag)
@ -143,7 +185,9 @@ def process_app_deployment_request(
else: else:
extra_build_args = [] # TODO: pull from request extra_build_args = [] # TODO: pull from request
logger.log(f"Building container image: {deployment_container_tag}") logger.log(f"Building container image: {deployment_container_tag}")
build_container_image(app, deployment_container_tag, extra_build_args, logger) build_container_image(
app, deployment_container_tag, extra_build_args, logger
)
logger.log("Build complete") logger.log("Build complete")
logger.log(f"Pushing container image: {deployment_container_tag}") logger.log(f"Pushing container image: {deployment_container_tag}")
push_container_image(deployment_dir, logger) push_container_image(deployment_dir, logger)
@ -158,17 +202,15 @@ def process_app_deployment_request(
logger.log("Requested app is already deployed, skipping build and image push") logger.log("Requested app is already deployed, skipping build and image push")
# 7. update config (if needed) # 7. update config (if needed)
if not deployment_record or file_hash(deployment_config_file) != deployment_record.attributes.meta.config: if (
not deployment_record
or file_hash(deployment_config_file) != deployment_record.attributes.meta.config
):
needs_k8s_deploy = True needs_k8s_deploy = True
# 8. update k8s deployment # 8. update k8s deployment
if needs_k8s_deploy: if needs_k8s_deploy:
deploy_to_k8s( deploy_to_k8s(deployment_record, deployment_dir, recreate_on_deploy, logger)
deployment_record,
deployment_dir,
recreate_on_deploy,
logger
)
logger.log("Publishing deployment to registry.") logger.log("Publishing deployment to registry.")
publish_deployment( publish_deployment(
@ -180,7 +222,7 @@ def process_app_deployment_request(
dns_lrn, dns_lrn,
deployment_dir, deployment_dir,
app_deployment_request, app_deployment_request,
logger logger,
) )
logger.log("Publication complete.") logger.log("Publication complete.")
logger.log("END - process_app_deployment_request") logger.log("END - process_app_deployment_request")
@ -197,38 +239,94 @@ def dump_known_requests(filename, requests, status="SEEN"):
return return
known_requests = load_known_requests(filename) known_requests = load_known_requests(filename)
for r in requests: for r in requests:
known_requests[r.id] = { known_requests[r.id] = {"createTime": r.createTime, "status": status}
"createTime": r.createTime,
"status": status
}
with open(filename, "w") as f: with open(filename, "w") as f:
json.dump(known_requests, f) json.dump(known_requests, f)
@click.command() @click.command()
@click.option("--kube-config", help="Provide a config file for a k8s deployment") @click.option("--kube-config", help="Provide a config file for a k8s deployment")
@click.option("--laconic-config", help="Provide a config file for laconicd", required=True) @click.option(
@click.option("--image-registry", help="Provide a container image registry url for this k8s cluster") "--laconic-config", help="Provide a config file for laconicd", required=True
@click.option("--deployment-parent-dir", help="Create deployment directories beneath this directory", required=True) )
@click.option(
"--image-registry",
help="Provide a container image registry url for this k8s cluster",
)
@click.option(
"--deployment-parent-dir",
help="Create deployment directories beneath this directory",
required=True,
)
@click.option("--request-id", help="The ApplicationDeploymentRequest to process") @click.option("--request-id", help="The ApplicationDeploymentRequest to process")
@click.option("--discover", help="Discover and process all pending ApplicationDeploymentRequests", is_flag=True, default=False) @click.option(
@click.option("--state-file", help="File to store state about previously seen requests.") "--discover",
@click.option("--only-update-state", help="Only update the state file, don't process any requests anything.", is_flag=True) help="Discover and process all pending ApplicationDeploymentRequests",
is_flag=True,
default=False,
)
@click.option(
"--state-file", help="File to store state about previously seen requests."
)
@click.option(
"--only-update-state",
help="Only update the state file, don't process any requests anything.",
is_flag=True,
)
@click.option("--dns-suffix", help="DNS domain to use eg, laconic.servesthe.world") @click.option("--dns-suffix", help="DNS domain to use eg, laconic.servesthe.world")
@click.option("--fqdn-policy", help="How to handle requests with an FQDN: prohibit, allow, preexisting", default="prohibit") @click.option(
"--fqdn-policy",
help="How to handle requests with an FQDN: prohibit, allow, preexisting",
default="prohibit",
)
@click.option("--record-namespace-dns", help="eg, lrn://laconic/dns") @click.option("--record-namespace-dns", help="eg, lrn://laconic/dns")
@click.option("--record-namespace-deployments", help="eg, lrn://laconic/deployments") @click.option("--record-namespace-deployments", help="eg, lrn://laconic/deployments")
@click.option("--dry-run", help="Don't do anything, just report what would be done.", is_flag=True) @click.option(
@click.option("--include-tags", help="Only include requests with matching tags (comma-separated).", default="") "--dry-run", help="Don't do anything, just report what would be done.", is_flag=True
@click.option("--exclude-tags", help="Exclude requests with matching tags (comma-separated).", default="") )
@click.option("--force-rebuild", help="Rebuild even if the image already exists.", is_flag=True) @click.option(
@click.option("--recreate-on-deploy", help="Remove and recreate deployments instead of updating them.", is_flag=True) "--include-tags",
@click.option("--log-dir", help="Output build/deployment logs to directory.", default=None) help="Only include requests with matching tags (comma-separated).",
default="",
)
@click.option(
"--exclude-tags",
help="Exclude requests with matching tags (comma-separated).",
default="",
)
@click.option(
"--force-rebuild", help="Rebuild even if the image already exists.", is_flag=True
)
@click.option(
"--recreate-on-deploy",
help="Remove and recreate deployments instead of updating them.",
is_flag=True,
)
@click.option(
"--log-dir", help="Output build/deployment logs to directory.", default=None
)
@click.pass_context @click.pass_context
def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_dir, # noqa: C901 def command( # noqa: C901
request_id, discover, state_file, only_update_state, ctx,
dns_suffix, fqdn_policy, record_namespace_dns, record_namespace_deployments, dry_run, kube_config,
include_tags, exclude_tags, force_rebuild, recreate_on_deploy, log_dir): laconic_config,
image_registry,
deployment_parent_dir,
request_id,
discover,
state_file,
only_update_state,
dns_suffix,
fqdn_policy,
record_namespace_dns,
record_namespace_deployments,
dry_run,
include_tags,
exclude_tags,
force_rebuild,
recreate_on_deploy,
log_dir,
):
if request_id and discover: if request_id and discover:
print("Cannot specify both --request-id and --discover", file=sys.stderr) print("Cannot specify both --request-id and --discover", file=sys.stderr)
sys.exit(2) sys.exit(2)
@ -242,26 +340,41 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
sys.exit(2) sys.exit(2)
if not only_update_state: if not only_update_state:
if not record_namespace_dns or not record_namespace_deployments or not dns_suffix: if (
print("--dns-suffix, --record-namespace-dns, and --record-namespace-deployments are all required", file=sys.stderr) not record_namespace_dns
or not record_namespace_deployments
or not dns_suffix
):
print(
"--dns-suffix, --record-namespace-dns, and --record-namespace-deployments are all required",
file=sys.stderr,
)
sys.exit(2) sys.exit(2)
if fqdn_policy not in ["prohibit", "allow", "preexisting"]: if fqdn_policy not in ["prohibit", "allow", "preexisting"]:
print("--fqdn-policy must be one of 'prohibit', 'allow', or 'preexisting'", file=sys.stderr) print(
"--fqdn-policy must be one of 'prohibit', 'allow', or 'preexisting'",
file=sys.stderr,
)
sys.exit(2) sys.exit(2)
main_logger = TimedLogger(file=sys.stderr)
try:
# Split CSV and clean up values. # Split CSV and clean up values.
include_tags = [tag.strip() for tag in include_tags.split(",") if tag] include_tags = [tag.strip() for tag in include_tags.split(",") if tag]
exclude_tags = [tag.strip() for tag in exclude_tags.split(",") if tag] exclude_tags = [tag.strip() for tag in exclude_tags.split(",") if tag]
laconic = LaconicRegistryClient(laconic_config) laconic = LaconicRegistryClient(laconic_config, log_file=sys.stderr)
# Find deployment requests. # Find deployment requests.
# single request # single request
if request_id: if request_id:
main_logger.log(f"Retrieving request {request_id}...")
requests = [laconic.get_record(request_id, require=True)] requests = [laconic.get_record(request_id, require=True)]
# all requests # all requests
elif discover: elif discover:
main_logger.log("Discovering deployment requests...")
requests = laconic.app_deployment_requests() requests = laconic.app_deployment_requests()
if only_update_state: if only_update_state:
@ -269,7 +382,11 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
dump_known_requests(state_file, requests) dump_known_requests(state_file, requests)
return return
if state_file:
main_logger.log(f"Loading known requests from {state_file}...")
previous_requests = load_known_requests(state_file) previous_requests = load_known_requests(state_file)
else:
previous_requests = {}
# Collapse related requests. # Collapse related requests.
requests.sort(key=lambda r: r.createTime) requests.sort(key=lambda r: r.createTime)
@ -277,45 +394,64 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
requests_by_name = {} requests_by_name = {}
skipped_by_name = {} skipped_by_name = {}
for r in requests: for r in requests:
status = None main_logger.log(f"BEGIN: Examining request {r.id}")
result = "PENDING"
try: try:
if r.id in previous_requests and previous_requests[r.id].get("status", "") != "RETRY": if (
print(f"Skipping request {r.id}, we've already seen it.") r.id in previous_requests
and previous_requests[r.id].get("status", "") != "RETRY"
):
main_logger.log(f"Skipping request {r.id}, we've already seen it.")
result = "SKIP"
continue continue
app = laconic.get_record(r.attributes.application) app = laconic.get_record(r.attributes.application)
if not app: if not app:
print(f"Skipping request {r.id}, cannot locate app.") main_logger.log(f"Skipping request {r.id}, cannot locate app.")
status = "SEEN" result = "ERROR"
continue continue
requested_name = r.attributes.dns requested_name = r.attributes.dns
if not requested_name: if not requested_name:
requested_name = generate_hostname_for_app(app) requested_name = generate_hostname_for_app(app)
print("Generating name %s for request %s." % (requested_name, r.id)) main_logger.log(
"Generating name %s for request %s." % (requested_name, r.id)
)
if requested_name in skipped_by_name or requested_name in requests_by_name: if (
print("Ignoring request %s, it has been superseded." % r.id) requested_name in skipped_by_name
or requested_name in requests_by_name
):
main_logger.log(
"Ignoring request %s, it has been superseded." % r.id
)
result = "SKIP"
continue continue
if skip_by_tag(r, include_tags, exclude_tags): if skip_by_tag(r, include_tags, exclude_tags):
print("Skipping request %s, filtered by tag (include %s, exclude %s, present %s)" % (r.id, main_logger.log(
include_tags, "Skipping request %s, filtered by tag (include %s, exclude %s, present %s)"
exclude_tags, % (r.id, include_tags, exclude_tags, r.attributes.tags)
r.attributes.tags)) )
skipped_by_name[requested_name] = r skipped_by_name[requested_name] = r
result = "SKIP"
continue continue
print("Found request %s to run application %s on %s." % (r.id, r.attributes.application, requested_name)) main_logger.log(
"Found pending request %s to run application %s on %s."
% (r.id, r.attributes.application, requested_name)
)
requests_by_name[requested_name] = r requests_by_name[requested_name] = r
except Exception as e: except Exception as e:
print(f"ERROR examining request {r.id}: " + str(e)) result = "ERROR"
status = "ERROR" main_logger.log(f"ERROR examining request {r.id}: " + str(e))
finally: finally:
if status: main_logger.log(f"DONE Examining request {r.id} with result {result}.")
dump_known_requests(state_file, [r], status) if result in ["ERROR"]:
dump_known_requests(state_file, [r], status=result)
# Find deployments. # Find deployments.
main_logger.log("Discovering existing app deployments...")
deployments = laconic.app_deployments() deployments = laconic.app_deployments()
deployments_by_request = {} deployments_by_request = {}
for d in deployments: for d in deployments:
@ -323,6 +459,7 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
deployments_by_request[d.attributes.request] = d deployments_by_request[d.attributes.request] = d
# Find removal requests. # Find removal requests.
main_logger.log("Discovering deployment removal and cancellation requests...")
cancellation_requests = {} cancellation_requests = {}
removal_requests = laconic.app_deployment_removal_requests() removal_requests = laconic.app_deployment_removal_requests()
for r in removal_requests: for r in removal_requests:
@ -331,23 +468,35 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
requests_to_execute = [] requests_to_execute = []
for r in requests_by_name.values(): for r in requests_by_name.values():
if r.id in cancellation_requests and match_owner(cancellation_requests[r.id], r): if r.id in cancellation_requests and match_owner(
print(f"Found deployment cancellation request for {r.id} at {cancellation_requests[r.id].id}") cancellation_requests[r.id], r
):
main_logger.log(
f"Found deployment cancellation request for {r.id} at {cancellation_requests[r.id].id}"
)
elif r.id in deployments_by_request: elif r.id in deployments_by_request:
print(f"Found satisfied request for {r.id} at {deployments_by_request[r.id].id}") main_logger.log(
f"Found satisfied request for {r.id} at {deployments_by_request[r.id].id}"
)
else: else:
if r.id not in previous_requests: if (
print(f"Request {r.id} needs to processed.") r.id in previous_requests
requests_to_execute.append(r) and previous_requests[r.id].get("status", "") != "RETRY"
else: ):
print( main_logger.log(
f"Skipping unsatisfied request {r.id} because we have seen it before." f"Skipping unsatisfied request {r.id} because we have seen it before."
) )
else:
main_logger.log(f"Request {r.id} needs to processed.")
requests_to_execute.append(r)
print("Found %d unsatisfied request(s) to process." % len(requests_to_execute)) main_logger.log(
"Found %d unsatisfied request(s) to process." % len(requests_to_execute)
)
if not dry_run: if not dry_run:
for r in requests_to_execute: for r in requests_to_execute:
main_logger.log(f"DEPLOYING {r.id}: BEGIN")
dump_known_requests(state_file, [r], "DEPLOYING") dump_known_requests(state_file, [r], "DEPLOYING")
status = "ERROR" status = "ERROR"
run_log_file = None run_log_file = None
@ -359,12 +508,16 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
if not os.path.exists(run_log_dir): if not os.path.exists(run_log_dir):
os.mkdir(run_log_dir) os.mkdir(run_log_dir)
run_log_file_path = os.path.join(run_log_dir, f"{run_id}.log") run_log_file_path = os.path.join(run_log_dir, f"{run_id}.log")
print(f"Directing deployment logs to: {run_log_file_path}") main_logger.log(
f"Directing deployment logs to: {run_log_file_path}"
)
run_log_file = open(run_log_file_path, "wt") run_log_file = open(run_log_file_path, "wt")
run_reg_client = LaconicRegistryClient(laconic_config, log_file=run_log_file) run_reg_client = LaconicRegistryClient(
laconic_config, log_file=run_log_file
)
logger = TimedLogger(run_id, run_log_file) build_logger = TimedLogger(run_id, run_log_file)
logger.log("Processing ...") build_logger.log("Processing ...")
process_app_deployment_request( process_app_deployment_request(
ctx, ctx,
run_reg_client, run_reg_client,
@ -378,14 +531,23 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
force_rebuild, force_rebuild,
fqdn_policy, fqdn_policy,
recreate_on_deploy, recreate_on_deploy,
logger build_logger,
) )
status = "DEPLOYED" status = "DEPLOYED"
except Exception as e: except Exception as e:
logger.log("ERROR: " + str(e)) main_logger.log(f"ERROR {r.id}:" + str(e))
build_logger.log("ERROR: " + str(e))
finally: finally:
if logger: main_logger.log(f"DEPLOYING {r.id}: END - {status}")
logger.log(f"DONE with status {status}", show_step_time=False, show_total_time=True) if build_logger:
build_logger.log(
f"DONE with status {status}",
show_step_time=False,
show_total_time=True,
)
dump_known_requests(state_file, [r], status) dump_known_requests(state_file, [r], status)
if run_log_file: if run_log_file:
run_log_file.close() run_log_file.close()
except Exception as e:
main_logger.log("UNCAUGHT ERROR:" + str(e))
raise e