86 Commits

Author	SHA1	Message	Date
Thomas E Lackey	75ff60752a	Require payment for app deployment requests. (#928) ... Adds three new options for deployment/undeployment: ``` "--min-required-payment", help="Requests must have a minimum payment to be processed", "--payment-address", help="The address to which payments should be made. Default is the current laconic account.", "--all-requests", help="Handle requests addressed to anyone (by default only requests to my payment address are examined).", ``` In this mode, requests should be designated for a particular address with the attribute `to` and include a `payment` attribute which is the tx hash for the payment. The deployer will confirm the payment (to the right account, right amount, not used before, etc.) and then proceed with the deployment or undeployment. Reviewed-on: #928 Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to> Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-08-21 14:39:20 +00:00
David Boreham	e56da7dcc1	Add support for k8s pod to node affinity and taint toleration (#917) ... Reviewed-on: #917 Reviewed-by: Thomas E Lackey <telackey@noreply.git.vdb.to> Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-08-15 20:32:58 +00:00
Thomas E Lackey	60d34217f8	More logging for webapp deployment (#923) ... ``` cerc-webapp-deployer: ############ DEPLOY ############# cerc-webapp-deployer: 2024-08-15 02:13:08.321991 - - 0:00:00.000031 (step): Discovering deployment requests... cerc-webapp-deployer: laconic -c /etc/config/laconic.yml registry record list --all --type ApplicationDeploymentRequest cerc-webapp-deployer: 2024-08-15 02:13:08.815428 - - 0:00:00.493420 (step): Loading known requests from /srv/deployments/autodeploy.state... cerc-webapp-deployer: 2024-08-15 02:13:08.815626 - - 0:00:00.000158 (step): BEGIN: Examining request bafyreigiltcdscwt7rqldnilo4ohrhgoulrlfceixde5ycewsym64sefgi cerc-webapp-deployer: 2024-08-15 02:13:08.815645 - - 0:00:00.000008 (step): Skipping request bafyreigiltcdscwt7rqldnilo4ohrhgoulrlfceixde5ycewsym64sefgi, we've already seen it. cerc-webapp-deployer: 2024-08-15 02:13:08.815653 - - 0:00:00.000005 (step): DONE Examining request bafyreigiltcdscwt7rqldnilo4ohrhgoulrlfceixde5ycewsym64sefgi with result SKIP. cerc-webapp-deployer: 2024-08-15 02:13:08.815664 - - 0:00:00.000005 (step): BEGIN: Examining request bafyreicoxippgdwab6cz72py4rgv63rvvbsea73y62hashlhqpcsxyfkue cerc-webapp-deployer: 2024-08-15 02:13:08.815674 - - 0:00:00.000006 (step): Skipping request bafyreicoxippgdwab6cz72py4rgv63rvvbsea73y62hashlhqpcsxyfkue, we've already seen it. cerc-webapp-deployer: 2024-08-15 02:13:08.815684 - - 0:00:00.000004 (step): DONE Examining request bafyreicoxippgdwab6cz72py4rgv63rvvbsea73y62hashlhqpcsxyfkue with result SKIP. cerc-webapp-deployer: 2024-08-15 02:13:08.815692 - - 0:00:00.000005 (step): BEGIN: Examining request bafyreih3gt44pvahnbg7ag26mlk3iie4s5m5znhygajja5dcovheti72ne cerc-webapp-deployer: 2024-08-15 02:13:08.815705 - - 0:00:00.000007 (step): Skipping request bafyreih3gt44pvahnbg7ag26mlk3iie4s5m5znhygajja5dcovheti72ne, we've already seen it. cerc-webapp-deployer: 2024-08-15 02:13:08.815714 - - 0:00:00.000005 (step): DONE Examining request bafyreih3gt44pvahnbg7ag26mlk3iie4s5m5znhygajja5dcovheti72ne with result SKIP. cerc-webapp-deployer: 2024-08-15 02:13:08.815724 - - 0:00:00.000004 (step): BEGIN: Examining request bafyreigjnbio47rug6x5tufzc6cwfcqpl3ck3xldzotrlz5bt663dh2pua cerc-webapp-deployer: 2024-08-15 02:13:08.815733 - - 0:00:00.000005 (step): Skipping request bafyreigjnbio47rug6x5tufzc6cwfcqpl3ck3xldzotrlz5bt663dh2pua, we've already seen it. cerc-webapp-deployer: 2024-08-15 02:13:08.815743 - - 0:00:00.000005 (step): DONE Examining request bafyreigjnbio47rug6x5tufzc6cwfcqpl3ck3xldzotrlz5bt663dh2pua with result SKIP. cerc-webapp-deployer: 2024-08-15 02:13:08.815751 - - 0:00:00.000004 (step): BEGIN: Examining request bafyreihsfno4s6lkxcp5a7g7pjj7kklrp3xaqo57mr2pz76nk3h4jukayy cerc-webapp-deployer: 2024-08-15 02:13:08.815761 - - 0:00:00.000006 (step): Skipping request bafyreihsfno4s6lkxcp5a7g7pjj7kklrp3xaqo57mr2pz76nk3h4jukayy, we've already seen it. cerc-webapp-deployer: 2024-08-15 02:13:08.815770 - - 0:00:00.000005 (step): DONE Examining request bafyreihsfno4s6lkxcp5a7g7pjj7kklrp3xaqo57mr2pz76nk3h4jukayy with result SKIP. cerc-webapp-deployer: 2024-08-15 02:13:08.815779 - - 0:00:00.000005 (step): BEGIN: Examining request bafyreicyfyj4ncmtuy5pain2rvc67v645cg2bbsiakizvhdiwvkx7asvdy cerc-webapp-deployer: 2024-08-15 02:13:08.815791 - - 0:00:00.000007 (step): Skipping request bafyreicyfyj4ncmtuy5pain2rvc67v645cg2bbsiakizvhdiwvkx7asvdy, we've already seen it. cerc-webapp-deployer: 2024-08-15 02:13:08.815800 - - 0:00:00.000004 (step): DONE Examining request bafyreicyfyj4ncmtuy5pain2rvc67v645cg2bbsiakizvhdiwvkx7asvdy with result SKIP. cerc-webapp-deployer: 2024-08-15 02:13:08.815808 - - 0:00:00.000004 (step): Discovering existing app deployments... cerc-webapp-deployer: laconic -c /etc/config/laconic.yml registry record list --all --type ApplicationDeploymentRecord cerc-webapp-deployer: 2024-08-15 02:13:09.330655 - - 0:00:00.514858 (step): Discovering deployment removal and cancellation requests... cerc-webapp-deployer: laconic -c /etc/config/laconic.yml registry record list --all --type ApplicationDeploymentRemovalRequest cerc-webapp-deployer: 2024-08-15 02:13:09.825145 - - 0:00:00.494460 (step): Found 0 unsatisfied request(s) to process. cerc-webapp-deployer: ############ DEPLOY SUCCESS ############# ``` Reviewed-on: #923 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-08-15 02:57:47 +00:00
Thomas E Lackey	952389abb0	Add option to recreate deployments rather than update them. (#920) ... cherry-pick from #912 Reviewed-on: #920 Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>	2024-08-14 20:14:40 +00:00
Thomas E Lackey	5c275aa622	Defensively handle errors examining app requests. (#922) ... Related to cerc-io/webapp-deployment-status-api#10 There are two issues in that. One is that the output probably changed recently, whether in the client or server, where no matching record is found by ID (Note this is specific to `laconic record get --id <v>` and does not seem to apply to the similar command to retrieve a record by name, `laconic name resolve <n>`). Rather than returning `[]` it is now returning `[ null ]`. This cause us to think there *was* an application record found, and we attempt to treat the `null` entry like an Application object. That's fixed by filtering out null responses, which is a good precaution for the deployer, though I think it makes sense to ask whether this new behavior by the client/server is correct. Seems suspicious. The other issue is that all the defensive checks we had in place to deal with broken/bad AppDeploymentRequests were around the _build_. This error was coming much earlier, merely when parsing and examining the request to see if it needed to be handled at all. I have now added similar defensive error handling around that portion of the code. Reviewed-on: #922 Reviewed-by: zramsay <zramsay@noreply.git.vdb.to> Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-08-14 18:04:31 +00:00
Thomas E Lackey	8576137557	Convert port to string. (#919) ... The str type check doesn't work if the port is a ruamel.yaml.scalarstring.SingleQuotedScalarString or ruamel.yaml.scalarstring.DoubleQuotedScalarString Reviewed-on: #919 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-08-14 00:25:35 +00:00
David Boreham	65c1cdf6b1	Fix crash if port has int type in yaml (#918) ... Reviewed-on: #918 Reviewed-by: Thomas E Lackey <telackey@noreply.git.vdb.to> Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-08-13 20:47:09 +00:00
David Boreham	265699bc38	Allow to disable kind cluster management for testing (#915) ... Reviewed-on: #915 Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-08-13 17:48:14 +00:00
Thomas E Lackey	6087e1cd31	Copy config under a volume for Docker (similar to a ConfigMap for K8S). (#914) ... This emulates the K8S ConfigMap behavior on Docker by using a regular volume. If a directory exists under `config/` which matches a named volume, the contents will be copied to the volume on `create` (provided the destination volume is empty). That is, rather than a ConfigMap, it is essentially a "config volume". For example, with a compose file like: ``` version: '3.7' services: caddy: image: cerc/caddy-ethcache:local restart: always volumes: - caddyconfig:/etc/caddy:ro volumes: caddyconfig: ``` And a directory: ``` ❯ ls stack-orchestrator/config/caddyconfig/ Caddyfile ``` After `laconic-so deploy create --spec-file caddy.yml --deployment-dir /srv/caddy` there will be: ``` ❯ ls /srv/caddy/data/caddyconfig Caddyfile ``` Mounted at `/etc/caddy` Reviewed-on: #914 Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to> Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-08-10 02:32:21 +00:00
Thomas E Lackey	1def279d26	Support multiple NodePorts, static NodePort mapping, and add 'replicas' spec option (#913) ... NodePort example: ``` network: ports: caddy: - 1234 - 32020:2020 ``` Replicas example: ``` replicas: 2 ``` This also adds an optimization for k8s where if a directory matching the name of a configmap exists in beneath config/ in the stack, its contents will be copied into the corresponding configmap. For example: ``` # Config files in the stack ❯ ls stack-orchestrator/config/caddyconfig Caddyfile Caddyfile.one-req-per-upstream-example # ConfigMap in the spec ❯ cat foo.yml | grep config ... configmaps: caddyconfig: ./configmaps/caddyconfig # Create the deployment ❯ laconic-so --stack ~/cerc/caddy-ethcache/stack-orchestrator/stacks/caddy-ethcache deploy create --spec-file foo.yml # The files from beneath config/<config_map_name> have been copied to the ConfigMap directory from the spec. ❯ ls deployment-001/configmaps/caddyconfig Caddyfile Caddyfile.one-req-per-upstream-example ``` Reviewed-on: #913 Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to> Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-08-09 02:32:06 +00:00
David Boreham	aef5986135	Allow gentx-files to be omitted	2024-08-07 14:11:06 -06:00
David Boreham	7590d6e237	Add stage 1 support	2024-08-07 11:28:10 -06:00
David Boreham	7d18334953	Mainnet-laconic stack fixes for laconicd2 (#904) ... Reviewed-on: #904	2024-07-31 13:51:28 +00:00
Thomas E Lackey	913c3a8557	Back to v2 now that we have a working webapp deployer build again. (#896) ... Reviewed-on: #896 Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to> Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-07-27 18:59:42 +00:00
Thomas E Lackey	2f5b0cdd13	Revert recent laconicd deployment changes to restore production webapp deployer function. (#895) ... Reviewed-on: #895	2024-07-27 17:04:03 +00:00
zramsay	80cff73344	crn --> lrn	2024-07-23 20:20:01 -04:00
zramsay	21b1270d27	fix lint	2024-07-23 20:16:16 -04:00
zramsay	008389dcd8	cns --> registry and other fixes	2024-07-23 20:10:06 -04:00
Roy Crihfield	36d4969b2d	Fixes for external stack deployment (#851) ... Fixes - stack path resolution for `build` - external stack path resolution for deployments - "extra" config detection - `deployment ports` command - `version` command in dist or source install (without build_tag.txt) - `setup-repos`, so it won't die when an existing repo is not at a branch or exact tag Used in cerc-io/fixturenet-eth-stacks#14 Reviewed-on: #851 Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>	2024-07-09 15:37:35 +00:00
David Boreham	62f7825ec2	Remove quotes from git config	2024-07-05 09:55:14 -06:00
David Boreham	6d24d4a7e6	Set github auth token if present (#868) ... Reviewed-on: #868 Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-07-05 12:27:22 +00:00
David Boreham	f06e5f9a2a	Don't try to tag remote images (#866) ... Reviewed-on: #866 Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-07-04 23:51:06 +00:00
David Boreham	c3a1402042	Derive the webapp host container id from stable data (#865) ... Reviewed-on: #865 Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-07-04 22:38:07 +00:00
David Boreham	ca5fffaed5	Add logging to webapp deployer (#863) ... Helps with diagnosing failures and odd behavior seen in the deployer in production. Reviewed-on: #863 Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-07-04 19:46:42 +00:00
David Boreham	bf1eccd486	Fix image tag name	2024-06-13 08:31:45 -06:00
David Boreham	3fb025b5c9	Make remote image tags unique to the deployment (#838) ... Reviewed-on: #838 Reviewed-by: Thomas E Lackey <telackey@noreply.git.vdb.to> Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-06-13 03:26:58 +00:00
David Boreham	13ce521d84	Fix config dir processing for external stacks (#810) ... Reviewed-on: #810 Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-04-23 21:47:20 +00:00
David Boreham	6e4dae9777	Add external stack support (#806) ... Reviewed-on: #806 Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-04-18 21:22:47 +00:00
Thomas E Lackey	9043a67c7c	Skip checks on requests we've already seen (#805) ... Reviewed-on: #805 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-04-15 20:27:35 +00:00
Thomas E Lackey	4126f2fc43	Add --fqdn-policy option to deploy-webapp-from-registry. (#802) ... This add a new option `--fqdn-policy` to the `deploy-webapp-from-registry`. The default policy, `prohibit` means that `ApplicationDeploymentRequests` which specify a FQDN will be rejected. The `allow` policy will cause them to be processed. The `preexisting` policy will only process them if an existing `DnsRecord` exists in the registry with the correct ownership. The latter would be useful in conjunction with a pre-checking scheme in the UI (eg, that the DNS entry is properly configured, the domain is under the control of the requestor, etc.) Only after all the checks were successful would the `DnsRecord` be created, allowing for `ApplicationDeploymentRequests` to use it. Reviewed-on: #802 Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to> Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-04-15 12:20:35 +00:00
Thomas E Lackey	d2442bcc9b	revert 5308ab1e4e (#788) ... revert Blind commit to fix laconic CLI calls after rename. (#784) `laconic cns` got renamed to `laconic registry` which breaks all the scripts and commands that use it. Reviewed-on: #784 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com> Reviewed-on: #788	2024-03-27 20:55:03 +00:00
Thomas E Lackey	5308ab1e4e	Blind commit to fix laconic CLI calls after rename. (#784) ... `laconic cns` got renamed to `laconic registry` which breaks all the scripts and commands that use it. Reviewed-on: #784 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-03-25 19:09:26 +00:00
Thomas E Lackey	523b5779be	Auto-detect which certificate to use (including wildcards). (#779) ... Rather than always requesting a certificate, attempt to re-use an existing certificate if it already exists in the k8s cluster. This includes matching to a wildcard certificate. Reviewed-on: #779 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-03-07 17:38:36 +00:00
Thomas E Lackey	10e2311a8b	Add timed logging for the webapp build (#771) ... Add lots of log and timer output to webapp builds. Reviewed-on: #771	2024-02-28 00:38:11 +00:00
Thomas E Lackey	a0413659f7	Check for existing tag in remote repo before building. (#764) ... webapps are meant to be build-once/deploy-many, but we were rebuilding them for every request. This changes that, so that we rebuild only for every unique ApplicationRecord. When we push the image, we now tag it according to its ApplicationRecord. We don't want to use that tag directly in the compose file for the deployment, however, as the deployment needs to be able to adjust to new builds w/o re-writing the file all the time. Instead, we use a per-deployment unique tag (same as before), we just update what image it references as needed. Reviewed-on: #764	2024-02-24 03:22:49 +00:00
Thomas E Lackey	2af6ffce77	Tweaks for running the container registry in k8s (#760) ... Minor tweaks for running the container-registry in k8s. The big change is not requiring --image-registry. Reviewed-on: #760 Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>	2024-02-22 21:11:06 +00:00
Thomas E Lackey	6bd77c893a	Even more logging fixes (#757) ... Hopefully the last one for a bit. This only output the cmdline if log_file is present (ie, not to stdout). It also fixes a bug where the log_file was not passed in one line. Reviewed-on: #757 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-22 01:24:44 +00:00
Thomas E Lackey	4a4d48ddb9	Fix error when logging exception. (#756) ... Reviewed-on: #756 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-22 00:11:06 +00:00
Thomas E Lackey	08438b1cd5	More logging for webapp deployments (#755) ... Reviewed-on: #755 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-21 23:48:52 +00:00
Thomas E Lackey	9f1dd284a5	Better error logging for registry deployments. (#754) ... We were missing errors related to registration, this should fix that. Reviewed-on: #754 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-21 20:12:52 +00:00
David Boreham	37b9500483	Support non-tls ingress for kind (#748) ... Reviewed-on: #748 Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-02-17 01:54:30 +00:00
David Boreham	d9bb6b3588	Test Database Stack (#737) ... Reviewed-on: #737	2024-02-15 05:26:29 +00:00
Thomas E Lackey	b22c72e715	For k8s, use provisioner-managed volumes when an absolute host path is not specified. (#741) ... In kind, when we bind-mount a host directory it is first mounted into the kind container at /mnt, then into the pod at the desired location. We accidentally picked this up for full-blown k8s, and were creating volumes at /mnt. This changes the behavior for both kind and regular k8s so that bind mounts are only allowed if a fully-qualified path is specified. If no path is specified at all, a default storageClass is assumed to be present, and the volume managed by a provisioner. Eg, for kind, the default provisioner is: https://github.com/rancher/local-path-provisioner ``` stack: test deploy-to: k8s-kind config: test-variable-1: test-value-1 network: ports: test: - '80' volumes: # this will be bind-mounted to a host-path test-data-bind: /srv/data # this will be managed by the k8s node test-data-auto: configmaps: test-config: ./configmap/test-config ``` Reviewed-on: #741 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-14 21:45:01 +00:00
Thomas E Lackey	c9444591f5	Fix default webapp port number. (#740) ... Reviewed-on: #740 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-09 01:20:41 +00:00
Thomas E Lackey	903f3b10e2	Add support for annotations and labels in spec. (#739) ... ``` stack: webapp-deployer-backend deploy-to: k8s annotations: foo.bar.annot/{name}: baz labels: a.b.c/{name}.blah: "value" ``` Reviewed-on: #739 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-09 00:11:07 +00:00
Thomas E Lackey	72ed2eb91a	Fix bad test in tag check. (#738) ... Reviewed-on: #738 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-08 20:38:41 +00:00
David Boreham	8be1e684e8	Process environment variables defined in compose files (#736) ... Reviewed-on: #736 Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-02-08 19:41:57 +00:00
Thomas E Lackey	3309782439	Refactor	2024-02-08 00:47:46 -06:00
Thomas E Lackey	4b3b3478e7	Switch to Docker-style limits	2024-02-08 00:43:41 -06:00
Thomas E Lackey	2a9955055c	debug	2024-02-07 16:56:35 -06:00