ansible-role-k8s

Ansible role for configuring k3s and rke2 kubernetes clusters

• https://docs.k3s.io/
• https://docs.rke2.io/
• https://kube-vip.io/
• https://github.com/sbstp/kubie
• https://kubernetes.io/docs/tasks/tools/
• https://helm.sh/

Requirements

There is an included helper script to install common tools scripts/get-kube-tools.sh

• yq required on the local system for the kubectl formatting task which places an updated kubeconfig in the local ~/.kube
• kubectl required on the local system for basic cluster mangement and application of locally stored manifests or secrets
• helm required on the local system for helm deployments that use locally stored value files, otherwise this is handled on the bootstrap node
• kubie recommened on the local system for context management after deployment

Setup

There is a helper script scripts/token-vault.sh which pre-generates a cluster token and places it in an encrypted vault file

Cluster Example

cluster hosts

[k8s_somecluster]
somecluster_control k8s_node_type=bootstrap
somecluster_agent_smith k8s_node_type=agent k8s_external_ip=x.x.x.x
somecluster_agent_jones k8s_node_type=agent k8s_external_ip=x.x.x.x

cluster tasks

- name: Setup k8s server node
  hosts: somehost
  become: true
  roles:
    - role: k8s
      k8s_type: rke2
      k8s_cluster_name: somecluster
      k8s_cluster_url: somecluster.somewhere
      k8s_cni_interface: enp1s0
      k8s_selinux: true

    - role: firewalld
      firewalld_add:
        - name: internal
          interfaces:
            - enp1s0
          masquerade: true
          forward: true
          interfaces:
            - enp1s0
          services:
            - dhcpv6-client
            - ssh
            - http
            - https
          ports:
            - 6443/tcp        # kubernetes API
            - 9345/tcp        # supervisor API
            - 10250/tcp       # kubelet metrics
            - 2379/tcp        # etcd client
            - 2380/tcp        # etcd peer
            - 30000-32767/tcp # NodePort range
            - 8472/udp        # canal/flannel vxlan
            - 9099/tcp        # canal health checks
                    
        - name: trusted
          sources:
            - 10.42.0.0/16
            - 10.43.0.0/16

        - name: public
          masquerade: true
          forward: true
          interfaces:
            - enp7s0
          services:
            - http
            - https

      firewalld_remove:
        - name: public
          interfaces:
            - enp1s0
          services:
            - dhcpv6-client
            - ssh

Retrieve kube config from an existing cluster

This task will retrieve and format the kubectl config for an existing cluster, this runs automatically during cluster creation.

k8s_cluster_name sets the cluster context k8s_cluster_url sets the server address

ansible-playbook -i prod/ site.yml --tags=k8s-get-config --limit=k8s_somecluster

Basic Cluster Interaction

kubie ctx <cluster-name>
kubectl get node -o wide
kubectl get pods,svc,ds --all-namespaces

Deployment and Removal

Deploy

ansible-playbook -i hosts site.yml --tags=firewalld,k8s --limit=k8s_somecluster

Adding a node, simply add the new host to the cluster group with its defined role and deploy

ansible-playbook -i hosts site.yml --tags=firewalld,k8s --limit=just_the_new_host

Remove firewall role

ansible-playbook -i hosts site.yml --tags=firewalld,k8s --extra-vars "firewall_action=remove" --limit=somehost

There is a task to completely destroy an existing cluster, this will ask for interactive user confirmation and should be used with caution.

ansible-playbook -i prod/ site.yml --tags=k8s --extra-vars 'k8s_action=destroy' --limit=some_innocent_cluster

Manual removal commands

/usr/local/bin/k3s-uninstall.sh
/usr/local/bin/k3s-agent-uninstall.sh

/usr/local/bin/rke2-uninstall.sh
/usr/local/bin/rke2-agent-uninstall.sh

Managing K3S Services

servers

systemctl status k3s.service
journalctl -u k3s.service -f

agents

systemctl status k3s-agent.service
journalctl -u k3s-agent -f

uninstall servers

/usr/local/bin/k3s-uninstall.sh

uninstall agents

/usr/local/bin/k3s-agent-uninstall.sh

Managing RKE2 Services

servers

systemctl status rke2-server.service
journalctl -u rke2-server -f

agents

systemctl status rke2-agent.service
journalctl -u rke2-agent -f

uninstall servers

/usr/bin/rke2-uninstall.sh

uninstall agents

/usr/local/bin/rke2-uninstall.sh

override default cannal options

# /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-canal
  namespace: kube-system
spec:
  valuesContent: |-
    flannel:
      iface: "eth1"

Enable flannels wireguard support under canal

kubectl rollout restart ds rke2-canal -n kube-system

# /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-canal
  namespace: kube-system
spec:
  valuesContent: |-
    flannel:
      backend: "wireguard"