Carlos Perelló Marín's blog

Day 2: GitOps Handoff — Bootstrapping ArgoCD on Talos with OpenTofu

2026-04-15T10:00:00+02:00

The Day 1 post ended with ArgoCD deployed as a Helm release inside 02-kubernetes/. It's running, but it can't do anything yet. It has no repository credentials, no decryption keys, and no root Application to tell it what to manage. "Running" is not "working."

This post covers the bootstrap resources that bridge Day 1 and Day 2: the credentials, the KSOPS decryption setup, and the root Application that kicks off the GitOps loop. Once that loop starts, OpenTofu steps back.

The Chicken-and-Egg Problem

ArgoCD manages applications from git repositories. To pull from a private repo, it needs credentials. To decrypt SOPS-encrypted manifests, it needs the cluster Age key. But those credentials and keys are themselves secrets that need to live somewhere.

The solution is straightforward: OpenTofu bootstraps the credentials as Kubernetes Secrets before ArgoCD tries to use them. ArgoCD doesn't manage its own credentials, at least not at bootstrap time. OpenTofu creates them, ArgoCD consumes them, and the two never overlap.

Everything below lives in argocd-bootstrap.tf.

Repository Credentials

Two private repos need to be registered: the platform gitops repo (controlled by Nubosas) and the tenant workload repo (controlled by the tenant). Both use a dedicated service account with repo-scoped GitHub tokens.

The tokens are stored SOPS-encrypted in secrets.enc.yaml and decrypted at apply time via the carlpett/sops provider that was declared back in Day 1:

data "sops_file" "secrets" {
  source_file = "${path.module}/environments/${var.environment}/secrets.enc.yaml"
}

Each repo is registered as a Kubernetes Secret with the label ArgoCD watches for:

resource "kubernetes_secret" "argocd_repo_platform" {
  metadata {
    name      = "repo-platform-gitops"
    namespace = "argocd"
    labels = {
      "argocd.argoproj.io/secret-type" = "repository"
    }
  }

  data = {
    type     = "git"
    url      = "https://github.com/example-org/platform-gitops.git"
    username = "example-bot"
    password = data.sops_file.secrets.data["github_token_platform"]
  }

  depends_on = [helm_release.argocd]
}

Same pattern for the tenant repo, different token. The depends_on ensures ArgoCD is running before we create secrets in its namespace.

This is the Day 1 encryption strategy paying off. The sops provider was declared but unused in the initial commit because no secrets were needed for Talos bootstrap. Now it's active, decrypting at apply time with the personal Age key.

KSOPS: Decrypting Secrets at Runtime

OpenTofu decrypts secrets at apply time. But ArgoCD needs to decrypt SOPS-encrypted manifests at sync time, inside the cluster, without the personal key.

That's where KSOPS comes in. It's a Kustomize plugin that intercepts SOPS-encrypted resources during ArgoCD's render phase and decrypts them using the cluster Age key.

ArgoCD's container image doesn't ship with KSOPS. The standard approach is an init container that downloads the binary into a shared volume before the repo-server starts.

The Init Container

repoServer:
  initContainers:
    - name: install-ksops
      image: alpine:3.21
      command: ["sh", "-c"]
      args:
        - |
          set -e
          wget -qO- https://github.com/viaduct-ai/kustomize-sops/releases/download/v4.4.0/ksops_4.4.0_Linux_x86_64.tar.gz \
            | tar xz -C /custom-tools
          chmod +x /custom-tools/ksops
          mkdir -p /custom-tools/plugin/viaduct.ai/v1/ksops
          cp /custom-tools/ksops /custom-tools/plugin/viaduct.ai/v1/ksops/ksops
      volumeMounts:
        - name: custom-tools
          mountPath: /custom-tools
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
        seccompProfile:
          type: RuntimeDefault

Two things worth calling out. First, the binary gets laid down twice: once as a flat ksops for Kustomize to invoke directly, and once inside plugin/viaduct.ai/v1/ksops/ksops so Kustomize can also discover it as an exec plugin via KUSTOMIZE_PLUGIN_HOME. Different code paths in Kustomize look for it in different places.

Second, the security context. This init container downloads and unpacks a single binary, so it has no reason to run as root, hold any capabilities, or escalate privileges. runAsUser: 65534 is nobody. The seccompProfile: RuntimeDefault restricts syscalls to the container runtime's default set. Defense in depth for a container that runs for two seconds, but it costs nothing.

Mounting the Binary and the Key

The KSOPS binary goes into /usr/local/bin/ksops on the repo-server via the shared volume, plus the plugin tree gets mounted under /home/argocd/ksops-plugin. The cluster Age key gets mounted as a directory so SOPS finds it at the path it expects:

repoServer:
  volumes:
    - name: custom-tools
      emptyDir: {}
    - name: sops-age-key
      secret:
        secretName: ksops-age-key
        defaultMode: 292  # 0444 — readable by argocd user (uid 999)
  volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/ksops
      subPath: ksops
    - name: custom-tools
      mountPath: /home/argocd/ksops-plugin
      subPath: plugin
    - name: sops-age-key
      mountPath: /home/argocd/.config/sops/age
      readOnly: true
  env:
    - name: SOPS_AGE_KEY_FILE
      value: /home/argocd/.config/sops/age/age.agekey
    - name: XDG_CONFIG_HOME
      value: /home/argocd/.config
    - name: KUSTOMIZE_PLUGIN_HOME
      value: /home/argocd/ksops-plugin

Three env vars matter here. SOPS_AGE_KEY_FILE tells SOPS exactly where the private key lives. XDG_CONFIG_HOME keeps SOPS's config search rooted under the argocd user's home. KUSTOMIZE_PLUGIN_HOME is what makes Kustomize discover KSOPS as an exec plugin. Miss any of these and the symptom is the same: opaque "decryption failed" or "plugin not found" errors at sync time.

The Age key Secret is created by OpenTofu alongside the repo credentials:

resource "kubernetes_secret" "ksops_age_key" {
  metadata {
    name      = "ksops-age-key"
    namespace = "argocd"
  }

  data = {
    "age.agekey" = data.sops_file.secrets.data["age_cluster_private_key"]
  }

  depends_on = [helm_release.argocd]
}

Notice: the cluster private key is itself SOPS-encrypted in secrets.enc.yaml, decrypted by OpenTofu using the personal key, then stored as a Kubernetes Secret for KSOPS. SOPS decrypting a key that enables more SOPS decryption.

Finally, Kustomize needs the alpha plugins flag to recognize KSOPS. This goes under configs.cm (the argocd-cm ConfigMap), not configs.params (the argocd-cmd-params-cm ConfigMap that drives CLI flags):

configs:
  params:
    "server.insecure": true
  cm:
    "kustomize.buildOptions": "--enable-alpha-plugins --enable-exec"

Easy to put in the wrong block — configs.params looks like the natural home for a "build option" but ArgoCD reads kustomize.buildOptions from the argocd-cm ConfigMap. Wrong block, silent no-op.

The Three-Layer Decryption Flow

Day 1 set up the two-key design. Here's how it completes:

At rest in git: secrets are encrypted with SOPS using Age public keys. Both the personal key and cluster key are recipients, so either can decrypt.
At bootstrap (tofu apply): the carlpett/sops provider decrypts secrets.enc.yaml using the personal Age key (available locally). The decrypted values become Kubernetes Secrets.
At runtime (ArgoCD sync): when ArgoCD reconciles a repo containing SOPS-encrypted manifests, KSOPS intercepts the render, reads the cluster Age key from the mounted Secret, and decrypts inline. The plaintext manifest is applied to the cluster. Plaintext never touches git.

Two keys, three layers, one encryption strategy designed from Day 1.

The Root Application

With credentials and decryption in place, the last step is telling ArgoCD what to manage. This is the App of Apps pattern: a single root Application that watches a directory for other Application definitions.

# root-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example-org/platform-gitops.git
    targetRevision: HEAD
    path: apps
    directory:
      recurse: true
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

directory.recurse: true is important. The apps/ directory uses subdirectories (apps/platform/, apps/tenants/) to organize Application definitions. Without recursion, ArgoCD only scans the top level and misses everything in subdirectories. I initially had this flat and had to add recursion when the directory structure grew.

prune: true means ArgoCD deletes resources that no longer exist in git. selfHeal: true means it reverts any manual changes. Together, they enforce that git is the single source of truth.

Why `null_resource` Is Ugly but Right

There's no clean way to apply a custom resource through the OpenTofu Kubernetes provider and have ArgoCD manage it afterwards. The kubernetes_manifest resource could create the Application, but then both OpenTofu and ArgoCD would be managing the same resource, fighting over the source of truth. The root Application needs to be created once and then owned entirely by ArgoCD.

The pragmatic answer is null_resource with local-exec:

resource "local_sensitive_file" "kubeconfig" {
  content  = talos_cluster_kubeconfig.cluster.kubeconfig_raw
  filename = "${path.module}/.kubeconfig"
}

resource "null_resource" "argocd_root_app" {
  triggers = {
    once = "bootstrap"
  }

  provisioner "local-exec" {
    command     = "kubectl apply -f ${path.module}/root-app.yaml"
    environment = {
      KUBECONFIG = local_sensitive_file.kubeconfig.filename
    }
  }

  depends_on = [
    helm_release.argocd,
    kubernetes_secret.argocd_repo_platform,
    kubernetes_secret.argocd_repo_tenant,
    kubernetes_secret.ksops_age_key,
  ]
}

triggers = { once = "bootstrap" } is a static value. It triggers on the first tofu apply and never again, because the trigger value never changes. Subsequent applies see the null_resource in state with the same trigger and skip it. This is a fire-and-forget bootstrap step.

The depends_on list is the full dependency chain: ArgoCD must be running, both repo credentials must exist, and the KSOPS key must be in place. Only then does kubectl apply create the root Application. After that, ArgoCD takes over.

Destroying the null_resource from OpenTofu state does NOT delete the ArgoCD Application from the cluster. It's a one-way handoff.

Multi-Tenancy: AppProject as the Authorization Boundary

Once the root Application syncs, it picks up everything in apps/, including the tenant onboarding resources. The natural question at this point: what stops a tenant from deploying into kube-system or another tenant's namespace?

The answer is the AppProject CRD. Every Application references a project, and the project defines what that Application is allowed to do.

The tenant AppProject:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: tenant-a
  namespace: argocd
spec:
  description: Tenant workloads
  sourceRepos:
    - https://github.com/example-tenant/tenant-gitops.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: tenant-a
  clusterResourceWhitelist: []
  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota
    - group: ""
      kind: LimitRange
  orphanedResources:
    warn: true

sourceRepos locks the tenant to their own repo. destinations restricts deployment to the tenant-a namespace only. clusterResourceWhitelist: [] blocks all cluster-scoped resources (Namespaces, ClusterRoles, CRDs). namespaceResourceBlacklist prevents the tenant from overriding the ResourceQuota and LimitRange that the platform manages.

If the tenant pushes a Deployment targeting kube-system to their gitops repo, the ArgoCD application controller rejects it at sync time. No workaround.

The Default Project Gap

The tenant AppProject was configured correctly. But I noticed the default project (which ships with ArgoCD) was wide open: sourceRepos: ['*'], destinations unrestricted, full cluster-scoped access.

Both the root Application and the tenant onboarding Application use project: default, which is fine since they source from the platform-controlled repo. The problem is lateral: if anyone creates an Application under project: default (misconfiguration, a future templating mistake, direct kubectl access), they get unrestricted access to the entire cluster from any repo.

The fix: lock the default project's sourceRepos to the platform repo only.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: default
  namespace: argocd
spec:
  description: Platform-managed resources only
  sourceRepos:
    - https://github.com/example-org/platform-gitops.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: '*'
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'

Still allows all destinations and cluster-scoped resources (the platform needs full control), but any Application in project: default can only pull from code Nubosas controls. Defense in depth for a gap that might never be exploited.

This lives in platform/default-project.yaml and gets deployed by a platform-config Application that the root app picks up automatically. The pattern scales: any future platform-level resource (NetworkPolicies, additional AppProject restrictions) goes into platform/ and is synced through the same path.

There's more to multi-tenancy security (Cilium NetworkPolicies for runtime network isolation, ArgoCD RBAC if tenants ever get UI access), but those are separate concerns for a future post.

The Full Dependency Chain

From Day 1 through Day 2, the dependency chain is explicit:

flowchart TD
    A["Talos machine secrets
+ node configs"] --> B["talos_machine_bootstrap
(etcd init)"]
    B --> C["talos_cluster_kubeconfig
(client certs)"]
    C --> D["helm_release.cilium
(CNI, pods can schedule)"]
    D -. "Day 2 starts" .-> E["helm_release.argocd"]
    E --> F1["kubernetes_secret
argocd_repo_* (repo creds)"]
    E --> F2["kubernetes_secret
ksops_age_key"]
    E --> F3["local_sensitive_file
kubeconfig"]
    F1 --> G["null_resource.argocd_root_app
(kubectl apply root-app.yaml)"]
    F2 --> G
    F3 --> G
    G -. "OpenTofu stops" .-> H["root Application
(auto-syncs apps/)"]
    H --> I1["platform-config
(locked default project)"]
    H --> I2["tenant-onboarding
(namespace + quota + AppProject)"]
    H --> I3["tenant Application
(tenant-gitops workloads)"]

    classDef day1 fill:#e8f0ff,stroke:#0063FF,color:#0A354F
    classDef day2 fill:#fff7e6,stroke:#C64350,color:#0A354F
    class A,B,C,D day1
    class E,F1,F2,F3,G,H,I1,I2,I3 day2

One tofu apply gets you from Talos API to a fully operational GitOps loop. After that, OpenTofu manages the cluster infrastructure (Talos configs, Cilium, ArgoCD itself) and ArgoCD manages everything that runs on top of it.

The boundary is the same principle from Day 0, applied one layer up: each tool owns what it can observe and control through an API. OpenTofu talks to the Talos and Kubernetes APIs. ArgoCD talks to git and the Kubernetes API. They don't overlap.

What's Next

The GitOps loop is running. Workloads can deploy. But the cluster still needs storage for stateful applications, which is where Rook-Ceph comes in. That's the subject of the next post.

Day 1: Taking Declarative Control of a Running Talos Cluster with OpenTofu

2026-04-01T10:00:00+02:00

The Day 0 bootstrapper exits the moment Talos starts listening on port 50000. The server has an OS. It's waiting for configuration. That's where OpenTofu takes over.

This post covers 02-kubernetes/, the layer that codifies the full cluster: 13 nodes across four roles, machine config generation and application, and Cilium managed as a Helm release. This is where the siderolabs/talos provider earns its place — proper declarative API, proper state, proper drift detection.

The Boundary

The siderolabs/talos provider talks directly to the Talos API on port 50000. Unlike the null_resource approach from 01-hardware/, it can actually observe state: read running configuration, apply patches, track whether a node needs updating. The moment a server is in maintenance mode waiting for its machine config, this layer owns it.

resource "talos_machine_configuration_apply" "node" {
  for_each = local.nodes

  client_configuration        = talos_machine_secrets.cluster.client_configuration
  machine_configuration_input = data.talos_machine_configuration.node[each.key].machine_configuration
  node                        = each.value.ip
  endpoint = each.value.role != "controlplane" ? local.first_controlplane_ip : null

  apply_mode = "no_reboot"
}

apply_mode = "no_reboot" is important for a running production cluster: config changes are applied live, and anything that requires a reboot is staged for the next maintenance window rather than rebooting immediately. Safe to run tofu apply without taking nodes down.

Node Roles and Patch Composition

The cluster has 13 nodes across four roles: control planes, workers, storage, and monitoring. Rather than one config file per node, machine configurations are built by composing patches:

config_patches = concat(
  [local.patch_common],                          # every node
  each.value.role == "controlplane" ? [local.patch_controlplane] : [],
  each.value.role == "monitoring"   ? [local.patch_monitoring_node, local.patch_monitoring_extras] : [],
  each.value.role == "storage"      ? [local.patch_storage_extras, local.patch_storage_label] : [],
  [local.patch_per_node[each.key]],              # hostname, IP, gateway, disk
)

The common patch sets cluster-wide settings: cluster name, pod and service subnets, CNI disabled (Cilium is installed separately), kube-proxy disabled (Cilium handles it). Role patches add what's specific to that role. The per-node patch sets hostname, static IP, gateway, and install disk.

This keeps the config readable and avoids repeating the same values across 13 node definitions.

Non-Control-Plane Node Routing

Worker, storage, and monitoring nodes are not directly reachable on port 50000 from outside the cluster network. The provider needs an entry point. The solution is the same as talosctl -e <cp-ip> -n <node-ip>: route through the first control plane.

endpoint = each.value.role != "controlplane" ? local.first_controlplane_ip : null

Control planes are reached directly. Everyone else goes through the first control plane as a relay. One line, handles the whole fleet.

NVMe Enumeration Is Not Deterministic

This caught me on one of the worker nodes. Talos detects the install disk by PCI bus discovery order, which is not guaranteed to be consistent across reboots or between physically identical servers. Two nodes with the same Samsung 512 GB NVMe drives had their disks enumerate in different orders.

The only safe way to confirm which device holds the STATE partition on a running node is:

talosctl get disks -n <node-ip>

For one worker, the STATE partition had landed on /dev/nvme1n1, not /dev/nvme0n1. One node also has a SATA disk rather than NVMe: a different hardware generation that ended up in the same cluster. Both are accounted for explicitly in the node metadata:

frodo = {
  role    = "worker"
  ip      = local.workers["frodo"]
  gateway = "..."
  disk    = "/dev/sda"  # SATA -- the only non-NVMe node
}

Assuming all nodes use the same disk path is a silent failure waiting to happen on the next upgrade.

Storage and Monitoring Node Specialization

Storage and monitoring nodes use Talos volume management to carve disk space beyond the system partition.

Storage nodes cap the EPHEMERAL volume at 50 GiB and claim the remaining space as a dedicated partition for GarageHQ data (an S3-compatible object store that handles its own HA across a 3-node cluster, independent of Rook-Ceph):

---
apiVersion: v1alpha1
kind: VolumeConfig
name: EPHEMERAL
provisioning:
  diskSelector:
    match: system_disk
  maxSize: 50GiB
---
apiVersion: v1alpha1
kind: UserVolumeConfig
name: garage-data
provisioning:
  diskSelector:
    match: system_disk
  minSize: 100GB
  grow: true

Monitoring nodes get a dedicated local storage volume carved from a secondary disk, plus a taint and label applied via machine config to keep general workloads off them:

patch_monitoring_node = yamlencode({
  machine = {
    nodeLabels = { pool = "monitoring" }
    nodeTaints = { dedicated = "monitoring:NoSchedule" }
  }
})

Both are machine config concerns, not Kubernetes concerns — the right layer for anything that affects how the node boots and presents itself.

Cilium: From Inline Manifest to Helm

Cilium was originally deployed as an inline manifest in the Talos machine config, which is how Talos bootstraps CNI before the Kubernetes API is available. It worked, but it meant Cilium was outside any management layer: no Helm release, no upgrade path, no values file.

Moving it to Helm under OpenTofu ownership required pre-annotating the existing resources with Helm ownership metadata before installing; otherwise Helm refuses to take over resources it didn't create:

helm template cilium cilium/cilium --version 1.16.3 -n kube-system \
  | kubectl annotate -f - meta.helm.sh/release-name=cilium \
      meta.helm.sh/release-namespace=kube-system --overwrite
helm template cilium cilium/cilium --version 1.16.3 -n kube-system \
  | kubectl label -f - app.kubernetes.io/managed-by=Helm --overwrite

Then there are two Talos-specific settings that the default Cilium Helm values get wrong.

First, Talos mounts cgroupv2 at /sys/fs/cgroup, not at the path Cilium expects. Disable Cilium's auto-mount and point it to the right location:

cgroup = {
  autoMount = { enabled = false }
  hostRoot  = "/sys/fs/cgroup"
}

Second, and less obvious: Talos's containerd-shim does not include SYS_MODULE in its capability bounding set. When the default Cilium values request it, runc returns EPERM and the init container crashes. The fix is to explicitly override the capability lists and drop SYS_MODULE:

securityContext = {
  capabilities = {
    ciliumAgent      = ["CHOWN", "KILL", "NET_ADMIN", "NET_RAW", "IPC_LOCK",
                        "SYS_ADMIN", "SYS_RESOURCE", "DAC_OVERRIDE", "FOWNER",
                        "SETGID", "SETUID"]
    cleanCiliumState = ["NET_ADMIN", "SYS_ADMIN", "SYS_RESOURCE"]
  }
}

Neither of these is documented anywhere obvious. Both are required. The pods crash silently until you read the container logs and work backwards from unable to apply caps: operation not permitted.

ArgoCD: The Last Piece of Day 1

ArgoCD lives in this layer for the same reason Cilium does: it's infrastructure, not a workload. Upgrading it is a version bump and a tofu apply, not an ArgoCD Application pointing at itself. Keeping both here means one tool owns all cluster-level components and one tofu apply gets you from Talos API to GitOps-ready.

resource "helm_release" "argocd" {
  name             = "argocd"
  repository       = "https://argoproj.github.io/argo-helm"
  chart            = "argo-cd"
  version          = "9.4.15"
  namespace        = "argocd"
  create_namespace = true

  depends_on = [helm_release.cilium]

  values = [yamlencode({
    global = {
      domain = "argocd.example.com"
    }
    configs = {
      params = {
        "server.insecure" = true
      }
    }
  })]
}

server.insecure = true is intentional: TLS terminates at the ingress layer, not inside ArgoCD. The ingress itself comes as a Day 2 Application once ArgoCD is running.

depends_on = [helm_release.cilium] enforces ordering: Cilium must be running before any pod can schedule. Without it, ArgoCD pods could land on nodes before the CNI is ready and get stuck in a networking failure that requires manual intervention.

Secrets in Git: SOPS and Age

Before handing off to ArgoCD, there's a practical problem: ArgoCD will need credentials. Database passwords, API tokens, private keys. Those have to live somewhere, and "somewhere" should be the same git repo as everything else.

The carlpett/sops provider was declared in versions.tf from the initial commit, alongside Talos and Helm. No secrets were needed at that point (the Talos provider generates its own machine secrets), but the encryption strategy was designed for the full Day 0 through Day 2 lifecycle.

The choice is SOPS with Age keys. Not Vault, not sealed-secrets. The reasoning is simple: SOPS is a local tool. There's no server to run, no controller that must be deployed before you can create your first secret. The decryption key is a single file. Encrypted values live in the same repo as the code that uses them, so git diff shows which secrets changed, git blame shows who changed them and when.

Two Keys, Two Purposes

The .sops.yaml at the repo root defines two creation rules:

creation_rules:
  - path_regex: environments/.*/secrets\.enc\.yaml$
    age: <personal-key>
  - path_regex: .*\.enc\.yaml$
    age: <personal-key>,<cluster-key>

The first key is mine, used locally during tofu apply to decrypt infrastructure secrets. The second is a cluster key that ArgoCD uses for runtime decryption in-cluster.

Infrastructure secrets (matched by the first rule) only need the personal key. Everything else, including gitops manifests that ArgoCD will consume, gets encrypted to both keys. One sops --encrypt call, two recipients, and the same file is decryptable both at my terminal and inside the cluster. If the cluster key is compromised, rotate it without touching the bootstrap key.

State Encryption (Separate Concern)

The OpenTofu state file is a separate encryption problem. It lives in Hetzner Object Storage (S3 backend) and contains every resource attribute, including computed values like kubeconfig contents. SOPS doesn't cover this; OpenTofu's native encryption block does:

encryption {
  key_provider "pbkdf2" "state" {
    passphrase = var.state_encryption_passphrase
  }
  method "aes_gcm" "state" {
    keys = key_provider.pbkdf2.state
  }
  state {
    method = method.aes_gcm.state
  }
}

PBKDF2 derives a key from a passphrase, AES-GCM encrypts the state at rest. The passphrase comes from an environment variable: password manager for local runs, Kubernetes Secret on the self-hosted runner.

Two encryption layers, two different scopes:

SOPS: encrypts secret values in git (credentials, tokens, private keys)
State encryption: encrypts the entire state file at rest (which contains all resource attributes, including things the Talos provider computes like kubeconfig contents)

Day 1 sets up both. Day 2 activates SOPS at runtime when ArgoCD starts decrypting secrets through KSOPS.

What This Layer Doesn't Touch

02-kubernetes/ stops once ArgoCD is running and has the credentials and encryption keys it needs to pull its first repo. The bootstrap — repository secrets, the KSOPS decryption setup, and the root Application that kicks off the GitOps loop — is the bridge between Day 1 and Day 2, and the subject of the next post.

The handoff point is clean: OpenTofu owns everything needed to run the cluster, ArgoCD owns everything that runs on top of it. The same boundary principle from Day 0, applied one layer up.

The Day 0 Bootstrapper: Replacing the OpenTofu Antipattern with a Proper Tool

2026-03-24T10:00:00+01:00

In the previous post I showed why using OpenTofu to bootstrap a bare-metal OS is the wrong tool for the job. The short version: the physical bootstrap sequence is imperative and one-directional: rescue mode, disk flash, reboot. OpenTofu is designed for declarative state backed by observable APIs. Forcing the two together with null_resource provisioners gives you something that works once and silently breaks everything after.

The clean boundary is the moment Talos starts listening on port 50000. From there, the siderolabs/talos provider gives you a real declarative API for machine configuration and cluster bootstrapping. Everything before that port opens belongs in a different kind of tool entirely.

This post is about that tool.

What It Needs to Do

The physical bootstrap sequence for a Hetzner dedicated server is:

Activate rescue mode via the Robot API
Reboot the server into rescue
Inspect disks and confirm before wiping
Flash Talos to the target disk
Reboot into Talos
Wait for the Talos API to answer on port 50000

It also needs to handle the case where a node is being reprovisioned: draining it from the cluster, wiping Talos state, and leaving it in maintenance mode for OpenTofu to re-apply the machine config.

None of this is declarative. It's a sequential script. So that's what it is: tools/talos-bootstrap.py, a Python script using the hetzner Robot API client, with two modes:

# Provision a fresh server
.venv/bin/python tools/talos-bootstrap.py --server-id 12345

# Drain, wipe, and re-provision a running node
.venv/bin/python tools/talos-bootstrap.py --server-id 12345 --reset

A full provision run looks like this:

$ .venv/bin/python tools/talos-bootstrap.py --server-id 12345

-- talos-bootstrap  server=12345  (203.0.113.189) --------------------

  [ok]   (203.0.113.189)

* Rescue mode
  - Rescue was already active -- re-activated to refresh password.
  - Password starts with 'Hn...'
  - Triggering hard reset...
  [ok] Reset triggered.

* Waiting for rescue SSH
  - Waiting for server to go down...
  - Waiting for SSH (rescue) .... [ok]

* Disk inspection

  Available disks:
    [0] /dev/nvme0n1  476.9G
    [1] /dev/nvme1n1  476.9G
  Select disk [0]: 1
  - Target: /dev/nvme1n1  (476.9G)
  [ok] /dev/nvme1n1 appears empty -- proceeding without confirmation.

* Flashing Talos v1.11.3 to /dev/nvme1n1
  - Source: https://github.com/siderolabs/talos/releases/download/v1.11.3/metal-amd64.raw.zst
  [ok] Written to /dev/nvme1n1.
  - Rebooting into Talos...

* Waiting for Talos API
  - Waiting for Talos API ... [ok]

-- done --------------------------------------------------------------

If any disk on the server has existing data, the confirmation step kicks in before anything is touched:

* Disk inspection

  Available disks:
    [0] /dev/nvme0n1  476.9G
    [1] /dev/nvme1n1  476.9G
  Select disk [0]: 1
  - Target: /dev/nvme1n1  (476.9G)

  [!] Existing data found on this server:
    nvme0n1  476.9G
      nvme0n1p2  1M
      nvme0n1p3  2G  [xfs]  label=BOOT
      nvme0n1p4  1M
    nvme1n1  476.9G
      nvme1n1p2  1M
      nvme1n1p3  2G  [xfs]  label=BOOT
      nvme1n1p4  1M

  Flashing Talos to /dev/nvme1n1 on a server with existing data.
  All data on /dev/nvme1n1 will be PERMANENTLY DESTROYED.

  Type the server ID (12345) to confirm:

[!!] Confirmation failed -- aborting.

Design Decisions Worth Explaining

Idempotency at the entry point

The first thing the provision flow does is check whether Talos is already answering on port 50000. If it is, there's nothing to do:

if port_open(server.ip, 50000):
    ok("Talos API already answering on port 50000, nothing to do.")
    print("  (Use --reset to drain and reprovision this node.)\n")
    sys.exit(0)

It's not full idempotency: if the server is mid-flash with no OS responding anywhere, the script will start from rescue again. But for the common case (re-running after a successful provision, checking a node that's already up), it exits cleanly rather than blowing up a running server.

Always re-activate rescue to get a fresh password

The hetzner package returns a cached rescue object. If rescue was activated in a previous session, server.rescue.password may return the password from that earlier activation, not necessarily the one the server actually booted with. Using a stale password means SSH authentication fails immediately (rc=5, permission denied).

The fix is to always call activate() regardless of whether rescue is already marked as active:

already_active = server.rescue.active
server.rescue.activate()
rescue_password = server.rescue.password
if already_active:
    info("Rescue was already active, re-activated to refresh password.")

Calling activate() on an already-active rescue resets it and returns a valid password. It's the only reliable way to get credentials you can actually use.

Hard reset, not soft

After rescue activation, the server needs to reboot into rescue mode. A soft reset (ACPI shutdown + boot) can hang if the current OS doesn't respond cleanly. For a bootstrap tool, we don't care about graceful shutdown. Hard reset every time.

Wait for SSH to go down before waiting for it to come back up

After triggering the reset, there's a window where the old OS is still running and port 22 is still open. If you immediately poll for SSH availability, you'll connect to the wrong system:

# Wait for the old SSH to disappear first
time.sleep(15)
deadline = time.time() + 120
while time.time() < deadline and port_open(server_ip, 22):
    time.sleep(5)
# Now wait for rescue SSH
if not wait_for_port(server_ip, 22, timeout=300, label="SSH (rescue)"):
    die("Timed out waiting for SSH.")

Without this, the script picks up the pre-reboot SSH session, authenticates with the rescue password against whatever was running before, and fails. The 15-second initial sleep gives the server time to begin shutting down before we start checking.

Disk safeguard checks all disks, not just the target

The wipe confirmation triggers if any disk on the server has existing data, not just the one selected as the flash target:

disks_with_data = [d for d in disks if disk_has_data(d)]
if disks_with_data:
    warn("Existing data found on this server:")
    for d in disks_with_data:
        print(fmt_disk(d, indent=2))
    print(f"\n  Flashing Talos to {dev} on a server with existing data.")
    answer = input(f"\n  Type the server ID ({server.number}) to confirm: ")

Installing on an empty second disk while there's data on the first is still reprovisioning a server with existing data. You should know about it before confirming. The confirmation requires typing the server ID, not just y: hard to misfire on a prod node.

The Reset Flow

The --reset flag handles reprovisioning a node that's still in the cluster. It:

Looks up the Kubernetes node name from the server's IP (or takes it via --node-name)
Drains the node (kubectl drain --ignore-daemonsets --delete-emptydir-data)
Removes it from the cluster (kubectl delete node)
Sends talosctl reset --graceful=false --reboot to wipe STATE and EPHEMERAL partitions
Waits for port 50000 to come back up in maintenance mode (Talos running, no machine config applied)

After that, tofu apply in 02-kubernetes/ re-applies the machine configuration and the node rejoins the cluster. The bootstrapper handles the physical lifecycle; OpenTofu handles the declarative configuration. The boundary stays clean.

If Talos isn't reachable at all (broken node, failed upgrade), the reset flow falls back to the full rescue bootstrap automatically.

What the Script Doesn't Do

It doesn't manage Talos upgrades. For a running node, talosctl upgrade handles version bumps without rescue mode: it downloads the new image and reboots in place. The bootstrapper is only for initial provisioning and full reprovisioning. If you're just upgrading a Talos version, don't reach for this.

It also doesn't apply machine configuration. That's OpenTofu's job. The script stops the moment Talos is listening on port 50000 (maintenance mode). From there, the siderolabs/talos provider takes over.

The Handoff

The 01-hardware/ directory in the reference repo still exists as an antipattern reference for the previous post, but nothing in it runs in production. The bootstrapper handles Day 0. When it exits, the cluster is exactly one tofu apply away from a configured, running node.

That tofu apply, the 02-kubernetes/ layer, is the subject of the next post.

The Bare-Metal Antipattern: Why Forcing OpenTofu to Bootstrap OS is a Trap

2026-03-11T10:00:00+01:00

Coming from a full cloud world where everything is an API, it felt natural to extend OpenTofu to cover our move to bare-metal. When we were automating the Turisapps migration — moving 850+ custom domains from DigitalOcean/Nomad to a bare-metal Kubernetes cluster on Hetzner — I wanted one tool, one state file, one tofu apply to go from empty server to running cluster.

The first red flag should have been that Hetzner doesn't even provide an official bare-metal provider — only a Cloud one. We had to rely on a community wrapper (silenium-dev/hetzner-robot) just to get OpenTofu to talk to the physical servers.

I ignored that flag. Here's what happened.

The Single-Tool Dream

The appeal was real. In cloud-land, tofu apply really does do everything: provision an EC2 instance, attach its EBS volume, configure its security groups, and register it with a load balancer. The entire lifecycle is API calls. If something is wrong, destroy and recreate. State matches reality because the cloud provider enforces it.

Bare metal is different. The physical server already exists — you've already paid Hetzner for it. There's no "create" API. There's just a server sitting in a rack, waiting to be told what to do.

But the tooling itch was strong. OpenTofu could talk to Hetzner Robot. OpenTofu could SSH. OpenTofu had null_resource. Surely we could wire it all together.

What "OpenTofu for Everything" Looks Like

Here's the actual code from the first attempt:

resource "hetzner-robot_boot" "rescue" {
  server_id        = var.server_id
  operating_system = "linux"
  architecture     = "64"
  active_profile   = "rescue"
}

data "external" "rescue_password" {
  depends_on = [hetzner-robot_boot.rescue]
  program = [
    "bash", "-c",
    "curl -sf -u '${var.robot_user}:${var.robot_password}' https://robot-ws.your-server.de/boot/${var.server_id}/rescue | python3 -c \"import json,sys; d=json.load(sys.stdin); print(json.dumps({'password': d['rescue']['password']}))\"",
  ]
}

resource "null_resource" "reboot_into_rescue" {
  depends_on = [hetzner-robot_boot.rescue]

  provisioner "local-exec" {
    command = <<-EOT
      curl -s -u '${var.robot_user}:${var.robot_password}' \
        -d 'type=sw' \
        https://robot-ws.your-server.de/reset/${var.server_id}
      sleep 60
      until nc -z ${var.server_ip} 22; do echo 'Waiting for SSH...'; sleep 10; done
    EOT
  }
}

resource "null_resource" "flash_talos" {
  depends_on = [null_resource.reboot_into_rescue]

  connection {
    type     = "ssh"
    user     = "root"
    password = data.external.rescue_password.result.password
    host     = var.server_ip
    timeout  = "10m"
  }

  provisioner "remote-exec" {
    inline = [
      "wget -O /tmp/talos.raw.zst https://github.com/siderolabs/talos/releases/download/v1.12.1/metal-amd64.raw.zst",
      "zstd -d -c /tmp/talos.raw.zst | dd of=/dev/nvme0n1 bs=4M status=progress",
      "sync"
    ]
  }
}

resource "null_resource" "reboot_into_talos" {
  depends_on = [null_resource.flash_talos]

  provisioner "local-exec" {
    command = <<-EOT
      curl -s -u '${var.robot_user}:${var.robot_password}' \
        -d 'type=sw' \
        https://robot-ws.your-server.de/reset/${var.server_id}
    EOT
  }
}

resource "null_resource" "wait_for_talos" {
  depends_on = [null_resource.reboot_into_talos]

  provisioner "local-exec" {
    command = "until nc -z ${var.server_ip} 50000; do echo 'Waiting for Talos API...'; sleep 10; done"
  }
}

It works. The first time.

In practice, getting even that first run to succeed required working around several provider bugs — the community silenium-dev/hetzner-robot provider returned empty values for both ipv4_address and password after rescue activation (a POST vs GET response parsing bug), had no reboot resource, and needed an external data source with an inline curl+python script just to read back the rescue password. Each workaround added another layer of imperative glue on top of the declarative facade. But eventually:

hetzner-robot_boot.rescue: Creating...
hetzner-robot_boot.rescue: Creation complete after 1s [id=2944755]
data.external.rescue_password: Reading...
null_resource.reboot_into_rescue: Creating...
null_resource.reboot_into_rescue: Provisioning with 'local-exec'...
null_resource.reboot_into_rescue (local-exec): Executing: ["/bin/sh" "-c" "curl -s -u '<robot_user>:<robot_password>' \
  -d 'type=sw' \
  https://robot-ws.your-server.de/reset/2944755
  ..."]
data.external.rescue_password: Read complete after 1s [id=-]
null_resource.reboot_into_rescue (local-exec): {"reset":{"server_ip":"203.0.113.189",...,"type":"sw"}}
null_resource.reboot_into_rescue (local-exec): Waiting for server to come back in rescue mode...
null_resource.reboot_into_rescue (local-exec): Connection to 203.0.113.189 port 22 [tcp/ssh] succeeded!
null_resource.reboot_into_rescue: Creation complete after 1m32s [id=...]
null_resource.flash_talos: Creating...
null_resource.flash_talos: Provisioning with 'remote-exec'...
null_resource.flash_talos (remote-exec): Connected!
null_resource.flash_talos (remote-exec): Downloading Talos...
null_resource.flash_talos (remote-exec): /tmp/talos.r 100% 188.71M  111MB/s in 1.7s
null_resource.flash_talos (remote-exec): Flashing NVMe...
null_resource.flash_talos (remote-exec): 4453302272 bytes (4.5 GB, 4.1 GiB) copied, 4.82233 s, 923 MB/s
null_resource.flash_talos: Creation complete after 8s [id=...]
null_resource.reboot_into_talos: Creation complete after 1s [id=...]
null_resource.wait_for_talos (local-exec): Connection to 203.0.113.189 port 50000 [tcp/*] succeeded!
null_resource.wait_for_talos: Creation complete after 55s [id=...]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

node_ip = "203.0.113.189"

Server flashed. Talos running. tofu apply succeeded.

Why It's a Trap

It's not idempotent.

Run tofu apply again on a healthy, running production server. OpenTofu sees null_resource.flash_talos in state. Nothing in the inputs changed. So it re-runs the provisioner — downloads Talos again, dds over the NVMe, reboots. Your production server is gone.

The only escape hatch is adding lifecycle { ignore_changes = all }. But at that point you've told OpenTofu to permanently ignore this resource. You've neutered the tool. You're no longer using infrastructure-as-code; you're using infrastructure-as-a-script-you-ran-once that happens to live in a .tf file.

State and reality are already decoupled.

OpenTofu state tracks API resource attributes. But there's no API to ask a physical server "what OS are you running?" before it's booted. The null_resource in state just records that the provisioner ran successfully. It has no idea if /dev/nvme0n1 actually contains Talos, or if the server rebooted cleanly, or if Talos came up healthy.

The polling hack has no exit.

until nc -z $IP 50000; do sleep 10; done

This runs on your local machine, blocking your terminal, with no timeout. If Talos never comes up — hardware failure, bad flash, wrong NVMe device — this loop runs forever. There's no error, no timeout, no recovery path. Kill it with Ctrl-C and OpenTofu doesn't even register a failure in state.

SSH mid-flight failures corrupt state.

The dd to flash the NVMe takes minutes. If the SSH connection drops during that window — network blip, rescue mode timeout, anything — OpenTofu marks null_resource.flash_talos as failed but the server might be 70% flashed. Now OpenTofu thinks the resource needs to be re-created, the server is in an unknown condition, and there's no clean recovery path short of manually intervening and wiping state.

The reboot timing hack.

nohup bash -c 'sleep 5 && reboot' > /dev/null 2>&1 &

Sleeping 5 seconds in the background so the SSH session can close cleanly before the reboot kills it. It works in practice. It also breaks silently when rescue mode is slow or sync takes longer than expected. This kind of timing hack has no business being in your infrastructure state.

The Root Cause

The fundamental mismatch is between declarative and imperative operations.

OpenTofu is designed for declarative infrastructure: describe the desired end state, and OpenTofu makes the API calls to get there. This works brilliantly for cloud resources because cloud providers model everything as observable, addressable state.

Flashing an OS onto a physical disk is imperative: it's a one-time action with side effects. There's no API to query afterwards. There's no way to declare "this disk should contain Talos v1.12.1" and have OpenTofu verify it.

When you use null_resource + provisioners to paper over this mismatch, you get something that looks like infrastructure-as-code but has none of its guarantees.

It Gets Worse on the Second Run

Running tofu apply a second time on the live server produced this:

hetzner-robot_boot.rescue: Modifying... [id=2944755]
hetzner-robot_boot.rescue: Modifications complete after 1s [id=2944755]
data.external.rescue_password: Read complete after 1s [id=-]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

No wipe, no reboot — so it looks safe. But look at what actually happened: the provider silently re-activated rescue mode on the running Talos server because it couldn't read its own state back correctly after the first apply (a bug in how the provider parses POST vs GET responses). The null_resource steps were skipped only because their IDs happened to be stable in state, not because OpenTofu verified Talos was still installed.

The server is now scheduled to boot into rescue mode on its next reboot. State says 1 changed, 0 destroyed and everything looks fine.

This is the real danger: not the dramatic wipe, but the silent inconsistency. The system looks healthy in state while the actual server is in a configuration that will fail the next time it restarts.

"Just Fix the Provider" Is the Wrong Answer

At this point you might think: patch the community provider, fix the POST response parsing, get password and ipv4_address back properly, and everything works. But that path leads somewhere worse.

The provider bug is a symptom, not the disease. Even a perfectly working provider can't solve the fundamental problem: bare metal bootstrapping is an imperative sequence of side effects. Rescue boot → reboot → flash → sync → reboot → wait. There is no desired end state to declare. There is no API that can tell OpenTofu "the NVMe currently contains Talos v1.12.1." You can make the provider return the right values, but you still end up with null_resource provisioners and local-exec curl scripts holding the whole thing together — a pipeline of shell commands wearing an OpenTofu costume.

You'd have all the fragility of a bash script with none of the debuggability, and the false reassurance of a state file that doesn't reflect reality.

The Fix: Know Where OpenTofu's Job Ends

The second run proved the point better than any contrived example could. Re-running tofu apply on a live Talos server silently re-activated rescue mode — because that's what's in state, and OpenTofu dutifully tries to maintain it. The rescue activation is a transient step in an imperative sequence, not a desired end state. Putting it in OpenTofu state means OpenTofu will try to enforce it permanently.

The entire physical bootstrap process — rescue activation, reboot, OS flash, reboot into Talos, wait for the API — is a one-way imperative sequence. None of it belongs in OpenTofu state. There is no meaningful "desired state" for any of these steps once the server is running.

The rule: if the desired state can't be observed through an API, don't put it in OpenTofu.

The clean boundary is the moment Talos starts listening on port 50000. From there, the siderolabs/talos provider gives you a proper declarative API for machine configuration and cluster bootstrapping. Everything before that port opens belongs in a dedicated bootstrapper: a purpose-built script that owns the physical lifecycle explicitly, runs once, and hands off to OpenTofu when there's an actual API to talk to.

That bootstrapper, and how it replaces everything in 01-hardware/, is the subject of the next post in this series.

What I’ve Been Building These Past Five Years

2025-09-13T00:00:00+02:00

It’s been a long time since my last post, and quite a lot has happened since launching Nubosas.

After a brief period without a project, I began a series of engagements as a Nubosas contractor.

The first was with Revieve, where I led the migration of two AWS accounts from console-managed infrastructure to Terraform-managed environments. I imported all existing resources, extended the setup with automation, and built CI/CD pipelines using GitHub Actions. The project also included deploying a MongoDB Atlas cluster to support new applications.

Next, I joined a still-active engagement with a Fortune 500 company, working on their Global Data Foundation team as a DataOps engineer. My focus has been supporting their AWS infrastructure and guiding the migration to Databricks on Azure, enabling a modern, scalable data platform across global regions.

In between these major projects I also handled a smaller migration from a single on-premises server to the cloud—proof that even modest workloads benefit from strong automation and careful planning.

These experiences have reinforced what I value most: building reliable, automated infrastructure that frees teams to innovate.

Today, through Nubosas, I focus on helping scaling SaaS businesses graduate from expensive public clouds (AWS/Azure/GCP) to their own highly automated, Private Sovereign Clouds using bare-metal Kubernetes. It brings Fortune 500 reliability without the cloud tax.

I’ll use this blog to share occasional technical insights and lessons learned. If you’d like to discuss a project or connect, you can reach me through Nubosas, on LinkedIn, or on X (Twitter).

Good practices: Minimum Docker Image Size

2020-04-29T09:00:00+02:00

Docker container are becoming more and more popular over time. It's a really simple way to encapsulate your application code with its dependencies and leave it ready to be deployed into different environments.

One of its key features are that it allows you to test your application together with the operating system that will be used to deploy it. The implication of this is that your tests will not only validate that your app updates don't break your tests, but also make quite simple to upgrade the operating system dependencies and validate it with regular app tests.

Let's say we have a Python project which uses PostgreSQL with this set of dependencies in a requirements.txt file:

psycopg2==2.8.5
requests==2.23.0

A simple Dockerfile for that app would be:

FROM python:3.8-slim

WORKDIR /app

RUN apt-get update && apt-get install -y --allow-unauthenticated \
    build-essential \
    libpq-dev

COPY run_app.sh run_app.sh
COPY my_app.py my_app.py
COPY requirements.txt requirements.txt

# Install app dependencies.
RUN pip install -r requirements.txt

CMD ["/app/run_app.sh"]

Which would produce an image size of 357MB starting from a base Python 3.8 image of 113MB. That means that each deployment needs to download 357MB from your docker registry to your production environment. Imagine how big would it become as your dependencies grow, specially when you start using external libraries or modules that need to be compiled, requiring dev libraries to create the container.

Is there anything we could do to reduce that size?

Yeah, and since the multi stage functionality addition in Docker is quite simple!

The way to approach this optimisation is to use one container only for the build and dependencies installation and then, in a fresh and new container, just copy the files you are interested on.

The way you implement this split depends on the technologies being used to run the service, in our case for this article, we use a Python based application, so we need a way to easily copy the installed dependencies. There are two different ways to achieve it: * Use pip to install all dependencies into an specific directory. * Use a virtualenv so the whole installation is isolated, included bin scripts.

The approach used on this article is the virtualenv one, but it's just a personal preference for simplicity.

A possible implementation would have first, a base stage with the common rules reused in different stages:

FROM python:3.8-slim as base

# PYTHONUNBUFFERED: https://docs.python.org/3.8/using/cmdline.html#envvar-PYTHONUNBUFFERED
ENV PYTHONUNBUFFERED 1
ENV VIRTUAL_ENV /srv/venv
ENV PYTHONDONTWRITEBYTECODE 1

WORKDIR /app

RUN pip install virtualenv

RUN apt-get update && apt-get install -y --allow-unauthenticated \
    libpq5

It basically initialises the environment, sets the final workdir, installs the virtualenv package that will used in the following stages and finally installs library dependencies required by our app, in this case, libpq5 for PostgreSQL.

Next stage would be used only for the build, it will include all development dependencies. It will start from previous base stage:

FROM base as compile-image

RUN apt-get update && apt-get install -y --allow-unauthenticated \
    build-essential \
    libpq-dev && \
    python -m virtualenv ${VIRTUAL_ENV}

ENV PATH "${VIRTUAL_ENV}/bin:${PATH}"

# Install app dependencies.
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt

To install the runtime dependencies for our app, it creates a virtualenv, which is used later to install the requirements.txt dependencies with pip. To simplify the python command execution, the PATH environment variable is updated to use the virtual env's commands by default.

Finally, is time to create the final image with just our application code and the required final libraries:

FROM base as build-image
ENV PATH "${VIRTUAL_ENV}/bin:${PATH}"

COPY --from=compile-image ${VIRTUAL_ENV} ${VIRTUAL_ENV}
COPY run_app.sh run_app.sh
COPY my_app.py my_app.py

CMD ["/app/run_app.sh"]

As you can see, this stage is copying the virtual environment we created in the previous stage which would have the final installed modules required to run our app and the code for our app. The postgres dependency requires libpq library to work, but we already have it installed from the base image, which allows us to share that installation between the dependencies installation stage and this final container stage.

Once this last stage is built, we end having a container of 157MB which is near 2.30 times smaller than the original container.