The Day 1 post ended with ArgoCD deployed as a Helm release inside 02-kubernetes/. It's running, but it can't do anything yet. It has no repository credentials, no decryption keys, and no root Application to tell it what to manage. "Running" is not "working."
This post covers the bootstrap resources that bridge Day 1 and Day 2: the credentials, the KSOPS decryption setup, and the root Application that kicks off the GitOps loop. Once that loop starts, OpenTofu steps back.
The Chicken-and-Egg Problem
ArgoCD manages applications from git repositories. To pull from a private repo, it needs credentials. To decrypt SOPS-encrypted manifests, it needs the cluster Age key. But those credentials and keys are themselves secrets that need to live somewhere.
The solution is straightforward: OpenTofu bootstraps the credentials as Kubernetes Secrets before ArgoCD tries to use them. ArgoCD doesn't manage its own credentials, at least not at bootstrap time. OpenTofu creates them, ArgoCD consumes them, and the two never overlap.
Everything below lives in argocd-bootstrap.tf.
Repository Credentials
Two private repos need to be registered: the platform gitops repo (controlled by Nubosas) and the tenant workload repo (controlled by the tenant). Both use a dedicated service account with repo-scoped GitHub tokens.
The tokens are stored SOPS-encrypted in secrets.enc.yaml and decrypted at apply time via the carlpett/sops provider that was declared back in Day 1:
data "sops_file" "secrets" {
source_file = "${path.module}/environments/${var.environment}/secrets.enc.yaml"
}
Each repo is registered as a Kubernetes Secret with the label ArgoCD watches for:
resource "kubernetes_secret" "argocd_repo_platform" {
metadata {
name = "repo-platform-gitops"
namespace = "argocd"
labels = {
"argocd.argoproj.io/secret-type" = "repository"
}
}
data = {
type = "git"
url = "https://github.com/example-org/platform-gitops.git"
username = "example-bot"
password = data.sops_file.secrets.data["github_token_platform"]
}
depends_on = [helm_release.argocd]
}
Same pattern for the tenant repo, different token. The depends_on ensures ArgoCD is running before we create secrets in its namespace.
This is the Day 1 encryption strategy paying off. The sops provider was declared but unused in the initial commit because no secrets were needed for Talos bootstrap. Now it's active, decrypting at apply time with the personal Age key.
KSOPS: Decrypting Secrets at Runtime
OpenTofu decrypts secrets at apply time. But ArgoCD needs to decrypt SOPS-encrypted manifests at sync time, inside the cluster, without the personal key.
That's where KSOPS comes in. It's a Kustomize plugin that intercepts SOPS-encrypted resources during ArgoCD's render phase and decrypts them using the cluster Age key.
ArgoCD's container image doesn't ship with KSOPS. The standard approach is an init container that downloads the binary into a shared volume before the repo-server starts.
The Init Container
repoServer:
initContainers:
- name: install-ksops
image: alpine:3.21
command: ["sh", "-c"]
args:
- |
set -e
wget -qO- https://github.com/viaduct-ai/kustomize-sops/releases/download/v4.4.0/ksops_4.4.0_Linux_x86_64.tar.gz \
| tar xz -C /custom-tools
chmod +x /custom-tools/ksops
mkdir -p /custom-tools/plugin/viaduct.ai/v1/ksops
cp /custom-tools/ksops /custom-tools/plugin/viaduct.ai/v1/ksops/ksops
volumeMounts:
- name: custom-tools
mountPath: /custom-tools
securityContext:
runAsNonRoot: true
runAsUser: 65534
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
seccompProfile:
type: RuntimeDefault
Two things worth calling out. First, the binary gets laid down twice: once as a flat ksops for Kustomize to invoke directly, and once inside plugin/viaduct.ai/v1/ksops/ksops so Kustomize can also discover it as an exec plugin via KUSTOMIZE_PLUGIN_HOME. Different code paths in Kustomize look for it in different places.
Second, the security context. This init container downloads and unpacks a single binary, so it has no reason to run as root, hold any capabilities, or escalate privileges. runAsUser: 65534 is nobody. The seccompProfile: RuntimeDefault restricts syscalls to the container runtime's default set. Defense in depth for a container that runs for two seconds, but it costs nothing.
Mounting the Binary and the Key
The KSOPS binary goes into /usr/local/bin/ksops on the repo-server via the shared volume, plus the plugin tree gets mounted under /home/argocd/ksops-plugin. The cluster Age key gets mounted as a directory so SOPS finds it at the path it expects:
repoServer:
volumes:
- name: custom-tools
emptyDir: {}
- name: sops-age-key
secret:
secretName: ksops-age-key
defaultMode: 292 # 0444 — readable by argocd user (uid 999)
volumeMounts:
- name: custom-tools
mountPath: /usr/local/bin/ksops
subPath: ksops
- name: custom-tools
mountPath: /home/argocd/ksops-plugin
subPath: plugin
- name: sops-age-key
mountPath: /home/argocd/.config/sops/age
readOnly: true
env:
- name: SOPS_AGE_KEY_FILE
value: /home/argocd/.config/sops/age/age.agekey
- name: XDG_CONFIG_HOME
value: /home/argocd/.config
- name: KUSTOMIZE_PLUGIN_HOME
value: /home/argocd/ksops-plugin
Three env vars matter here. SOPS_AGE_KEY_FILE tells SOPS exactly where the private key lives. XDG_CONFIG_HOME keeps SOPS's config search rooted under the argocd user's home. KUSTOMIZE_PLUGIN_HOME is what makes Kustomize discover KSOPS as an exec plugin. Miss any of these and the symptom is the same: opaque "decryption failed" or "plugin not found" errors at sync time.
The Age key Secret is created by OpenTofu alongside the repo credentials:
resource "kubernetes_secret" "ksops_age_key" {
metadata {
name = "ksops-age-key"
namespace = "argocd"
}
data = {
"age.agekey" = data.sops_file.secrets.data["age_cluster_private_key"]
}
depends_on = [helm_release.argocd]
}
Notice: the cluster private key is itself SOPS-encrypted in secrets.enc.yaml, decrypted by OpenTofu using the personal key, then stored as a Kubernetes Secret for KSOPS. SOPS decrypting a key that enables more SOPS decryption.
Finally, Kustomize needs the alpha plugins flag to recognize KSOPS. This goes under configs.cm (the argocd-cm ConfigMap), not configs.params (the argocd-cmd-params-cm ConfigMap that drives CLI flags):
configs:
params:
"server.insecure": true
cm:
"kustomize.buildOptions": "--enable-alpha-plugins --enable-exec"
Easy to put in the wrong block — configs.params looks like the natural home for a "build option" but ArgoCD reads kustomize.buildOptions from the argocd-cm ConfigMap. Wrong block, silent no-op.
The Three-Layer Decryption Flow
Day 1 set up the two-key design. Here's how it completes:
-
At rest in git: secrets are encrypted with SOPS using Age public keys. Both the personal key and cluster key are recipients, so either can decrypt.
-
At bootstrap (tofu apply): the carlpett/sops provider decrypts secrets.enc.yaml using the personal Age key (available locally). The decrypted values become Kubernetes Secrets.
-
At runtime (ArgoCD sync): when ArgoCD reconciles a repo containing SOPS-encrypted manifests, KSOPS intercepts the render, reads the cluster Age key from the mounted Secret, and decrypts inline. The plaintext manifest is applied to the cluster. Plaintext never touches git.
Two keys, three layers, one encryption strategy designed from Day 1.
The Root Application
With credentials and decryption in place, the last step is telling ArgoCD what to manage. This is the App of Apps pattern: a single root Application that watches a directory for other Application definitions.
# root-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/example-org/platform-gitops.git
targetRevision: HEAD
path: apps
directory:
recurse: true
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
directory.recurse: true is important. The apps/ directory uses subdirectories (apps/platform/, apps/tenants/) to organize Application definitions. Without recursion, ArgoCD only scans the top level and misses everything in subdirectories. I initially had this flat and had to add recursion when the directory structure grew.
prune: true means ArgoCD deletes resources that no longer exist in git. selfHeal: true means it reverts any manual changes. Together, they enforce that git is the single source of truth.
Why null_resource Is Ugly but Right
There's no clean way to apply a custom resource through the OpenTofu Kubernetes provider and have ArgoCD manage it afterwards. The kubernetes_manifest resource could create the Application, but then both OpenTofu and ArgoCD would be managing the same resource, fighting over the source of truth. The root Application needs to be created once and then owned entirely by ArgoCD.
The pragmatic answer is null_resource with local-exec:
resource "local_sensitive_file" "kubeconfig" {
content = talos_cluster_kubeconfig.cluster.kubeconfig_raw
filename = "${path.module}/.kubeconfig"
}
resource "null_resource" "argocd_root_app" {
triggers = {
once = "bootstrap"
}
provisioner "local-exec" {
command = "kubectl apply -f ${path.module}/root-app.yaml"
environment = {
KUBECONFIG = local_sensitive_file.kubeconfig.filename
}
}
depends_on = [
helm_release.argocd,
kubernetes_secret.argocd_repo_platform,
kubernetes_secret.argocd_repo_tenant,
kubernetes_secret.ksops_age_key,
]
}
triggers = { once = "bootstrap" } is a static value. It triggers on the first tofu apply and never again, because the trigger value never changes. Subsequent applies see the null_resource in state with the same trigger and skip it. This is a fire-and-forget bootstrap step.
The depends_on list is the full dependency chain: ArgoCD must be running, both repo credentials must exist, and the KSOPS key must be in place. Only then does kubectl apply create the root Application. After that, ArgoCD takes over.
Destroying the null_resource from OpenTofu state does NOT delete the ArgoCD Application from the cluster. It's a one-way handoff.
Multi-Tenancy: AppProject as the Authorization Boundary
Once the root Application syncs, it picks up everything in apps/, including the tenant onboarding resources. The natural question at this point: what stops a tenant from deploying into kube-system or another tenant's namespace?
The answer is the AppProject CRD. Every Application references a project, and the project defines what that Application is allowed to do.
The tenant AppProject:
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: tenant-a
namespace: argocd
spec:
description: Tenant workloads
sourceRepos:
- https://github.com/example-tenant/tenant-gitops.git
destinations:
- server: https://kubernetes.default.svc
namespace: tenant-a
clusterResourceWhitelist: []
namespaceResourceBlacklist:
- group: ""
kind: ResourceQuota
- group: ""
kind: LimitRange
orphanedResources:
warn: true
sourceRepos locks the tenant to their own repo. destinations restricts deployment to the tenant-a namespace only. clusterResourceWhitelist: [] blocks all cluster-scoped resources (Namespaces, ClusterRoles, CRDs). namespaceResourceBlacklist prevents the tenant from overriding the ResourceQuota and LimitRange that the platform manages.
If the tenant pushes a Deployment targeting kube-system to their gitops repo, the ArgoCD application controller rejects it at sync time. No workaround.
The Default Project Gap
The tenant AppProject was configured correctly. But I noticed the default project (which ships with ArgoCD) was wide open: sourceRepos: ['*'], destinations unrestricted, full cluster-scoped access.
Both the root Application and the tenant onboarding Application use project: default, which is fine since they source from the platform-controlled repo. The problem is lateral: if anyone creates an Application under project: default (misconfiguration, a future templating mistake, direct kubectl access), they get unrestricted access to the entire cluster from any repo.
The fix: lock the default project's sourceRepos to the platform repo only.
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: default
namespace: argocd
spec:
description: Platform-managed resources only
sourceRepos:
- https://github.com/example-org/platform-gitops.git
destinations:
- server: https://kubernetes.default.svc
namespace: '*'
clusterResourceWhitelist:
- group: '*'
kind: '*'
Still allows all destinations and cluster-scoped resources (the platform needs full control), but any Application in project: default can only pull from code Nubosas controls. Defense in depth for a gap that might never be exploited.
This lives in platform/default-project.yaml and gets deployed by a platform-config Application that the root app picks up automatically. The pattern scales: any future platform-level resource (NetworkPolicies, additional AppProject restrictions) goes into platform/ and is synced through the same path.
There's more to multi-tenancy security (Cilium NetworkPolicies for runtime network isolation, ArgoCD RBAC if tenants ever get UI access), but those are separate concerns for a future post.
The Full Dependency Chain
From Day 1 through Day 2, the dependency chain is explicit:
Talos machine secrets + node configs (Day 1)
|
talos_machine_bootstrap (etcd init)
|
talos_cluster_kubeconfig (client certs)
|
helm_release.cilium (CNI -- pods can schedule)
|
--- Day 2 starts here ---
|
helm_release.argocd (ArgoCD running)
|
kubernetes_secret.argocd_repo_* (repo credentials)
kubernetes_secret.ksops_age_key (decryption key)
local_sensitive_file.kubeconfig (kubectl access)
|
null_resource.argocd_root_app (kubectl apply root-app.yaml)
|
--- OpenTofu stops here ---
|
root Application auto-syncs apps/ from platform-gitops
|
platform-config Application --> deploys platform/ (locked-down default project)
tenant-onboarding Application --> deploys namespace, quota, AppProject
tenant Application --> deploys workloads from tenant-gitops
One tofu apply gets you from Talos API to a fully operational GitOps loop. After that, OpenTofu manages the cluster infrastructure (Talos configs, Cilium, ArgoCD itself) and ArgoCD manages everything that runs on top of it.
The boundary is the same principle from Day 0, applied one layer up: each tool owns what it can observe and control through an API. OpenTofu talks to the Talos and Kubernetes APIs. ArgoCD talks to git and the Kubernetes API. They don't overlap.
What's Next
The GitOps loop is running. Workloads can deploy. But the cluster still needs storage for stateful applications, which is where Rook-Ceph comes in. That's the subject of the next post.