Skip to main content

Application delivery with KubeVela — installed, defined, not yet load-bearing

This page answers a question the repo layout raises but no single section answers: the cluster runs KubeVela, so why does every app look like plain Kustomize? The short version — and this page exists to keep the docs honest about it — is that KubeVela is installed and configured but not yet delivering a single application. The controller runs, two custom TraitDefinitions exist, and there are zero Application resources anywhere in Git. Every one of the ~46 workloads on the talos cluster and all three on the edge cluster ships through the Kustomize base/overlay pattern described in components/.

The page covers both delivery models side by side, what the KubeVela layer actually consists of today, and what would have to happen before an app could move over. The actors:

ActorWhere it lives in docs
Flux (delivers both models)foundation/flux
Kustomize bases, overlays, components, shared resourcescomponents/
KubeVela / vela-core (the OAM controller)platform/kubevela
Envoy Gateway (what the httproute-int trait targets)platform/envoy-gateway
http-internal (the Kustomize equivalent of that trait)components/http-internal
TrueNAS NFS shares (what the truenas-volume trait targets)components/storage
cert-manager (issues the admission webhook cert)platform/cert-manager

The two models, side by side​

Both models end at the same place: raw Kubernetes resources reconciled into the cluster. The difference is who renders them — kustomize build at commit time versus the vela-core controller expanding CUE templates at apply time.

What ships apps today: Kustomize, three layers deep​

The pattern is uniform across the repo and is the one the GitOps flow page walks end to end:

  1. k8s/apps/base/<app>/ — the cluster-agnostic core: a Deployment, a Service, a kustomization.yaml. Nothing about networks, storage classes, or hostnames.
  2. k8s/apps/<cluster>/<app>/ — the per-cluster overlay (talos/ for almost everything, edge/ for the three internet-facing workloads). This is where the app gains a namespace, an HTTPRoute (or the http-internal variant), PVCs, network policies, SOPS-encrypted secrets, and components: references into k8s/components/<cluster>/ for cross-cutting behaviors — backups, CNPG databases, valkey, workspace.
  3. k8s/resources/<cluster>/ — shared cluster resources referenced by many overlays. The relevant one here is k8s/resources/talos/storage/<share>/, which defines a pre-bound PersistentVolume/PersistentVolumeClaim pair per TrueNAS NFS export (archive, audio, downloads, gaming, images, reading, stash, videos) against 192.168.104.200:/mnt/zima/<share> — see components/storage.

Delivery is per-app: each entry in k8s/clusters/talos/ pairs a Flux Kustomization with a cosign-verified OCIRepository (e.g. oci://gitea.web.kueber.eu/johnny/oci-talos-app-jellyfin, tag edge), gated on dependsOn: infra-configs. Forty-six app entries follow this shape on talos alone.

This model has no abstraction layer — an app's overlay says kind: HTTPRoute and kind: PersistentVolumeClaim in plain YAML — but it also has no runtime indirection: what kustomize build prints is exactly what the cluster runs, which is what makes the CI dry-run in the GitOps flow meaningful.

What KubeVela adds: the installed layer​

KubeVela runs only on the talos cluster (there is no kubevela entry in k8s/clusters/edge/). Two Flux paths deliver it:

The controller — k8s/clusters/talos/kubevela.yaml points a Flux Kustomization at k8s/platform/talos/controllers/kubevela/, which layers the base HelmRelease (vela-core 1.11.0 from https://kubevela.github.io/charts, namespace kubevela-system) with a values ConfigMap. Notable, all verifiable in kubevela-values.yaml and helm-patch.yaml:

  • Multicluster is disabled — the values file comments: "No joined clusters — the cluster-gateway APIService only serves remote-cluster management and its helm-hook-patched caBundle repeatedly wedged upgrades."
  • The admission webhook is enabled with failurePolicy: Fail, certificates via cert-manager. Flux driftDetection deliberately ignores the caBundle fields on the validating/mutating webhook configurations and on the v1alpha1.cluster.core.oam.dev APIService, because vela injects those at runtime and the chart's Cg== placeholder is rejected by Kubernetes ≥ 1.36.
  • Built-in definitions (webservice, k8s-objects, …) are enabled.
  • VelaUX (the dashboard) is enabled with one replica, but as a ClusterIP service with ingress: enabled: false and no HTTPRoute anywhere in Git — reaching it means kubectl port-forward. It has deliberately not been wired into Envoy Gateway.

The custom definitions — k8s/platform/talos/configs/kubevela/ ships two TraitDefinitions into kubevela-system, applied by the same infra-configs Kustomization that carries all platform configuration. There are no custom ComponentDefinitions — only the chart's built-ins.

httproute-int — ingress as a trait​

A CUE template that, attached to a Deployment/StatefulSet/DaemonSet component, emits an HTTPRoute for the component's Service. It is the OAM mirror of the http-internal Kustomize component. Its current state betrays its age:

outputs: httproute: {
  apiVersion: "gateway.networking.k8s.io/v1"
  kind:       "HTTPRoute"
  spec: {
    parentRefs: [{
      name:      "twingate"            // ← this Gateway no longer exists
      namespace: "envoy-gateway-system"
    }]
    ...

The trait hard-codes a parent Gateway named twingate — a leftover from the pre-NetBird era. The Gateways actually defined in k8s/platform/talos/configs/envoy-gateway/ today are internal and public. The trait also declares servicePort and paths parameters that the template never uses (the path match is hard-coded to /). Any Application using this trait as-is would produce an HTTPRoute that attaches to nothing.

truenas-volume — NFS media shares as a trait​

The more interesting one: given a share name (the same eight-share enum as k8s/resources/talos/storage/) and a list of mount points, it emits a pre-bound PV/PVC pair and patches the workload's volumes/volumeMounts in one declaration — collapsing what the Kustomize model spreads across a storage/ sub-kustomization, a namePrefix, and a manual patch in the app overlay. It is a genuinely better ergonomic for the *arr/media class of apps.

It has also drifted: the trait pins the NFS server to 192.168.100.200, while every PV under k8s/resources/talos/storage/ — the ones actually mounted by Jellyfin, stash, and the rest — points at 192.168.104.200. Same ZFS box, different subnet than when the trait was written.

How KubeVela coexists with Flux​

There is no conflict between the two, by construction:

  • Flux delivers KubeVela itself — the HelmRelease and the TraitDefinitions are ordinary GitOps-managed resources, renovated and drift-detected like every other controller in platform/.
  • If an Application existed, Flux would deliver that too. The division of labor is: Flux applies the Application CR from Git; vela-core expands it into child resources and tracks them with its own resource tracker. Flux only prunes objects it applied itself, so vela-owned children are invisible to Flux's garbage collection, and vice versa.
  • Nothing depends on KubeVela. The infra-configs Kustomization's dependsOn list has its kubevela entry commented out — the platform config layer, and therefore every app, becomes ready without waiting for vela-core to be healthy. If the controller wedged tomorrow, no deployment would notice.

That last point is the honest summary of adoption: the platform was installed and shaped (multicluster trimmed off, webhooks integrated with cert-manager, upgrade-wedging caBundles worked around), two traits were prototyped against real pain points (internal ingress, NFS media mounts) — and then delivery stayed on Kustomize, where CI can kustomize build and server-side dry-run every manifest before merge. A CUE template that renders in-controller has no equivalent pre-merge check in the current pipeline.

What can break, and where to look​

SymptomMost likely causeWhere to look first
HelmRelease/kubevela not ready, upgrade loopwebhook/APIService caBundle drift patterns no longer match the chart's object nameshelm-patch.yaml under k8s/platform/talos/controllers/kubevela/
Pods rejected cluster-wide on applyvela admission webhook down with failurePolicy: Failkubevela-system pods; the webhook only intercepts OAM types, so blast radius is limited to those
infra-configs fails on the trait manifestsvela-core CRDs missing (the dependsOn: kubevela gate is commented out)k8s/clusters/talos/infrastructure.yaml; flux logs --kind=Kustomization
A future Application deploys but its route attaches nowherehttproute-int still points at the removed twingate Gatewayk8s/platform/talos/configs/kubevela/trait-httproute-int.yaml
A future Application mounts an empty/hanging NFS volumetruenas-volume pins the stale 192.168.100.200 server addressk8s/platform/talos/configs/kubevela/trait-truenas-volume.yaml vs k8s/resources/talos/storage/
VelaUX unreachableintentional — ClusterIP, no ingress, no HTTPRoutekubectl port-forward into kubevela-system

If an app ever moves over​

Not a plan, just the checklist this page's findings imply: fix the two drifted values in the traits (Gateway name, NFS server), pick one media app whose overlay is mostly storage-plus-route boilerplate, express it as an Application committed to Git next to its siblings, and give CI a way to validate CUE-rendered output (e.g. vela dry-run) so the GitOps flow's pre-merge guarantees survive the abstraction. Until then, KubeVela remains what the repo shows: a well-groomed runway with no planes on it.