Archivist

Every account you ever had, on one timeline — built from the exports the platforms owe you.

Archivist is a first-party, multi-user archive of external accounts. It ingests the official data-export ZIPs the platforms hand out (Reddit, ChatGPT, Claude, Twitter, Instagram, Facebook, Google Takeout, WhatsApp, TikTok, Telegram, Snapchat, Spotify, LinkedIn, Discord), normalises every record into one canonical event and shows it all on a searchable, filterable timeline with facets, a per-day histogram and the attachments inline. Go backend, SvelteKit frontend embedded in the binary, PostgreSQL for the events, an S3-compatible store for the original ZIPs and the extracted media. Every import ends with a coverage report of what was mapped and what landed in <source>.unknown, and because the original archive is preserved write-once, a later provider improvement reaches old data through Reprocess.

Alternatives considered

Self Hosted

Tool	Open Source	Full Features	Notes
Timeliner	Yes	Partly	Single-user CLI that pulls live APIs with per-service OAuth keys; superseded by Timelinize
Timelinize	Yes	Partly	Desktop app around one local database per person; no multi-user server, no gateway in front
The platforms' own "download your data" pages	No	Partly	The exports exist anyway; without Archivist they are a pile of ZIPs with fourteen different schemas

Archivist takes the export ZIP as the sole ingestion path: it is the one artefact every platform is obliged to hand over, it needs no API keys or OAuth pulls, and the archive stays complete after the account is closed.

Installation

Architecture

• Deployment: Single archivist deployment in the archivist namespace, one replica (in-process import worker, in-memory rate limits), Recreate strategy, container port 8080 with /readyz (database and blob store included) and /healthz probes
• Image: gitea.web.kueber.eu/johnny/archivist, a release version pinned by digest and pulled with the regcred-docker-gitea pull secret (namespace label secrets.docker-gitea); the archivist repo's release workflow builds, pushes and signs it on a version tag, there is no upstream chart
• Database: CNPG PostgreSQL cluster from the cnpg component, 10Gi; DATABASE_URL is the uri key of the generated cnpg-app Secret and MIGRATE=1 applies the goose migrations on boot
• Object storage: a SILO StatefulSet from the shared minio component with a 50Gi longhorn-object-encrypted volume; the server targets it with BLOB_S3_* on the cluster-local minio service and creates the archivist bucket itself
• Staging: DATA_DIR is the 20Gi archivist-staging PVC (longhorn-object-single-encrypted); uploaded ZIP parts wait there until the worker has preserved them in the bucket
• Networking: ClusterIP service on 8080, HTTPRoute on the internal gateway at archivist.int.kueber.eu; TRUSTED_PROXIES=1 for the single Envoy hop. Login is the app's own OpenID Connect against Keycloak, no gateway policy in front
• Resources: app requests 50m CPU and 256Mi with a 1Gi limit and GOMEMLIMIT=900MiB; SILO 192Mi with a 384Mi limit; CNPG 256Mi with a 512Mi limit; all three tier-2

Security

• Runs as uid 10001 (the image's app user), runAsNonRoot: true, read-only root filesystem with an emptyDir at /tmp, all capabilities dropped, RuntimeDefault seccomp, no service account token; SILO runs as uid 1000 with the same drops
• Login through Keycloak (authorization code with PKCE, ID token signature, issuer, audience, expiry and nonce verified): only users carrying the archivist.user role (mapped to the family group) pass the client's flow, and each Keycloak identity is linked to one Archivist account of its own (opaque server-side sessions, __Host- cookies, login throttle in Postgres; password login remains for accounts that set one, Argon2id). Tenant isolation is enforced in every repository method
• Ingress only from the internal gateway's Envoy pods; no egress beyond the app-network-policy baseline — the Keycloak discovery and token exchange ride its gateway allowance, and the app never calls a platform
• Untrusted archives are vetted before they are opened (central-directory and entry-count checks, per-member and total byte limits, bounded JSON nesting), streamed rather than extracted, and staged through os.Root
• The CNPG, SILO and staging volumes are Longhorn-encrypted with keys kept next to the respective component (cnpg/, minio/); the MinIO root credentials, the restic password and the workspace encryption key are SOPS-encrypted, the Keycloak client secret arrives in workspace-outputs
• Kyverno's gitea-johnny cosign policy lists this image among the first-party builds that must be signed with the homelab key

Updates

A new version is a signed tag in the archivist repo; its release workflow publishes the image and Renovate turns it into a version bump plus digest move in the manifest, auto-merged overnight. SILO and the CNPG image are digest-pinned through Renovate like every other component. Migrations are forward-only and run on boot.

Data Management

• Database: CNPG cluster, pg_dump via the backups-cnpg PreBackupPod
• Objects: the SILO data volume is annotated k8up.io/backup: "true" by the component; it holds the preserved original ZIPs and all extracted media, so it is primary data and the backup runs as uid 1000 to read it
• Staging: not backed up; an upload that has not been preserved yet is re-uploadable
• Backups: k8up Schedules to Hetzner and Backblaze via restic, bucket suffix -archivist-bf6e20b8
• Deleting an import removes its events, archive and job; deleting an account removes staged uploads, blobs and the user row with its cascades

User Management

Keycloak decides who may log in (client archivist, realm kueber.eu, created by the app's workspace with spec.oidc, gated on the family group, callback /api/auth/oidc/keycloak/callback). The first Continue with Keycloak creates the Archivist account, or attaches to the password account with the same verified email; password signup is closed (ALLOW_SIGNUP=0), and an account created through Keycloak has no password until its owner sets one on the account page. There is no admin role and no password reset by email.

Configuration Management

Everything is environment: listen address, DATA_DIR, MIGRATE and log format in the base, tier and resources, TRUSTED_PROXIES, ALLOW_SIGNUP and GOMEMLIMIT in the talos patch, DATABASE_URL from the generated CNPG Secret, BLOB_S3_* from the SILO Secret, ARCHIVIST_OIDC_* next to the workspace with the client secret from workspace-outputs. Upload size, quotas and session lifetimes keep the server defaults (8 GiB per import, 3 active imports and 50 GiB of archives per user, 30-day sessions with a 14-day idle timeout).

Usage

Open archivist.int.kueber.eu from the LAN or NetBird and continue with Keycloak; the first login is the account. Request an export from a platform (the in-app /help/exports page explains each one), pick the source, drop the ZIP — all parts at once for platforms that split large exports — and watch the import progress; the coverage report says what was captured. The timeline filters by source, category, type, date range and full-text search, days are drawn in the browser's time zone, and media plays inline with seeking. Reprocess re-runs the current providers over a preserved archive after a provider improvement; Delete removes an import and its events.

Cluster-specific deviations from the above live in the per-cluster README — see k8s/apps/archivist/talos/README.md.

Cluster Deployment

App URLs

• archivist.int.kueber.eu

Depends on

• CloudNative-PG
• Envoy Gateway
• k8up backups
• Longhorn

Archivist — Talos cluster

Deviations from defaults

• Internal gateway only (archivist.int.kueber.eu) — the archive is the private history of every account its users ever had; nothing about it belongs on the internet. LAN and NetBird reach the internal VIP, the Crossplane workspace writes the int.kueber.eu record. Exports are uploaded from a desktop anyway.
• Login through Keycloak, in the app — Archivist 0.1.1 brings its own OpenID Connect login, so the app talks to Keycloak itself (workspace/patch-archivist.yaml: provider keycloak, issuer https://keycloak.web.kueber.eu/realms/kueber.eu, client archivist, secret from workspace-outputs, callback https://archivist.int.kueber.eu/api/auth/oidc/keycloak/callback) and no gateway SecurityPolicy sits in front any more — one login, not two. The client is created by the TofuWorkspace (workspace/patch-oidc.yaml, gated on archivist.user for the family group), and that role gate is the allow-list: a first Keycloak login creates the Archivist account (or attaches to the password account with the same verified email), so ALLOW_SIGNUP is 0 from the start and only governs the password signup form. Discovery runs at boot and the token exchange goes to the public hostname, both through the network baseline's gateway allowance; an unreachable Keycloak stops the server with a clear error.
• First-party image from the private Gitea registry — gitea.web.kueber.eu/johnny/archivist:0.1.0, digest-pinned, pulled via regcred-docker-gitea (namespace label secrets.docker-gitea). The archivist repo's release workflow builds, pushes and cosign-signs the image on a version tag (git tag -s vX.Y.Z && git push --tags publishes X.Y.Z, latest and sha-<12>, with SBOM and provenance as BuildKit attestations); Renovate bumps the version and moves the digest. The repo signs with its own COSIGN_PRIVATE_KEY, not the Flux key, so the verify-gitea-johnny-images audit lists it until the two keys are the same.
• Blobs in an app-local SILO, uploads on a staging volume — the shipped deployment's shape: BLOB_S3_* points at the minio component on the cluster-local service (media is streamed through the app, no browser ever sees the endpoint, so no public route as voyage needs). The server creates the archivist bucket at boot. The data volume is widened to 50Gi: original export ZIPs are preserved write-once and run to several GB each. DATA_DIR is the 20Gi archivist-staging PVC on longhorn-object-single-encrypted: an upload is the only copy of itself until the worker has preserved it in the bucket, so it has to survive a restart, but it is re-uploadable, so one replica and no backup. Both volumes unlock with the namespace's fixed object-longhorn key. BLOB_S3_SSE stays off (SILO rejects it without a KMS); the LUKS volumes cover encryption at rest.
• TRUSTED_PROXIES=1 — Envoy is the single hop and the only peer the ingress policy admits, so the login throttle and the per-IP rate limits key on the client address it forwards rather than on one shared bucket.
• No internet egress — export ZIPs are the only ingestion path; the app never calls a platform. Postgres and SILO are intra-namespace, the token exchange is Envoy's. No egress policy beyond the baseline, and no ingress rule for SILO because only this namespace talks to it.
• 10Gi CNPG volume, tier-2 everywhere — events plus their tsvector and the consumption histories (Spotify, YouTube) outgrow the component's 2Gi quickly. App, database and object store all run tier-2; the app's requests (256Mi, 1Gi limit, GOMEMLIMIT=900MiB) are a first estimate to re-measure after the first real imports.
• Backups run as uid 1000 — the SILO data volume is the only PVC k8up sees (the component's template annotation), owned by the minio image's user; backups/patch-schedule.yaml runs the job as that identity. The database goes out as the pg_dump PreBackupPod. Restic bucket suffix -archivist-bf6e20b8.
• Metrics open on the pod, closed at the edge — /metrics carries no METRICS_TOKEN because the only peers that can reach port 8080 are the internal gateway (behind the OIDC policy) and vmagent, which scrapes through the Service annotations.

Bootstrap

1. Open https://archivist.int.kueber.eu and choose Continue with Keycloak — the first login creates the account, an ordinary one, there is no admin role.
2. Re-measure the app's memory against the 256Mi request after the first large imports.

Kubernetes Metadata

• Image: gitea.web.kueber.eu/johnny/archivist:0.1.1@sha256:8e471b8c581ec9f678ca1986588d9853a9bb1d75b95028fc4bc4d1d51e3d2c4b
• Image: pgsty/silo:RELEASE.2026-10-15T00-00-00Z@sha256:aaac6bce13eb99316b3c88bd20143fc3c38ab607585256a411a095c44ee57f62

Rendered manifests (kustomize build)

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/force: enabled
  labels:
    app.kubernetes.io/name: archivist
  name: archivist
  namespace: archivist
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: archivist
  strategy:
    rollingUpdate: null
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: archivist
    spec:
      automountServiceAccountToken: false
      containers:
        - env:
            - name: GOMEMLIMIT
              value: 900MiB
            - name: TRUSTED_PROXIES
              value: '1'
            - name: ALLOW_SIGNUP
              value: '0'
            - name: ARCHIVIST_OIDC_PROVIDERS
              value: keycloak
            - name: ARCHIVIST_OIDC_KEYCLOAK_NAME
              value: Keycloak
            - name: ARCHIVIST_OIDC_KEYCLOAK_ISSUER
              value: https://keycloak.web.kueber.eu/realms/kueber.eu
            - name: ARCHIVIST_OIDC_KEYCLOAK_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  key: client-id
                  name: workspace-outputs
            - name: ARCHIVIST_OIDC_KEYCLOAK_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  key: client-secret
                  name: workspace-outputs
            - name: ARCHIVIST_OIDC_KEYCLOAK_REDIRECT_URL
              value: https://archivist.int.kueber.eu/api/auth/oidc/keycloak/callback
            - name: BLOB_S3_ENDPOINT
              value: http://minio:9000
            - name: BLOB_S3_BUCKET
              value: archivist
            - name: BLOB_S3_REGION
              value: us-east-1
            - name: BLOB_S3_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: MINIO_ROOT_USER
                  name: minio
            - name: BLOB_S3_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  key: MINIO_ROOT_PASSWORD
                  name: minio
            - name: BLOB_S3_SSE
              value: '0'
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  key: uri
                  name: cnpg-app
            - name: ADDR
              value: :8080
            - name: DATA_DIR
              value: /data
            - name: MIGRATE
              value: '1'
            - name: LOG_FORMAT
              value: json
          image: gitea.web.kueber.eu/johnny/archivist:0.1.1@sha256:8e471b8c581ec9f678ca1986588d9853a9bb1d75b95028fc4bc4d1d51e3d2c4b
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: web
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          name: archivist
          ports:
            - containerPort: 8080
              name: web
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /readyz
              port: web
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 3
          resources:
            limits:
              memory: 1Gi
            requests:
              cpu: 50m
              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: true
          volumeMounts:
            - mountPath: /data
              name: staging
            - mountPath: /tmp
              name: tmp
      imagePullSecrets:
        - name: regcred-docker-gitea
      priorityClassName: tier-2
      securityContext:
        fsGroup: 10001
        fsGroupChangePolicy: OnRootMismatch
        runAsGroup: 10001
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile:
          type: RuntimeDefault
      volumes:
        - name: staging
          persistentVolumeClaim:
            claimName: archivist-staging
        - emptyDir: {}
          name: tmp
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/force: enabled
  name: minio
  namespace: archivist
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: minio
  serviceName: minio
  template:
    metadata:
      labels:
        app.kubernetes.io/name: minio
    spec:
      containers:
        - args:
            - server
            - /data
            - --console-address
            - :9001
          envFrom:
            - secretRef:
                name: minio
          image: pgsty/silo:RELEASE.2026-10-15T00-00-00Z@sha256:aaac6bce13eb99316b3c88bd20143fc3c38ab607585256a411a095c44ee57f62
          livenessProbe:
            httpGet:
              path: /minio/health/live
              port: s3
            initialDelaySeconds: 10
            periodSeconds: 10
          name: minio
          ports:
            - containerPort: 9000
              name: s3
            - containerPort: 9001
              name: console
          readinessProbe:
            httpGet:
              path: /minio/health/ready
              port: s3
            initialDelaySeconds: 3
            periodSeconds: 5
          resources:
            limits:
              memory: 384Mi
            requests:
              cpu: 25m
              memory: 192Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
          volumeMounts:
            - mountPath: /data
              name: data
      priorityClassName: tier-2
      securityContext:
        fsGroup: 1000
        fsGroupChangePolicy: OnRootMismatch
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
  volumeClaimTemplates:
    - metadata:
        annotations:
          k8up.io/backup: 'true'
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
        storageClassName: longhorn-object-encrypted