Installing Mastodon 4.6 on Kubernetes

In September 2025 I already wrote up how to install Mastodon on Kubernetes. That blog no longer exists, and the old guide survives only in the Web Archive. A text about backups that only exists as somebody else's copy is at least staying on topic. My own instance ran on that setup until March 2026, when I moved Mastodon off Kubernetes on cost grounds. Almost everything in the old guide held up over that time. One part was wrong all the same, and it was the most important one: Postgres as a Deployment with a PVC next to it.

This is the whole guide rewritten: the configuration as it last ran on that instance, with the version pins brought up to today. Finished, tested, and the reason I know where the sharp edges are. You do not need to know the old guide, everything is here from an empty namespace to a tested backup. Where something changed, I explain it at the point where the decision gets made.

Same rules as last time: no Helm, everything by hand, because I want to understand what is running. Mine ran as GitOps through ArgoCD, but every file here works just as well with kubectl apply.

My setup#

Three nodes with 4 GB RAM, 2 cores and 20 GB SSD each, plus a load balancer. The guide is written against Kubernetes 1.36. The persistent volumes are network volumes from the provider's storage class, the local SSD only holds the system and images. Envoy Gateway for the traffic, cert-manager for certificates, kubeseal for secrets. No Elasticsearch, it does not pay off at this size. I explain further down why the 2025 nginx ingress is gone.

The versions this guide is written against:

Valkey instead of Redis is a deliberate choice. Mastodon documents Redis 7.0 and newer, and Valkey is a protocol compatible fork of it. The environment variables are still called REDIS_*, only the image changes.

Tools#

You need kubeseal locally for the secrets. On macOS:

brew install kubeseal

On Linux:

export KUBESEAL_VERSION="0.38.4"
curl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"
tar -xvzf kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Verify with kubeseal --version. You also need Docker or Podman locally: the Mastodon keys further down are generated in a throwaway container.

Namespace and controllers#

kubectl create namespace mastodon

The Sealed Secrets controller, if you do not have one yet:

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.38.4/controller.yaml

This manifest puts the controller into kube-system as sealed-secrets-controller, which is exactly what the kubeseal calls further down assume. If your controller lives somewhere else, say from a Helm chart in its own namespace, adjust --controller-namespace and --controller-name there.

cert-manager, if it is not running yet:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.0/cert-manager.yaml

kubectl -n cert-manager rollout status deployment/cert-manager-webhook

And CloudNativePG, the Postgres operator. The --server-side flag is not optional here, the CRDs are too large for the classic annotation:

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.0.yaml

kubectl -n cnpg-system rollout status deployment/cnpg-controller-manager

The database#

What was wrong with the Deployment#

This is the part I got wrong in 2025. Back then it was a Deployment with replicas: 1, a PVC, and a pg_dump CronJob at four in the morning. That runs fine for months and has three problems, all of which surface at the least convenient time.

A Deployment rolls out with RollingUpdate by default, starting the new pod before the old one is gone. On a ReadWriteOnce volume the rollout then sits there until somebody intervenes. Kubernetes also sends SIGTERM on shutdown, which Postgres reads as a smart shutdown: wait until all clients disconnect on their own. Sidekiq does not disconnect, so after 30 seconds comes SIGKILL and on the next start a crash recovery. And a nightly dump means a mistake at 14:00 costs you half a day of federation.

The CNPG cluster#

The operator takes all of that off your hands. First the credentials for the backup store:

kubectl create secret generic mastodon-backup-creds \
  --namespace mastodon \
  --from-literal=ACCESS_KEY_ID=your_key \
  --from-literal=ACCESS_SECRET_KEY=your_secret \
  --dry-run=client -o yaml > backup-creds.yaml

kubeseal --controller-namespace kube-system \
  --controller-name sealed-secrets-controller \
  --format yaml < backup-creds.yaml > sealed-backup-creds.yaml

kubectl apply -f sealed-backup-creds.yaml && rm backup-creds.yaml

Then the cluster:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: mastodon-db
  namespace: mastodon
spec:
  instances: 2
  imageName: ghcr.io/cloudnative-pg/postgresql:18.4

  # On three small nodes, "preferred" is the difference between
  # "running" and "Pending, because the rule cannot be satisfied"
  affinity:
    enablePodAntiAffinity: true
    topologyKey: kubernetes.io/hostname
    podAntiAffinityType: preferred

  storage:
    size: 20Gi
  walStorage:
    size: 10Gi

  postgresql:
    parameters:
      shared_buffers: "256MB"
      effective_cache_size: "768MB"
      work_mem: "8MB"
      maintenance_work_mem: "128MB"

  bootstrap:
    initdb:
      database: mastodon_production
      owner: mastodon

  backup:
    barmanObjectStore:
      destinationPath: s3://your-bucket/mastodon
      endpointURL: https://your-s3-endpoint
      s3Credentials:
        accessKeyId:
          name: mastodon-backup-creds
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: mastodon-backup-creds
          key: ACCESS_SECRET_KEY
      wal:
        compression: gzip
      data:
        compression: gzip
    retentionPolicy: "30d"

  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      memory: "1Gi"

Four points in there need explaining.

walStorage as its own volume separates WAL from data, and the difference shows when one of the two fills up. Data volume full: writes fail, but Postgres keeps running, the WAL archive stays intact, and you grow the volume at your leisure. WAL volume full, classically because archiving to object storage is stuck: Postgres stops hard with a PANIC, but the data next to it stays untouched. On a shared volume, both cases end in the hard variant.

The backup block does continuous WAL archiving: instead of one state per night, you can restore to any point in time since the last base backup. A note on versions, because this is moving right now: this built in barmanObjectStore form has been deprecated since CNPG 1.26, and the 1.30 release notes say it will be removed in 1.31. On 1.30 it still works and is still the default. If you are installing anything newer, start with the Barman Cloud plugin right away instead of migrating twice. And for this setup it means: plan the move to the plugin before you lift the operator to 1.31.

You do not have to copy instances: 2. On three nodes with 4 GB each, the second instance costs memory that Sidekiq would happily use, and it tightens scheduling: with a node down or in maintenance, both Postgres pods compete with everything else for the remaining two. With instances: 1 you still get backups, point in time recovery, controlled updates and declarative configuration, just no automatic failover when a node dies. For a personal instance that is a legitimate trade, and I see no shame in running it that way.

The fourth point only shows if you go looking for it: there is no CREATE EXTENSION anywhere. Mastodon 4.6 needs nothing beyond plpgsql, which every Postgres database has out of the box. Do not take my word for it, check the enable_extension lines in the db/schema.rb of your version, that is where I look before setting up. If an extension ever shows up there, postInitApplicationSQL under initdb is the place to create it, and since Postgres 13 extensions marked trusted can also be created by the database owner during the migration, no superuser needed.

Automatic backups#

The scheduled base backup that goes with it:

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: mastodon-db-daily
  namespace: mastodon
spec:
  # Six fields, not five. The first one is seconds.
  schedule: "0 0 3 * * *"
  backupOwnerReference: self
  cluster:
    name: mastodon-db

The cron expression caught me on the first attempt. CNPG uses a Go library with a seconds field, so a five field expression copied from a normal CronJob means something other than you intended.

Services and credentials#

The operator now creates three services: mastodon-db-rw always points at the current primary, mastodon-db-ro at the replicas, mastodon-db-r at all of them. That is exactly what a handwritten Service with a fixed selector cannot do, because after a failover it would point at the wrong pod.

The operator generates the application user's password itself and stores it in the mastodon-db-app secret. It never gets copied anywhere: the deployments further down read it straight from there via valueFrom. That leaves exactly one source of truth, and if the operator ever rotates the password, nothing quietly keeps running with a stale copy. In case you still want to connect with psql by hand:

kubectl -n mastodon get secret mastodon-db-app \
  -o jsonpath='{.data.password}' | base64 -d

Valkey#

For Mastodon, Valkey holds more than the cache: the Sidekiq queues live there too. Lose those and you lose in-flight jobs, meaning outgoing deliveries, push notifications and media processing. So it gets a StatefulSet with its own volume, not a Deployment with a PVC attached like I had in 2025.

apiVersion: v1
kind: Service
metadata:
  name: valkey
  namespace: mastodon
spec:
  clusterIP: None
  selector:
    app: valkey
  ports:
    - port: 6379
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: valkey
  namespace: mastodon
spec:
  serviceName: valkey
  replicas: 1
  selector:
    matchLabels:
      app: valkey
  template:
    metadata:
      labels:
        app: valkey
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: valkey
          image: valkey/valkey:9.1-alpine
          args: ["--appendonly", "yes", "--appendfsync", "everysec"]
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: data
              mountPath: /data
          readinessProbe:
            exec:
              command: ["valkey-cli", "ping"]
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: "100m"
              memory: "256Mi"
            limits:
              memory: "512Mi"
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 5Gi

The grace period is set to 60 seconds because Valkey needs time to write its data out on shutdown, and the 30 second default is tight for that.

Secrets#

These values get generated once and never changed. SECRET_KEY_BASE encrypts sessions, OTP_SECRET is tied to two factor authentication, the VAPID keys to push notifications. Swap them later and you throw away every login, every 2FA enrolment and every push subscription. For LOCAL_DOMAIN this holds even more strictly: the domain is baked into the identity of every account, it is the name every other server knows you by. Change it later and the federation for the old identity is gone for good, and nothing can repair that.

# Active Record encryption keys
docker run --rm ghcr.io/mastodon/mastodon:v4.6.4 \
  bin/rails db:encryption:init

# SECRET_KEY_BASE and OTP_SECRET, run twice
docker run --rm ghcr.io/mastodon/mastodon:v4.6.4 bundle exec rake secret

# VAPID key pair
docker run --rm ghcr.io/mastodon/mastodon:v4.6.4 \
  bundle exec rake mastodon:webpush:generate_vapid_key

Those go into the secret. DB_HOST points at the operator's -rw service, and there is deliberately no DB_PASS: that comes straight from mastodon-db-app via valueFrom in a moment:

apiVersion: v1
kind: Secret
metadata:
  name: mastodon-env
  namespace: mastodon
type: Opaque
stringData:
  LOCAL_DOMAIN: "your-domain.ch"
  SINGLE_USER_MODE: "true"
  RAILS_ENV: "production"

  SECRET_KEY_BASE: "..."
  OTP_SECRET: "..."
  VAPID_PRIVATE_KEY: "..."
  VAPID_PUBLIC_KEY: "..."
  ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY: "..."
  ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT: "..."
  ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY: "..."

  DB_HOST: "mastodon-db-rw"
  DB_NAME: "mastodon_production"
  DB_USER: "mastodon"

  REDIS_HOST: "valkey"
  REDIS_PORT: "6379"
  CACHE_REDIS_URL: "redis://valkey:6379/1"

  SMTP_SERVER: "smtp.example.ch"
  SMTP_PORT: "587"
  SMTP_LOGIN: "mail@your-domain.ch"
  SMTP_PASSWORD: "..."
  SMTP_FROM_ADDRESS: "mail@your-domain.ch"

  S3_ENABLED: "true"
  S3_BUCKET: "your-bucket"
  S3_REGION: "..."
  S3_ENDPOINT: "https://your-s3-endpoint"
  S3_FORCE_PATH_STYLE: "true"
  S3_ALIAS_HOST: "cdn.your-domain.ch"
  AWS_ACCESS_KEY_ID: "..."
  AWS_SECRET_ACCESS_KEY: "..."

CACHE_REDIS_URL moves the Rails cache into database 1. That way you can flush the cache in an emergency without touching the Sidekiq queues in database 0. One limit of this double duty is worth knowing: the eviction policy applies per instance, not per database. The queues only tolerate noeviction, the default, so the cache here must never be capped with allkeys-lru either. As long as memory holds out this does not matter. Once the instance outgrows it, the cache moves into its own small Valkey.

Seal it and delete the plaintext file:

kubeseal --controller-namespace kube-system \
  --controller-name sealed-secrets-controller \
  --format yaml < mastodon-env.yaml > sealed-mastodon-env.yaml

kubectl apply -f sealed-mastodon-env.yaml
rm mastodon-env.yaml

Migrations#

The schema has to exist before the first start. As a Job, not an init container, so it does not run again on every pod restart:

apiVersion: batch/v1
kind: Job
metadata:
  name: mastodon-db-migrate
  namespace: mastodon
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: ghcr.io/mastodon/mastodon:v4.6.4
          command: ["bundle", "exec", "rails", "db:migrate"]
          envFrom:
            - secretRef:
                name: mastodon-env
          env:
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: mastodon-db-app
                  key: password

This is the first appearance of the pattern for the database password: envFrom pulls everything from mastodon-env, and the single env entry takes DB_PASS straight from the operator secret. The same block appears from now on in every manifest that talks to the database. If your mastodon-env still carries an old DB_PASS: individual env entries beat envFrom, the direct reference wins.

On the first run the job replays Mastodon's entire migration history, a few hundred migrations, so give it a few minutes.

You need this same job after every upgrade, and since Jobs are immutable, delete the old one first: kubectl -n mastodon delete job mastodon-db-migrate, then apply again. Check the release notes too. Bigger upgrades split their migrations: the part before the rollout runs with SKIP_POST_DEPLOYMENT_MIGRATIONS=true, the rest after it.

Web, streaming and Sidekiq#

Web#

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mastodon-web
  namespace: mastodon
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mastodon-web
  template:
    metadata:
      labels:
        app: mastodon-web
    spec:
      containers:
        - name: web
          image: ghcr.io/mastodon/mastodon:v4.6.4
          command: ["bundle", "exec", "puma", "-C", "config/puma.rb"]
          envFrom:
            - secretRef:
                name: mastodon-env
          env:
            - name: WEB_CONCURRENCY
              value: "1"
            - name: MAX_THREADS
              value: "3"
            - name: DB_POOL
              value: "5"
            - name: MALLOC_ARENA_MAX
              value: "2"
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: mastodon-db-app
                  key: password
          ports:
            - containerPort: 3000
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 20
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 60
            periodSeconds: 30
            timeoutSeconds: 10
            failureThreshold: 5
          resources:
            requests:
              cpu: "300m"
              memory: "600Mi"
            limits:
              memory: "1.2Gi"

The PVC for /mastodon/public/system from my old guide is gone here. With S3_ENABLED: true the media live in object storage, the local directory is not needed, and that also removes the init container that used to fix up permissions.

The readiness probe was missing entirely in 2025. Without it the gateway sends requests to a pod where Puma is still starting. There is now also a liveness probe, deliberately sluggish: a Puma that is genuinely hung is still hung after five failed checks 30 seconds apart, and one that is merely busy gets enough slack before anyone restarts it for no reason.

The resources follow a pattern that runs through every manifest in this guide, the Postgres cluster included: limits for memory, none for CPU. Memory is not compressible, and without a limit a growing process eventually takes the node down with it. CPU is different: when it gets scarce, the scheduler divides it up according to the requests, and a CPU limit would throttle Puma even while the node has nothing else to do.

Streaming#

Streaming is a separate process with its own image:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mastodon-streaming
  namespace: mastodon
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mastodon-streaming
  template:
    metadata:
      labels:
        app: mastodon-streaming
    spec:
      containers:
        - name: streaming
          image: ghcr.io/mastodon/mastodon-streaming:v4.6.4
          envFrom:
            - secretRef:
                name: mastodon-env
          env:
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: mastodon-db-app
                  key: password
          ports:
            - containerPort: 4000
          readinessProbe:
            httpGet:
              path: /api/v1/streaming/health
              port: 4000
            initialDelaySeconds: 10
          livenessProbe:
            httpGet:
              path: /api/v1/streaming/health
              port: 4000
            initialDelaySeconds: 30
            periodSeconds: 30
            timeoutSeconds: 10
            failureThreshold: 5
          resources:
            requests:
              cpu: "100m"
              memory: "200Mi"
            limits:
              memory: "400Mi"

Sidekiq and the scheduler#

Sidekiq does the federation work. Two values matter on shutdown here: Kubernetes waits up to terminationGracePeriodSeconds after SIGTERM, but Sidekiq has its own timeout and by default pushes unfinished jobs back onto the queue after 25 seconds. Only the -t 110 below makes use of the full 120 seconds. Nothing is lost without the flag, the jobs simply run again later, they just start over from the beginning:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mastodon-sidekiq
  namespace: mastodon
spec:
  selector:
    matchLabels:
      app: mastodon-sidekiq
  template:
    metadata:
      labels:
        app: mastodon-sidekiq
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: sidekiq
          image: ghcr.io/mastodon/mastodon:v4.6.4
          command:
            - bundle
            - exec
            - sidekiq
            - -c
            - "5"
            - -t
            - "110"
            - -q
            - default,8
            - -q
            - push,6
            - -q
            - ingress,4
            - -q
            - mailers,2
            - -q
            - pull,1
          envFrom:
            - secretRef:
                name: mastodon-env
          env:
            - name: DB_POOL
              value: "5"
            - name: MALLOC_ARENA_MAX
              value: "2"
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: mastodon-db-app
                  key: password
          resources:
            requests:
              cpu: "500m"
              memory: "300Mi"
            limits:
              memory: "600Mi"

One queue is deliberately missing from that list. Mastodon's documentation is blunt about scheduler: it must never run in more than one process at a time. The deployment above is about to get an autoscaler, so the scheduler gets its own deployment with exactly one replica and no autoscaling:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mastodon-sidekiq-scheduler
  namespace: mastodon
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mastodon-sidekiq-scheduler
  template:
    metadata:
      labels:
        app: mastodon-sidekiq-scheduler
    spec:
      containers:
        - name: sidekiq
          image: ghcr.io/mastodon/mastodon:v4.6.4
          command: ["bundle", "exec", "sidekiq", "-c", "1", "-q", "scheduler"]
          envFrom:
            - secretRef:
                name: mastodon-env
          env:
            - name: DB_POOL
              value: "2"
            - name: MALLOC_ARENA_MAX
              value: "2"
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: mastodon-db-app
                  key: password
          resources:
            requests:
              cpu: "50m"
              memory: "300Mi"
            limits:
              memory: "512Mi"

The scheduler triggers the periodic work: publishing scheduled posts, refreshing trends, cleaning up old data. Run it twice and all of that happens twice, including your scheduled posts going out twice.

Autoscaling and connections#

Plus the autoscaler for the main worker, so that under load you get several small pods instead of one large one:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: mastodon-sidekiq
  namespace: mastodon
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: mastodon-sidekiq
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80

This is why the Sidekiq deployment above has no replicas field. Once the HPA manages the count, every repeated kubectl apply and every ArgoCD sync would write the value from the file back and flatten the scaling again. Without the field the deployment starts with one replica, and after that the HPA has the last word.

Scaling on CPU has a catch: during federation Sidekiq spends most of its time waiting on other people's servers, so it is often not CPU bound. Queue length would be the better signal, but that costs you a custom metrics adapter. For a small instance CPU is good enough, you should just know it is an approximation.

One more thing about DB_POOL: the value applies per process. One web pod with five, up to three Sidekiq pods with five each, the scheduler with its two, plus streaming with its default pool of ten and the occasional migration job: around forty connections, which the Postgres default of 100 covers with room to spare. If you ever run significantly more processes, do not just raise max_connections, every connection costs memory in a database container that has little to spare, put PgBouncer in front instead. That adds one easily missed line: in transaction pooling mode PREPARED_STATEMENTS must be false, otherwise Rails' prepared statements collide with shared connections.

Services and the gateway#

Why Gateway API#

apiVersion: v1
kind: Service
metadata:
  name: mastodon-web
  namespace: mastodon
spec:
  selector:
    app: mastodon-web
  ports:
    - port: 80
      targetPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: mastodon-streaming
  namespace: mastodon
spec:
  selector:
    app: mastodon-streaming
  ports:
    - port: 4000
      targetPort: 4000

In 2025 this section was an nginx Ingress, and I would have happily kept it. But the Kubernetes project announced the retirement of ingress-nginx in November 2025 and stopped maintenance in March 2026. Existing installations keep running, only no patches are coming anymore, including for the next CVE. I am not willing to run the component that parses every byte from the open internet on those terms.

The question of what replaces it has an official answer: Gateway API instead of Ingress. As the implementation I picked Envoy Gateway. CNCF project, Envoy underneath, and it installs like everything else in this guide, one manifest, no Helm:

kubectl apply --server-side -f \
  https://github.com/envoyproxy/gateway/releases/download/v1.8.3/install.yaml

kubectl -n envoy-gateway-system rollout status deployment/envoy-gateway

--server-side for the same reason as with CNPG, and the manifest already brings the Gateway API CRDs (v1.5.1) with it, you do not install those separately.

Adjusting cert-manager#

A word on compatibility, since Kubernetes 1.36 is what I put at the top: Envoy Gateway v1.8 is officially tested against 1.32 through 1.35, and 1.36 was not in the matrix yet when I wrote this. It does not depend on any API that changed in 1.36, and the next release should close the gap. If you want to stay strictly inside the tested matrix, stay on Kubernetes 1.35, which is supported until February 2027 and behaves exactly the same in this guide.

cert-manager can answer HTTP-01 challenges through a Gateway, it just does not look for Gateways by default. The switch is called --enable-gateway-api:

kubectl -n cert-manager patch deployment cert-manager --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--enable-gateway-api"}]'

The patch restarts the controller, which it needs anyway to see the new CRDs. Since cert-manager above comes from the static manifest, the flag is gone again on the next re-apply, so it belongs in your copy in Git.

The ClusterIssuer changes too. The old one solved HTTP-01 through an Ingress, this one builds a temporary HTTPRoute per challenge:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: mail@your-domain.ch
    privateKeySecretRef:
      name: letsencrypt-prod-account
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: mastodon
                namespace: mastodon
                kind: Gateway

The gateway and the routes#

Then the gateway itself:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: mastodon
  namespace: mastodon
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  gatewayClassName: envoy
  listeners:
    - name: http
      protocol: HTTP
      port: 80
    - name: https
      protocol: HTTPS
      port: 443
      hostname: your-domain.ch
      tls:
        mode: Terminate
        certificateRefs:
          - name: your-domain-tls

The annotation does what it used to do on the Ingress: cert-manager sees the Gateway, issues the certificate for the listener's hostname and writes it into the secret the listener references.

Two routes attach to this. The first one only redirects, from the http listener to https:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mastodon-redirect
  namespace: mastodon
spec:
  parentRefs:
    - name: mastodon
      sectionName: http
  rules:
    - filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            statusCode: 301

The second one does the actual routing:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mastodon
  namespace: mastodon
spec:
  parentRefs:
    - name: mastodon
      sectionName: https
  hostnames:
    - your-domain.ch
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api/v1/streaming
      backendRefs:
        - name: mastodon-streaming
          port: 4000
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: mastodon-web
          port: 80

A few details, for once all of them pleasant. The order of the rules looks meaningful and is not: Gateway API specifies that the longest matching path wins, however you sort them. With nginx that was true as well, but only as an implementation detail. Here it is contract. If you are wondering whether the redirect route swallows Let's Encrypt's challenges: it does not. cert-manager's challenge route matches a longer path, and longer wins, also by spec. proxy-body-size has no successor here, on purpose: Envoy streams request bodies instead of buffering them, so there is no default limit to raise and no 413 in the middle of a video upload. And websockets for streaming work without any extra configuration.

LoadBalancer and DNS#

Envoy Gateway creates its own LoadBalancer service for the gateway in envoy-gateway-system. That only works if your cluster can give LoadBalancer services an external IP: on managed providers the load balancer from my setup at the top takes care of it, on bare metal you need something like MetalLB, otherwise the address simply stays empty. Your DNS record points there from now on:

kubectl -n mastodon get gateway mastodon

The ADDRESS column is what goes into DNS, and PROGRAMMED should say True. If you are coming from the ingress setup: build the gateway and routes alongside it, switch DNS, and only remove ingress-nginx once no traffic shows up in its logs anymore.

The first account#

With SINGLE_USER_MODE the registration form stays closed, and as long as no account exists the front page has nothing to point at. So create the owner account directly in the web pod:

kubectl -n mastodon exec deploy/mastodon-web -- \
  bin/tootctl accounts create dima \
  --email mail@your-domain.ch --confirmed --role Owner

The command prints a temporary password. Log in with it, change it, set up 2FA, done.

Cleaning up media#

Without cleanup, object storage grows without bound, because every federated file gets cached.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mastodon-media-cleanup
  namespace: mastodon
spec:
  schedule: "0 4 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: cleanup
              image: ghcr.io/mastodon/mastodon:v4.6.4
              envFrom:
                - secretRef:
                    name: mastodon-env
              env:
                - name: DB_PASS
                  valueFrom:
                    secretKeyRef:
                      name: mastodon-db-app
                      key: password
              command:
                - /bin/bash
                - -c
                - |
                  bin/tootctl media remove --days=7
                  bin/tootctl media remove-orphans
                  bin/tootctl preview_cards remove --days=7
                  bin/tootctl statuses remove --days=14
                  bin/tootctl accounts prune

Four in the morning is deliberate, an hour after the base backup, so the two do not overlap. concurrencyPolicy: Forbid was missing in my old version. If a run takes longer than 24 hours, the next one otherwise starts alongside it, and two parallel cleanup jobs against the same database are not a good idea.

The backup that stopped hurting#

In 2025 this was the longest and most unpleasant section of the whole guide. I had built my own Docker image: Ubuntu with AWS CLI, rclone and the Postgres client, plus a bash script with error checks after every step and a registry secret so the cluster could pull the image. It worked, and it was a lot of moving parts for a job that is really standard.

The ScheduledBackup above replaces all of it. The database goes to object storage continuously through WAL archiving, the base backup runs at night, and retentionPolicy handles how long things are kept.

The media no longer need their own backup if they already live in S3. What they need is versioning or replication on the provider side, and that is a bucket setting rather than a nightly rclone sync.

You can check the state like this:

kubectl -n mastodon get backups
kubectl -n mastodon get cluster mastodon-db \
  -o jsonpath='{.status.firstRecoverabilityPoint}'

And the part I did not have before: restoring to a point in time. It creates a new cluster while the old one keeps running untouched, so you can compare and then switch over:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: mastodon-db-restore
  namespace: mastodon
spec:
  instances: 1
  imageName: ghcr.io/cloudnative-pg/postgresql:18.4
  storage:
    size: 20Gi
  walStorage:
    size: 10Gi
  bootstrap:
    recovery:
      source: mastodon-db
      recoveryTarget:
        targetTime: "2026-07-27 14:00:00+02"
  externalClusters:
    - name: mastodon-db
      barmanObjectStore:
        destinationPath: s3://your-bucket/mastodon
        endpointURL: https://your-s3-endpoint
        s3Credentials:
          accessKeyId:
            name: mastodon-backup-creds
            key: ACCESS_KEY_ID
          secretAccessKey:
            name: mastodon-backup-creds
            key: ACCESS_SECRET_KEY

The restored cluster has no backup block, and that is on purpose: it is a rehearsal, so it has no business writing to object storage. If you ever promote one of these to the real thing, give it its own destinationPath or serverName before you add backups back. Point it at the old path and it archives its WAL straight over the history of the cluster you were trying to rescue.

If you are coming from the old guide#

If Postgres is still running as a Deployment for you: CNPG can import from an existing database as part of bootstrapping, instead of you juggling pg_dump and pg_restore.

spec:
  bootstrap:
    initdb:
      database: mastodon_production
      owner: mastodon
      import:
        type: microservice
        databases:
          - mastodon_production
        source:
          externalCluster: old-instance
  externalClusters:
    - name: old-instance
      connectionParameters:
        host: postgres
        user: mastodon
        dbname: mastodon_production
      password:
        name: mastodon-env
        key: DB_PASS

One quirk you need to know: the import runs exactly once, when the cluster is created. The only way to get a second pass is to delete the new cluster and create it again. So the sequence looks like this: create the cluster with import and do not touch Mastodon, that is the dress rehearsal. Compare row counts on the large tables against the old database, then delete the new cluster again. For the real move, scale web, streaming and Sidekiq to zero, downtime starts here, and create the cluster with import once more, this time without a gap. Point DB_HOST at mastodon-db-rw in the secret, scale back up, watch the logs. While you are at it, delete the old DB_PASS from the secret, the new manifests read the password straight from mastodon-db-app.

Leave the old deployment and its PVC in place for a week afterwards, even if it annoys you. Delete the way back only once you are sure you will not need it.

What changed since 2025#

What I did not change: Sealed Secrets, skipping Helm, no Elasticsearch on small nodes, the Sidekiq autoscaler, and the nightly media cleanup. That was already right.

And one warning that belongs here for honesty: two Postgres instances protect you from a dead node, not from a DELETE without a WHERE. That replicates cleanly to both. High availability and backups solve two different problems, and only one of them can be fixed at three in the morning with a coffee.

If you rebuild this and get stuck somewhere, write to me. Last time the best corrections came from people who simply tried what I had written down.