Installing Mastodon 4.6 on Kubernetes

In September 2025 I already wrote up how to install Mastodon on Kubernetes. That guide has been running my own instance ever since, and I still stand behind almost all of it. One part was wrong anyway, and it was the most important one: Postgres as a Deployment with a PVC next to it.

This is the whole guide rewritten. You do not need to know the old one, everything is here from an empty namespace to a tested backup. Where something changed, I explain it at the point where the decision gets made.

Same rules as last time: no Helm, everything by hand, because I want to understand what is running. I use GitOps with ArgoCD, but every file here works just as well with kubectl apply.

My setup

Kubernetes 1.33, three nodes with 4 GB RAM, 2 cores and 20 GB SSD each, plus a load balancer. Nginx ingress, cert-manager for certificates, kubeseal for secrets. No Elasticsearch, it does not pay off at this size.

The versions I am using while writing this:

Valkey instead of Redis is a deliberate choice. Mastodon documents Redis 7.0 and newer, and Valkey is a protocol compatible fork of it. The environment variables are still called REDIS_*, only the image changes.

Tools

You need kubeseal locally for the secrets. On macOS:

brew install kubeseal

On Linux:

export KUBESEAL_VERSION="0.38.4"
curl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"
tar -xvzf kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Verify with kubeseal --version.

Namespace and controllers

kubectl create namespace mastodon

The Sealed Secrets controller, if you do not have one yet:

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.38.4/controller.yaml

And the Postgres operator. The --server-side flag is not optional here, the CRDs are too large for the classic annotation:

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.0.yaml

kubectl -n cnpg-system rollout status deployment/cnpg-controller-manager

The database

This is the part I got wrong in 2025. Back then it was a Deployment with replicas: 1, a PVC, and a pg_dump CronJob at four in the morning. That runs fine for months and has three problems, all of which surface at the least convenient time.

A Deployment rolls out with RollingUpdate by default, starting the new pod before the old one is gone. On a ReadWriteOnce volume the rollout then sits there until somebody intervenes. Kubernetes also sends SIGTERM on shutdown, which Postgres reads as a smart shutdown: wait until all clients disconnect on their own. Sidekiq does not disconnect, so after 30 seconds comes SIGKILL and on the next start a crash recovery. And a nightly dump means a mistake at 14:00 costs you half a day of federation.

The operator takes all of that off your hands. First the credentials for the backup store:

kubectl create secret generic mastodon-backup-creds \
  --namespace mastodon \
  --from-literal=ACCESS_KEY_ID=your_key \
  --from-literal=ACCESS_SECRET_KEY=your_secret \
  --dry-run=client -o yaml > backup-creds.yaml

kubeseal --controller-namespace kube-system \
  --controller-name sealed-secrets-controller \
  --format yaml < backup-creds.yaml > sealed-backup-creds.yaml

kubectl apply -f sealed-backup-creds.yaml && rm backup-creds.yaml

Then the cluster:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: mastodon-db
  namespace: mastodon
spec:
  instances: 2
  imageName: ghcr.io/cloudnative-pg/postgresql:18.4

  # On three small nodes, "preferred" is the difference between
  # "running" and "Pending, because the rule cannot be satisfied"
  affinity:
    enablePodAntiAffinity: true
    topologyKey: kubernetes.io/hostname
    podAntiAffinityType: preferred

  storage:
    size: 20Gi
  walStorage:
    size: 10Gi

  postgresql:
    parameters:
      shared_buffers: "512MB"
      effective_cache_size: "1536MB"
      max_connections: "200"
      work_mem: "8MB"
      maintenance_work_mem: "128MB"

  bootstrap:
    initdb:
      database: mastodon_production
      owner: mastodon
      postInitApplicationSQL:
        - CREATE EXTENSION IF NOT EXISTS pg_trgm;

  backup:
    barmanObjectStore:
      destinationPath: s3://your-bucket/mastodon
      endpointURL: https://your-s3-endpoint
      s3Credentials:
        accessKeyId:
          name: mastodon-backup-creds
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: mastodon-backup-creds
          key: ACCESS_SECRET_KEY
      wal:
        compression: gzip
      data:
        compression: gzip
    retentionPolicy: "30d"

  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      memory: "1Gi"

Four things in there need explaining.

walStorage as its own volume separates WAL from data. If the data volume fills up while the WAL volume still has room, your odds of a clean outcome are considerably better.

postInitApplicationSQL creates pg_trgm, which Mastodon needs for search. The application user under CNPG is deliberately not a superuser, and since Postgres 13 pg_trgm is marked trusted, so the database owner may create it. Which extensions your Mastodon version also insists on is in its db/schema.rb, and I look there before setting up rather than finding out during the migration.

The backup block does continuous WAL archiving. Not one state per night, but any point in time since the last base backup. A note on versions, because this is moving right now: as of CNPG 1.26 this built in barmanObjectStore form is deprecated in favour of the Barman Cloud plugin. It is still the default and it works, but if you are reading this later, check the documentation first on whether to go straight to the plugin.

And instances: 2 is a decision, not a requirement. On three nodes with 4 GB each, the second instance costs memory that Sidekiq would happily use. With instances: 1 you still get backups, point in time recovery, controlled updates and declarative configuration, just no automatic failover when a node dies. For a personal instance that is a legitimate trade, and I see no shame in running it that way.

The scheduled base backup that goes with it:

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: mastodon-db-daily
  namespace: mastodon
spec:
  # Six fields, not five. The first one is seconds.
  schedule: "0 0 3 * * *"
  backupOwnerReference: self
  cluster:
    name: mastodon-db

The cron expression caught me on the first attempt. CNPG uses a Go library with a seconds field, so a five field expression copied from a normal CronJob means something other than you intended.

The operator now creates three services: mastodon-db-rw always points at the current primary, mastodon-db-ro at the replicas, mastodon-db-r at all of them. That is exactly what a hand written Service with a fixed selector cannot do, because after a failover it would point at the wrong pod.

The operator generates the application user's password itself:

kubectl -n mastodon get secret mastodon-db-app \
  -o jsonpath='{.data.password}' | base64 -d

Valkey

For Mastodon, Valkey holds more than the cache: the Sidekiq queues live there too. Lose those and you lose in flight jobs, meaning outgoing deliveries, push notifications and media processing. So it gets a StatefulSet with its own volume, not a Deployment with a PVC attached like I had in 2025.

apiVersion: v1
kind: Service
metadata:
  name: valkey
  namespace: mastodon
spec:
  clusterIP: None
  selector:
    app: valkey
  ports:
    - port: 6379
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: valkey
  namespace: mastodon
spec:
  serviceName: valkey
  replicas: 1
  selector:
    matchLabels:
      app: valkey
  template:
    metadata:
      labels:
        app: valkey
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: valkey
          image: valkey/valkey:9.1-alpine
          args: ["--appendonly", "yes", "--appendfsync", "everysec"]
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: data
              mountPath: /data
          readinessProbe:
            exec:
              command: ["valkey-cli", "ping"]
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: "100m"
              memory: "256Mi"
            limits:
              memory: "512Mi"
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 5Gi

The 60 second grace period is not an arbitrary number. Valkey should be able to write its data out on shutdown, and the 30 second default is tight for that.

Secrets

These values get generated once and never changed. SECRET_KEY_BASE encrypts sessions, OTP_SECRET is tied to two factor authentication, the VAPID keys to push notifications. Swap them later and you throw away every login, every 2FA enrolment and every push subscription.

# Active Record encryption keys
docker run --rm ghcr.io/mastodon/mastodon:v4.6.4 \
  bin/rails db:encryption:init

# SECRET_KEY_BASE and OTP_SECRET, run twice
docker run --rm ghcr.io/mastodon/mastodon:v4.6.4 bundle exec rake secret

# VAPID key pair
docker run --rm ghcr.io/mastodon/mastodon:v4.6.4 \
  bundle exec rake mastodon:webpush:generate_vapid_key

Those go into the secret. DB_HOST points at the operator's -rw service:

apiVersion: v1
kind: Secret
metadata:
  name: mastodon-env
  namespace: mastodon
type: Opaque
stringData:
  LOCAL_DOMAIN: "your-domain.ch"
  WEB_DOMAIN: "your-domain.ch"
  SINGLE_USER_MODE: "true"
  RAILS_ENV: "production"

  SECRET_KEY_BASE: "..."
  OTP_SECRET: "..."
  VAPID_PRIVATE_KEY: "..."
  VAPID_PUBLIC_KEY: "..."
  ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY: "..."
  ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT: "..."
  ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY: "..."

  DB_HOST: "mastodon-db-rw"
  DB_NAME: "mastodon_production"
  DB_USER: "mastodon"
  DB_PASS: "generated_by_the_operator"

  REDIS_HOST: "valkey"
  REDIS_PORT: "6379"
  CACHE_REDIS_URL: "redis://valkey:6379/1"
  RAILS_CACHE_STORE: "redis_cache_store"

  SMTP_SERVER: "smtp.example.ch"
  SMTP_PORT: "587"
  SMTP_LOGIN: "mail@your-domain.ch"
  SMTP_PASSWORD: "..."
  SMTP_FROM_ADDRESS: "mail@your-domain.ch"

  S3_ENABLED: "true"
  S3_BUCKET: "your-bucket"
  S3_REGION: "..."
  S3_ENDPOINT: "https://your-s3-endpoint"
  S3_FORCE_PATH_STYLE: "true"
  S3_ALIAS_HOST: "cdn.your-domain.ch"
  AWS_ACCESS_KEY_ID: "..."
  AWS_SECRET_ACCESS_KEY: "..."

Seal it and delete the plaintext file:

kubeseal --controller-namespace kube-system \
  --controller-name sealed-secrets-controller \
  --format yaml < mastodon-env.yaml > sealed-mastodon-env.yaml

kubectl apply -f sealed-mastodon-env.yaml
rm mastodon-env.yaml

Migrations

The schema has to exist before the first start. As a Job, not an init container, so it does not run again on every pod restart:

apiVersion: batch/v1
kind: Job
metadata:
  name: mastodon-db-migrate
  namespace: mastodon
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: ghcr.io/mastodon/mastodon:v4.6.4
          command: ["bundle", "exec", "rails", "db:migrate"]
          envFrom:
            - secretRef:
                name: mastodon-env

You need this same job after every upgrade. Check the release notes first on whether the migration should run before or after the rollout.

Web, streaming and Sidekiq

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mastodon-web
  namespace: mastodon
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mastodon-web
  template:
    metadata:
      labels:
        app: mastodon-web
    spec:
      containers:
        - name: web
          image: ghcr.io/mastodon/mastodon:v4.6.4
          command: ["bundle", "exec", "puma", "-C", "config/puma.rb"]
          envFrom:
            - secretRef:
                name: mastodon-env
          env:
            - name: WEB_CONCURRENCY
              value: "1"
            - name: MAX_THREADS
              value: "3"
            - name: DB_POOL
              value: "5"
            - name: MALLOC_ARENA_MAX
              value: "2"
          ports:
            - containerPort: 3000
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 20
            periodSeconds: 10
          resources:
            requests:
              cpu: "300m"
              memory: "600Mi"
            limits:
              memory: "1.2Gi"

The PVC for /mastodon/public/system from my old guide is gone here. With S3_ENABLED: true the media live in object storage, the local directory is not needed, and that also removes the init container that used to fix up permissions.

The readiness probe was missing entirely in 2025. Without it the ingress sends requests to a pod where Puma is still starting.

Streaming is a separate process with its own image:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mastodon-streaming
  namespace: mastodon
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mastodon-streaming
  template:
    metadata:
      labels:
        app: mastodon-streaming
    spec:
      containers:
        - name: streaming
          image: ghcr.io/mastodon/mastodon-streaming:v4.6.4
          envFrom:
            - secretRef:
                name: mastodon-env
          ports:
            - containerPort: 4000
          readinessProbe:
            httpGet:
              path: /api/v1/streaming/health
              port: 4000
            initialDelaySeconds: 10
          resources:
            requests:
              cpu: "100m"
              memory: "200Mi"
            limits:
              memory: "400Mi"

Sidekiq does the federation work. The grace period matters more here than for the other services, because otherwise running jobs get cut off mid delivery:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mastodon-sidekiq
  namespace: mastodon
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mastodon-sidekiq
  template:
    metadata:
      labels:
        app: mastodon-sidekiq
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: sidekiq
          image: ghcr.io/mastodon/mastodon:v4.6.4
          command:
            - bundle
            - exec
            - sidekiq
            - -c
            - "5"
            - -q
            - default,8
            - -q
            - push,6
            - -q
            - ingress,4
            - -q
            - mailers,2
            - -q
            - pull,1
            - -q
            - scheduler,1
          envFrom:
            - secretRef:
                name: mastodon-env
          env:
            - name: DB_POOL
              value: "5"
            - name: MALLOC_ARENA_MAX
              value: "2"
          resources:
            requests:
              cpu: "500m"
              memory: "300Mi"
            limits:
              memory: "600Mi"

Plus the autoscaler, so that under load you get several small pods instead of one large one:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: mastodon-sidekiq
  namespace: mastodon
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: mastodon-sidekiq
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80

Being honest about scaling on CPU: during federation Sidekiq spends a lot of time waiting on other people's servers, so it is often not CPU bound. Queue latency would be the better signal, but that costs you a custom metrics adapter. For a small instance CPU is good enough, you should just know it is an approximation.

A word on DB_POOL: the value applies per process. One web pod with five, three Sidekiq pods with five each, plus connections for migrations, and it adds up faster than you expect. That is why max_connections: 200 is set above instead of the default 100. If you later put PgBouncer in front, there is one easily missed line to add: in transaction pooling mode PREPARED_STATEMENTS must be false, otherwise Rails' prepared statements collide with shared connections.

Services and ingress

apiVersion: v1
kind: Service
metadata:
  name: mastodon-web
  namespace: mastodon
spec:
  selector:
    app: mastodon-web
  ports:
    - port: 80
      targetPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: mastodon-streaming
  namespace: mastodon
spec:
  selector:
    app: mastodon-streaming
  ports:
    - port: 4000
      targetPort: 4000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mastodon
  namespace: mastodon
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - your-domain.ch
      secretName: your-domain-tls
  rules:
    - host: your-domain.ch
      http:
        paths:
          - path: /api/v1/streaming
            pathType: Prefix
            backend:
              service:
                name: mastodon-streaming
                port:
                  number: 4000
          - path: /
            pathType: Prefix
            backend:
              service:
                name: mastodon-web
                port:
                  number: 80

Two details: the streaming path comes before /, otherwise the more general rule matches first. And proxy-body-size needs raising, because nginx's default is too small for media uploads.

Cleaning up media

Without cleanup, object storage grows without bound, because every federated file gets cached.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mastodon-media-cleanup
  namespace: mastodon
spec:
  schedule: "0 3 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: cleanup
              image: ghcr.io/mastodon/mastodon:v4.6.4
              envFrom:
                - secretRef:
                    name: mastodon-env
              command:
                - /bin/bash
                - -c
                - |
                  bin/tootctl media remove --days=7
                  bin/tootctl media remove-orphans
                  bin/tootctl preview_cards remove --days=7
                  bin/tootctl statuses remove --days=14
                  bin/tootctl accounts prune

concurrencyPolicy: Forbid was missing in my old version. If a run takes longer than 24 hours, the next one otherwise starts alongside it, and two parallel cleanup jobs against the same database are not a good idea.

The backup that stopped hurting

In 2025 this was the longest and most unpleasant section of the whole guide. I had built my own Docker image: Ubuntu with AWS CLI, rclone and the Postgres client, plus a bash script with error checks after every step and a registry secret so the cluster could pull the image. It worked, and it was a lot of moving parts for a job that is really standard.

The ScheduledBackup above replaces all of it. The database goes to object storage continuously through WAL archiving, the base backup runs at night, and retentionPolicy handles how long things are kept.

The media no longer need their own backup if they already live in S3. What they need is versioning or replication on the provider side, and that is a bucket setting rather than a nightly rclone sync.

You can check the state like this:

kubectl -n mastodon get backups
kubectl -n mastodon get cluster mastodon-db \
  -o jsonpath='{.status.firstRecoverabilityPoint}'

And the part I did not have before: restoring to a point in time. It creates a new cluster while the old one keeps running untouched, so you can compare and then switch over:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: mastodon-db-restore
  namespace: mastodon
spec:
  instances: 1
  imageName: ghcr.io/cloudnative-pg/postgresql:18.4
  storage:
    size: 20Gi
  bootstrap:
    recovery:
      source: mastodon-db
      recoveryTarget:
        targetTime: "2026-07-27 14:00:00+02"
  externalClusters:
    - name: mastodon-db
      barmanObjectStore:
        destinationPath: s3://your-bucket/mastodon
        endpointURL: https://your-s3-endpoint
        s3Credentials:
          accessKeyId:
            name: mastodon-backup-creds
            key: ACCESS_KEY_ID
          secretAccessKey:
            name: mastodon-backup-creds
            key: ACCESS_SECRET_KEY

If you are coming from the old guide

If Postgres is still running as a Deployment for you: CNPG can import from an existing database as part of bootstrapping, instead of you juggling pg_dump and pg_restore.

spec:
  bootstrap:
    initdb:
      import:
        type: microservice
        databases:
          - mastodon_production
        source:
          externalCluster: old-instance
  externalClusters:
    - name: old-instance
      connectionParameters:
        host: postgres
        user: mastodon
        dbname: mastodon_production
      password:
        name: mastodon-env
        key: DB_PASS

The sequence: create the cluster with import and do not touch Mastodon yet. Compare row counts on the large tables against the old database. Scale web, streaming and Sidekiq to zero, and real downtime starts here. Repeat the import to close the gap. Point DB_HOST at mastodon-db-rw in the secret, scale back up, watch the logs.

Leave the old deployment and its PVC alone for a week afterwards. Do not delete it. The way back has to exist until you are sure you do not need it.

What changed since 2025

What I did not change: Sealed Secrets, skipping Helm, no Elasticsearch on small nodes, the Sidekiq autoscaler, and the nightly media cleanup. That was already right.

And one warning that belongs here for honesty: two Postgres instances protect you from a dead node, not from a DELETE without a WHERE. That replicates cleanly to both. High availability and backups solve two different problems, and only one of them can be fixed at three in the morning with a coffee.

If you rebuild this and get stuck somewhere, write to me. Last time the best corrections came from people who simply tried what I had written down.