Flux postBuild Broke My PL/pgSQL — One Annotation Fixed It

I spent hours debugging why my Postgres trigger function was failing with a syntax error. The function worked fine when I ran it manually. It worked in staging. But through FluxCD GitOps, the SQL got silently mangled before it ever reached the database.

Here’s what happened and the one-line fix.

The Setup

We use FluxCD to manage a Kubernetes cluster via GitOps. The Flux Kustomization has postBuild.substituteFrom enabled — this lets us use ${DOMAIN} style variables in YAML manifests, and Flux replaces them with actual values from a ConfigMap at apply time.

# Flux Kustomization
spec:
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars

This works great for Helm values, IngressRoutes, and certificates where you want ${DOMAIN} replaced with bvt.example.com.

The Problem

We had a ConfigMap containing a shell script and a PL/pgSQL trigger function:

apiVersion: v1
kind: ConfigMap
metadata:
  name: schema-scripts
data:
  run-scripts.sh: |
    #!/bin/bash
    if [ -z "${DB_USERNAME}" ]; then
      echo "Error: DB_USERNAME must be set"
      exit 1
    fi
    psql "postgres://${DB_USERNAME}:${DB_PASSWORD}@db:5432/mydb" -f /scripts/trigger.sql

  trigger.sql: |
    CREATE OR REPLACE FUNCTION set_default_slug()
    RETURNS TRIGGER AS $$
    BEGIN
      IF NEW.metadata IS NULL THEN
        NEW.metadata = '{"slug": "default"}'::jsonb;
      END IF;
      RETURN NEW;
    END;
    $$ LANGUAGE plpgsql;

The Job that mounted this ConfigMap kept failing. The errors were confusing:

Error: DB_USERNAME and DB_PASSWORD must be set

And when we fixed that, the SQL failed:

ERROR: syntax error at or near "$"
LINE 2: RETURNS TRIGGER AS $

The Investigation

I checked the ConfigMap on the cluster:

kubectl get configmap schema-scripts -o jsonpath='{.data.run-scripts\.sh}'

Output:

if [ -z "" ] || [ -z "" ]; then

The ${DB_USERNAME} and ${DB_PASSWORD} were replaced with empty strings. Flux’s postBuild saw ${DB_USERNAME} and tried to substitute it from the cluster-vars ConfigMap. Since those variables don’t exist in cluster-vars, they became empty.

For the SQL:

kubectl get configmap schema-scripts -o jsonpath='{.data.trigger\.sql}'

Output:

RETURNS TRIGGER AS $
$ LANGUAGE plpgsql;

The PL/pgSQL $$ delimiter (standard PostgreSQL quoting for function bodies) was converted to a single $. Flux treats $$ as an escape sequence meaning “literal $”.

The Failed Attempts

Attempt 1: Escape with $${} syntax

Flux docs say $${VAR} produces ${VAR}. So I tried $${DB_USERNAME} which correctly produced ${DB_USERNAME} in the shell script. But $$ in the SQL still got converted to $.

Attempt 2: Use $func$ instead of $$

PostgreSQL allows named delimiters like $func$. But Flux also mangled $func$ into $ — it strips content between $...$ patterns.

Attempt 3: Use $$$$ (four dollars)

Theory: $$ → $, so $$$$ → $$. In practice, Flux processes it in a single pass and I got $ again.

Attempt 4: Base64 encode the SQL

Store the SQL as base64 in the ConfigMap, decode at runtime. Works, but ugly and unmaintainable. Nobody wants to review base64-encoded SQL in a PR.

The Fix

One annotation on the ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: schema-scripts
  annotations:
    kustomize.toolkit.fluxcd.io/substitute: disabled
data:
  run-scripts.sh: |
    #!/bin/bash
    if [ -z "${DB_USERNAME}" ]; then
      ...

That’s it. Flux skips variable substitution for this specific resource. The $$ stays as $$. The ${DB_USERNAME} stays as ${DB_USERNAME} (resolved by bash at runtime, not by Flux).

Everything else in the Kustomization still gets variable substitution. Only this ConfigMap is excluded.

The Rules

Flux postBuild substitution:

If your ConfigMap contains shell scripts, SQL, or any language that uses $ syntax — and you have postBuild enabled on the Kustomization — add the annotation:

annotations:
  kustomize.toolkit.fluxcd.io/substitute: disabled

Docs: Flux postBuild Variable Substitution

Takeaway

The substitution is useful for domain names and environment-specific values. But it’s a global text processor — it doesn’t understand YAML structure, shell syntax, or SQL. It just pattern-matches ${...} and $$ across all text in all resources.

When you mix infrastructure variables with application code in the same Kustomization, use the substitute: disabled annotation on resources that contain code. Keep the variables for resources that need them.