Crossplane is a cloud-native Kubernetes add-on that lets you provision and manage infrastructure using Kubernetes manifests. It can provision resources like databases, message queues, and object storage buckets across various cloud providers, including AWS, GCP, and Azure.

When Crossplane provisions these external resources, it often needs to store the connection secrets for these resources in Kubernetes Secrets. These secrets typically contain sensitive information like API keys, usernames, passwords, and hostnames. By default, Crossplane creates these secrets in the same namespace where the CompositeResourceDefinition (CRD) and CompositeResource (XR) are defined, which is usually a control plane namespace (e.g., crossplane-system).

However, applications that need to consume these provisioned resources usually reside in separate application namespaces. To allow these applications to use the provisioned infrastructure, the connection secrets need to be accessible from their respective namespaces. Propagating these secrets from the control plane namespace to application namespaces is a common requirement.

Here’s how you can propagate connection secrets from Crossplane to your application namespaces:

The simplest approach, though not scalable or maintainable, is to manually copy the secrets.

  1. Identify the Secret: Find the secret created by Crossplane in the control plane namespace. It’s usually named after the CompositeResource (XR) with a -conn suffix. For example, if your XR is my-rds-instance, the secret might be my-rds-instance-conn.

    kubectl get secrets -n crossplane-system
    
  2. Copy the Secret: Use kubectl get and kubectl apply to copy the secret to the target application namespace.

    kubectl get secret my-rds-instance-conn -n crossplane-system -o yaml | \
    sed "s/namespace: crossplane-system/namespace: my-app-namespace/" | \
    kubectl apply -n my-app-namespace -f -
    

Why it works: This command retrieves the secret’s YAML definition, changes the namespace field to the application’s namespace, and then applies the modified YAML to create the secret in the new namespace.

Drawback: This is a manual process. If the original secret is updated by Crossplane (e.g., due to rotation or re-provisioning), you’d have to manually re-copy the updated secret. This is error-prone and doesn’t scale.

A more robust and automated approach involves using Kubernetes’ built-in mechanisms for managing secrets across namespaces. One effective pattern is to use a Secret aggregation or replication strategy.

Option 2.1: Using a Custom Controller or Operator

You can build or use an existing operator that watches for secrets in the control plane namespace and automatically copies or creates them in specified application namespaces. This is the most flexible but requires more development effort.

Option 2.2: Using kubectl with a Watcher Script

For a less custom, but still automated, solution, you can write a simple script that watches for secret changes and propagates them.

Example Script (Conceptual - requires jq and kubectl):

#!/bin/bash

CONTROL_PLANE_NS="crossplane-system"
APP_NS="my-app-namespace"
SECRET_NAME_PREFIX="my-rds-instance" # Prefix of the secret you want to propagate

# Function to propagate a secret
propagate_secret() {
  local secret_name=$1
  local ns=$2
  local target_ns=$3

  echo "Checking secret: $secret_name in namespace: $ns"
  if kubectl get secret "$secret_name" -n "$ns" > /dev/null 2>&1; then
    echo "Secret '$secret_name' found in '$ns'. Propagating to '$target_ns'."
    if kubectl get secret "$secret_name" -n "$target_ns" > /dev/null 2>&1; then
      echo "Secret '$secret_name' already exists in '$target_ns'. Deleting and recreating."
      kubectl delete secret "$secret_name" -n "$target_ns"
    fi
    kubectl get secret "$secret_name" -n "$ns" -o yaml | \
    sed "s/namespace: $ns/namespace: $target_ns/" | \
    kubectl apply -n "$target_ns" -f -
    echo "Secret '$secret_name' propagated to '$target_ns'."
  else
    echo "Secret '$secret_name' not found in '$ns'."
  fi
}

# Initial propagation
SECRET_TO_PROPAGATE="${SECRET_NAME_PREFIX}-conn"
propagate_secret "$SECRET_TO_PROPAGATE" "$CONTROL_PLANE_NS" "$APP_NS"

# Watch for changes (simplified, a real watcher would use 'kubectl get --watch')
echo "Watching for changes to secrets with prefix '$SECRET_NAME_PREFIX' in '$CONTROL_PLANE_NS'..."
# In a real-world scenario, you'd use `kubectl get secrets -n $CONTROL_PLANE_NS --watch`
# and parse the output to detect additions/modifications.
# For demonstration, we'll just re-run periodically.
while true; do
  sleep 60 # Check every minute
  # A more sophisticated watcher would detect new/updated secrets
  # and only re-propagate if necessary.
  # For simplicity here, we just re-run the propagation check.
  propagate_secret "$SECRET_TO_PROPAGATE" "$CONTROL_PLANE_NS" "$APP_NS"
done

To use this:

  1. Save the script (e.g., propagate_secrets.sh).
  2. Make it executable: chmod +x propagate_secrets.sh.
  3. Run it in your cluster: kubectl apply -f your-script-deployment.yaml (or run it directly in a pod).

Why it works: This script periodically checks for a specific secret in the control plane namespace. If found, it ensures a copy exists in the application namespace, deleting and recreating it if it already exists to pick up any updates. A proper watcher using kubectl watch would be more efficient.

Option 2.3: Using GitOps and Secret Management Tools

Tools like Argo CD, Flux, or HashiCorp Vault can be integrated to manage this propagation.

  • Argo CD/Flux: You can define a Secret resource in your Git repository that references the secret in the control plane namespace (using source_secret or similar mechanisms if supported by your GitOps tool) and specifies the target namespace.
  • HashiCorp Vault: You can use Vault’s Kubernetes secrets engine to dynamically generate or manage secrets, and then inject them into application pods or make them available as Kubernetes Secrets in the application namespaces.

Method 3: Crossplane’s Secret Resource Configuration

Crossplane itself offers a way to manage how secrets are created and where they are stored. When you define a CompositeResource (XR), you can specify the connectionSecret configuration.

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xsqsqueues.mycompany.com
spec:
  group: mycompany.com
  names:
    kind: XSQSQueue
    plural: xsqsqueues
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              # ... other spec properties ...
              compositionSelector:
                type: object
                properties:
                  matchLabels:
                    type: object
                    additionalProperties:
                      type: string
              writeConnectionSecretTo: # <-- This is what you configure
                type: object
                properties:
                  name: my-sqs-queue-conn # Default name for the secret
                  namespace: crossplane-system # Default namespace for the secret
            required:
            - compositionSelector
        required:
        - spec
---
apiVersion: mycompany.com/v1alpha1
kind: XSQSQueue
metadata:
  name: my-application-queue
  namespace: my-app-namespace # XR can be in app namespace too!
spec:
  compositionSelector:
    matchLabels:
      provider: aws
  parameters:
    # ... SQS queue parameters ...
  writeConnectionSecretTo: # <-- Override or specify here
    name: my-application-queue-conn
    namespace: my-app-namespace # <-- Target application namespace directly

Why it works: By setting writeConnectionSecretTo.namespace directly in the CompositeResource (XR) definition, you instruct Crossplane to create the connection secret in the specified application namespace from the outset. This eliminates the need for separate propagation steps.

Important Considerations:

  • RBAC: Ensure that the Crossplane service account has permissions to create secrets in the target application namespaces. You might need to grant it create, get, list, watch, and update permissions on secrets in those namespaces.
  • Secret Naming: Be consistent with secret naming. Using the XR’s name with a -conn suffix is a common convention.
  • Security: Secrets contain sensitive credentials. Handle them with care. Avoid committing them directly to Git unless encrypted. Use tools like Sealed Secrets or SOPS for managing secrets in GitOps workflows.
  • Dynamic Propagation: If your application namespaces are created dynamically, you’ll need a more sophisticated solution (like a custom controller or an event-driven watcher) to ensure secrets are propagated to new namespaces as they appear.

The most idiomatic and maintainable approach is often to define the writeConnectionSecretTo.namespace directly within the CompositeResource (XR) definition, ensuring the secret lands in the application namespace from the start. If your XR is already defined in the control plane namespace and you want to propagate to multiple application namespaces, a dedicated controller or a robust GitOps workflow becomes essential.

The next challenge you’ll likely encounter is ensuring that your applications are correctly configured to use these secrets, which often involves annotating Deployments or Pods to inject the secret values as environment variables or volumes.

Want structured learning?

Take the full Crossplane course →