Crossplane is a cloud-native Kubernetes add-on that lets you provision and manage infrastructure using Kubernetes manifests. It can provision resources like databases, message queues, and object storage buckets across various cloud providers, including AWS, GCP, and Azure.
When Crossplane provisions these external resources, it often needs to store the connection secrets for these resources in Kubernetes Secrets. These secrets typically contain sensitive information like API keys, usernames, passwords, and hostnames. By default, Crossplane creates these secrets in the same namespace where the CompositeResourceDefinition (CRD) and CompositeResource (XR) are defined, which is usually a control plane namespace (e.g., crossplane-system).
However, applications that need to consume these provisioned resources usually reside in separate application namespaces. To allow these applications to use the provisioned infrastructure, the connection secrets need to be accessible from their respective namespaces. Propagating these secrets from the control plane namespace to application namespaces is a common requirement.
Here’s how you can propagate connection secrets from Crossplane to your application namespaces:
Method 1: Manual Secret Copying (Not Recommended for Production)
The simplest approach, though not scalable or maintainable, is to manually copy the secrets.
-
Identify the Secret: Find the secret created by Crossplane in the control plane namespace. It’s usually named after the
CompositeResource(XR) with a-connsuffix. For example, if your XR ismy-rds-instance, the secret might bemy-rds-instance-conn.kubectl get secrets -n crossplane-system -
Copy the Secret: Use
kubectl getandkubectl applyto copy the secret to the target application namespace.kubectl get secret my-rds-instance-conn -n crossplane-system -o yaml | \ sed "s/namespace: crossplane-system/namespace: my-app-namespace/" | \ kubectl apply -n my-app-namespace -f -
Why it works: This command retrieves the secret’s YAML definition, changes the namespace field to the application’s namespace, and then applies the modified YAML to create the secret in the new namespace.
Drawback: This is a manual process. If the original secret is updated by Crossplane (e.g., due to rotation or re-provisioning), you’d have to manually re-copy the updated secret. This is error-prone and doesn’t scale.
Method 2: Using Kubernetes Secret Aggregation (Recommended)
A more robust and automated approach involves using Kubernetes’ built-in mechanisms for managing secrets across namespaces. One effective pattern is to use a Secret aggregation or replication strategy.
Option 2.1: Using a Custom Controller or Operator
You can build or use an existing operator that watches for secrets in the control plane namespace and automatically copies or creates them in specified application namespaces. This is the most flexible but requires more development effort.
Option 2.2: Using kubectl with a Watcher Script
For a less custom, but still automated, solution, you can write a simple script that watches for secret changes and propagates them.
Example Script (Conceptual - requires jq and kubectl):
#!/bin/bash
CONTROL_PLANE_NS="crossplane-system"
APP_NS="my-app-namespace"
SECRET_NAME_PREFIX="my-rds-instance" # Prefix of the secret you want to propagate
# Function to propagate a secret
propagate_secret() {
local secret_name=$1
local ns=$2
local target_ns=$3
echo "Checking secret: $secret_name in namespace: $ns"
if kubectl get secret "$secret_name" -n "$ns" > /dev/null 2>&1; then
echo "Secret '$secret_name' found in '$ns'. Propagating to '$target_ns'."
if kubectl get secret "$secret_name" -n "$target_ns" > /dev/null 2>&1; then
echo "Secret '$secret_name' already exists in '$target_ns'. Deleting and recreating."
kubectl delete secret "$secret_name" -n "$target_ns"
fi
kubectl get secret "$secret_name" -n "$ns" -o yaml | \
sed "s/namespace: $ns/namespace: $target_ns/" | \
kubectl apply -n "$target_ns" -f -
echo "Secret '$secret_name' propagated to '$target_ns'."
else
echo "Secret '$secret_name' not found in '$ns'."
fi
}
# Initial propagation
SECRET_TO_PROPAGATE="${SECRET_NAME_PREFIX}-conn"
propagate_secret "$SECRET_TO_PROPAGATE" "$CONTROL_PLANE_NS" "$APP_NS"
# Watch for changes (simplified, a real watcher would use 'kubectl get --watch')
echo "Watching for changes to secrets with prefix '$SECRET_NAME_PREFIX' in '$CONTROL_PLANE_NS'..."
# In a real-world scenario, you'd use `kubectl get secrets -n $CONTROL_PLANE_NS --watch`
# and parse the output to detect additions/modifications.
# For demonstration, we'll just re-run periodically.
while true; do
sleep 60 # Check every minute
# A more sophisticated watcher would detect new/updated secrets
# and only re-propagate if necessary.
# For simplicity here, we just re-run the propagation check.
propagate_secret "$SECRET_TO_PROPAGATE" "$CONTROL_PLANE_NS" "$APP_NS"
done
To use this:
- Save the script (e.g.,
propagate_secrets.sh). - Make it executable:
chmod +x propagate_secrets.sh. - Run it in your cluster:
kubectl apply -f your-script-deployment.yaml(or run it directly in a pod).
Why it works: This script periodically checks for a specific secret in the control plane namespace. If found, it ensures a copy exists in the application namespace, deleting and recreating it if it already exists to pick up any updates. A proper watcher using kubectl watch would be more efficient.
Option 2.3: Using GitOps and Secret Management Tools
Tools like Argo CD, Flux, or HashiCorp Vault can be integrated to manage this propagation.
- Argo CD/Flux: You can define a
Secretresource in your Git repository that references the secret in the control plane namespace (usingsource_secretor similar mechanisms if supported by your GitOps tool) and specifies the target namespace. - HashiCorp Vault: You can use Vault’s Kubernetes secrets engine to dynamically generate or manage secrets, and then inject them into application pods or make them available as Kubernetes Secrets in the application namespaces.
Method 3: Crossplane’s Secret Resource Configuration
Crossplane itself offers a way to manage how secrets are created and where they are stored. When you define a CompositeResource (XR), you can specify the connectionSecret configuration.
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
name: xsqsqueues.mycompany.com
spec:
group: mycompany.com
names:
kind: XSQSQueue
plural: xsqsqueues
versions:
- name: v1alpha1
served: true
referenceable: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
# ... other spec properties ...
compositionSelector:
type: object
properties:
matchLabels:
type: object
additionalProperties:
type: string
writeConnectionSecretTo: # <-- This is what you configure
type: object
properties:
name: my-sqs-queue-conn # Default name for the secret
namespace: crossplane-system # Default namespace for the secret
required:
- compositionSelector
required:
- spec
---
apiVersion: mycompany.com/v1alpha1
kind: XSQSQueue
metadata:
name: my-application-queue
namespace: my-app-namespace # XR can be in app namespace too!
spec:
compositionSelector:
matchLabels:
provider: aws
parameters:
# ... SQS queue parameters ...
writeConnectionSecretTo: # <-- Override or specify here
name: my-application-queue-conn
namespace: my-app-namespace # <-- Target application namespace directly
Why it works: By setting writeConnectionSecretTo.namespace directly in the CompositeResource (XR) definition, you instruct Crossplane to create the connection secret in the specified application namespace from the outset. This eliminates the need for separate propagation steps.
Important Considerations:
- RBAC: Ensure that the Crossplane service account has permissions to create secrets in the target application namespaces. You might need to grant it
create,get,list,watch, andupdatepermissions onsecretsin those namespaces. - Secret Naming: Be consistent with secret naming. Using the XR’s name with a
-connsuffix is a common convention. - Security: Secrets contain sensitive credentials. Handle them with care. Avoid committing them directly to Git unless encrypted. Use tools like Sealed Secrets or SOPS for managing secrets in GitOps workflows.
- Dynamic Propagation: If your application namespaces are created dynamically, you’ll need a more sophisticated solution (like a custom controller or an event-driven watcher) to ensure secrets are propagated to new namespaces as they appear.
The most idiomatic and maintainable approach is often to define the writeConnectionSecretTo.namespace directly within the CompositeResource (XR) definition, ensuring the secret lands in the application namespace from the start. If your XR is already defined in the control plane namespace and you want to propagate to multiple application namespaces, a dedicated controller or a robust GitOps workflow becomes essential.
The next challenge you’ll likely encounter is ensuring that your applications are correctly configured to use these secrets, which often involves annotating Deployments or Pods to inject the secret values as environment variables or volumes.