Prometheus doesn’t actually scrape metrics; it pulls them from targets that expose an HTTP endpoint.

Here’s how you’d set up Prometheus to scrape metrics from Crossplane:

The Goal: Getting Crossplane’s Internal State into Prometheus

Crossplane, being a control plane, is constantly observing the state of your managed resources and comparing it to your desired state. This internal activity generates a lot of useful metrics that tell you about reconciliation loops, resource counts, API server interactions, and more. Prometheus is the de facto standard for collecting and alerting on these kinds of operational metrics in Kubernetes.

How Crossplane Exposes Metrics

Crossplane components (like the crossplane core controller and any provider controllers) expose their metrics on a /metrics HTTP endpoint. This endpoint adheres to the OpenMetrics format, which Prometheus understands.

Setting Up Prometheus to Scrape Crossplane

  1. Ensure Crossplane is Running: You need a running Crossplane installation. This typically involves a crossplane-system namespace and pods for the core Crossplane components and any installed providers.

  2. Prometheus Operator and ServiceMonitor: The most common way to manage Prometheus in Kubernetes is via the Prometheus Operator. This operator uses custom resources like ServiceMonitor to tell Prometheus which services to scrape.

    Here’s a sample ServiceMonitor definition that targets Crossplane’s core metrics. You’ll likely need to adjust namespaceSelector and selector based on your Crossplane installation.

    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: crossplane-core
      namespace: monitoring # The namespace where your Prometheus Operator is installed
      labels:
        release: prometheus # This label should match the label selector for Prometheus in your Prometheus Operator config
    spec:
      namespaceSelector:
        matchNames:
          - crossplane-system # The namespace where Crossplane is installed
      selector:
        matchLabels:
          app: crossplane # This label should match the labels on the Crossplane service
      endpoints:
        - port: metrics
          interval: 15s # How often to scrape metrics
          path: /metrics # The path where metrics are exposed
    
    • metadata.namespace: This must be the namespace where your Prometheus Operator is deployed and configured to watch for ServiceMonitor resources.
    • metadata.labels: The release: prometheus label (or whatever you’ve configured for your Prometheus instance) is crucial. The Prometheus Operator uses this to associate ServiceMonitor resources with specific Prometheus instances.
    • spec.namespaceSelector.matchNames: This tells the operator to look for services in the crossplane-system namespace (or wherever your Crossplane pods are running).
    • spec.selector.matchLabels: This is how the ServiceMonitor finds the correct Kubernetes Service object that points to the Crossplane pods. You’ll need to inspect your Crossplane service’s labels (e.g., kubectl get svc -n crossplane-system). Often, it’s something like app: crossplane or component: crossplane.
    • spec.endpoints:
      • port: This must match the name of the port in the Kubernetes Service that exposes the metrics endpoint. For Crossplane, this is typically named metrics.
      • interval: How frequently Prometheus should scrape this endpoint. 15s is a common default.
      • path: The HTTP path where metrics are exposed. For most Kubernetes applications, this is /metrics.
  3. Apply the ServiceMonitor: Save the YAML above to a file (e.g., crossplane-servicemonitor.yaml) and apply it:

    kubectl apply -f crossplane-servicemonitor.yaml
    
  4. Verify in Prometheus UI: Once applied, your Prometheus instance (managed by the operator) should automatically discover and start scraping the Crossplane metrics. You can verify this in the Prometheus UI:

    • Navigate to your Prometheus UI (often accessible via port-forwarding or an Ingress).
    • Go to Status -> Targets.
    • You should see an entry for crossplane-core (or whatever you named your ServiceMonitor) with a state of UP.

What You Can Scrape

By default, the crossplane core controller exposes metrics like:

  • crossplane_controller_runtime_reconcile_errors_total: Number of reconciliation errors.
  • crossplane_controller_runtime_reconcile_total: Total number of reconciliation attempts.
  • crossplane_controller_runtime_reconcile_duration_seconds: Latency of reconciliation loops.
  • crossplane_managed_resource_count: Number of managed resources by kind and provider.
  • crossplane_api_request_total: API requests made to external providers.

If you have providers installed (e.g., AWS, Azure, GCP), they will also expose their own provider-specific metrics, which you can target with separate ServiceMonitor resources, adjusting the selector and port accordingly.

The Counterintuitive Part: Not Just the Core

Many users assume that Prometheus scraping is just about the crossplane pod. However, each provider you install (e.g., provider-aws, provider-azure) runs as its own set of pods and exposes its own /metrics endpoint. To scrape metrics from, say, provider-aws, you’d create a separate ServiceMonitor targeting the service associated with the provider-aws pods, which will have different labels.

What’s Next?

After successfully scraping Crossplane metrics, the next logical step is to create Prometheus alerts based on these metrics to proactively detect issues.

Want structured learning?

Take the full Crossplane course →