Prometheus doesn’t actually scrape metrics; it pulls them from targets that expose an HTTP endpoint.
Here’s how you’d set up Prometheus to scrape metrics from Crossplane:
The Goal: Getting Crossplane’s Internal State into Prometheus
Crossplane, being a control plane, is constantly observing the state of your managed resources and comparing it to your desired state. This internal activity generates a lot of useful metrics that tell you about reconciliation loops, resource counts, API server interactions, and more. Prometheus is the de facto standard for collecting and alerting on these kinds of operational metrics in Kubernetes.
How Crossplane Exposes Metrics
Crossplane components (like the crossplane core controller and any provider controllers) expose their metrics on a /metrics HTTP endpoint. This endpoint adheres to the OpenMetrics format, which Prometheus understands.
Setting Up Prometheus to Scrape Crossplane
-
Ensure Crossplane is Running: You need a running Crossplane installation. This typically involves a
crossplane-systemnamespace and pods for the core Crossplane components and any installed providers. -
Prometheus Operator and
ServiceMonitor: The most common way to manage Prometheus in Kubernetes is via the Prometheus Operator. This operator uses custom resources likeServiceMonitorto tell Prometheus which services to scrape.Here’s a sample
ServiceMonitordefinition that targets Crossplane’s core metrics. You’ll likely need to adjustnamespaceSelectorandselectorbased on your Crossplane installation.apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: crossplane-core namespace: monitoring # The namespace where your Prometheus Operator is installed labels: release: prometheus # This label should match the label selector for Prometheus in your Prometheus Operator config spec: namespaceSelector: matchNames: - crossplane-system # The namespace where Crossplane is installed selector: matchLabels: app: crossplane # This label should match the labels on the Crossplane service endpoints: - port: metrics interval: 15s # How often to scrape metrics path: /metrics # The path where metrics are exposedmetadata.namespace: This must be the namespace where your Prometheus Operator is deployed and configured to watch forServiceMonitorresources.metadata.labels: Therelease: prometheuslabel (or whatever you’ve configured for your Prometheus instance) is crucial. The Prometheus Operator uses this to associateServiceMonitorresources with specific Prometheus instances.spec.namespaceSelector.matchNames: This tells the operator to look for services in thecrossplane-systemnamespace (or wherever your Crossplane pods are running).spec.selector.matchLabels: This is how theServiceMonitorfinds the correct KubernetesServiceobject that points to the Crossplane pods. You’ll need to inspect your Crossplane service’s labels (e.g.,kubectl get svc -n crossplane-system). Often, it’s something likeapp: crossplaneorcomponent: crossplane.spec.endpoints:port: This must match thenameof the port in the KubernetesServicethat exposes the metrics endpoint. For Crossplane, this is typically namedmetrics.interval: How frequently Prometheus should scrape this endpoint.15sis a common default.path: The HTTP path where metrics are exposed. For most Kubernetes applications, this is/metrics.
-
Apply the
ServiceMonitor: Save the YAML above to a file (e.g.,crossplane-servicemonitor.yaml) and apply it:kubectl apply -f crossplane-servicemonitor.yaml -
Verify in Prometheus UI: Once applied, your Prometheus instance (managed by the operator) should automatically discover and start scraping the Crossplane metrics. You can verify this in the Prometheus UI:
- Navigate to your Prometheus UI (often accessible via port-forwarding or an Ingress).
- Go to
Status->Targets. - You should see an entry for
crossplane-core(or whatever you named yourServiceMonitor) with a state ofUP.
What You Can Scrape
By default, the crossplane core controller exposes metrics like:
crossplane_controller_runtime_reconcile_errors_total: Number of reconciliation errors.crossplane_controller_runtime_reconcile_total: Total number of reconciliation attempts.crossplane_controller_runtime_reconcile_duration_seconds: Latency of reconciliation loops.crossplane_managed_resource_count: Number of managed resources by kind and provider.crossplane_api_request_total: API requests made to external providers.
If you have providers installed (e.g., AWS, Azure, GCP), they will also expose their own provider-specific metrics, which you can target with separate ServiceMonitor resources, adjusting the selector and port accordingly.
The Counterintuitive Part: Not Just the Core
Many users assume that Prometheus scraping is just about the crossplane pod. However, each provider you install (e.g., provider-aws, provider-azure) runs as its own set of pods and exposes its own /metrics endpoint. To scrape metrics from, say, provider-aws, you’d create a separate ServiceMonitor targeting the service associated with the provider-aws pods, which will have different labels.
What’s Next?
After successfully scraping Crossplane metrics, the next logical step is to create Prometheus alerts based on these metrics to proactively detect issues.