Azure diagnostic logs are a treasure trove of operational data, but by default, they’re ephemeral. This guide explains how to make them persistent and accessible for analysis, whether you’re shipping them to a Storage Account for long-term retention or a Security Information and Event Management (SIEM) system for threat detection.
The most surprising thing about Azure diagnostic logs is that they aren’t logs in the traditional sense; they’re events emitted by Azure resources that are then captured and routed. This distinction is crucial because it means you’re not just "turning on logging" but rather configuring an event subscription.
Let’s see this in action. Imagine you have a Virtual Machine and want to capture its boot diagnostics.
First, navigate to your Virtual Machine in the Azure portal.
Under "Monitoring," select "Diagnostic settings."
Click "Turn on diagnostics."
You’ll see options to send logs and metrics. For this example, we’ll focus on logs.
You can choose to send to a "Storage account," "Log Analytics workspace," or "Event Hubs." We’ll select "Storage account" for demonstration.
If you don’t have a storage account, you’ll need to create one. Choose "Create new," give it a name (e.g., diagstoracct20230101), select a region, and click "OK."
Back in the diagnostic settings, select the "Storage account" you just created.
Now, you need to specify which logs to capture. For VM boot diagnostics, you’d typically select VMGuestOSBootLogs and VMGuestOSPlatformLogs. You can also select metrics if desired.
Click "Save."
Once saved, Azure will begin sending the selected diagnostic logs to the specified storage account. You can then access these logs by navigating to the Storage Account, going to "Containers" under "Blob service," and looking for a container named insights-logs-vmgl-vmguestosbootlogs (the exact name might vary slightly based on the log category). Inside, you’ll find a hierarchical structure of folders representing the subscription, resource group, resource type, resource name, and date, with .json files containing the actual log entries.
The core problem diagnostic logs solve is the lack of historical, queryable operational data for Azure resources. Without them, troubleshooting failures, auditing changes, or understanding performance degradation becomes a guessing game. By routing these logs, you create a persistent audit trail and a data source for advanced analytics.
Internally, diagnostic settings leverage Azure Event Grid or Azure Monitor’s data collection infrastructure. When a resource emits a diagnostic event, it’s captured and then routed according to your configured diagnostic setting. This routing can be thought of as a subscription to a specific type of event emitted by a resource.
The key levers you control are:
- Resource: Which Azure resource’s logs you want to capture (VMs, App Services, Key Vaults, etc.).
- Log Categories: The specific types of events to collect from a resource (e.g.,
VMGuestOSBootLogs,ApplicationGatewayAccessLog,KeyVaultLogs). Each resource type has its own set of available categories. - Destination: Where the logs go.
- Storage Account: For long-term, cost-effective archival. Access is via Blob Storage.
- Log Analytics Workspace: For immediate querying and analysis using Kusto Query Language (KQL). This is the most powerful option for real-time troubleshooting.
- Event Hubs: For streaming logs to third-party SIEMs (like Splunk, QRadar) or custom applications.
When configuring for a SIEM via Event Hubs, you’re essentially setting up a bridge. Azure Monitor collects the logs, and Event Hubs acts as a highly scalable, durable buffer that your SIEM can ingest from. The SIEM then performs its analysis, correlation, and alerting. You’ll need to create an Event Hubs Namespace and an Event Hub within it, then grant the Azure Monitor service principal permissions to send data to that hub.
A common point of confusion is the difference between "Diagnostic settings" and "Resource logs." Diagnostic settings are the mechanism for exporting logs, while "resource logs" are the data being exported. You configure diagnostic settings to export specific resource logs.
The next hurdle you’ll likely encounter is effectively querying and analyzing the vast amounts of data you’ve collected, especially if you’ve opted for a Storage Account and need to build your own analysis pipeline.