Log Centralization: ELK vs. Splunk vs. Loki

The surprising truth about log aggregation is that the "best" stack isn’t about more features, but about the least friction for your specific team and use case.

Let’s see Loki in action. Imagine you’re running a few microservices, each spitting out logs to stdout. You’ve got Docker Compose set up, and you want to see what’s happening.

First, you need a promtail agent running alongside your services. This is the "tailer" that reads your logs. Here’s a snippet of its config:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker-containers
    static_configs:
      - targets:
          - localhost
        labels:
          job: my-app
          __path__: /var/lib/docker/containers/*/*-json.log
    pipeline_stages:
      - docker:
          container_id_regex: (.*)
      - labels:
          container_id:
          app_name:
            source: labels.job
      - timestamp:
          source: time
          format: RFC3339Nano
      - json:
          expressions:
            message:
      - logfmt:
          mapping:
            level:

This promtail config tells it to:

• Listen on port 9080 (for Prometheus metrics, not logs themselves).
• Store its current read position in /tmp/positions.yaml so it doesn’t re-read old logs.
• Send logs to your Loki instance at http://loki:3100.
• Scrape logs from Docker containers (/var/lib/docker/containers/*/*-json.log).
• Use the docker stage to extract container metadata.
• Create labels like job and container_id.
• Parse timestamps using RFC3339Nano format.
• Extract the actual log message using a json stage.
• Parse logfmt for a level field.

Your Loki instance itself is surprisingly simple. Here’s a minimal loki.yaml:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      directory: /loki-data

ingester:
  chunk_block_size: 262144
  chunk_idle_period: 1h
  chunk_retain_period: 48h

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

This Loki config:

• Disables authentication (for simplicity here).
• Listens on port 3100.
• Stores data (/loki-data) and index files locally on the filesystem.
• Sets up chunking parameters for how logs are batched in storage.
• Uses boltdb-shipper for indexing, which is good for smaller deployments, and stores index data locally.

Now, with promtail running in your Docker environment and Loki up, you can use Grafana to query. If your promtail is scraping logs with labels job="my-app" and container_id="some-id", you can query in Grafana like this:

{job="my-app"}

This will show you all logs from containers labeled job="my-app". You can refine it:

{job="my-app", container_id="some-id"}

Or filter by content:

{job="my-app"} | json | level="error"

This query asks for logs from my-app that, after being parsed as JSON, have a level field equal to error.

The core problem Loki solves is separating log indexing from log storage. Unlike ELK where Elasticsearch is both the index and the storage, Loki only indexes metadata (labels). The actual log content is stored in object storage (like S3, GCS, or even just the local filesystem for small setups). This dramatically reduces the cost and complexity of running the system because you’re not indexing every single word in every log line. You’re only indexing the labels you define.

Most people don’t realize that Loki’s query language, LogQL, is heavily inspired by Prometheus’s PromQL. This means if you’re already familiar with querying metrics in Prometheus, you’ll find LogQL quite intuitive. The | operator acts as a pipe, allowing you to apply transformations and filters after the initial label selection. For instance, | json parses the log line as JSON, | logfmt parses it as logfmt, and | regexp "<your_regex>" can extract specific patterns. You can even perform aggregation using LogQL, similar to PromQL, but on log streams.

The next concept you’ll likely encounter is advanced label management and the impact of label cardinality on query performance.