Structured logging in Flask for production is less about what you log and more about how you format it to make it machine-readable and queryable.
Here’s a Flask app with basic logging:
from flask import Flask, request
import logging
app = Flask(__name__)
# Configure basic logging
logging.basicConfig(level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
@app.route('/')
def index():
app.logger.info(f"Request received for path: {request.path}")
return "Hello, World!"
if __name__ == '__main__':
app.run(debug=False)
When you hit /, you’ll see something like:
2023-10-27 10:30:00,123 - app - INFO - Request received for path: /
This is functional, but it’s hard to filter by path or extract request.path programmatically when you have thousands of log lines.
Structured logging turns that text into a JSON object, making it easy to ingest into log aggregation systems like Elasticsearch, Splunk, or Datadog. Each log event becomes a distinct record with key-value pairs.
Let’s integrate python-json-logger for structured output.
First, install the library:
pip install python-json-logger
Now, modify the Flask app:
from flask import Flask, request
import logging
from pythonjsonlogger import jsonlogger
app = Flask(__name__)
# Configure structured logging
formatter = jsonlogger.JsonFormatter()
logHandler = logging.StreamHandler()
logHandler.setFormatter(formatter)
app.logger.handlers.clear() # Remove default handlers
app.logger.addHandler(logHandler)
app.logger.setLevel(logging.INFO)
@app.route('/')
def index():
app.logger.info("Request received", extra={'path': request.path, 'method': request.method})
return "Hello, World!"
if __name__ == '__main__':
app.run(debug=False)
When you hit / now, the output will be a JSON string:
{"asctime": "2023-10-27 10:35:00,456", "name": "app", "levelname": "INFO", "message": "Request received", "path": "/", "method": "GET"}
This JSON output is invaluable. You can now easily query for all GET requests to the / path, or filter logs by a specific request_id if you add that as an extra field.
The core idea is to pass a dictionary as the extra argument to your logging calls. The JsonFormatter then merges these extra fields into the final JSON log record. You can add any context you need: user IDs, request IDs, performance timings, specific application states.
For example, to add a unique request ID:
import uuid
from flask import Flask, request
import logging
from pythonjsonlogger import jsonlogger
app = Flask(__name__)
formatter = jsonlogger.JsonFormatter()
logHandler = logging.StreamHandler()
logHandler.setFormatter(formatter)
app.logger.handlers.clear()
app.logger.addHandler(logHandler)
app.logger.setLevel(logging.INFO)
@app.before_request
def add_request_id():
g.request_id = str(uuid.uuid4())
app.logger.info("Request started", extra={'request_id': g.request_id})
@app.after_request
def log_request(response):
app.logger.info("Request finished", extra={'request_id': g.request_id, 'status_code': response.status_code})
return response
@app.route('/')
def index():
app.logger.info("Processing index route", extra={'request_id': g.request_id})
return "Hello, World!"
if __name__ == '__main__':
# Use Gunicorn for production to see logs in action
# Example: gunicorn -w 4 your_app:app
app.run(debug=False)
In this snippet, g.request_id is a Flask global object that’s available within the request context. The before_request and after_request decorators ensure that a unique ID is generated for each incoming request and logged at the start and end. This allows you to trace a single request’s journey through your application.
The JsonFormatter can be customized to include or exclude specific fields, or to rename them. For instance, if you want to ensure levelname is always the top-level key for severity, you can subclass JsonFormatter and override process_log_record.
A common pattern for production is to log to stdout or stderr and let your container orchestrator (like Kubernetes) or process manager (like systemd or supervisord) handle redirecting these streams to your chosen log aggregation service. This decouples your application from the logging infrastructure.
The real power comes when you start adding context that’s specific to your application’s domain. Instead of just logging "User logged in," you log {"message": "User logged in", "user_id": 123, "email": "test@example.com", "ip_address": "192.168.1.1"}. This is a game-changer for debugging and monitoring.
Most people don’t realize that the extra dictionary in Python’s logging can contain arbitrary Python objects, not just simple strings or numbers. The JsonFormatter will attempt to serialize these. However, if you have complex objects, you might need to define a custom serializer or ensure they have a __str__ or __repr__ method that produces a useful string representation. For instance, logging a datetime object directly will result in its ISO 8601 string representation by default, which is usually what you want.
The next step is to integrate this with a robust log shipper and a centralized logging platform.