Chaos engineering is the practice of proactively injecting failures into a distributed system to uncover weaknesses before they impact real users. The most surprising true thing about it is that the goal isn’t to break things, but to understand them better.
Imagine a system with three microservices: Auth, UserDB, and Profile. Auth authenticates users, UserDB stores user credentials, and Profile manages user profile data. When a user logs in, Auth talks to UserDB to verify credentials, then to Profile to fetch profile information.
Here’s a simplified Python example using requests to simulate this:
import requests
import random
import time
AUTH_URL = "http://localhost:5001/auth"
USERDB_URL = "http://localhost:5002/user"
PROFILE_URL = "http://localhost:5003/profile"
def login_user(username, password):
try:
# Step 1: Authenticate
auth_response = requests.post(AUTH_URL, json={"username": username, "password": password})
auth_response.raise_for_status() # Will raise an exception for bad status codes
auth_token = auth_response.json()["token"]
print(f"Authentication successful, token: {auth_token[:8]}...")
# Step 2: Get user ID (simulated, in reality auth might return this)
user_response = requests.get(USERDB_URL, headers={"Authorization": f"Bearer {auth_token}"})
user_response.raise_for_status()
user_id = user_response.json()["user_id"]
print(f"User ID retrieved: {user_id}")
# Step 3: Get profile
profile_response = requests.get(PROFILE_URL, params={"user_id": user_id})
profile_response.raise_for_status()
profile_data = profile_response.json()
print(f"Profile fetched for user {user_id}: {profile_data}")
return True, "Login successful"
except requests.exceptions.RequestException as e:
print(f"Login failed: {e}")
return False, str(e)
if __name__ == "__main__":
# Simulate some login attempts
for _ in range(5):
success, message = login_user("testuser", "password123")
if success:
print("User login flow completed successfully.")
else:
print(f"User login flow failed: {message}")
time.sleep(random.uniform(0.5, 2.0))
This login_user function represents a critical user journey. Auth might have a timeout of 2 seconds to UserDB, and UserDB might have a timeout of 1 second to Profile. If any of these requests fail or take too long, the entire login process fails.
Chaos engineering injects controlled failures into this system to see how it behaves. You might introduce:
- Network Latency: Slow down requests between
AuthandUserDBby 500ms. - Service Unavailability: Temporarily stop the
Profileservice. - Resource Exhaustion: Simulate high CPU or memory usage on the
UserDBinstance. - Error Injection: Make the
UserDBreturn a 500 Internal Server Error for 1% of requests.
These experiments are typically run in a staging or even production environment (with careful controls and blast radius limitations). Tools like Gremlin, Chaos Mesh, or LitmusChaos automate the injection of these failures.
The core problem chaos engineering solves is the "it works on my machine" or "it worked in staging" fallacy. Distributed systems are complex; dependencies, network issues, and cascading failures are inevitable. You can’t test every permutation of failure manually. Chaos engineering provides a systematic way to discover these weak points.
You control chaos experiments through:
- Experiments: The specific type of failure to inject (e.g.,
network-latency,process-kill,cpu-usage). - Targeting: Which hosts, pods, or services the experiment applies to. This is crucial for controlling the blast radius.
- Scope: The percentage of targets or requests affected.
- Duration: How long the failure should persist.
- Rollback: How to automatically or manually revert the injected fault.
When running a latency experiment between Auth and UserDB, you might configure it like this:
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: auth-userdb-latency
spec:
action: delay
mode: one
selector:
labelSelectors:
app: auth-service # Target the Auth service pods
delay:
latency: "500ms"
duration: "5m"
subscribers:
- podSelector:
labelSelectors:
app: userdb-service # Target the UserDB service pods
port: 5002 # The port UserDB listens on
This experiment targets pods labeled app: auth-service and injects a 500ms network delay for traffic directed at pods labeled app: userdb-service on port 5002 for 5 minutes. The mode: one ensures it affects only one pod at a time, minimizing impact.
The real magic happens when you observe your system’s response. Does Auth have a circuit breaker that trips when UserDB is slow? Does it retry gracefully? Does the entire login flow time out, or does it degrade gracefully, perhaps allowing login but with a delay in profile loading? The goal is to find failures, document them, fix the underlying issues (e.g., implement proper retries with backoff, add circuit breakers, optimize dependencies), and then re-run the experiment to confirm the fix.
A common misconception is that chaos engineering is only for large, sophisticated teams. However, even a small team running a few targeted experiments on a critical user flow can uncover significant vulnerabilities. For instance, many teams don’t realize that their database connection pool size is too small to handle a sudden spike in requests caused by a downstream service becoming sluggish, leading to connection timeouts that cascade.
After successfully running latency experiments and confirming your system handles them gracefully, you might then explore more complex scenarios, like simulating a leader election failure in a distributed consensus system.