Articles

Identify and Automate Toil to Free Up Engineering Capacity (2026)

April 16, 2026 6 min read

Automating toil is often framed as a way to reduce manual work, but its true power lies in revealing the hidden assumptions and fragility of your systems.

Let’s say you’re running a service that needs to process user-uploaded images. A common workflow might look like this:

  1. User Uploads Image: A user submits an image via your web app.
  2. Store Raw Image: The image is saved to a raw S3 bucket.
  3. Trigger Thumbnail Generation: An S3 event notification triggers a Lambda function.
  4. Generate Thumbnails: The Lambda function resizes the image into several thumbnail sizes.
  5. Store Thumbnails: The generated thumbnails are saved to a different S3 bucket.
  6. Update Database: A record is created or updated in your database with URLs to the original image and its thumbnails.

Here’s how this might look in practice, without the toil automation:

Initial Setup (Conceptual):

  • S3 Buckets:
    • my-app-raw-images (private)
    • my-app-thumbnails (publicly readable)
  • Lambda Function (generateThumbnails):
    • Triggered by s3:ObjectCreated:* on my-app-raw-images/*.
    • Uses imagemagick (or a similar library) to resize.
    • Writes output to my-app-thumbnails.
  • Database:
    • Table images with columns id, original_url, thumbnail_url_small, thumbnail_url_medium.

The "Toil" Emerges:

Imagine the generateThumbnails Lambda function starts failing intermittently. You get alerts: "Lambda function generateThumbnails timed out."

  • Initial Triage: You check CloudWatch logs. You see errors like ImageMagick: unable to open image '...'. Or maybe MemoryError.
  • The "Fix": Your immediate thought is to increase the Lambda memory. You go to the Lambda console, change the memory from 128MB to 256MB. This might work for a while.
  • The Next Problem: Now, images are being processed, but the database isn’t being updated consistently. You get alerts: "Database record missing for image ID 12345." You find the Lambda function did generate the thumbnails, but it failed to write to the database after that. The error? RDS connection refused.
  • The "Fix": You check your RDS instance. It’s maxed out on connections. You increase the max_connections parameter from 100 to 150.

This is toil: repetitive, manual interventions to keep a system limping along. It’s not just about the manual steps; it’s about the systemic issues these interventions mask.

The Real System in Action (with Automation in Mind):

Let’s re-imagine this with a focus on identifying and automating the toil, which forces us to build a more robust system from the start.

1. Identify the Core Process: Image upload and thumbnail generation.

2. Identify Potential Failure Points (and the Toil they cause):

  • Image Processing Library Crashes: (Toil: Manually restarting processes, increasing memory/CPU, debugging library versions).
  • S3 Bucket Permissions: (Toil: Manually checking and correcting IAM policies).
  • Lambda Function Configuration Errors: (Toil: Manually adjusting timeouts, memory, environment variables).
  • Database Connection Exhaustion: (Toil: Manually increasing DB connection limits, restarting DB instances).
  • Network Issues between Services: (Toil: Manually checking security groups, VPC peering, NACLs).
  • Idempotency Failures: (Toil: Manually deduplicating records, cleaning up partial processing).

3. Automate the Detection of Toil:

Instead of waiting for alerts, build automated checks that would cause toil if they failed.

  • Synthetic Image Upload: A scheduled job that uploads a small, known image.
    • Check: Does the image appear in the raw bucket? Is a thumbnail generated? Is a DB record created?
    • Automation: If any step fails, create a Jira ticket with specific diagnostic info.
  • Database Connection Pool Health: A background job that periodically checks the active connection count against a threshold (e.g., 80% of max_connections).
    • Check: SELECT count(*) FROM pg_stat_activity WHERE state = 'active';
    • Automation: If the count exceeds the threshold, automatically create a ticket or even trigger an alert to a specific on-call engineer.

4. Automate the Remediation of Toil:

This is where the real engineering capacity is freed.

  • Image Processing Failures (e.g., imagemagick crashes):
    • Diagnosis: CloudWatch logs show exit code 137 (often OOM killer).
    • Toil: Manually increasing Lambda memory.
    • Automation: Implement a self-healing mechanism. If a Lambda function processing an image fails with an OOM error, automatically increase its memory by 128MB for the next run, up to a safe maximum (e.g., 1024MB).
      # Example AWS CLI command to update Lambda configuration
aws lambda update-function-configuration \
    --function-name generateThumbnails \
    --memory-size 256 \
    --region us-east-1
    • Why it works: The system self-adjusts resource allocation based on observed workload demands, preventing manual intervention and service degradation.
  • Database Connection Exhaustion:
    • Diagnosis: pg_stat_activity shows state = 'active' count approaching max_connections.
    • Toil: Manually increasing max_connections or restarting the DB.
    • Automation: Create a Lambda function that monitors connection counts. If the count exceeds 80% of max_connections for 5 minutes, it automatically increments max_connections by 10 (up to a defined safe limit, e.g., 200) and sends a notification.
      -- SQL to check connection count
SELECT count(*) FROM pg_stat_activity WHERE state = 'active';

      # Example AWS CLI command to update RDS parameter group
# (This is more complex, often involving parameter group updates and instance reboots)
# For simplicity, imagine a script that updates a parameter group and triggers a blue/green deployment or rollback.
    • Why it works: The system proactively scales a critical resource before it becomes a bottleneck, averting service disruption.
  • Idempotency Issues (e.g., duplicate DB entries):
    • Diagnosis: Application logs show multiple identical entries for the same image ID.
    • Toil: Manually deleting duplicates, writing a one-off script to clean up.
    • Automation: Implement a "write-ahead log" or a unique transaction ID. Before writing to the database, check if a record with that transaction ID (or image ID and operation type) already exists. If so, skip the write and log a "duplicate operation" message.
      -- Example check before insert
INSERT INTO images (id, original_url, thumbnail_url_small, ...)
SELECT uuid_generate_v4(), '...', '...', ...
WHERE NOT EXISTS (
    SELECT 1 FROM images WHERE transaction_id = '...' -- or some other unique identifier
);
    • Why it works: Ensures operations are atomic and can be safely retried without causing data corruption or duplication.

The Hidden Model:

When you automate toil, you’re not just building a script to click buttons for you. You’re forced to:

  1. Understand the exact failure modes: What specifically goes wrong? What are the symptoms?
  2. Quantify the problem: What are the thresholds? What are the metrics? (e.g., memory usage, connection count, latency).
  3. Define remediation steps precisely: What exact command or API call fixes it? What are the safe limits?
  4. Build observable systems: If you can’t measure it, you can’t automate its recovery. This means better logging, metrics, and tracing.

The "toil" is often a symptom of an underlying architectural weakness. Automating it reveals and fixes that weakness, leading to a more resilient and scalable system.

The next error you’ll hit after fixing these transient issues is often related to the rate at which these events occur, leading you to optimize the underlying processing or implement more sophisticated queuing and backpressure mechanisms.

Want structured learning?

Take the full DevOps & Platform Engineering course →