The most surprising thing about waiting for resources to be ready after a CDK deploy is that the CDK itself often doesn’t know they’re ready, even when your application does.
Let’s watch a real deployment. We’ll deploy a simple Lambda function that writes to an S3 bucket.
First, our CDK code:
from aws_cdk import (
aws_lambda as _lambda,
aws_s3 as s3,
Stack,
Duration
)
from constructs import Construct
class LambdaS3Stack(Stack):
def __init__(self, scope: Construct, construct_id: str) -> None:
super().__init__(scope, construct_id)
bucket = s3.Bucket(self, "MyBucket",
versioned=True,
removal_policy=RemovalPolicy.DESTROY) # Be careful with REMOVE_DESTROY in prod!
my_lambda = _lambda.Function(self, "MyLambda",
runtime=_lambda.Runtime.PYTHON_3_9,
handler="index.handler",
code=_lambda.Code.from_asset("lambda"),
environment={
"BUCKET_NAME": bucket.bucket_name
},
timeout=Duration.seconds(30))
bucket.grant_read_write(my_lambda)
And our Lambda code (lambda/index.py):
import boto3
import os
s3 = boto3.client('s3')
bucket_name = os.environ['BUCKET_NAME']
def handler(event, context):
try:
s3.put_object(Bucket=bucket_name, Key='hello.txt', Body='World')
print(f"Successfully wrote to {bucket_name}/hello.txt")
return {
'statusCode': 200,
'body': 'Success!'
}
except Exception as e:
print(f"Error writing to S3: {e}")
return {
'statusCode': 500,
'body': str(e)
}
Now, let’s deploy this with cdk deploy --require-approval never.
cdk deploy --require-approval never
The CDK will report that the stack is deploying. It will show the S3 bucket and the Lambda function being created. Once it says Stack XXXXXXX-LambdaS3Stack IS COMPLETE, you might think everything is ready.
But if you immediately try to invoke the Lambda function, you might see an error. Why? Because the "completion" reported by cdk deploy is based on CloudFormation’s assessment of resource creation, not resource readiness for specific operations.
For our Lambda function to successfully write to S3, the S3 bucket needs to be fully provisioned and accessible. While CloudFormation might mark the S3 bucket as CREATE_COMPLETE relatively quickly, there’s a small window where it’s still becoming globally available or its policies are propagating. The Lambda service, when it tries to execute your function, queries for the bucket’s details. If that query happens in that tiny propagation window, it can fail.
This is especially true for cross-region resources or when dealing with services that have complex internal dependencies. The CDK, by default, relies on CloudFormation’s stack events. CloudFormation’s CREATE_COMPLETE signal is often optimistic.
To handle this, we need to add explicit checks or delays. One common approach is to use a Lambda function or a Custom Resource within the CDK itself to poll for the dependent resource’s readiness.
Here’s how you might add a simple delay within the CDK deployment, though this is a blunt instrument:
from aws_cdk import (
aws_lambda as _lambda,
aws_s3 as s3,
Stack,
Duration,
RemovalPolicy
)
from constructs import Construct
import aws_cdk.custom_resources as cr
import time
class LambdaS3Stack(Stack):
def __init__(self, scope: Construct, construct_id: str) -> None:
super().__init__(scope, construct_id)
bucket = s3.Bucket(self, "MyBucket",
versioned=True,
removal_policy=RemovalPolicy.DESTROY)
my_lambda = _lambda.Function(self, "MyLambda",
runtime=_lambda.Runtime.PYTHON_3_9,
handler="index.handler",
code=_lambda.Code.from_asset("lambda"),
environment={
"BUCKET_NAME": bucket.bucket_name
},
timeout=Duration.seconds(30))
bucket.grant_read_write(my_lambda)
# Add a simple wait after bucket creation.
# This is often not needed for S3 but demonstrates the concept.
# A more robust solution would use a custom resource to poll.
wait_resource = cr.AwsCustomResource(
self, "WaitForBucketReadiness",
on_create=cr.AwsSdkCall(
service="S3",
action="listBuckets", # A no-op that forces an S3 API call
parameters={},
physical_resource_id=cr.PhysicalResourceId.of("BucketReadinessCheck")
),
policy=cr.AwsCustomResourcePolicy.from_sdk_calls(
resources=cr.AwsCustomResource.ANY_RESOURCE
),
install_role_policy=False # We're only making S3 calls
)
# Ensure the Lambda depends on this wait resource, which depends on the bucket
my_lambda.node.add_dependency(wait_resource)
wait_resource.node.add_dependency(bucket)
In this modified code, we introduce an AwsCustomResource. The on_create action is a simple S3 API call (listBuckets). This call will fail if the S3 service is not ready to respond to API requests for the bucket. By making the Lambda function depend on this custom resource, and the custom resource depend on the bucket, we effectively insert a wait. If listBuckets fails due to the bucket not being fully ready, the custom resource will fail, and thus the stack deployment will fail, prompting a retry.
The true power here is in understanding that CloudFormation’s CREATE_COMPLETE is a signal about the resource definition being accepted, not necessarily its operational readiness. Services like S3, Lambda, or even IAM roles can take a few extra seconds to propagate their state across AWS’s distributed infrastructure.
The most robust pattern is to use a Custom Resource that actively polls the dependent resource until a specific condition is met. For S3, this might involve trying to head_bucket or list_objects_v2 until it succeeds. For other services, like RDS or ECS, you might check status fields in their respective API responses. The AwsCustomResource is your tool for this polling.
The next hurdle you’ll encounter is managing dependencies between resources that aren’t directly linked in the CDK construct tree, or when a resource needs to be ready for a specific type of operation (e.g., a database needs to be ready for writes, not just creation).