Deploying C# Worker Services to production with Docker and Kubernetes isn’t about just getting your app running; it’s about building a resilient, scalable system that can handle real-world demands. The most surprising thing about this whole process is how much of the "magic" is actually just well-understood distributed systems principles, applied with meticulous configuration.
Let’s see a worker service in action, not in a dotnet run scenario, but as a containerized process managed by Kubernetes.
Consider a simple C# Worker Service that polls a queue for messages and processes them.
// Program.cs
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using System;
using System.Threading;
using System.Threading.Tasks;
public class Program
{
public static void Main(string[] args)
{
CreateHostBuilder(args).Build().Run();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) =>
{
services.AddHostedService<MyWorkerService>();
// Register any other necessary services (e.g., queue clients, database contexts)
});
}
// MyWorkerService.cs
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using System;
using System.Threading;
using System.Threading.Tasks;
public class MyWorkerService : BackgroundService
{
private readonly ILogger<MyWorkerService> _logger;
// Inject your queue client or other dependencies here
public MyWorkerService(ILogger<MyWorkerService> logger /*, IQueueClient queueClient */)
{
_logger = logger;
// _queueClient = queueClient;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
_logger.LogInformation("Worker Service is starting.");
while (!stoppingToken.IsCancellationRequested)
{
_logger.LogInformation("Worker Service is doing work at: {time}", DateTimeOffset.Now);
// Simulate work: poll queue, process message, etc.
// await _queueClient.ProcessNextMessageAsync();
await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken); // Poll every 5 seconds
}
_logger.LogInformation("Worker Service is stopping.");
}
}
To get this into production, we’ll containerize it.
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /app
COPY *.csproj ./
RUN dotnet restore
COPY . ./
RUN dotnet publish -c Release -o out
FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS runtime
WORKDIR /app
COPY --from=build /app/out .
ENTRYPOINT ["dotnet", "MyWorkerService.dll"]
This Dockerfile builds our application in a larger SDK image and then copies the published output into a smaller runtime image, creating an efficient container.
Now, let’s deploy this to Kubernetes. We’ll need a deployment.yaml file.
deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-worker-service
labels:
app: worker
spec:
replicas: 3 # Start with 3 instances for HA
selector:
matchLabels:
app: worker
template:
metadata:
labels:
app: worker
spec:
containers:
- name: worker
image: your-dockerhub-username/my-worker-service:latest # Replace with your image
ports:
- containerPort: 80 # If your service exposes any HTTP endpoints for health checks
env:
- name: MESSAGE_QUEUE_CONNECTION_STRING
valueFrom:
secretKeyRef:
name: queue-secrets
key: connection-string
resources:
requests:
memory: "64Mi"
cpu: "250m" # 1/4 of a CPU core
limits:
memory: "128Mi"
cpu: "500m" # 1/2 of a CPU core
This Deployment tells Kubernetes to run 3 replicas of our worker container. It specifies the image to use, sets environment variables (like a connection string fetched from a Secret for security), and critically, defines resource requests and limits. These limits prevent a runaway process from consuming all node resources and ensure predictable performance.
For resilience, we also need a Service to expose our worker (e.g., for health checks) and potentially a HorizontalPodAutoscaler (HPA) to automatically adjust the number of replicas.
service.yaml (optional, for health checks):
apiVersion: v1
kind: Service
metadata:
name: worker-service
spec:
selector:
app: worker
ports:
- protocol: TCP
port: 80
targetPort: 80
hpa.yaml (for autoscaling):
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: worker-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-worker-service
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70 # Scale up when CPU utilization hits 70%
The HPA monitors the CPU utilization of the pods managed by the my-worker-service deployment. When the average utilization across all replicas exceeds 70%, Kubernetes will automatically scale up the number of replicas, up to a maximum of 10. Conversely, if utilization drops, it will scale down to the minimum of 3.
The core problem this entire setup solves is moving from a single-instance, manually managed application to a distributed, self-healing system. Kubernetes handles pod scheduling, restarts on failure, and scaling. Docker provides a consistent, isolated environment for your application. The C# Worker Service’s BackgroundService pattern is designed precisely for this kind of long-running, asynchronous workload within a container.
The most impactful aspect of managing worker services in Kubernetes, beyond basic deployment, is understanding the lifecycle and readiness probes. While the BackgroundService handles the application’s internal looping, Kubernetes needs to know when your actual work is ready to be processed and when it’s safe to terminate a pod. You’d typically implement a small ASP.NET Core endpoint within your worker service (even if it’s just for health checks) and configure livenessProbe and readinessProbe in your deployment. A livenessProbe failing tells Kubernetes the container is unhealthy and needs restarting. A readinessProbe failing tells Kubernetes that the pod is not yet ready to receive traffic (or in the case of a worker, perhaps not ready to start processing new tasks, e.g., it’s still initializing its connection to a message queue).
The next concept you’ll grapple with is managing state and distributed coordination, especially if your workers need to ensure exactly-once processing or avoid duplicate work.