Argo Workflows doesn’t actually run your containerized tasks directly; it delegates that responsibility to a plugin executor, and you can swap that plugin out to run your tasks anywhere.
Let’s see it in action. Imagine you have a simple workflow that just echoes "hello":
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: echo-workflow-
spec:
entrypoint: echo
templates:
- name: echo
container:
image: alpine:latest
command: ["echo", "hello"]
When this workflow runs, Argo submits a WorkflowRun object. The WorkflowController watches for these objects. It then looks at the spec.executor field on the WorkflowRun. If it’s not specified, it defaults to the kubernetes executor. This executor tells the controller to create a standard Kubernetes Pod for the task. The Pod runs the specified alpine:latest image and executes the echo "hello" command. The output of the command is captured by the WorkflowController and stored as part of the WorkflowRun status.
But what if you want to run that alpine container not on your Kubernetes cluster, but somewhere else? Like on a remote machine, or even on a different cloud provider’s compute instances? That’s where custom plugin executors come in. You tell Argo to use a different executor by specifying it in the WorkflowRun or in the Argo Workflows ConfigMap.
Here’s how you’d specify an executor in the ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: argo-workflows-controller-configmap
namespace: argo
data:
executor: "my-custom-executor"
Now, whenever a WorkflowRun is created without an explicit spec.executor, Argo will attempt to communicate with whatever service is registered as my-custom-executor.
The core idea is that Argo Workflows itself is a workflow orchestrator, not a task runner. It defines the DAG, manages state, handles retries, and collects artifacts. The actual execution of the containerized task is offloaded. The default kubernetes executor is just one implementation of this offloading.
To build your own executor, you’d typically create a service that listens for requests from the Argo WorkflowController. This service needs to understand a specific protocol that Argo expects. Argo Workflows uses a gRPC API for inter-process communication between the controller and executors. Your custom executor service would implement this gRPC API.
When Argo wants to run a task using your custom executor, it sends a gRPC request to your executor service. This request contains all the details needed to run the task: the container image, command, arguments, environment variables, volume mounts, etc. Your executor service then takes this information and translates it into the native execution environment it manages.
For example, if your custom executor is designed to run tasks on AWS EC2 instances, it would use the AWS SDK to:
- Launch an EC2 instance.
- Pull the specified container image onto that instance.
- Run the container using Docker or containerd.
- Stream the logs back to Argo.
- Report the task’s exit status.
The most surprising truth about plugin executors is that they don’t have to run on Kubernetes at all. While the default kubernetes executor obviously does, you could, in theory, run your custom executor service on a completely separate set of machines, or even on your laptop, and have Argo Workflows delegate task execution to it. The WorkflowController just needs to be able to reach your executor service over the network via gRPC.
The exact gRPC service definition for executors is part of the Argo Workflows codebase, specifically in the pkg/apis/workflow/v1alpha1/executor.proto file. This file defines the Executor service with methods like RunPod and GetPodLogs. Your custom executor must implement these methods. The RunPod method receives a RunPodRequest which contains a Pod spec, and your executor is responsible for creating and managing the execution of that pod in its target environment, returning a RunPodResult with the pod’s status and lifecycle events.
The critical piece of information that most users miss is that the spec.executor field in a WorkflowRun overrides the global default defined in the ConfigMap. This means you can have a single Argo Workflows installation that uses different executors for different workflows, or even different tasks within the same workflow, by specifying the desired executor directly in the WorkflowRun manifest.
Once you’ve mastered custom executors, the next logical step is exploring how to manage the lifecycle and scaling of those external execution environments, perhaps using tools like Terraform or Ansible to provision and configure the compute resources your custom executors rely on.