Argo Workflows lets you define complex, multi-step processes, but sometimes a single, monolithic workflow becomes unwieldy. That’s where nested workflows come in. Instead of cramming everything into one massive Workflow object, you can break it down into smaller, manageable sub-workflows, each orchestrated by a parent workflow.
Imagine you have a CI/CD pipeline. You might have a "Build" stage, a "Test" stage, and a "Deploy" stage. In Argo, you could represent each of these as a separate Workflow template. Then, a "Main Pipeline" workflow would simply call these templates sequentially.
Here’s a simplified example of how you might define a parent workflow that triggers a "build" sub-workflow:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: parent-workflow
spec:
entrypoint: main
templates:
- name: main
steps:
- - name: trigger-build
template: build-workflow-template
- name: build-workflow-template
dag:
tasks:
- name: build-task
template: build-sub-workflow
- name: build-sub-workflow
workflow:
spec:
entrypoint: build-logic
templates:
- name: build-logic
container:
image: ubuntu:20.04
command: ["bash", "-c"]
args: ["echo 'Building the application...' && sleep 30"]
In this structure:
parent-workflowis the top-level workflow.- Its
maintemplate usesstepsto define sequential execution. - The
trigger-buildstep callsbuild-workflow-template. build-workflow-templateis adagthat contains a single task,build-task.- Crucially,
build-task’stemplateisbuild-sub-workflow, which is defined inline using theworkflowfield. This inlineworkflowis the nested workflow. - The nested
build-sub-workflowhas its ownentrypoint(build-logic) and executes a simple containerized command.
When parent-workflow runs, it will trigger the build-sub-workflow. Argo handles the creation and execution of this nested workflow as a child process. The output and status of the nested workflow are then reported back to the parent.
The real power of nesting comes from using WorkflowTemplates for your sub-workflows. This allows you to define reusable components that can be invoked multiple times from different parent workflows or even within the same parent workflow.
Consider this refined example using a WorkflowTemplate:
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: build-template
spec:
entrypoint: build-logic
templates:
- name: build-logic
container:
image: ubuntu:20.04
command: ["bash", "-c"]
args: ["echo 'Building the application...' && sleep 30"]
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: parent-workflow-with-template
spec:
entrypoint: main
templates:
- name: main
steps:
- - name: trigger-build
template: build-template # Referencing the WorkflowTemplate
arguments:
parameters:
- name: build-version
value: "1.0.0" # Example of passing parameters
Here, build-template is a standalone WorkflowTemplate. The parent-workflow-with-template then references build-template within its main step. When trigger-build executes, Argo creates a new Workflow instance based on the build-template. This new workflow is effectively nested. You can pass parameters to these nested workflows, making them dynamic and configurable.
You can also nest workflows deeply. A parent workflow can trigger a sub-workflow, which in turn can trigger another sub-workflow. Argo’s UI will visually represent these nested structures, making it easier to follow the execution flow. The argo get and argo list commands will also show the hierarchical relationship, often by appending the sub-workflow’s name or ID to the parent’s.
One subtle but powerful aspect of nested workflows is how they manage resource isolation and lifecycle. Each nested Workflow runs as its own independent Kubernetes object. This means they have their own service accounts, secrets, and resource quotas, which can be defined independently of the parent workflow. If a nested workflow fails, it doesn’t necessarily bring down the entire parent; the parent can be configured to handle sub-workflow failures gracefully (e.g., by retrying or executing an alternative path).
The execution of a nested workflow is initiated by a task in the parent workflow that specifies a template of type workflow. This can be an inline workflow definition or, more commonly, a reference to a WorkflowTemplate or even another Workflow resource. When Argo encounters this, it doesn’t execute the container directly within the parent’s context. Instead, it dynamically creates a new Workflow resource in Kubernetes, populating its spec from the specified template. This new Workflow then runs according to its own entrypoint and templates. The parent workflow task then waits for this new Workflow to complete, capturing its exit code and artifacts.
Passing artifacts between nested workflows is a common pattern. If a nested workflow produces an artifact (e.g., a compiled binary), the parent workflow can define a task that depends on the completion of the nested workflow and then access that artifact, perhaps to pass it to a subsequent nested workflow for testing or deployment. This is managed through Argo’s artifact passing mechanisms, where outputs from a child workflow can be explicitly declared and then consumed as inputs by subsequent tasks in the parent.
The primary benefit of nested workflows is modularity and reusability. By breaking down complex processes into smaller, self-contained workflows, you improve maintainability, testability, and the ability to share workflow components across different projects or pipelines. It allows you to treat entire workflows as atomic units that can be invoked and orchestrated, much like calling a function in traditional programming.
A common point of confusion is when to use a step template that calls a WorkflowTemplate versus a step template that defines an inline workflow. The former is for reusable, independently defined workflows, while the latter is for defining a nested workflow directly within the context of its parent. For most practical scenarios involving complex pipelines, using WorkflowTemplates for your nested components is the recommended and more scalable approach.
The next logical step after mastering nested workflows is to explore how Argo Workflows integrates with GitOps principles, allowing you to manage your entire workflow definitions as code within a Git repository.