Envoy’s TCP proxy filter is surprisingly flexible, allowing you to not just route raw TCP traffic but also to inject custom logic and transform connections on the fly.
Let’s see it in action. Imagine we have a simple backend service listening on port 9000. We want to expose it via Envoy, but with a twist: we want to add a simple prefix to every request received before forwarding it.
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.tcp_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
stat_prefix: tcp_proxy
cluster: tcp_backend
# This is where the magic happens for transformation
transforms:
- name: envoy.wasm
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.common.wasm.v3.PluginConfig
name: tcp_prefixer
root_id: tcp_prefixer_root
vm_config:
runtime: envoy.wasm.runtime.v8
code:
local:
filename: /etc/envoy/wasm/tcp_prefixer.wasm
configuration:
"@type": type.googleapis.com/google.protobuf.BytesValue
value: "ewogICAgImFkZGl0aW9uYWxfY29uZmlnIjogImNvb2xfcHJlZml4OiI=" # Base64 encoded: {"additional_config": "cool_prefix:"}
clusters:
- name: tcp_backend
connect_timeout: 0.25s
type: LOGICAL_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: tcp_backend
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 9000
In this configuration, we’ve set up a listener on port 10000. The tcp_proxy filter is configured to forward traffic to a cluster named tcp_backend. The crucial part is the transforms section, which uses the envoy.wasm filter. This allows us to load a WebAssembly (Wasm) plugin.
The Wasm plugin, tcp_prefixer.wasm, is loaded from /etc/envoy/wasm/tcp_prefixer.wasm. It receives configuration via configuration.value, which is a base64 encoded JSON string. In this case, it’s {"additional_config": "cool_prefix:"}. This Wasm plugin will intercept the incoming TCP data, prepend cool_prefix:, and then pass the modified data downstream.
To make this work, you’d need a Wasm plugin compiled for Envoy. A simple Wasm plugin written in Rust (and compiled to .wasm) might look something like this conceptually:
use envoy_sdk::hostcalls;
use envoy_sdk::hostcalls::log_level;
use envoy_sdk::hostcalls::log;
use envoy_sdk::hostcalls::Buffer;
use envoy_sdk::hostcalls::StreamSender;
use envoy_sdk::hostcalls::StreamReceiver;
use envoy_sdk::hostcalls::StreamResponse;
use envoy_sdk::hostcalls::StreamShared;
use envoy_sdk::hostcalls::stream_response_handler;
use envoy_sdk::hostcalls::StreamResponseHandler;
use envoy_sdk::hostcalls::FilterConfig;
use envoy_sdk::hostcalls::FilterKind;
use envoy_sdk::hostcalls::FilterManager;
use envoy_sdk::hostcalls::FilterResponse;
use envoy_sdk::hostcalls::FilterStatus;
use envoy_sdk::hostcalls::FilterData;
use envoy_sdk::hostcalls::StreamContext;
#[no_mangle]
pub extern "C" fn envoy_filter_init(
filter_manager: &mut dyn FilterManager,
) -> bool {
filter_manager.register_stream_filter(
"tcp_prefixer",
FilterKind::Read,
Box::new(TcpPrefixerFilter),
);
true
}
struct TcpPrefixerFilter;
impl StreamSender for TcpPrefixerFilter {
fn on_data(
&mut self,
context: &mut dyn StreamContext,
data: FilterData,
) -> FilterStatus {
let prefix = "cool_prefix:"; // This would ideally be read from config
let mut buffer = prefix.as_bytes().to_vec();
buffer.extend_from_slice(data.as_ref());
context.send_data(FilterData::from(buffer));
FilterStatus::Continue
}
}
impl StreamReceiver for TcpPrefixerFilter {
// Implement other necessary methods like on_new_connection, on_end_stream, etc.
// For simplicity, we're only focusing on data transformation here.
}
// You'd also need to implement `StreamResponseHandler` and `FilterConfig`
// and potentially `StreamShared` depending on the complexity.
When you send data to Envoy on port 10000, say hello world, the Wasm plugin intercepts it. It prepends cool_prefix: to create cool_prefix:hello world. This modified data is then sent to the actual backend service on port 9000. The backend receives cool_prefix:hello world and processes it.
The core problem this solves is adding application-level logic to raw TCP streams without modifying the backend service itself. You can implement features like:
- Request/Response Transformation: Adding headers, modifying payloads, filtering data.
- Protocol Translation (Limited): While not a full protocol translator, you can manipulate data to fit slightly different protocol expectations.
- Security Enhancements: Injecting authentication tokens, masking sensitive data.
- Observability: Adding custom tracing information or metadata to TCP flows.
The transforms field in the TcpProxy filter is the key. It allows chaining multiple filters, including Wasm plugins, directly within the TCP proxy flow. Each transform operates on the stream sequentially. The envoy.wasm filter is particularly powerful because it allows you to run custom code written in various languages (Go, Rust, C++, etc.) compiled to Wasm. The configuration field allows passing dynamic settings to your Wasm plugin, making it highly configurable without recompiling.
What many people miss is that the Wasm plugin can both read from the upstream (backend) and write to the downstream (client). This means you can not only modify requests going to the backend but also modify responses coming from the backend before they reach the client. This bidirectional transformation capability unlocks a vast range of use cases for customizing TCP traffic.
The next logical step is to explore how to chain multiple Wasm filters or combine Wasm filters with other network filters like envoy.filters.network.http_connection_manager if you decide to eventually move to HTTP.