Envoy’s gRPC-Bridge component is failing because it’s unable to translate gRPC requests into a format that a non-gRPC backend can understand, or vice-versa, usually due to a mismatch in how the request is structured or how it’s being terminated.
Common Causes and Fixes
-
Mismatched Content-Type Header: The gRPC-Bridge relies on the
content-typeheader to determine if a request is gRPC. If it’s missing or incorrect, Envoy won’t attempt the translation.- Diagnosis: Inspect the request headers arriving at Envoy. Use
curl -vto see headers, or Envoy’s access logs if configured to log them. Look forcontent-type: application/grpcorcontent-type: application/grpc+protofor gRPC requests. - Fix: Ensure the client sending the gRPC request sets the
content-typeheader correctly. If you’re bridging from gRPC to HTTP/1.1, Envoy’s gRPC-Bridge filter will usually set this correctly on the outgoing request to the backend. If you are bridging from HTTP/1.1 to gRPC, ensure the client is explicitly settingcontent-type: application/grpc. - Why it works: The gRPC-Bridge filter specifically looks for this header to identify and process gRPC requests. Without it, it treats them as regular HTTP requests and bypasses the translation logic.
- Diagnosis: Inspect the request headers arriving at Envoy. Use
-
Incorrect
grpc-timeoutHeader Format: gRPC uses a specificgrpc-timeoutheader for request timeouts. If this header is malformed or missing when expected, the gRPC client might abort the request before it even reaches Envoy, or Envoy might not interpret the timeout correctly.- Diagnosis: Check client-side logs for gRPC timeout errors. If the request reaches Envoy, inspect its headers in access logs or via
curl -v. The header should be in the formatt.valuewheretis a unit (s, m, h) andvalueis a number (e.g.,10s,5m). - Fix: Ensure the gRPC client correctly formats the
grpc-timeoutheader. For example,req.add_header("grpc-timeout", "10s"). - Why it works: This header is a standard gRPC mechanism for clients to communicate their desired timeout to the server. Envoy leverages this for gRPC-Bridge translation.
- Diagnosis: Check client-side logs for gRPC timeout errors. If the request reaches Envoy, inspect its headers in access logs or via
-
Missing
grpc-messageorgrpc-statusin Backend Response (HTTP to gRPC): When bridging HTTP/1.1 to gRPC, Envoy expects the backend to return gRPC-style error information if an error occurs.- Diagnosis: Examine Envoy’s response headers and body when an error is returned from the backend. Look for
grpc-status(an integer) andgrpc-message(a string) headers. - Fix: Configure your HTTP/1.1 backend to return
grpc-statusandgrpc-messageheaders on error responses. For instance, for an "unimplemented" error, the backend should returngrpc-status: 12andgrpc-message: "Method not implemented". - Why it works: Envoy uses these headers to reconstruct a gRPC error response to the original gRPC client, maintaining the gRPC error contract.
- Diagnosis: Examine Envoy’s response headers and body when an error is returned from the backend. Look for
-
HTTP/1.1 Backend Not Handling gRPC Payload Encoding: gRPC messages are typically length-prefixed and may be compressed. If the HTTP/1.1 backend doesn’t expect or correctly parse this, it will fail.
- Diagnosis: Observe the raw HTTP request received by the backend. Look for a length prefix (a single byte,
0x00for uncompressed,0x01for compressed) followed by the message payload. - Fix: If bridging HTTP to gRPC, ensure your backend understands the gRPC framing (length prefix + payload). If bridging gRPC to HTTP, ensure Envoy is configured to decompress if necessary and that the backend can handle the raw protobuf payload. Envoy’s
grpc_bridgefilter handles the framing. - Why it works: gRPC has a specific framing protocol for its payloads. Envoy’s gRPC-Bridge filter adds this framing when sending to a gRPC backend and can remove it when receiving from an HTTP/1.1 backend.
- Diagnosis: Observe the raw HTTP request received by the backend. Look for a length prefix (a single byte,
-
Incorrect
content-encodingHeader for Compressed Payloads: gRPC supports compression. If compression is used, thecontent-encodingheader must be set correctly.- Diagnosis: Inspect the
content-encodingheader in requests or responses. Common values aregziporidentity(no compression). - Fix: Ensure that if compression is enabled in Envoy’s gRPC-Bridge configuration (e.g.,
grpc_codec.compressororgrpc_codec.decompressor), thecontent-encodingheader is correctly set on the wire and that both Envoy and the backend can handle the specified encoding. If you’re seeinggrpc-encodingheaders, ensure they match what Envoy expects for compression. - Why it works: The
content-encodingheader signals to the receiver whether and how the payload has been compressed, allowing it to decompress the data before processing.
- Diagnosis: Inspect the
-
TLS Termination Mismatch or Client Authentication Failure: If gRPC is over TLS, and Envoy is configured to terminate TLS, but the backend expects TLS, or vice-versa, it will cause connection failures.
- Diagnosis: Check Envoy’s TLS configuration (
tls_context) and the backend’s TLS requirements. Look for TLS handshake errors in Envoy logs or backend logs. Ensure certificates are valid and trusted by both ends. - Fix: Ensure Envoy’s TLS configuration matches the backend’s requirements. If Envoy terminates TLS, the backend should expect plain HTTP/1.1. If Envoy is acting as a pass-through or if the backend requires TLS, configure
tls_contextaccordingly for outbound connections. - Why it works: TLS ensures secure communication. Mismatched configurations prevent the secure channel from being established, leading to communication failure.
- Diagnosis: Check Envoy’s TLS configuration (
The next error you’ll likely encounter after fixing these is a 503 Service Unavailable if the upstream cluster is misconfigured or unhealthy, or a 502 Bad Gateway if Envoy successfully forwards the request but the upstream returns an invalid response.