Repository navigation
Bigquery Python Library: AppendRows bidi-streaming RPC sends empty x-goog-request-params routing header, causing intermittent InvalidArgument: Cannot route on empty project id ''` #16650
Description
Determine this is the right repository
- I determined this is the correct repository in which to report this bug.
Summary of the issue
Context
Using BigQueryWriteClient.append_rows() to write data via the Storage Write API default stream (_default), with multiple sequential calls through the same client. This is in a multiprocessing setup where each worker creates its own BigQueryWriteClient() and makes many append_rows calls.
Expected Behavior:
append_rows should populate the x-goog-request-params gRPC routing header with the write_stream resource name (which contains the project ID), similar to how other methods in the same client do it:
# create_write_stream — sets routing correctly
metadata = ... + (gapic_v1.routing_header.to_grpc_metadata((("parent", request.parent),)),)Actual Behavior:
append_rows sends an empty routing header because it's a bidi-streaming RPC and the request iterator hasn't been consumed when metadata is set:
# client.py, append_rows() — all versions 2.26.0 through 2.37.0
metadata = tuple(metadata) + (gapic_v1.routing_header.to_grpc_metadata(()),)This produces x-goog-request-params: ''. The first several calls may succeed via the gateway's fallback routing, but after a gRPC reconnection (idle timeout, load balancing), the gateway cannot determine the target project and returns:
google.api_core.exceptions.InvalidArgument: 400 Cannot route on empty project id ''
Reproduction:
from google.cloud.bigquery_storage_v1 import BigQueryWriteClient
from google.api_core.gapic_v1 import routing_header
import inspect
# Verify empty routing in source
source = inspect.getsource(BigQueryWriteClient.append_rows)
assert "to_grpc_metadata(())" in source
# Verify it produces an empty header
assert routing_header.to_grpc_metadata(()) == ("x-goog-request-params", "")Workaround:
Pass routing metadata explicitly:
metadata = (routing_header.to_grpc_metadata((("write_stream", stream_name),)),)
response = client.append_rows(requests=iter([request]), metadata=metadata)Environment:
google-cloud-bigquery-storage: 2.26.0 through 2.37.0 (all affected)- Python 3.9
API client name and version
No response
Reproduction steps: code
file: main.py
def reproduce():
# complete code hereReproduction steps: supporting files
file: mydata.csv
alpha,1,3
beta,2,5
Reproduction steps: actual results
file: output.txtmydata.csv
Calculated: foo
Reproduction steps: expected results
file: output.txtmydata.csv
Calculated: bar
OS & version + platform
No response
Python environment
No response
Python dependencies
No response
Additional context
No response
Activity
[-]`AppendRows` bidi-streaming RPC sends empty `x-goog-request-params` routing header, causing intermittent `InvalidArgument: Cannot route on empty project id ''``[/-][+]Bigquery Python Library: `AppendRows` bidi-streaming RPC sends empty `x-goog-request-params` routing header, causing intermittent `InvalidArgument: Cannot route on empty project id ''``[/+]erezinman commentedon Apr 14, 2026
Note that actual reproduction is difficult because the initial gRPC connection gets routed correctly through the server's default/fallback behavior. But when the connection is re-established (idle timeout, load balancing, server-side reconnect), the
gateway needs the routing header to know where to send the request. With an empty header, it can't determine the project.
daniel-sanche commentedon Jul 31, 2026
For most packages, this would be something that needs to be configured in the service configuration. I think bigquery is different though, so this may have to be addressed on the client side. Can you take a look at this @chalmerlowe ?
daniel-sanche commentedon Oct 7, 2026
I put together a generator-level fix for this: #18591
We shouldn't be emitting empty metadata
fix(gapic-generator): do not emit empty routing metadata on client-st…
daniel-sanche commentedon Oct 8, 2026
The PR has been merged in the generator, this should be fixed after the next update