ALTS Handshaker Blocks Worker Threads

Affected commit: c33d01624d
Sink: source/extensions/transport_sockets/alts/alts_proxy.cc:121

Summary

A downstream ALTS peer sends one byte to trigger synchronous waiting for the handshaker service. If the handshaker withholds its response, each connection blocks an Envoy worker for up to 30 seconds. Repeated connections exhaust all workers and stall unrelated traffic.

Detail

At alts_proxy.cc:29, the deadline is hardcoded:

constexpr absl::Duration AltsClientContextDeadline = absl::Seconds(30);

At line 44, this deadline is applied to the gRPC client context:

client_context->set_deadline(absl::ToChronoTime(absl::Now() + AltsClientContextDeadline));

At lines 115-122, the handshake performs synchronous blocking I/O on the worker thread:

if (!stream_->Write(request)) {
  return absl::UnavailableError(...);
}
HandshakerResp response;
if (!stream_->Read(&response)) {
  return absl::InternalError(...);
}

The stream_->Read(&response) call blocks the Envoy worker thread until the handshaker service responds or the 30-second deadline expires. Since Envoy runs a fixed number of worker threads (typically equal to CPU cores), blocking all workers stalls the entire proxy.

Reproduce

#!/bin/bash
set -e

git clone --depth 1 https://github.com/envoyproxy/envoy.git /tmp/envoy-alts
cd /tmp/envoy-alts
echo "HEAD: $(git rev-parse HEAD)"

# Verify hardcoded 30-second deadline
echo "=== Hardcoded 30-second deadline ==="
grep -n 'AltsClientContextDeadline' source/extensions/transport_sockets/alts/alts_proxy.cc

# Verify synchronous blocking Read on worker thread
echo ""
echo "=== Synchronous blocking I/O (alts_proxy.cc:113-128) ==="
sed -n '113,128p' source/extensions/transport_sockets/alts/alts_proxy.cc

# Verify the deadline is not configurable (TODO comment)
echo ""
echo "=== Deadline not configurable ==="
grep -n 'TODO.*deadline.*configurable\|TODO.*Make this deadline' source/extensions/transport_sockets/alts/alts_proxy.cc

rm -rf /tmp/envoy-alts

Expected output (line numbers may shift):

HEAD: <resolved-HEAD>
=== Hardcoded 30-second deadline ===
29:constexpr absl::Duration AltsClientContextDeadline = absl::Seconds(30);
44:  client_context->set_deadline(absl::ToChronoTime(absl::Now() + AltsClientContextDeadline));

=== Synchronous blocking I/O (alts_proxy.cc:113-128) ===
  // Send the StartServerHandshakeReq message to the handshaker service and wait
  // for the response.
  if (!stream_->Write(request)) {
    return absl::UnavailableError(
        "Failed to write server start to handshaker service. ...");
  }
  HandshakerResp response;
  if (!stream_->Read(&response)) {
    return absl::InternalError("Failed to read server start response from handshaker service.");
  }
  ...

=== Deadline not configurable ===
28:// TODO(matthewstevenson88): Make this deadline configurable.

A downstream peer initiating ALTS handshakes blocks one Envoy worker per connection for up to 30 seconds when the handshaker service is slow or unresponsive. With a default Envoy deployment using one worker per CPU core, an attacker opening as many concurrent ALTS connections as there are workers stalls all request processing.

Credit

Zheng Yu @ Depthfirst