Principal Engineer — Cloud Data Plane & Traffic Processing

About the Role

We are building the inline cloud data plane that powers a next-generation AI security platform. This service terminates TLS, inspects and processes enterprise AI traffic in real time, and applies policy, classification, orchestration, auditing, and analytics — at millisecond latencies under production SaaS load across multiple global regions.

We are looking for a hands-on Principal Engineer to own the architecture, design, and implementation of this data plane end to end.

What You’ll Do

• Architect and build the inline cloud data plane that processes enterprise AI traffic in real time.
• Own the request path end to end: TLS termination, HTTP/2 and HTTP/3 handling, policy evaluation, classification hooks, upstream routing, and response streaming.
• Design multi-tenant, multi-region PoP architecture with strict p50 / p95 / p99 latency budgets and availability SLOs.
• Design and implement request routing, service discovery, resiliency, timeout, retry, circuit-breaking, and backpressure strategies on the hot path.
• Optimize the platform for added latency, throughput, connection density, TLS handshakes/sec, and cost per million requests.
• Drive cost and operational efficiency across the data plane, including capacity planning, autoscaling behavior, infrastructure utilization, and cost per unit of traffic.
• Integrate the data plane with policy enforcement, classification, orchestration, auditing, analytics, and other core platform services.
• Define the control-plane integration model for configuration distribution, policy propagation, service discovery, tenant isolation, and safe rollout of data-plane changes.
• Build distributed tracing, metrics, and end-to-end observability tuned for inline request-path debugging.
• Architect Kubernetes-based deployments, horizontal autoscaling, rolling upgrades, and multi-region disaster recovery for the data plane.
• Drive technical decisions on proxy technology selection (Envoy vs. custom vs. hybrid), extension model (filters, WASM, native), and control-plane integration.
• Mentor engineers and set the engineering bar for inline-service development, incident response, and operational excellence.

Required Qualifications

• 10+ years of software engineering experience, with at least 5 years building or operating an inline, customer-traffic, low-latency SaaS data plane (proxy, gateway, edge, CDN, SASE/SWG, or equivalent).
• Production experience with a programmable L7 proxy — Envoy, Nginx/OpenResty, HAProxy, Squid, or a comparable in-house proxy — including writing filters, modules, or extensions.
• Deep hands-on expertise with HTTP/1.1, HTTP/2, HTTP/3 (QUIC), gRPC, WebSockets, TLS (termination, MITM/SSL-inspection, SNI, session resumption, cert management at scale), and streaming architectures.
• Proven track record of hitting hard SLOs on a live data plane: peak RPS, p99 added latency, TLS handshakes/sec, concurrent connections — you can quote your numbers.
• Strong expertise in multi-tenant, multi-region service design, including PoP architecture, global load balancing, and failover.
• Deep understanding of Kubernetes and cloud-native application design as it applies to inline, stateful / long-lived-connection workloads.
• Strong systems-programming skills in Go (preferred), Rust, C++, or Java.
• Experience with observability platforms — OpenTelemetry, Prometheus, Grafana, or Datadog — for high-cardinality, request-path observability.
• Excellent architectural, debugging, and performance-optimization skills under production load.

Preferred Qualifications

• Experience building or extending Envoy (filters, xDS control plane), API gateways (Kong, Apigee, Tyk), service mesh (Istio, Linkerd, Consul Connect), or edge/CDN platforms.
• Production experience with SWG, SASE, CASB, ZTNA, or forward/reverse proxy products.
• Experience with AI/LLM traffic patterns (streaming responses, long-lived SSE/WebSocket connections, token-level inspection).
• Experience designing systems.