Senior Software Engineer - Distributed Systems

About the Role

Serving inference at scale is a scheduling problem. Requests arrive with wildly different shapes and latency expectations, GPUs are heterogeneous and expensive, and the difference between a platform that's fast and one that's economical usually comes down to how well work gets placed. That system is what you'll own.

You'll design and build the scheduling and routing layer of our platform: how requests get admitted, prioritized, batched, and placed across a heterogeneous multi-cloud GPU fleet, under real multi-tenant load and real latency commitments. This is core-systems work with a clean slate — you'll be making the foundational architectural decisions, not maintaining someone else's, and the quality of those decisions will show up directly in our margins and our customers' latency numbers.

You'll work close to the metal and close to the math. Some days that means reasoning about queueing behavior and control loops on a whiteboard; other days it means profiling Go or Rust until the tail latency comes down. We're a small team, so you'll own systems end-to-end — design, implementation, rollout, and the production reality afterward.

Responsibilities

• Own the scheduling and routing layer — design and build request admission, prioritization, batching, and placement across a heterogeneous GPU fleet spanning multiple clouds and accelerator types.
• Engineer for latency and utilization at once — drive down tail latency while driving up fleet utilization; these fight each other, and resolving that tension well is the job.
• Model the system, not just code it — apply queueing theory, control theory, and load-shedding principles to make the platform behave predictably under bursty, multi-tenant traffic.
• Build multi-tenant fairness and isolation — ensure priority guarantees and SLO commitments hold when the fleet is saturated and customers are competing for the same capacity.
• Own it in production — instrument, observe, and debug distributed behavior in a live system; carry your designs through rollout and real-world load.
• Set the technical bar — make foundational architecture decisions, write the design docs that anchor them, and raise the engineering standard of everyone around you.

Qualifications

This is a senior individual-contributor role. We're looking for someone who has built systems like this before and can operate independently from the first week.

• Substantial experience building and operating large-scale distributed systems in production — you've owned something load-bearing, not just contributed to it.
• Track record of designing core systems from zero to one, and living with the consequences of your architectural decisions.
• Hands-on experience with scheduling, load balancing, request routing, or resource allocation systems.
• Strong systems fundamentals — operating systems, networking, concurrency — and the ability to reason quantitatively about system behavior using queueing theory, control theory, or similar.
• Fluency in a performance-sensitive language (Go, Rust, or C++), with the profiling and optimization instincts that come from actually chasing latency in production.
• Comfort with GPU infrastructure and LLM inference fundamentals — batching, KV cache behavior, throughput/latency tradeoffs; deep expertise here is a plus, but strong distributed-systems judgment matters more.
• Clear technical writing — you can make a hard design decision legible to people who weren't in your head.
• An AI-native way of working — you use AI tools daily and have your own view of how they.