How to Start Learning Inference Engineering (Why It's Everywhere)

There is a green book going around X right now, Inference Engineering by @philipkiely

I am reading it too. But before jumping into the book, it is worth clearing the assumptions the book does not stop to clear.

What it did well is give a scattered pile of problems, GPU memory, batching, latency, cost per token, one name and one map. That name is now spreading faster than the guidance. People want into this field, and almost nobody answers the entry questions:

Where do you actually start? How much machine learning is genuinely mandatory? Is this a separate career, or a corner of the one you already have?

That is the gap this covers. What the field is, what it contains, what you need before you begin, and the order to learn it in. No code, no walkthrough.

The confusion

Ask five engineers what inference engineering is and you get two answers. Half say it is MLOps with a GPU attached. The other half say it is a niche for CUDA people and quietly exclude themselves.

Both are wrong, and the second one is expensive, because it stops people from starting.

MLOps optimises pipelines and reliability. Inference engineering optimises a different unit: useful tokens per second, per GPU, per dollar. That single difference changes what you measure, what you tune, and what counts as a win. A pipeline that never fails but serves 40 tokens per second on an H100 is an MLOps success and an inference failure.

It is not a subfield of AI engineering either. It sits underneath it. AI engineering asks what the model should do. Inference engineering asks what it costs to let it.

What it actually is

Philip Kiely, who wrote the field's first real textbook, frames it in one line:

Doing inference well requires three layers: Runtime, Infrastructure, and Tooling.

Runtime is one GPU, made as fast as it can be. Infrastructure is many GPUs, across machines and clouds, made as cheap as they can be. Tooling is the instrumentation that makes the other two debuggable, because you cannot optimise a system you cannot see.

The problem all three solve is the same one. A trained model is a fixed artifact. You cannot make it smarter without retraining it, but you can make it dramatically cheaper and faster to run, and the gap between a naive deployment and a tuned one is not ten percent. It is multiples.

Why the field is heating up

Inference now consumes about two thirds of AI compute, up from roughly a third in 2023, according to Deloitte figures cited in @chriszeoli 's survey of the field. Training happens once. Serving happens every time a user types.

@baseten, which publishes the textbook and also sells inference infrastructure, projects a million inference engineers within a couple of years. Treat that as an interested party's forecast rather than a fact.

Here is the part nobody tells beginners. In August 2026, Zeoli hand-counted the public careers boards and found roughly 58 open inference engineering roles, with 24 of them at just OpenAI and Anthropic. He calls it a floor, since several large employers run job portals that defeat counting. Posted ranges at those two labs reach $850,000.

So: enormous leverage, extreme compensation, and a visible market of a few dozen listed jobs concentrated in a handful of US labs. Those numbers do not transfer to every market, mine included. Learn this because the skill pays off inside whatever job you already have, not because there is a queue of recruiters waiting.

What is actually mandatory

This is where most people stall, so let me be blunt about the ordering.

Personally I focus on the basics of LLMs and a bit of GPUs, and that has been enough to be useful. The people who tell you to learn CUDA first are describing the last ten percent of the field as if it were the entrance.

What "enough GPU" means, concretely. HBM is the slow tier, and it is the one you fight. Weights stream out of it on every decode step, which is why decode is memory-bound while prefill, doing dense matrix work, is compute-bound. Arithmetic intensity tells you which side you are on: FLOPs done over bytes fetched, measured against the card's own peak compute over peak bandwidth. Below the card's number, more FLOPs buy you nothing.

That is the whole prerequisite. CUDA you need to read, not write: enough to follow a profiler trace and see what fusing two kernels saves.

The map

1. 01Runtime covers inference mechanics, the KV cache, arithmetic intensity, kernel selection and fusion, the serving engines (vLLM with PagedAttention and continuous batching, SGLang with RadixAttention, TensorRT-LLM), quantization (FP8, INT8, INT4, and the algorithms GPTQ, AWQ, SmoothQuant), speculative decoding, prefix caching, model parallelism, and prefill/decode disaggregation.
2. 02Infrastructure covers GPU architecture and memory hierarchy, hardware generations, multi-instance GPU, containerisation, autoscaling and cold starts, routing and queueing, multi-cloud capacity, and cost estimation.
3. 03Tooling covers benchmarking and profiling, observability, and client-side concerns like streaming and async protocols. It is the smallest layer and the one people skip, which is why so many optimisation efforts are guesses.

The three engines, and what separates them. You will meet vLLM, SGLang and TensorRT-LLM immediately, and they are easy to mistake for interchangeable. They are not. Each one bets on a different idea.

• →vLLM pages the KV cache. Instead of reserving a contiguous block big enough for the longest possible response, it hands out small fixed-size pages on demand, the way an operating system does. Reservation wasted most of what it held. Paging cut that to a few percent, which means more requests fit on the same card.
• →SGLang shares the cache between requests. It keeps prefixes in a radix tree, so a system prompt or a few-shot block that a thousand requests have in common gets computed once instead of a thousand times. When your traffic repeats itself, this is the bigger win.
• →TensorRT-LLM compiles. Rather than a Python scheduler driving PyTorch, it builds the model ahead of time into an execution plan tuned to one specific NVIDIA GPU, with kernels fused and selected in advance. You get the highest raw throughput and you pay for it in operational rigidity, because changing the model or the hardware means building again.

If you want to read more technical perspective about Inference Engineering then read this article by @TheAhmadOsman:

What the top labs are actually working on

If you want to know where this field is heading, watch what the labs optimise when budget is not the constraint.

Disaggregation. Prefill and decode have opposite performance profiles, so the frontier runs them on separate GPU pools and ships the KV cache between them. It matters most under high load, which is exactly where unit cost gets decided.

Speculative decoding, in production rather than in papers. A small draft model proposes tokens and the large model verifies them in parallel. Production benchmarks across vLLM, SGLang and TensorRT-LLM report 2x to 6.5x throughput gains at low to medium concurrency. DeepSeek-V3 built Multi-Token Prediction into the model itself and reports around 1.8x at acceptance rates above 80%.

Their own serving stack. OpenAI credits work on GPT-5.6 Sol's serving path with roughly 20% lower serving cost from GPU kernel improvements and 15% better token generation from improved speculative decoding.

The metric itself. Peak tokens per second is out. What labs optimise now is useful throughput under time-to-first-token and time-per-output-token constraints, which is a harder target than a single throughput number.

The gap between a stock deployment and a tuned one is commonly 3x to 4x. Closing that gap is the job.

The sequence

Before any of the three layers, the fundamentals. Neural networks, deep learning, and enough linear algebra to read a matrix multiply without flinching. Those are assumed rather than taught here, and if you are already shipping with LLMs you probably have them.

Then the part that actually carries into inference: transformer architecture, the attention mechanism, how decoding works, the prefill and decode split, and the KV cache. Learn those five properly.

That is enough theory. Not most of a masters, not a reading list of forty papers. Five topics, and then the book takes over and you follow Kiely's chapter order.

Once you are inside the three layers, the conventional advice is to start at the runtime layer, because that is where the leverage is. I would not.

Start with tooling, for about a week. Learn to measure time to first token, time between tokens, throughput and GPU utilisation on something you already run. Without those numbers, every optimisation you attempt afterwards is superstition.

Then go to runtime, and stay there. This is the bulk of the work and the bulk of the value. Deploy vLLM. Watch what continuous batching does to your throughput. Quantize a model and measure what you lost. This layer alone will make you useful.

Then infrastructure, when your problem becomes many GPUs rather than one. Most engineers reach this later than they expect, and some never need it.

The order matters because each layer's wins are invisible without the previous one's measurements.

Where to start this week

Three resources, all free.

See my previous post about inference engineering resources:

So, concretely.

RI have not worked every layer of this stack in production, and multi-cloud infrastructure is still ahead of me rather than behind me. But the entrance is not guarded the way it looks.

You do not need to be the person who writes the kernels. You need to be the person who knows why the GPU is idle and how to optimize.

Inference Engineering also impacts UX!

Follow me @kmeanskaran