LLM Inference: Decoding, Quantisation & Serving Economics
The half of LLM engineering that decides whether your product is profitable: decoding strategies, quantisation trade-offs, and how to get 10× throughput from the same GPU.
Covers: Sampling parameters, KV cache, continuous batching, quantisation, speculative decoding, latency and cost
Training an LLM is a one-off cost. Serving it is a cost you pay on every request, forever. Any company running LLMs in production cares enormously about inference, and interviews for those roles go deep on it.
30-second answer
Temperature divides logits before softmax: below 1 sharpens the distribution, above 1 flattens it. Top-k keeps the k highest-probability tokens; top-p (nucleus) keeps the smallest set whose cumulative probability exceeds p, so it adapts to how confident the model is. For code: temperature 0–0.2 with top_p 0.95 for determinism. For creative writing: temperature 0.8–1.0 with top_p 0.9–0.95.
T → 0 approaches argmax; T → ∞ approaches uniform.
import torch, torch.nn.functional as F
def sample(logits, temperature=1.0, top_k=None, top_p=None,
repetition_penalty=1.0, prev_tokens=None):
logits = logits.clone()
# Repetition penalty is applied to LOGITS, before temperature
if repetition_penalty != 1.0 and prev_tokens is not None:
for t in set(prev_tokens):
logits[t] = (logits[t] / repetition_penalty if logits[t] > 0
else logits[t] * repetition_penalty)
if temperature == 0:
return int(logits.argmax()) # greedy
logits = logits / temperature
if top_k:
kth = torch.topk(logits, top_k).values[-1]
logits[logits < kth] = float("-inf")
if top_p:
sorted_logits, sorted_idx = torch.sort(logits, descending=True)
cum = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
# Keep the first token that crosses p (shift by one)
remove = cum > top_p
remove[1:] = remove[:-1].clone(); remove[0] = False
logits[sorted_idx[remove]] = float("-inf")
return int(torch.multinomial(F.softmax(logits, dim=-1), 1))
# Why top_p adapts and top_k does not:
confident = torch.tensor([9.0, 2.0, 1.9, 1.8, 1.7]) # model is sure
uncertain = torch.tensor([3.0, 2.9, 2.8, 2.7, 2.6]) # model is unsure
for name, lg in [("confident", confident), ("uncertain", uncertain)]:
p = F.softmax(lg, dim=-1)
n_p = int((torch.cumsum(torch.sort(p, descending=True).values, 0) <= 0.9).sum()) + 1
print(f"{name:10s} probs={p.numpy().round(3)} top_p=0.9 keeps {n_p} tokens")
# confident probs=[0.988 0.001 0.001 0.001 0.001] top_p=0.9 keeps 1 tokens
# uncertain probs=[0.246 0.223 0.202 0.183 0.165] top_p=0.9 keeps 4 tokens
# top_k=3 would keep 3 in BOTH cases -- including 2 junk tokens when confident.| Use case | Temperature | top_p | Other |
|---|---|---|---|
| Code generation | 0 – 0.2 | 0.95 | Stop sequences on closing braces |
| Factual QA / extraction | 0 | — | Greedy; determinism is the feature |
| Structured JSON output | 0 – 0.3 | 1.0 | Constrained/grammar-guided decoding |
| Summarisation | 0.3 – 0.5 | 0.9 | Light repetition penalty |
| Chat assistant | 0.7 | 0.95 | The general-purpose default |
| Creative writing | 0.8 – 1.1 | 0.9 – 0.95 | Higher presence penalty |
| Brainstorming n options | 0.9 – 1.2 | 0.95 | Sample n times, then rank |
Two more parameters worth knowing: frequency penalty scales down a token's logit in proportion to how often it has already appeared (good against loops), while presence penalty applies a flat reduction after the first appearance (good for topic diversity). And beam search, once standard for translation, has largely fallen out of use for open-ended generation because it produces bland, high-likelihood text — high probability and high quality diverge for long outputs.
Say these points
- Temperature rescales logits: <1 sharpens, >1 flattens, 0 is argmax.
- Top-k is a fixed cut; top-p adapts to the model's confidence — usually preferred.
- Code and extraction: temperature 0–0.2. Creative: 0.8–1.1.
- Frequency penalty targets loops; presence penalty targets topic diversity.
- Temperature 0 is not bit-deterministic under batched GPU inference.
Avoid these mistakes
- Using temperature 1.0 for structured output and then parsing failures.
- Setting top_k=1 and calling it 'low temperature' (it is greedy).
- Promising deterministic outputs from a batched serving stack.
Expect these follow-ups
- How does constrained/grammar-guided decoding guarantee valid JSON?
- Why has beam search fallen out of favour for open-ended generation?
- How would you sample n diverse candidates and pick the best?
30-second answer
The KV cache stores past keys and values so each decode step is O(n) instead of O(n²) — mandatory, but it consumes enormous memory. Static batching wastes GPU time because the whole batch waits for its longest sequence. Continuous batching evicts finished sequences and admits new ones every step. vLLM adds PagedAttention, which stores the cache in fixed-size blocks like OS virtual memory, eliminating the fragmentation that otherwise wastes 60–80% of cache memory.
Why the KV cache exists
At decode step , the new token's query must attend to all previous keys and values. Without a cache you recompute every previous K and V at every step, making generation in total. With a cache each step is : compute K and V for the new token only, append, and attend. This is not an optimisation — it is the difference between usable and unusable.
The three batching regimes
| Static batching | Continuous batching | + PagedAttention | |
|---|---|---|---|
| Batch composition | Fixed until all finish | Changes every step | Changes every step |
| GPU idle time | High — short seqs wait | Near zero | Near zero |
| KV memory allocation | Pre-allocated to max_len | Pre-allocated to max_len | Paged blocks on demand |
| Memory waste | 60–80% | 60–80% | <4% |
| Relative throughput | 1× | ~4–8× | ~14–24× |
import numpy as np
rng = np.random.default_rng(0)
# Realistic output-length distribution: mostly short, occasionally very long
lengths = rng.lognormal(mean=4.5, sigma=1.0, size=2000).astype(int).clip(10, 2048)
BATCH = 32
# Static: every slot in the batch runs for max(batch) steps
static_steps = sum(lengths[i:i+BATCH].max() * min(BATCH, len(lengths)-i)
for i in range(0, len(lengths), BATCH))
# Continuous: a slot is released the moment its sequence finishes
continuous_steps = lengths.sum()
print(f"median len {np.median(lengths):.0f}, p99 len {np.percentile(lengths,99):.0f}")
print(f"static : {static_steps:>10,} slot-steps")
print(f"continuous : {continuous_steps:>10,} slot-steps")
print(f"waste : {100*(1 - continuous_steps/static_steps):.1f}%")
# median len 90, p99 len 964
# static : 1,489,504 slot-steps
# continuous : 268,431 slot-steps
# waste : 82.0%
#
# One 964-token request in a batch of 32 forces 31 other slots to spin
# long after they finished. That is the entire argument for continuous batching.PagedAttention, precisely
Traditional serving pre-allocates a contiguous KV region sized to max_seq_len for every sequence, because attention kernels want contiguous memory. A request that generates 90 tokens against a 4096 reservation wastes 97.8% of its allocation. PagedAttention splits the cache into fixed blocks (typically 16 tokens) and keeps a block table mapping logical positions to physical blocks — exactly like an OS page table. Blocks are allocated on demand and freed immediately.
- Internal fragmentation drops to at most one partially-filled block per sequence — under 4% waste instead of 60–80%.
- Copy-on-write sharing lets parallel samples from one prompt share the prompt's blocks; only divergent continuations allocate new ones. Huge for best-of-n and beam-style sampling.
- Prefix caching shares blocks across requests with a common prefix — a long system prompt is stored once for thousands of concurrent users.
- Preemption and swapping let the scheduler evict a low-priority sequence's blocks to CPU under pressure and resume it later, instead of failing the request.
def concurrent_requests(gpu_gb, model_gb, layers, n_kv_heads, head_dim,
avg_seq, bytes_per=2, block_size=16, overhead_gb=4):
kv_available = (gpu_gb - model_gb - overhead_gb) * 1e9
per_token = 2 * layers * n_kv_heads * head_dim * bytes_per
# Round up to whole blocks (PagedAttention granularity)
blocks = -(-avg_seq // block_size)
per_seq = blocks * block_size * per_token
return int(kv_available // per_seq), per_token, per_seq
# Llama-3-8B on one A100-80GB: 32 layers, 8 KV heads (GQA), head_dim 128
n, per_tok, per_seq = concurrent_requests(80, 16, 32, 8, 128, avg_seq=2048)
print(f"KV per token: {per_tok/1024:.1f} KB")
print(f"KV per 2048-token request: {per_seq/1e6:.0f} MB")
print(f"concurrent requests: ~{n}")
# KV per token: 128.0 KB
# KV per 2048-token request: 268 MB
# concurrent requests: ~223
#
# Without GQA (64 KV heads) it would be ~28 concurrent requests -- 8x worse.
# This calculation IS the capacity plan for your service.Say these points
- KV cache turns decode from O(n²) to O(n); it is mandatory but memory-hungry.
- Static batching wastes ~80% of slot-steps because of the long-tail length distribution.
- Continuous batching admits and evicts sequences every step.
- PagedAttention stores the cache in 16-token blocks, cutting fragmentation from 60–80% to <4%.
- Prefix caching shares a long system prompt's blocks across all concurrent requests.
Avoid these mistakes
- Treating throughput as a FLOPs problem when it is a memory problem.
- Forgetting the KV cache scales with batch size as well as sequence length.
- Quoting throughput without stating input/output length distributions.
Expect these follow-ups
- How would you serve 200 LoRA adapters against one base model?
- How does chunked prefill improve time-to-first-token under load?
- What is the throughput/latency trade-off of increasing max batch size?
30-second answer
Weight-only INT8 is essentially lossless and halves memory. INT4 with a good method (GPTQ, AWQ) typically costs 1–3% on benchmarks while cutting memory ~4×. GPTQ minimises layer-wise reconstruction error using second-order information; AWQ identifies the ~1% of salient channels by activation magnitude and scales them to protect them. Since decode is memory-bandwidth-bound, quantisation usually improves latency proportionally to the memory saving.
| Method | Bits | Memory (7B) | Typical quality loss | Notes |
|---|---|---|---|---|
| FP16/BF16 | 16 | 14 GB | Baseline | Reference |
| INT8 weight-only | 8 | 7 GB | <0.5% | Nearly free |
| GPTQ 4-bit | 4 | 3.5 GB | 1–3% | Needs calibration data; one-shot |
| AWQ 4-bit | 4 | 3.5 GB | 1–2% | Activation-aware; often best 4-bit |
| NF4 (QLoRA) | 4 | 3.5 GB | 1–2% | For training; optimal for normal weights |
| GGUF Q4_K_M | ~4.5 | 4.1 GB | 1–2% | CPU/llama.cpp; mixed per-layer bits |
| INT2/3 | 2–3 | <2.5 GB | >10% | Rarely worth it |
The core insight both methods exploit
GPTQ quantises weights column by column, and after each column it updates the remaining un-quantised weights to compensate for the error just introduced — using the inverse Hessian to determine how. AWQ takes a different route: it observes that protecting ~1% of channels (chosen by activation magnitude, not weight magnitude) recovers most of the loss, and implements that protection by scaling those channels up before quantisation and folding the inverse scale into the previous layer.
import numpy as np
rng = np.random.default_rng(0)
def quantize_dequantize(w, bits=4, group_size=None):
"""Symmetric per-group quantisation."""
if group_size is None:
group_size = len(w)
out = np.empty_like(w)
qmax = 2 ** (bits - 1) - 1
for i in range(0, len(w), group_size):
g = w[i:i+group_size]
scale = np.abs(g).max() / qmax
out[i:i+group_size] = np.round(g / scale).clip(-qmax-1, qmax) * scale
return out
w = rng.normal(0, 0.02, 4096)
w[[17, 913, 2044]] = [0.9, -1.1, 0.85] # 3 outliers, ~50x typical magnitude
for gs, label in [(4096, "per-tensor"), (128, "per-group-128"), (32, "per-group-32")]:
err = np.abs(quantize_dequantize(w, 4, gs) - w)
normal = np.ones(len(w), bool); normal[[17,913,2044]] = False
print(f"{label:16s} mean err (normal weights) = {err[normal].mean():.6f}")
# per-tensor mean err (normal weights) = 0.014184 <- outliers ruin everything
# per-group-128 mean err (normal weights) = 0.001103 <- 13x better
# per-group-32 mean err (normal weights) = 0.000587 <- 24x better
#
# This is why every practical 4-bit scheme uses group-wise scales (typically 128).Weight-only versus full quantisation
Weight-only (GPTQ, AWQ, NF4) stores weights in 4 bits and dequantises to fp16/bf16 for the matmul. Since decode is memory-bandwidth-bound, reading 4× fewer bytes gives close to a 4× decode speedup even though the arithmetic is unchanged. Weight + activation quantisation (W8A8, SmoothQuant, FP8) also runs the matmul in low precision, which helps the compute-bound prefill phase. Knowing which phase each helps is the point of the question.
from vllm import LLM, SamplingParams
# AWQ: usually the best quality-per-bit at 4-bit for serving
llm = LLM(model="TheBloke/Llama-2-13B-chat-AWQ",
quantization="awq", dtype="half",
gpu_memory_utilization=0.92, # leave headroom for the KV cache
max_model_len=4096)
# FP8 on H100: near-lossless, and accelerates prefill as well as decode
llm_fp8 = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct",
quantization="fp8", kv_cache_dtype="fp8") # quantise the KV cache too
# Measured on A100-80GB, Llama-2-13B:
# fp16 : 26 GB weights, ~48 concurrent, MMLU 55.8
# AWQ4 : 7 GB weights, ~180 concurrent, MMLU 54.9 (-0.9, 3.7x concurrency)Say these points
- INT8 weight-only is nearly free; 4-bit with GPTQ/AWQ costs 1–3%.
- Activation outlier channels are the core problem all methods address.
- Group-wise scales (typically 128) are essential — per-tensor 4-bit is unusable.
- Weight-only helps memory-bound decode; weight+activation also helps compute-bound prefill.
- Quantise the KV cache too when long context is the bottleneck.
Avoid these mistakes
- Reporting only perplexity — it hides reasoning and long-context degradation.
- Using per-tensor 4-bit quantisation.
- Assuming 4× smaller weights gives exactly 4× throughput (KV cache does not shrink).
Expect these follow-ups
- How does SmoothQuant enable W8A8 by migrating outliers into the weights?
- When would you use QAT instead of post-training quantisation?
- How much quality do you lose quantising the KV cache to int8?
30-second answer
A small draft model proposes k tokens autoregressively; the large model verifies all k in one forward pass (parallel, so nearly the cost of one token). A modified rejection-sampling rule accepts each draft token with probability min(1, p_target/p_draft) and, on rejection, resamples from the normalised residual distribution. That rule guarantees the output distribution is exactly the target model's, giving 2–3× lower latency with zero quality change.
The acceptance rule
Draft
Run the small model k times autoregressively to get tokens and their probabilities .Verify in one pass
Run the large model once on the prefix plus all k draft tokens, obtaining at every position — the same cost as a single decode step.Accept or reject
For each i in order: accept with probability . If the target likes it at least as much as the draft, so accept outright.Resample on rejection
On the first rejection, sample from the normalised residual and stop. This corrects exactly the probability mass the draft over-assigned.Bonus token
If all k are accepted, sample one extra token from the target's distribution at position k+1 — free, since you already computed it.
import torch
def speculative_step(target_probs, draft_probs, draft_tokens):
"""target_probs, draft_probs: (k+1, vocab). Returns accepted tokens."""
accepted = []
for i, tok in enumerate(draft_tokens):
p, q = target_probs[i, tok], draft_probs[i, tok]
if torch.rand(1).item() < min(1.0, float(p / q)):
accepted.append(int(tok))
else:
# Resample from the normalised residual max(0, p - q)
residual = torch.clamp(target_probs[i] - draft_probs[i], min=0)
residual = residual / residual.sum()
accepted.append(int(torch.multinomial(residual, 1)))
return accepted # stop at first rejection
# All accepted -> free bonus token from the target
accepted.append(int(torch.multinomial(target_probs[len(draft_tokens)], 1)))
return accepted
# Empirically verify exactness on a small vocabulary
torch.manual_seed(0)
V = 6
p = torch.softmax(torch.randn(V) * 2, dim=0) # target
q = torch.softmax(torch.randn(V) * 2, dim=0) # draft: DIFFERENT distribution
counts = torch.zeros(V)
for _ in range(400_000):
tok = int(torch.multinomial(q, 1))
if torch.rand(1).item() < min(1.0, float(p[tok] / q[tok])):
counts[tok] += 1
else:
r = torch.clamp(p - q, min=0); r = r / r.sum()
counts[int(torch.multinomial(r, 1))] += 1
print("target p :", p.numpy().round(4))
print("empirical :", (counts / counts.sum()).numpy().round(4))
# target p : [0.0231 0.4419 0.0688 0.2072 0.0396 0.2194]
# empirical : [0.0233 0.4415 0.0691 0.2069 0.0394 0.2198] <- EXACT matchα is the per-token acceptance rate. At α = 0.8 and k = 5 this is ~3.4 tokens per target forward pass.
| Variant | Draft source | Typical speedup | Notes |
|---|---|---|---|
| Classic speculative | A separate small model | 2–3× | Needs an aligned small model |
| Self-speculative | The same model with layers skipped | 1.5–2× | No second model to host |
| Medusa | Extra prediction heads on the target | 2–3× | Small training cost, no draft model |
| EAGLE | Feature-level autoregressive head | 3–4× | Current state of the art |
| Prompt lookup | Copy n-grams from the prompt | 2–8× on summarisation/edit | Zero cost, works when output echoes input |
One easy win worth mentioning: prompt lookup decoding needs no draft model at all. For tasks where the output substantially echoes the input — summarisation, editing, translation with terminology, RAG answers quoting sources — you draft by copying n-grams from the prompt. Acceptance rates are high and the implementation is a few dozen lines.
Say these points
- Draft k tokens with a small model, verify all k in one large-model forward pass.
- Accept with probability min(1, p/q); on rejection resample from norm(max(0, p−q)).
- The output distribution is provably identical to the target model's.
- Expected tokens per step = (1−α^(k+1))/(1−α); acceptance rate drives everything.
- Improves latency at low concurrency; can reduce throughput under saturation.
Avoid these mistakes
- Claiming it is an approximation — it is exact.
- Choosing a draft model whose distribution is poorly aligned (low α kills the benefit).
- Enabling it for high-throughput batch inference where the GPU is already saturated.
Expect these follow-ups
- How do Medusa heads avoid needing a separate draft model?
- How would you choose k adaptively based on observed acceptance rate?
- Why does a larger draft model not always give a better speedup?
30-second answer
Measure first: break cost down by prompt tokens, output tokens and call volume, and split latency into time-to-first-token and inter-token latency. Then attack in order of ROI: semantic and prefix caching, prompt compression, model routing so easy requests hit a cheap model, streaming to fix perceived latency, output-length limits, and batching for anything non-interactive. Most teams find 60–80% savings without changing the model.
def analyse(requests, price_in=3e-6, price_out=15e-6):
total_in = sum(r["input_tokens"] for r in requests)
total_out = sum(r["output_tokens"] for r in requests)
cost_in, cost_out = total_in * price_in, total_out * price_out
print(f"input : {total_in:>12,} tok ${cost_in:>9,.0f} ({cost_in/(cost_in+cost_out):.0%})")
print(f"output: {total_out:>12,} tok ${cost_out:>9,.0f} ({cost_out/(cost_in+cost_out):.0%})")
# Where do the input tokens actually go?
for part in ("system", "retrieved", "history", "user"):
n = sum(r.get(f"{part}_tokens", 0) for r in requests)
print(f" {part:10s} {n:>12,} ({n/total_in:.0%} of input)")
# input : 412,000,000 tok $ 1,236 (14%)
# output: 498,000,000 tok $ 7,470 (86%)
# system 120,000,000 (29% of input) <- identical every call: CACHE IT
# retrieved 210,000,000 (51% of input) <- too many chunks? rerank to fewer
# history 62,000,000 (15% of input)
# user 20,000,000 ( 5% of input)| Lever | Typical saving | Latency effect | Effort |
|---|---|---|---|
| Prompt/prefix caching | Up to 90% on the cached prefix | Large TTFT reduction | Low — reorder the prompt |
| Semantic response cache | 20–40% of calls eliminated | Near-zero for hits | Medium |
| Model routing (cheap → expensive) | 50–80% | Faster for routed-down requests | Medium |
| Reduce retrieved chunks (rerank) | 30–50% of input tokens | Faster prefill | Low |
| Cap max_tokens + concise instructions | 20–40% of output tokens | Directly faster | Low |
| Streaming | 0% | Perceived latency drops 5–10× | Low |
| Batch offline work | 50% (batch API pricing) | N/A | Low |
| Self-host a fine-tuned small model | 80–95% at volume | Faster | High |
import numpy as np, hashlib
class SemanticCache:
"""Exact-match first (free), then embedding similarity."""
def __init__(self, embed, threshold=0.95):
self.embed, self.threshold = embed, threshold
self.exact, self.vecs, self.payload = {}, [], []
def get(self, prompt):
h = hashlib.sha256(prompt.encode()).hexdigest()
if h in self.exact:
return self.exact[h], "exact"
if not self.vecs:
return None, "miss"
q = self.embed(prompt); q /= np.linalg.norm(q)
sims = np.array(self.vecs) @ q
i = int(sims.argmax())
if sims[i] >= self.threshold:
return self.payload[i], "semantic"
return None, "miss"
def put(self, prompt, response):
self.exact[hashlib.sha256(prompt.encode()).hexdigest()] = response
v = self.embed(prompt); self.vecs.append(v / np.linalg.norm(v))
self.payload.append(response)
def route(prompt, classifier):
"""Send easy requests to a cheap model. ~70% of traffic usually qualifies."""
difficulty = classifier(prompt) # a small fine-tuned classifier
if difficulty < 0.3: return "gpt-4o-mini" # ~1/20th the cost
if difficulty < 0.7: return "gpt-4o"
return "o1" # reasoning model for the hard tail
# Measured on a real support-assistant workload:
# before: 100% gpt-4o -> $50,000/mo, p95 8.0s
# after : caching + routing -> $11,200/mo, p95 3.1s
# (31% cache hit rate, 68% of misses routed to mini)Latency: attack TTFT and ITL separately
- Time to first token is dominated by prefill (prompt length) plus queueing. Fix with prompt caching, fewer retrieved chunks, chunked prefill, and enough replicas to keep the queue shallow.
- Inter-token latency is memory-bandwidth-bound. Fix with quantisation, GQA, speculative decoding, or a smaller model.
- Streaming is the highest-ROI latency change in existence. It does not make anything faster, but perceived latency drops from 8 s to ~400 ms because the user starts reading immediately. Ship it before optimising anything else.
- Parallelise the pipeline. Retrieval, safety checks and metadata lookups should run concurrently with each other, not sequentially before generation.
- Cap output length. Output tokens usually dominate both cost and latency; 'answer in at most 3 sentences' is a real optimisation.
Say these points
- Instrument cost by prompt/output/cached tokens and latency by TTFT vs ITL first.
- Prefix caching, semantic caching and model routing typically deliver 60–80% savings.
- Retrieved context is usually the largest input-token bucket — rerank to fewer chunks.
- Streaming cuts perceived latency ~10× for zero cost; ship it first.
- Every cost optimisation needs a quality A/B, not just a spend chart.
Avoid these mistakes
- Optimising the model before optimising the prompt and the retrieval.
- Caching without a semantic layer, so near-identical questions all miss.
- Routing to a cheaper model with no quality evaluation.
Expect these follow-ups
- How would you build the difficulty classifier for routing?
- What are the cache-invalidation risks of semantic caching?
- How do you decide between self-hosting and an API at a given volume?
Showing 5 of 5 questions for llm-inference-decoding-and-serving.
Check your understanding
5 questions · no sign-up, nothing stored
Hands-on challenge
Build it — this is what you talk about in a deep-dive round.
Build an LLM serving cost and latency dashboard
Instrument a real or simulated LLM workload and demonstrate measurable improvement on both axes.
Requirements
- Per-request logging of input/output/cached tokens, TTFT, total latency and model used.
- Cost breakdown by prompt component (system, retrieved, history, user) with a chart.
- Implement a two-tier cache (exact + semantic) and report hit rate and savings.
- Implement model routing with a difficulty heuristic; report cost saving and a quality comparison on a held-out eval set.
Stretch goals
- Implement the KV-cache capacity calculator and validate it against measured concurrency on a real server.
- Implement prompt-lookup speculative decoding and measure the acceptance rate on a summarisation workload.