LLM Fundamentals: Tokenisation, Pre-training & Scaling Laws

The practical problems that show up in production

1. 1

   Language cost asymmetry

   English averages ~1.3 tokens per word. Hindi, Thai, Burmese and many others can average 3–5× more because their scripts are underrepresented in the merge training corpus. The same request literally costs several times more in some languages — a genuine fairness and unit-economics issue, not a curiosity.
2. 2

   Number tokenisation breaks arithmetic

   Depending on the tokenizer, 1234 might be one token, or 12 + 34, or 1 + 234. The model must learn arithmetic over an inconsistent representation. Newer tokenizers deliberately split digits individually, which measurably improves arithmetic.
3. 3

   Trailing whitespace changes outputs

   "The answer is" and "The answer is " tokenise differently, because most tokens embed a leading space. Ending a prompt with a space frequently degrades completions. This is a real, reproducible API gotcha.
4. 4

   Structured formats are token-expensive

   JSON's braces, quotes and repeated keys cost tokens on every record. For long lists, a compact format can cut token usage 30–50% for identical information — a direct cost saving at scale.
5. 5

   Character-level tasks are hard

   'How many r's in strawberry?' is difficult because the model sees a few subword tokens, not letters. It is a tokenisation artefact, not a reasoning failure — a useful thing to be able to explain to a stakeholder.

When an encoder is still the right call

• Embeddings for retrieval. A 110M-parameter bidirectional encoder produces excellent sentence embeddings at a fraction of a decoder's cost. Nearly every production RAG system uses an encoder for the embedding model.
• High-volume classification. Classifying 50M documents a day: a fine-tuned DeBERTa costs cents; calling an LLM costs thousands of dollars and adds latency.
• Token-level tasks. NER and extractive QA benefit genuinely from seeing tokens on both sides.
• Reranking. A cross-encoder that reads query and document jointly is the standard second stage in retrieval, and bidirectional attention is exactly what makes it strong.
• Strict latency budgets. Sub-10ms inference is achievable with a small encoder and effectively impossible with a generative model.

For completeness: T5-style encoder-decoders remain strong for seq2seq tasks with a fixed input-output structure (translation, summarisation) because the encoder can attend bidirectionally over the source while the decoder generates. They lost ground mostly because a single decoder-only model with a good prompt is simpler to operate, not because the architecture is worse.

The structure matters: two separate terms means you can be bottlenecked by either parameters or data. Kaplan's original analysis used a fixed LR schedule across all runs, which systematically disadvantaged the longer (more data) runs. Chinchilla re-ran with per-run cosine schedules and the conclusion flipped.

The caveats to raise

• Scaling laws are fits, not physics. They hold impressively over many orders of magnitude but extrapolation far beyond the fitted range is an assumption, not a guarantee.
• Data quality shifts the curve. Deduplication, filtering and curriculum can be worth an effective 2× in data. The laws assume a fixed data distribution.
• Data is finite. High-quality public text is estimated in the tens of trillions of tokens. Beyond that you need synthetic data, multiple epochs (which degrade in value) or new modalities.
• MoE changes the accounting. A mixture-of-experts model has many more parameters than active parameters per token, so N in the FLOP formula is the active count, not the total.
• 'Emergent abilities' are partly a metric artefact. Sharp jumps often disappear under continuous metrics — the underlying capability improves smoothly, but a thresholded metric like exact-match only registers it late.

The cost argument for not just using a bigger window

Prefill cost is linear in context length and attention is quadratic in it. A 100k-token prompt is roughly 100× the prefill cost and noticeably slower time-to-first-token compared with a 1k prompt built by good retrieval — and it is frequently less accurate because of the middle-of-context degradation. 'Just use the long-context model' is usually the expensive and worse option.

• Always pin the system prompt. The most common production bug is a truncation routine that drops the instructions and the assistant silently changes behaviour.
• Use prompt caching. Anthropic and OpenAI both cache stable prefixes, so putting the invariant system prompt and long documents first can cut cost by up to 90% on repeated calls.
• Store structured memory separately. User preferences and durable facts belong in a database, not re-derived from a transcript every session.
• Track token usage as a product metric. Tokens per conversation is a cost line item; alert on it the way you alert on latency.

The mechanical difference: an LLM produces a distribution over next tokens; an embedding model produces one vector for the whole input. A generative model's hidden states are not good similarity embeddings out of the box — they are optimised for prediction, not for a metric space. That is why dedicated embedding models exist and why mean-pooling GPT hidden states underperforms.

Finally, mention Matryoshka representation learning: models trained so that the first 256 of 1536 dimensions are themselves a usable embedding. This lets you run a cheap 256-d first pass over the whole corpus and rerank the top candidates with full dimensions — meaningful index-cost savings with negligible quality loss.