ModelRefs / What are Large Language Models? — Tutorial
What are Large Language Models? — Tutorial
Understand how LLMs work, what they can and can't do, and how to use them via API. Covers A next-token predictor trained at massive scale.
Overview
Understand how LLMs work, what they can and can't do, and how to use them via API
Level: Intermediate. Estimated reading time: 25 minutes.
A next-token predictor trained at massive scale
A Large Language Model is a neural network (almost always transformer-based) trained to predict the next token in a sequence. Given "The capital of France is", it assigns probabilities to all tokens in its vocabulary and samples the most likely one — "Paris".
The 'large' refers to scale: billions to hundreds of billions of parameters, trained on trillions of tokens of web text, books, and code. Scale unlocks capabilities that don't appear in smaller models: multi-step reasoning, in-context learning, instruction following.
Emergent capabilities and what they mean
Capabilities not present in small models appear suddenly at larger scales — this is called emergence. Examples: few-shot learning (GPT-3 performing tasks from examples in the prompt), chain-of-thought reasoning (solving math by thinking step-by-step), code generation, and world knowledge.
These capabilities arise from the same pretraining objective (next-token prediction) applied at scale, not from task-specific training. Post-training (instruction fine-tuning + RLHF) shapes how the model applies these capabilities to user requests.
What LLMs cannot do — the stochastic parrot problem
LLMs do not 'understand' in the human sense — they are statistical models of token sequences. Common failure modes:
Hallucination: confidently generating plausible-sounding but false information. LLMs don't have a 'don't know' state by default.
No persistent memory: each conversation starts fresh unless you include history in the context window.
Knowledge cutoff: training data ends at a date; the model has no access to events after that.
Counting/arithmetic: transformer attention doesn't naturally handle exact computation — use tool calling for math.
Use retrieval (RAG), tool calling, and grounding to mitigate these limitations.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to What are Large Language Models? — Tutorial.