A large language model (LLM) is a deep learning model trained on enormous amounts of text to predict what comes next in a sequence. To answer a question, it breaks your prompt into small pieces called tokens, converts them into numbers, uses a neural network to estimate which token is most likely to follow, picks one, and repeats. A paragraph of fluent text is the result of that single step performed again and again.
IBM calls LLMs “giant statistical prediction machines” that repeatedly predict the next word in a sequence. That description sounds almost too simple to explain a chatbot that drafts essays and debugs code, and it leaves out how much structure is hidden inside the prediction. This article walks through the full pipeline, runs a small sampling example, explains how these models are trained, and covers why they sometimes state falsehoods with total confidence.
What Is a Large Language Model?
IBM defines LLMs as deep learning models trained on very large amounts of data that can understand and generate natural language and other content across many tasks. Most are built on the transformer architecture, introduced in 2017 in the paper “Attention Is All You Need.”
The “large” refers mainly to the number of parameters, the adjustable internal numbers that the model tunes during training. IBM notes that LLMs can have billions or even trillions of them. GPT-3, released in 2020, had 175 billion. Smaller language models with far fewer parameters exist for devices with limited resources.
An LLM is a neural network, so everything in machine learning applies: it starts with random parameters, measures its prediction error with a loss function, and improves through backpropagation and gradient descent. What makes it a language model is the data it learns from and the task it practices.
Step 1: Turning Text Into Tokens
Models cannot read letters directly. The first step is tokenization: splitting text into units called tokens. IBM explains that tokens can be whole words, parts of words, or single characters. Splitting into subword pieces lets a model handle rare or brand-new words by assembling them from familiar parts, instead of treating every unfamiliar word as unknown.
Each token is mapped to an ID number, and each ID is then converted into an embedding, a list of numbers that represents the token’s meaning in a form the network can work with. Tokens with related meanings tend to end up with similar embeddings. Because word order matters (“dog bites man” differs from “man bites dog”), the model also adds positional encodings that record where each token sits in the sequence.
Step 2: Attention Decides What Matters
The core innovation of the transformer is self-attention. In the sentence “The trophy didn’t fit in the suitcase because it was too big,” the word “it” refers to the trophy, and a good model needs to work that out. Self-attention lets every token look at every other token in the input and decide how relevant each one is.
IBM describes the mechanism this way: each token is projected into a query, a key, and a value vector. Comparing a token’s query with the other tokens’ keys produces attention weights, which determine how much each token’s value contributes to the updated representation. The model can focus on relevant context, including distant words, and give little weight to the rest.
Transformers also process many tokens in parallel, unlike older recurrent networks that handled a sequence one step at a time. IBM points to that parallelism as what made training on very large datasets practical. A real model stacks many layers of attention and other neural network components, each refining the representation of every token.
Step 3: Predicting the Next Token
After passing through all the layers, the model produces a score, called a logit, for every token in its vocabulary. These raw scores are converted by a function called softmax into probabilities that sum to 1.
Suppose the prompt is “me gusta mucho” (Spanish for “I like very much”). A model might assign scores such as 12.5 to viajar (to travel), 8.2 to jugar (to play), and -3.1 to dormir (to sleep). After softmax, nearly all the probability sits on viajar, a little on jugar, and almost none on dormir.
The model then selects one token, appends it to the input, and runs the whole process again to predict the token after that. This loop is why LLMs produce text one piece at a time. IBM describes generation as exactly this: tokenize the prompt, convert to embeddings, then generate output token by token by estimating probabilities for the next one.
It is fair to ask why such a simple objective produces useful behavior. To predict the next token well across billions of varied sentences, the network has to absorb grammar, common facts, writing conventions, and patterns of reasoning, because all of them help reduce prediction error. The skills are a side effect of practicing prediction at scale. They are also imperfect, which matters later when we reach hallucinations.
Step 4: Choosing the Token (Sampling)
The model could always pick the single most probable token, but that tends to produce repetitive, flat text. Instead, systems usually sample from the probabilities, with settings that control how adventurous the choice is.
Temperature divides the logits before softmax. A value below 1 widens the gaps between scores, so the top token dominates and output becomes more predictable. A value above 1 narrows the gaps, flattening the distribution so less likely tokens get a real chance.
Top-p (nucleus) sampling sorts tokens by probability and keeps only the smallest group whose combined probability reaches a threshold such as 0.9. The next token is sampled from that group alone, which cuts off the long tail of unlikely choices.
IBM notes that settings such as temperature and top-k or top-p sampling control randomness and focus. A common rule of thumb is a low temperature for factual or code tasks and a higher one for creative writing.
A runnable example
This short script uses the example scores above and shows how temperature and top-p reshape the choice, in plain Python with no libraries:
python
import math
import random
# Scores (logits) a model might assign to candidate next words
# after the prompt "me gusta mucho" (I like ... a lot)
logits = {"viajar": 12.5, "jugar": 8.2, "dormir": -3.1}
def softmax(scores, temperature=1.0):
scaled = [s / temperature for s in scores.values()]
top = max(scaled) # subtract max for numerical stability
exps = [math.exp(s - top) for s in scaled]
total = sum(exps)
return {word: e / total for word, e in zip(scores, exps)}
def top_p_filter(probs, p):
kept, running = {}, 0.0
for word, prob in sorted(probs.items(), key=lambda kv: -kv[1]):
kept[word] = prob
running += prob
if running >= p:
break
total = sum(kept.values())
return {w: pr / total for w, pr in kept.items()} # renormalize
for t in (0.5, 1.0, 2.0):
probs = softmax(logits, t)
print(f"temperature {t}: " + ", ".join(f"{w}={pr:.4f}" for w, pr in probs.items()))
random.seed(42)
probs = softmax(logits, 2.0)
nucleus = top_p_filter(probs, 0.9)
print("top-p 0.9 candidates at t=2:", {w: round(pr, 3) for w, pr in nucleus.items()})
draws = random.choices(list(nucleus), weights=list(nucleus.values()), k=1000)
print("1000 samples:", {w: draws.count(w) for w in nucleus})Running it gives this output:
temperature 0.5: viajar=0.9998, jugar=0.0002, dormir=0.0000
temperature 1.0: viajar=0.9866, jugar=0.0134, dormir=0.0000
temperature 2.0: viajar=0.8953, jugar=0.1043, dormir=0.0004
top-p 0.9 candidates at t=2: {'viajar': 0.896, 'jugar': 0.104}
1000 samples: {'viajar': 888, 'jugar': 112}At low temperature the model nearly always says viajar. At temperature 2, jugar gets about a 10 percent chance, and top-p still discards dormir, whose probability is 0.04 percent. The final line shows what that looks like in practice: across 1,000 draws, roughly one in nine comes out as the second choice.
How an LLM Is Trained
Training happens in stages, and each shapes a different part of the behavior.
Pretraining
In pretraining, the model learns from billions or trillions of words drawn from books, websites, code, and other sources, after the data has been cleaned. IBM describes this as self-supervised learning: the model finds patterns in unlabeled text, and the text itself supplies the answers. Hide the next token, ask the model to guess it, measure the error, adjust the parameters, and repeat across the entire dataset. Nobody has to label anything by hand.
Pretraining gives the model broad knowledge of grammar, facts, writing styles, and code patterns. A purely pretrained model is good at continuing text, but it is not necessarily good at following instructions or answering helpfully.
Fine-tuning and alignment
IBM lists several adaptation methods:
- Supervised fine-tuning on smaller, labeled datasets for particular tasks or domains.
- Instruction tuning to improve how well the model follows user instructions.
- Reinforcement learning from human feedback (RLHF) to align behavior and style with what people prefer.
- Reasoning-focused training that encourages multi-step problem solving.
The 2022 InstructGPT paper from OpenAI is a well-known demonstration of RLHF. Researchers first fine-tuned GPT-3 on human-written examples of desired behavior, then collected human rankings of model outputs and used them in reinforcement learning. In human evaluations, labelers preferred outputs from the 1.3-billion-parameter InstructGPT over those from the 175-billion-parameter GPT-3, despite the roughly hundredfold size difference. The authors also reported gains in truthfulness and fewer toxic outputs, while noting that the model still made simple mistakes.
Context Windows and Retrieval
An LLM can only consider a limited amount of text at once, called its context window, measured in tokens. IBM notes that newer models have windows of hundreds of thousands of tokens. Anything outside the window is invisible to the model during that response.
A technique called retrieval-augmented generation (RAG) works around this. Relevant documents are fetched from an external source and added to the prompt, so the model can draw on fresh or private information without being retrained.
Why LLMs Hallucinate
Because an LLM generates the statistically plausible next token, not a verified fact, it can produce fluent text that is false. IBM lists hallucinations among the main limitations, describing them as plausible but false or misleading output. NIST’s generative AI risk profile (AI 600-1) also lists hallucination among its 12 risk categories.
Several features of the design make this likely:
- The training objective rewards plausible continuation, not truth.
- The model has no built-in lookup of facts. Whatever it “knows” is encoded in its parameters, and those are lossy.
- Its knowledge stops at the end of its training data unless connected to search or retrieval.
- When asked about something obscure, it may still produce a confident-sounding answer because fluency does not signal certainty.
Other Limits and Risks
- Bias: IBM notes that LLMs can reflect and amplify biases found in their training data.
- Resource demands: training and running large models requires substantial computing power, energy, and cost.
- Inconsistency: because output is sampled, the same prompt can produce different answers.
- Limited transparency: with billions of parameters, explaining why a model produced a specific sentence is hard.
IBM suggests mitigation through AI governance, benchmarks, red-teaming, and fairness evaluations.
Using LLMs Well
- Be specific. State the audience, format, and constraints. The model only sees what you give it.
- Supply source material. Pasting in the relevant text works much like retrieval and reduces guessing.
- Verify important claims. Check names, numbers, quotations, and citations independently, especially in medical, legal, and financial contexts.
- Iterate. Treat the first response as a draft and refine it with follow-up instructions.
- Mind your data. Avoid pasting confidential or sensitive material into tools whose privacy terms you have not reviewed.
Conclusion
A large language model is a transformer-based neural network trained on vast text to predict the next token. Tokenization turns text into numbers, attention weighs which parts of the context matter, the network scores every possible next token, and a sampling step picks one. Repeat that loop and you get a paragraph. Pretraining supplies broad language ability, and fine-tuning with human feedback makes the model better at following instructions.
Understanding that mechanism explains both the power and the weakness. Prediction at massive scale produces remarkably fluent and useful text, but fluency is not accuracy. The best way to use an LLM is as a fast, capable drafting and reasoning assistant whose claims you check when the stakes are real.