I wanted to understand reasoning models below the API level. Instead of only calling a hosted model and observing that it “thinks,” I worked through the mechanics that make the behavior possible. The goal was not to train a frontier model. It was to build a small, inspectable system where I could trace every transition from prompt to tokens, rewards, gradients, and updated behavior.
Why I Studied Reasoning Models From First Principles
A reasoning model is still an autoregressive language model. It predicts one token at a time. The interesting part is the machinery around that model: prompting it to spend tokens on intermediate work, sampling multiple candidate solutions, checking answers with verifiers, scoring trajectories, and updating the policy toward better behavior.
This perspective helped me separate three ways to improve a reasoning system:
- Inference-time scaling: spend more compute while answering, without changing model weights.
- Reinforcement learning: use rewards to update the model toward more successful reasoning trajectories.
- Distillation: train a smaller model on high-quality reasoning generated by a stronger teacher.
The Atomic Unit: Next-Token Generation
The first part of my learning was rebuilding the generation path around a pretrained Qwen3 model. A prompt is tokenized into IDs, mapped to embedding vectors, passed through the transformer, and projected into one logit for every token in the vocabulary. Only the logits from the final sequence position are needed to choose the next token.
prompt → token IDs: [B, T] → embeddings: [B, T, C] → transformer blocks → logits: [B, T, vocab_size] → take logits[:, -1, :] → sample next token → append token and repeat
The shape transitions gave me a concrete understanding of what the model is doing. With a hidden dimension of 1,024 and a vocabulary of 151,936 tokens, the output head converts the final 1,024-dimensional hidden state into 151,936 candidate scores. Those scores become probabilities after normalization.
Why the KV cache matters
Without caching, generating every new token recomputes attention over all earlier tokens. The key-value cache stores the attention keys and values from previous positions so each decoding step only computes the new position. This changes generation from repeatedly rebuilding the full prefix to incrementally extending it.
Without KV cache
For token t, the model reprocesses tokens 1 through t. Simple, but increasingly expensive as the sequence grows.
With KV cache
Past attention states are reused. The model processes only the new token while attending to cached history.
with torch.inference_mode():
for token_id in generate_with_cache(
model=model,
token_ids=input_ids,
max_new_tokens=512,
temperature=0.7,
top_p=0.9,
):
generated.append(token_id.item())
I also learned why generation wrappers are useful. The low-level generator should focus on model execution and token selection. A wrapper can handle prompt rendering, streaming decoded text, end-of-sequence behavior, and user-facing return values. That separation made the code easier to test and extend.
Evaluation and Verifiers
A model producing a long explanation is not evidence that it solved the problem. To improve reasoning, I first needed a reliable way to measure correctness. For math tasks, this meant extracting the final answer, normalizing formatting, and checking mathematical equivalence against a reference solution.
| Stage | Purpose | Failure it prevents |
|---|---|---|
| Answer extraction | Locate the model's final answer inside a longer response. | Grading the entire chain of thought as if it were the answer. |
| Normalization | Remove superficial formatting differences. | Treating 1/2 and 0.5 as unrelated strings. |
| Equivalence check | Determine whether two mathematical expressions mean the same thing. | Rewarding only exact text matches. |
| Dataset evaluation | Aggregate accuracy across many examples. | Drawing conclusions from a few hand-picked prompts. |
Inference-Time Scaling
Inference-time scaling improves results without updating model weights. Instead of accepting the first greedy answer, the system spends more computation exploring candidate solutions and selecting a better one.
Temperature and top-p sampling
Temperature rescales logits before sampling. Lower values concentrate probability on high-scoring tokens; higher values flatten the distribution and increase diversity. Top-p sampling limits selection to the smallest set of tokens whose cumulative probability reaches a threshold.
scaled_logits = next_token_logits / temperature probs = torch.softmax(scaled_logits, dim=-1) sorted_probs, sorted_idx = torch.sort(probs, descending=True) cumulative = torch.cumsum(sorted_probs, dim=-1) keep = cumulative <= top_p keep[..., 0] = True
The important insight was that diversity is useful when the system can evaluate candidates. Randomness by itself does not improve reasoning; it creates a search space. A verifier or voting mechanism must then identify which samples are promising.
Self-consistency
Self-consistency runs several independent reasoning trajectories, extracts their final answers, and selects the answer that appears most often. Different samples may take different paths, but correct reasoning often converges on the same result.
Single-path decoding
- Low cost
- Low latency
- One early mistake can determine the final answer
Self-consistency
- Higher compute cost
- Explores multiple reasoning paths
- Voting can reduce variance
Self-Refinement and Confidence
Self-refinement adds an iterative loop: generate a draft, critique it, create a repair plan, and produce a revised answer. This gave me a practical model of how an agent can use the same language model in different roles while preserving explicit state between steps.
draft = generate(problem) critique = generate(make_critique_prompt(problem, draft)) revision = generate(make_revision_prompt(problem, draft, critique)) score = verifier(problem, revision)
I also worked through token probabilities and log probabilities. Sequence probability becomes extremely small because it multiplies many token probabilities together. Log probabilities turn multiplication into addition and are therefore more numerically stable and easier to compare.
log p(sequence) = Σ log p(token_t | token_<t)
One caution is that confidence is not correctness. A model may assign high probability to a familiar but wrong pattern. Confidence signals are useful features, but they should be combined with verifiers, consistency checks, or external tools.
Training With RLVR and GRPO
The reinforcement-learning section connected generation to actual parameter updates. In reinforcement learning with verifiable rewards, the model samples several responses to the same prompt, receives an automatically computed reward, and learns to increase the likelihood of stronger responses.
GRPO uses the relative performance of responses sampled for the same prompt. Instead of requiring a separately trained value model, it normalizes rewards within the response group to produce advantages.
advantage_i = (reward_i - mean(group_rewards))
/ (std(group_rewards) + epsilon)
A response that scores above the group average receives a positive advantage; a below-average response receives a negative one. The policy update then increases the probability of useful trajectories and decreases the probability of weaker ones.
Stability mechanisms
I learned that a working RL objective needs guardrails. Clipped policy ratios prevent a single update from changing the policy too aggressively. A KL term discourages the trained policy from drifting too far from a reference model. Format rewards can teach the model to emit structured reasoning tags, but they must not overpower correctness rewards.
- Policy ratio: measures how much the probability of a sampled token changed under the new policy.
- Clipping: limits the contribution of excessively large probability changes.
- KL regularization: penalizes large divergence from a stable reference policy.
- Entropy tracking: helps detect whether the model is becoming too deterministic or unstable.
Distilling Reasoning Behavior
Distillation shifts the problem from online reinforcement learning to supervised learning. A stronger teacher generates high-quality reasoning traces, those traces become a training dataset, and a smaller student model learns to imitate the teacher's behavior.
The workflow helped me understand that dataset construction is the central engineering problem:
- Generate several candidate solutions from a teacher.
- Verify correctness and reject weak or malformed outputs.
- Format the prompt and response using the student's tokenizer and chat template.
- Filter examples by sequence length and quality.
- Train with next-token prediction and monitor validation loss.
- Evaluate the student using the same verifier-based pipeline.
Distillation can make reasoning cheaper at inference time because the student internalizes patterns that previously required a larger model or more expensive search. The tradeoff is that the student may inherit the teacher's mistakes and become specialized to the distribution of generated examples.
What I Can Now Build and Explain
Working through the system end to end changed how I think about reasoning models. I no longer see them as a single architecture feature. I see them as a pipeline that combines model execution, search, verification, optimization, and data engineering.
| Area | What I learned to implement or explain |
|---|---|
| Inference | Autoregressive generation, streaming, temperature, top-p sampling, KV caching, and the relationship between hidden states, logits, and tokens. |
| Evaluation | Answer extraction, normalization, mathematical equivalence, verifier design, and regression-style benchmark evaluation. |
| Test-time compute | Chain-of-thought prompting, multiple rollouts, self-consistency, confidence scoring, critique, and revision loops. |
| Post-training | RLVR, group-relative advantages, GRPO policy loss, clipping, KL control, format rewards, and training metrics. |
| Model compression | Teacher-generated reasoning datasets, supervised distillation, filtering, loss computation, and student evaluation. |
The most valuable result was not a single accuracy number. It was developing a first-principles mental model that lets me debug a reasoning system layer by layer: Is generation too deterministic? Is the verifier wrong? Are rewards sparse? Is the policy drifting? Is the dataset teaching the intended behavior? Those questions turn a vague “reasoning model” into an engineering system I can inspect and improve.