← Model Anatomy

Glossary

Every term the guide clicks through to. Definitions are plain-language but precise — the same ones that pop up when you hover a term in a module.

Last reviewed November 2026

Ablation
An experiment that removes or changes one part of a model or method to see how much that part matters.
Abliteration
Removing a model's refusal behavior by finding the direction in its activations that triggers refusal and projecting it out of the weights.
Abstention / escalation
A model's ability to decline a decision and hand it to a person instead of answering. Three numbers make it real: how often it abstains, how accurate it is on the cases it does answer (selective accuracy), and whether the cases it abstained on are the ones it would have gotten wrong. An escalation path that is configured but never fires in evaluation has not been shown to work.
Active parameters
The parameters actually used for a given token: the always-on core (embeddings, attention, norms, any shared expert) + the k routed experts + the router. In a dense model, active = total.
AdamW
The standard optimizer for training transformers: adaptive per-parameter step sizes from running averages of the gradient and its square, with weight decay applied separately. It keeps two values per parameter.
See also: Muon, Optimizer state
Attention
The sub-layer where each token gathers information from earlier tokens: it compares its query with every earlier key, turns the scores into weights with a softmax, and takes the weighted sum of their values.
See also: Query, key and value (Q/K/V), Attention head, KV-cache
Attention head
One of several attention computations run in parallel in a layer, each with its own query projection and its own (or shared) key and value projections, so different heads can track different relationships. Config keys: num_attention_heads, num_key_value_heads.
See also: MHA, GQA, head_dim
Attention sink
A token, usually the first, that receives a large share of attention regardless of content, acting as a place for heads to put weight they don't need. It matters for streaming and compression, which must keep it, and gated attention reduces it.
See also: Gated attention
Attention sparsity
Skipping most token pairs in attention: computing attention over only a selected subset of earlier tokens (DSA, CSA). Distinct from parameter sparsity (mixture of experts). "Sparsity" on its own is ambiguous; this is the attention kind.
See also: Parameter sparsity, DSA, CSA (Compressed Sparse Attention)
Auxiliary-loss-free balancing
The DeepSeek-line way to keep mixture-of-experts experts evenly used: add a per-expert bias to the router's scores, adjusted during training, instead of relying on an auxiliary load-balancing loss. The bias changes only which experts are selected, not the weights applied to their outputs. DeepSeek-V3 still keeps a very small sequence-level balance loss as a backstop, so the quality cost is much smaller, not zero. Config key: topk_method = noaux_tc.
See also: Router, Routing collapse
AWQ / GPTQ
Two common methods for quantizing weights to about 4 bits after training. GPTQ corrects each layer's rounding errors using calibration data; AWQ protects the weights that matter most to activations. The output is a smaller derivative of the same model.
See also: Quantization, Derivative model
Base architecture
The root pre-training architecture a model derives from: the structural fingerprint the detector matches a candidate against (for example deepseek_v3, qwen3, llama3, gemma, glm4). Distinct from a base model (a checkpoint before post-training) and from a model's publisher (whoever uploaded it). ModelDNA's detector recognizes 42 base architectures as of November 2026. Config key: model_type.
See also: Base model, Fingerprinting, Provenance / lineage
Base model
A model straight out of pre-training (or mid-training), before post-training. It continues text rather than following instructions, and is the usual starting point for others to build on.
See also: Base architecture, Pre-training
Benchmark contamination
Test questions appearing in a model's training data, which inflates its scores. Decontamination removes them from training data; a contamination check looks for signs the model memorized them.
BF16
16-bit brain floating point: the same numeric range as 32-bit floats with less precision, at half the memory. It is the standard precision for mixed-precision training, where the master weights and optimizer state are usually kept in FP32.
See also: FP8-native
Calibration
Whether a model's stated confidence means what it says. A forecaster who says "70% chance of rain" is calibrated if it rains on about 70 of every 100 such days. This is not the same as accuracy: a model can be accurate and badly calibrated (always says 95%, right 70% of the time), or calibrated and often wrong. Calibration is the property you need to automate against a threshold.
Catastrophic forgetting / replay
A model losing general ability while training on a narrow mix. Replay mixes some general data back in to prevent it.
Causal mask
The rule in a generative model that each token can attend only to itself and earlier tokens, never later ones, so the model learns to predict what comes next.
See also: Attention
Checkpoint
A saved copy of a model's weights at one point in training. "The released checkpoint" is the set of files you download.
config.json
The file in a Hugging Face model repository that declares the architecture: layer count, widths, head counts, expert counts, position settings and model_type. It is the first thing ModelDNA reads, and its keys are named in many entries here.
See also: Base architecture, Fingerprinting
Context length
The most tokens a model can attend to at once, for example 128K or 1M. Training at long lengths costs more per token than training at short ones.
Context parallelism
Splitting one long sequence across several GPUs, so a context too long for one GPU's memory can still be trained on or served.
Continued pre-training
Resuming next-token training on a released model, often on new languages or domains. It shifts the weights more than fine-tuning does, but the result is still a derivative of the original.
See also: Mid-training, Pre-training, Derivative model
CSA (Compressed Sparse Attention)
DeepSeek-V4's sparse attention: it first compresses the KV-cache so each entry covers a few tokens (4 in DeepSeek-V4-Flash), then a lightning indexer picks the top-k compressed entries (512) for the main attention. DeepSeek-V4 alternates it with HCA, which compresses far more heavily and attends to everything.
See also: HCA (Heavily Compressed Attention), DSA, Attention sparsity
d_model
The width of the residual stream, the single most important size number in a model. Layer count, head count and FFN width are chosen separately, often in conventional ratios to it. Config key: hidden_size.
See also: Residual stream
Decode
The second phase of serving a request: the model generates the reply one token at a time, each step reading the whole KV-cache. Decode speed is usually limited by memory bandwidth rather than arithmetic, which is why speculative decoding and smaller caches help.
See also: Prefill, KV-cache, Speculative decoding
Dense model
A model that uses every parameter for every token; the opposite of a mixture-of-experts model, which activates only a few experts per token.
Derivative model
A license term for a model made from another: fine-tuned, merged, quantized or, under some licenses, trained on its outputs. A derivative may be bound by the original's terms, depending on the license: Llama-style and OpenRAIL licenses carry restrictions forward, while Apache 2.0 and MIT carry little beyond attribution. Read the specific license.
See also: Distillation, Open weights
Diffusion language model
A model that generates by repeatedly refining a whole block of text rather than producing one token at a time.
Distillation
Training one model to imitate another. Classic distillation trains on the teacher's output probabilities (logits), which needs a shared or compatible tokenizer, itself a provenance clue. The looser modern sense trains on the teacher's text outputs or synthetic data. Some licenses forbid it or make the result a derivative.
See also: Derivative model, Tokenizer / vocabulary, Logit
DPO
Direct preference optimization: training directly on pairs of preferred and rejected responses, without a separate reward model or reinforcement-learning loop. A common, cheaper alternative to RLHF.
See also: RLHF
DSA
DeepSeek Sparse Attention (DeepSeek-V3.2): a cheap lightning indexer scores every earlier token, and the main attention runs only over the top-k it selects (2,048 in V3.2). The indexer is still quadratic but light; the expensive part is no longer. DeepSeek-V4 moved on to CSA, which selects among compressed entries instead of single tokens.
See also: CSA (Compressed Sparse Attention), HCA (Heavily Compressed Attention), Attention sparsity
Embedding
The learned vector a model looks up for each token before the first layer, and more generally any vector representation of text. The embedding table has one row per vocabulary entry. Config key: vocab_size.
See also: Token, Tied embeddings / LM head
Evaluation harness
Everything around a model in a benchmark: prompts, few-shot examples, sampling settings and grading. Scores from different harnesses aren't directly comparable.
Expert
One of the many small FFN sub-networks in a MoE layer. A router selects the top-k experts per token; only those compute (plus any always-on shared expert).
FFN
Feed-forward network: the per-token 'thinking' sub-layer of a transformer block, applied to each token independently. In modern models it uses the SwiGLU variant and holds the majority of the weights.
Fingerprinting
Identifying a model's lineage from features it can't easily hide: config shape, tensor names and sizes, tokenizer, and the weights themselves. ModelDNA matches these against known base architectures.
See also: Base architecture, Provenance / lineage, Weight forensics
FLOPs-per-token
The arithmetic done per token. Rule of thumb: about 2 × N operations per generated token for the forward pass and about 6 × N per training token, where N is the active parameter count (all parameters in a dense model). Attention adds a term that grows with context length, which is why long context costs more even at a fixed parameter count.
See also: Active parameters, MFU (model FLOPs utilization)
FP8-native
A model trained in 8-bit floating point and released with FP8 weights, rather than trained at higher precision and quantized afterwards. Master weights, optimizer state and some sensitive operations usually stay at higher precision. Its error profile differs from that of post-hoc quantization.
See also: BF16, Quantization
fused_qkv
Packing the query, key and value projections into one matrix multiply: a kernel efficiency choice. It does not change the KV-cache.
See also: Query, key and value (Q/K/V), KV-cache
Gated attention
A learned gate between 0 and 1 applied to attention's output. In the Qwen3-Next form it depends on the query, is computed per head, and is applied before the output projection; it is credited with reducing attention sinks and improving training stability. Config key: use_gated_attn (ERNIE-5.5).
See also: Attention sink
GGUF
The single-file model format used by llama.cpp and most local-inference tools, holding quantized weights plus metadata such as the tokenizer and architecture. Converting to GGUF changes the files but not the underlying lineage.
See also: safetensors, Quantization
GPU-hour (H100-hour)
One GPU running for one hour, the standard unit of training cost. Stating it in H100-hours makes estimates comparable across clusters.
GQA
Grouped-query attention: several query heads share one key-value head, so the KV-cache shrinks by the group ratio (for example 4:1). The most common choice because the quality cost is small. Config: num_key_value_heads smaller than num_attention_heads.
See also: MHA, MQA, MLA
GRPO
Group relative policy optimization (DeepSeekMath, 2024): a reinforcement-learning method that samples several answers per prompt and scores each against the group's average, so it needs no separate value model. Widely used for reasoning models.
See also: Reinforcement learning (RL), RLVR
HCA (Heavily Compressed Attention)
Heavily Compressed Attention: each layer keeps one key-value entry per block of tokens (128 in DeepSeek-V4 and ERNIE-5.5) and attends densely to all of those compressed entries, with no sparse selection. Paired with CSA or MLA-plus-DSA layers so precise lookups remain possible. Detected from compress_ratios values of 64 or more.
See also: CSA (Compressed Sparse Attention), DSA, KV-cache
head_dim
The width of a single attention head. KV-cache per token, per layer = 2 (key and value) × number of KV heads × head_dim × bytes per element (2 for BF16). The formula does not apply to MLA, which caches a latent instead. Config keys: head_dim, num_key_value_heads.
See also: KV-cache, GQA
Hybrid model
A model that mixes layer types, such as a few full-attention layers among sliding-window, linear-attention or state-space layers, to get long-context efficiency while keeping some exact recall. The layer pattern is itself a fingerprint.
See also: SWA, Linear attention / Gated DeltaNet, SSM
Hyper-connections (HC) / mHC
Hyper-connections replace the single residual stream with several parallel streams (four in DeepSeek-V4 and ERNIE-5.5) mixed by learned weights. The manifold-constrained version, mHC, keeps the mixing matrix doubly stochastic (in the Birkhoff polytope) with a Sinkhorn projection, which keeps the signal stable through depth. Config keys: hc_mult, hc_sinkhorn_iters.
See also: Residual stream
Inference engine
The software that loads a model and serves it efficiently, batching requests, managing the KV-cache and running optimized kernels. vLLM and SGLang are the common open ones.
kv_lora_rank
The dimension of MLA's cached latent: the compressed key-value summary stored per token (for example 512). A small decoupled RoPE key (typically 64) is cached alongside it. Independent of head count.
See also: MLA
KV-cache
The cached Keys and Values of every past token, kept during generation so each new token can attend to the history. It grows with context length and is usually the real wall for long context — not parameter count. Queries are never cached.
See also: MLA, GQA, head_dim, Prefill
Latent MoE
Mixture-of-experts where the experts work in a narrower space than the residual stream, with projections in and out. It cuts the cost of moving tokens to and from experts.
LayerNorm
The original transformer normalization: subtract each vector's mean, divide by its standard deviation, then apply a learned gain and bias. Most current models use the cheaper RMSNorm instead.
See also: RMSNorm
Learning-rate schedule / warm-up
How the training step size changes over a run. Warm-up raises it gradually at the start; resuming training on a released model needs a new warm-up, because too high a rate can undo what the model learned.
See also: WSD schedule
Linear attention / Gated DeltaNet
Attention variants that replace the growing KV-cache with a fixed-size state updated at each token, so cost grows linearly with length. Gated DeltaNet adds a decay gate and a delta-rule update; Qwen3-Next mixes it with standard attention layers, and Kimi Linear uses a refinement of it. Related to, but not quite, a state-space model.
See also: SSM, Hybrid model
Logit
The raw score a language model assigns to every token in its vocabulary at each step, before anything turns those scores into probabilities. Higher means more likely; only the differences between scores matter.
Logprob
The logarithm of a token's probability. Raw logprobs come straight from the model; processed logprobs come after the server applies sampling settings such as temperature and truncation. For any decision your code acts on, read the raw ones, and check your engine's setting (in vLLM, logprobs_mode).
See also: Logit, Temperature (sampling)
LoRA / adapter
Low-rank adaptation: instead of updating a full weight matrix, train two small matrices whose product is the update. The result is an adapter that can be swapped in and out. Adapters are often merged into the base weights, after which they can't be swapped but leave a detectable low-rank difference from the base.
See also: Weight forensics, SFT (supervised fine-tuning)
Memorization
A model reproducing specific training text, such as a URL or an ID, rather than producing something merely plausible in its place.
MFU (model FLOPs utilization)
The share of a GPU's peak arithmetic a training run actually achieves. Well-tuned large dense runs report about 38 to 46% (Llama 3 405B, PaLM); mixture-of-experts runs usually sit lower. The rest is lost to communication, memory traffic and waiting.
See also: GPU-hour (H100-hour), FLOPs-per-token
MHA
Multi-head attention: the original design, where every query head has its own key and value heads. The most expressive of the classic designs and the largest KV-cache; GQA and MQA shrink it, and MLA's authors report matching or beating it. Config: num_key_value_heads equals num_attention_heads.
See also: GQA, MQA, MLA
Mid-training
Continued training of a pre-trained model on a narrower mix, such as code, math or long documents, before any instruction tuning, to strengthen chosen abilities.
See also: Continued pre-training, WSD schedule
MLA
Multi-head latent attention (DeepSeek line): compress each token's keys and values into one small latent vector (kv_lora_rank, for example 512) and cache that, plus a small separate RoPE key (64 values), so 576 values per token per layer. Optimized kernels don't rebuild per-head keys and values; the up-projections are absorbed into the query and output matrices. The saving depends on the baseline: about 28× against full MHA with 64 heads of 128 dimensions; DeepSeek-V2's paper reported a 93.3% cut against its own DeepSeek 67B.
See also: kv_lora_rank, MHA, GQA, KV-cache
Model card
The README a publisher writes for a model: what it is, how it was trained, intended use, license and benchmark claims. It is the publisher's account, which ModelDNA checks against the config and weights.
See also: Provenance / lineage, config.json
Model merging
Combining two or more models' weights into one without further training, by averaging or interpolating them (SLERP), resolving conflicts between their changes (TIES, DARE), or stacking layers from different models (frankenmerges). Merges inherit their parents' structure and leave traceable weight patterns.
See also: Weight forensics, Derivative model
MoE
Mixture-of-Experts: replace the one big FFN with many smaller expert FFNs plus a router that sends each token to only a few. The model holds huge total parameters while using few active ones per token — parameter sparsity.
See also: Expert, Router, Top-k routing, Shared expert, Latent MoE, Upcycling
MQA
Multi-query attention: the extreme of GQA where all query heads share a single Key/Value head. Smallest KV-cache, but a larger quality cost, so it's less common than GQA.
See also: MHA, GQA, MLA
MTP
Multi-token prediction: training a model to predict several future tokens, not just the next one. In DeepSeek-V3 it is first a training objective that densifies the learning signal, built as small sequential transformer modules; the same modules can then serve as a drafter for speculative decoding. Earlier designs (Meta, 2024) used independent parallel output heads.
See also: Speculative decoding
Muon
An optimizer that orthogonalizes each weight matrix's update with a few Newton–Schulz iterations. It applies only to the 2-D hidden weight matrices; embeddings, the output head and norms usually stay on AdamW. Moonshot AI reported about 2× the compute efficiency of AdamW and found that "a model that is both Muon-pretrained and Muon-finetuned outperforms others." Used in production by Kimi K2 (as MuonClip), DeepSeek-V4 and ERNIE-5.5.
See also: AdamW, Optimizer state
NTK-aware scaling
A RoPE context-extension trick that raises the rotation base instead of squeezing positions, so high-frequency dimensions (fine local order) are barely changed while low-frequency ones stretch to cover longer contexts. YaRN builds on it.
See also: YaRN, rope_theta
Open weights
A model whose trained weights can be downloaded. Not the same as open source: the training data and code may still be closed, and the license may restrict use.
Optimizer state
Per-parameter running statistics an optimizer keeps between steps: two values per parameter for AdamW, one (momentum) for Muon on the matrices it covers, with AdamW's two on the rest. Often larger than the weights themselves and rarely released, which makes continuing someone else's training harder.
See also: Muon, AdamW
Out-of-distribution (OOD)
Input unlike anything a model was trained on. Models typically keep their confidence when they leave familiar ground, so accuracy falls while certainty does not — and nothing errors or warns. It matters most for guardrails and routing, whose whole job is to catch inputs nobody anticipated.
Parameter sparsity
Using only a fraction of the weights per token: a router sends each token to a few expert FFNs (mixture of experts). Distinct from attention sparsity (DSA). "Sparsity" on its own is ambiguous; this is the parameter kind.
See also: Attention sparsity, MoE, Sparsity ratio (SR)
Parity test
Checking that a reimplementation of a model produces the same outputs as the original on the same inputs, before trusting it for training or serving.
Partial rotary
Applying RoPE to only part of each head's dimensions. The unrotated dimensions carry no position signal at all: they match on content alone, which is the usual reason for leaving them out, and they are sometimes credited with better behavior at long lengths. Config key: partial_rotary_factor.
See also: RoPE
Post-training
Everything after pre- and mid-training that turns a base model into a usable assistant: supervised fine-tuning, reinforcement learning, and preference and safety tuning.
Pre-norm
Placing the normalization before each sub-layer rather than after it, which is what made very deep transformer stacks trainable. It is the dominant placement, not a universal one: Gemma 2 and 3 normalize both before and after each sub-layer, and OLMo 2 normalizes the sub-layer's output inside the residual branch. Those differences are structural fingerprints.
See also: RMSNorm, LayerNorm
Pre-registration
Writing down predictions, and what result would refute them, before running a test, so the outcome can't be reinterpreted afterwards.
Pre-training
The first and usually most expensive training stage: next-token prediction on trillions of tokens of general text. It builds the model's knowledge and general ability. Historically everything after it cost one or two orders of magnitude less, though reinforcement-learning budgets for reasoning models have grown substantially.
See also: Mid-training, Post-training, Continued pre-training
Prefill
Two senses. In serving, prefill is the first phase of a request: the model processes the whole prompt in parallel and fills the KV-cache, before decode produces the reply token by token. In testing, prefilling a response means starting the model's reply with given text and letting it continue, to see what it does from a chosen point.
See also: Decode, KV-cache
Provenance / lineage
Where a model came from: which base it was built on, by whom, and through what steps (fine-tuning, merging, quantization, re-upload). Lineage is the family tree; provenance is the documented history behind it.
See also: Fingerprinting, Base architecture, Weight forensics
Pruning / layer removal
Making a model smaller by removing weights, attention heads, experts or whole layers, usually followed by some retraining. A pruned model keeps its parent's remaining weights, so the lineage stays detectable.
See also: Weight forensics
Quantization
Storing weights in fewer bits, such as 8 or 4, to cut memory and speed up serving at some cost in fidelity. NVFP4 and MXFP4 are 4-bit formats with a shared scale per small block. Two quantized builds of one model can behave measurably differently.
Query, key and value (Q/K/V)
The three vectors attention computes for each token. The query asks what this token is looking for, the key advertises what a token offers, and the value is what gets passed along when a key matches. Keys and values of past tokens are what the KV-cache stores.
See also: Attention, KV-cache, fused_qkv
Re-upload / renaming
Publishing an existing model's weights under a new name or organization, sometimes with a new card. Identical tensors give it away, whatever the name says.
See also: Weight forensics, Provenance / lineage
Reasoning model / thinking tokens
A model trained to write out intermediate reasoning (thinking tokens) before its answer, usually with reinforcement learning. Many can switch it on or off, and each lab names the control differently, so the mode must be recorded with every score.
See also: RLVR, System 1 / System 2
Reinforcement learning (RL)
Training on the model's own attempts (rollouts), each scored by a reward. With verifiable tasks, such as math answers or code that must pass tests, the reward can be checked automatically.
Residual stream
The running sum every sub-layer reads from and writes to — the 'highway' that lets information and gradients skip the full depth of the network. Its width is d_model.
Reward model
A model trained to score responses, standing in for human judgment during reinforcement learning. Its blind spots become the policy's loopholes (reward hacking).
See also: RLHF, RLVR
RLHF
Reinforcement learning from human feedback: people compare model responses, a reward model learns their preferences, and the model is trained with reinforcement learning to score well on it.
See also: Reward model, DPO, Reinforcement learning (RL)
RLVR
Reinforcement learning with verifiable rewards: the reward comes from an automatic check, such as a math answer or code that must pass tests, instead of a learned reward model.
See also: Reinforcement learning (RL), GRPO
RMSNorm
Root-mean-square normalization: a cheaper normalization than LayerNorm that rescales by the RMS with no mean subtraction and one learned gain vector (one value per dimension). It is the dominant choice in current models. Config key: rms_norm_eps.
See also: LayerNorm, Pre-norm
RoPE
Rotary position embedding: encode token order by rotating each query and key (in 2-D pairs of dimensions) by an angle proportional to position. Because attention is a dot product, scores then depend on relative distance, for fully rotated heads. Config key: rope_theta.
See also: rope_theta, Partial rotary, YaRN
rope_theta
The base hyper-parameter that sets the spread of RoPE's rotation frequencies. Increasing it is one lever for extending context length.
Router
The small linear layer in a MoE that scores the experts for each token and selects the top-k. Poor routing that favors a few experts is 'routing collapse'.
Routing collapse
A mixture-of-experts failure where the router sends most tokens to a few experts, leaving the rest undertrained. Load-balancing losses and auxiliary-loss-free biases exist to prevent it.
See also: Router, Auxiliary-loss-free balancing
safetensors
The standard weight format on Hugging Face. It holds only tensors, so loading it can't run code, and its header lists every tensor's name, shape and type. Older .bin and .pt files are Python pickles, which can execute code on load. GGUF dominates local inference.
See also: GGUF
SFT (supervised fine-tuning)
Training on example prompts paired with good responses, so the model learns a format and a behavior. Usually the first post-training step.
Shared expert
An expert that is active for every token, carrying common computation so the routed experts can specialize. Many mixture-of-experts models have none (Mixtral, Qwen3-MoE), so its presence or absence is a structural tell. Config key: n_shared_experts.
See also: Expert, MoE
Softmax
The function that turns a list of scores into probabilities that are positive and sum to 1, exaggerating the gaps between them. Attention uses it to turn match scores into weights; the output layer uses it to turn logits into next-token probabilities.
See also: Logit, Attention
Sparsity ratio (SR)
Total parameters ÷ active parameters: how sparse a model's parameter use is (for example 60×). A dense model is exactly 1×, the baseline. It is a parameter-usage headline, not a speed or cost ratio, because it ignores memory bandwidth.
See also: Active parameters, Top-k routing
Speculative decoding
Speeding up generation by drafting several tokens cheaply, then having the full model verify them in one pass. MTP modules often serve as the drafter.
See also: MTP, Decode
SSM
State-space model (e.g. Mamba): an O(n) sequence-mixing alternative to softmax attention, often interleaved with attention layers in a hybrid stack.
See also: Linear attention / Gated DeltaNet, Hybrid model
SWA
Sliding-window attention: a layer attends only to the last W tokens. It reliably cuts attention compute to O(W); it cuts KV-cache memory only if the serving stack evicts tokens outside the window. Usually deployed as a hybrid with a few full-attention 'global' layers.
See also: Hybrid model, KV-cache
SwiGLU
The modern feed-forward (FFN) variant: three matrices (gate, up, down) with a SiLU-gated product — down(SiLU(gate·x) ⊙ (up·x)). It's where most of a dense model's parameters live (~70%), which is why MoE targets the FFN.
System 1 / System 2
Kahneman's two modes of thought: System 2 is slow and deliberate and shows its working; System 1 is fast and automatic, and the judgment arrives whole. Among language models, reasoning models with long thinking traces are the System 2 analogy, and a chat model answering without reasoning is closer to System 1. ModelDNA uses "System 1 model" for a narrower category: a model that reads the input in one pass and returns a typed answer plus a confidence, without writing any text.
See also: Reasoning model / thinking tokens, Calibration
Temperature (sampling)
A setting that divides a model's scores before they become probabilities. Below 1 makes the top choice look more certain; above 1 flattens the choices. It shapes how generated text reads and says nothing about how sure the model really is.
See also: Truncation (top_k / top_p), Logprob
Tied embeddings / LM head
The LM head is the final layer that turns the residual stream into a score for every token. Some models reuse the input embedding table for it (tied embeddings), which saves parameters and is a structural tell. Config key: tie_word_embeddings.
See also: Embedding, Logit
Tier-1 / Tier-2
ModelDNA's evidence tiers, numbered by strength. Tier-1 is measured behavior: what a model does, from actually running it (agentic, worldview and safety tests), always dated. Tier-2 is facts derived from config and structure: what a model is, read from its config and weights. Most of this guide is Tier-2 because structure can be read without running the model. A Tier-2 structural fact (such as a sparsity ratio) is never presented as a Tier-1 performance claim (such as speed).
Token
The unit a model reads and writes: a word, part of a word, a punctuation mark or a byte sequence, as set by its tokenizer. Lengths, prices and training budgets are counted in tokens.
See also: Tokenizer / vocabulary
Tokenizer / vocabulary
The fixed set of text pieces (tokens) a model reads and writes, and the rules for splitting text into them. Vocabulary size is the number of distinct tokens.
See also: Token, Tokenizer extension / transplant
Tokenizer extension / transplant
Adding tokens to a model's vocabulary, or swapping in another model's tokenizer and retraining the embeddings. The untouched rows of the embedding table still match the parent.
See also: Tokenizer / vocabulary, Embedding
Top-k routing
How many experts (k) a mixture-of-experts router activates per token, out of N in the layer. Sparsity depends on k relative to N: 2 of 8 (Mixtral) is far denser than 8 of 256 (DeepSeek-V3) or 10 of 512 (ERNIE-5.5). The sparsity ratio gives the headline number. Not the same thing as top_k in sampling. Config keys: num_experts_per_tok, n_routed_experts.
See also: Sparsity ratio (SR), Router, Truncation (top_k / top_p)
Transformer block
One repeated unit of a transformer: a normalization, an attention sub-layer, another normalization and an FFN (or mixture-of-experts) sub-layer, each adding its output back into the residual stream. A model is a stack of these. Config key: num_hidden_layers.
See also: Attention, FFN, Residual stream, Pre-norm
Truncation (top_k / top_p)
Sampling settings that drop all but the most likely tokens before one is picked: top_k keeps a fixed number, top_p keeps the smallest set covering a share of the probability. Dropped tokens become impossible, which can silently remove a valid answer from a decision.
See also: Top-k routing, Temperature (sampling)
Upcycling
Turning a trained dense model into a mixture-of-experts model by copying its FFN into each expert and continuing training. A major lineage path: the experts start out nearly identical, which can show in the weights.
See also: MoE, Weight forensics
Weight forensics
Reading the weight tensors directly (not just the config) to answer what the config can't: a per-tensor diff against a known base (is this really its own model, or a renamed copy?), or detecting modifications like abliteration. The deepest structural read — still 'structure,' just at the weight layer rather than the config.
See also: Fingerprinting, LoRA / adapter, Re-upload / renaming
WSD schedule
Warmup-stable-decay: a learning-rate schedule that warms up, holds a constant rate for most of training, then decays quickly at the end. Checkpoints from the stable phase can be resumed easily, and mid-training often runs during the decay.
See also: Learning-rate schedule / warm-up, Mid-training
YaRN
A context-extension scheme: NTK-aware interpolation that stretches low RoPE frequencies while preserving high ones, blended with a smooth ramp and an attention-temperature correction. It usually comes with a short fine-tune at the extended length; without one it still degrades more gracefully than naive interpolation. Config: rope_scaling type yarn.
See also: RoPE, NTK-aware scaling, rope_theta