CVG-010 · Model internals · April 2026

PLV at Full-Attention Boundaries

Hive architecture / probe notes

Early-exit agreement should peak where Qwen consolidates global context

Abstract. Qwen 3.5-27B is hybrid: every fourth layer is full attention, the other three are DeltaNet linear attention with rolling recurrent state. DeltaNet layers do not produce useful intermediate logits; they keep compressed state. Prediction: layerwise next-token agreement (p_agree) should be better when exiting at full-attention boundaries (layers 4, 8, …, 64) than at arbitrary DeltaNet layers, because those are the checkpoints where the model consolidates global context.

Hypothesis

Exiting a hybrid transformer at a linear-attention layer asks a language-model head to read a recurrent compression it was never trained to decode. Full-attention layers re-materialize global context. If partial-layer viewing (PLV) is going to work as a speculative or early-exit signal, the exit set should be those boundaries, not an even layer grid.

Setup

Sweep A: 30 short prompts, last-token next-token p_agree, full-attn layers versus a DeltaNet control set, two head strategies (final_norm+lm_head, and next-layer input layernorm then final_norm+lm_head). Sweep B: 12 long prompts, 1984 tokens, plus a 52M-param regression adapter per probed layer.

Why it matters to the hive

If the prediction holds, early-exit and speculative decoding for this hybrid should skip DeltaNet exits entirely. That is a structural constraint, not a hyperparameter. It also warns against treating “layer N of 64” as a uniform depth: three-quarters of the depth is a different algorithm. Probe JSON in the hive tree is the evidence log; this paper is the claim the evidence was gathered to test.

Source report. Summarized from docs/architecture/PLV-FULL-ATTN-BOUNDARIES.md, probe JSON in the hive tree. This page is the paper. The report remains the primary.