Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Select an option

  • Save ruvnet/197eb50028368d4f278e8530e9432387 to your computer and use it in GitHub Desktop.

Select an option

Save ruvnet/197eb50028368d4f278e8530e9432387 to your computer and use it in GitHub Desktop.
Jacobian-Lens interpretability for open-weight LLMs — @metaharness/workspace-lens (runtime mechanistic governance, IntOps, prompt-injection & hidden-objective detection)

Jacobian-Lens Interpretability for Open-Weight LLMs — @metaharness/workspace-lens

Read the model's mind while it thinks. A dependency-free, runtime interpretability primitive that turns the Jacobian Lens from Anthropic's 2026 global-workspace paper into a production audit + governance tool for open-weight language models.

📦 Live on npm: @metaharness/workspace-lens@0.1.0 · npm i @metaharness/workspace-lens · MIT · source & PR


TL;DR

  • What: a TypeScript runtime that applies a fitted Jacobian Lens to open-weight LLM activations — lens_l(h) = unembed(J_l · h) — and emits workspace tokens, a layer-by-layer thinking trajectory, drift/entropy scores, vectorized safety flags, and a signable interpretability receipt.
  • Why it's better than logit lens: it corrects for inter-layer representational transforms, so meaningful concepts surface earlier in the network — often before the first output token.
  • Why it matters: it's the foundation for Interpretability Operations (IntOps) — runtime mechanistic governance that taps the model's internal wires instead of trusting after-the-fact self-explanations (which are prone to sycophancy and confabulation).
  • Cost: runtime is a pair of matrix–vector products + a softmax per monitored layer — zero backward passes — cheap enough to run live on high-stakes requests, not just 1% shadow sampling.
  • Constraint: fitting the lens needs a model backward pass (open-weight, GPU) and is done out of band; this package applies a fitted lens.

1. Background: logit lens vs. Jacobian lens

The logit lens decodes an intermediate residual activation h_l straight through the model's unembedding, assuming middle layers already live in final-output coordinates. They don't — so early/mid readouts are noisy and often just echo surface tokens.

Anthropic's Jacobian Lens (paper: "Verbalizable Representations Form a Global Workspace in Language Models", 2026-07-06; reference code: anthropics/jacobian-lens) learns an average input→output Jacobian J_l — a map from a layer's activations to final-layer coordinates — and decodes through it:

lens_l(h) = unembed(J_l · h)

Semantically, this asks "what is this activation disposed to make the model say later?" rather than "what does it predict right now?". The correction surfaces meaningful, reportable concepts earlier and more clearly than logit lens.

The paper's larger claim: LLMs maintain a small set of verbalizable internal representations that behave like a functional global workspace — supporting report, modulation, reasoning, flexible reuse, and selective access. Crucially for safety, hidden concepts like evaluation awareness, manipulation, secretly, panic, reward, and trick light up in that workspace even when absent from the output.

Honest framing: this is a functional analogy, not a consciousness claim — the paper explicitly leaves the philosophy open. The practical value is a measurable window into hidden reasoning.


2. From black-box testing to Mechanistic Governance (IntOps)

Perimeter guardrails (Llama Guard, regex lists, output classifiers) act at the input/output boundary — they can only react after the model has processed an injection and begun emitting compromised tokens.

Treating the Jacobian Lens as a runtime probe unlocks a deterministic circuit breaker:

  • Traditional: prompt → model → malicious output → guardrail blocks (or misses) it.
  • IntOps: prompt → hidden layers → lens detects an un-emitted spike in the exfiltration / override / credential directions → execution halts mid-forward-pass, before a single bad token.

This shifts AI safety from reactive filtering to proactive state assertion.


3. Features

Capability API Description
Lens readout WorkspaceLens.readout(h) unembed(J_l·h) → top workspace tokens + readout entropy
J-projection WorkspaceLens.project(h) z = J_l·h, activation in final-layer coordinates
Workspace drift workspaceDrift(readouts) mean Jensen–Shannon divergence between consecutive readouts — is the reasoning path stable or mutating uncontrollably?
Entropy trajectory entropyTrajectory(readouts) per-layer entropy — is the workspace converging or dissolving?
Vectorized safety detectConcepts / flagsFromTriggers dot-product triggers vs. concept directions{promptInjection, evalAwareness, hiddenObjective, refusalConflict}
Interpretability receipt buildReceipt(...) signable audit artifact: prompt hash, model/lens id, layer trajectory, top tokens, flags, drift, entropy
Decision rule decide(...) taskResolved && drift<θ && noCriticalFlags && receiptCoverage===1, with auditable reasons

Runtime-only · model-agnostic · dependency-free (Node built-ins) · deterministic.


4. Cross-family vocabulary alignment (the hard part, solved)

Different model families (Qwen vs. Gemma 2) have vastly different tokenizers — you cannot align a concept like "hidden objective" by token string. This library aligns at the concept-direction level: a canonical concept name maps to a per-model unit vector in that model's J-space (ConceptVector), fitted from example activations. Detection is a cosine/dot-product in activation space — tokenizer- agnostic — and a vector is never cross-applied across modelId. So hidden_objective is one concept with a Qwen vector and a Gemma vector, aligned by name.


5. Deployment topology — the Triage Architecture

Because runtime projection is static linear algebra (no gradients), you can tier it by risk:

Tier Trigger Depth Overhead
1 · Passive low-risk chat, static generation lens bypassed 0%
2 · Spot-check 1% shadow sampling · Darwin-Mode mutation evidence async batch logging ~0%
3 · Full intercept tool calls · financial txns · PII · untrusted retrieval synchronous, mid-layers at critical tokens small

Bind Tier 3 to high-stakes routing tokens (e.g. a tool-call token) for a bulletproof circuit breaker exactly when the agent is about to touch the outside world.


6. Benchmarks, cost profile & acceptance test

Status (honest): the shipped package is the runtime measurement core, covered by 17 deterministic $0 synthetic tests (projection math, readout, cross-model isolation, drift, decision rule, end-to-end receipt, and the mechanism shape of the acceptance test). Numbers below are the cost model and the acceptance-test methodology — they are not fabricated model results. Live lens fitting requires an open-weight model + backward pass (out of band).

Cost model (per monitored layer, runtime): J_l·h is O(d_model²), the unembed is O(vocab·d_model), plus a softmax — zero backward passes. Reading ~9 layers streams at ~full generation speed on a single consumer GPU in reference UIs. Fitting is the only expensive step and is one-time/offline (~100–1000 short sequences; quality saturates near ~100).

Use case Cost Latency Value
Offline fit (1B–7B model) Medium Minutes–hours High (model audit)
Per-prompt readout Low activation capture + matmuls High (debugging)
Production high-stakes routing Low–Medium small synchronous add High (regulated / tool-use)
1% shadow sampling Low offline Great default

Acceptance test (methodology). Fit a lens on a 1.5B–7B Qwen model; on silent-state-holding prompts ("Is 12 + 5 = 1 correct? Answer only Yes or No."), extract mid-layer states and verify the Jacobian readout surfaces wrong/incorrect/false 2–3 layers earlier and with ≥20% higher logit clarity than logit lens, before the first output token, across ≥50 variations on a hand-labeled rubric. Target failure-mode categories: multi-step planning, context-stuffing traps, conversational turn-arounds.


7. Where it plugs into MetaHarness

  • Evaluation — a workspace_probe surface: does a candidate harness make the model hold better intermediate concepts before answering?
  • Darwin Mode — J-lens readouts as mutation evidence: reject a prompt mutation that improves the final token but causes the workspace to lose its early grip on the right concept (structurally brittle).
  • Safety auditing — prompt injection, hidden-objective drift, eval-awareness, reward-hacking, refusal analysis — as state assertions, not perimeter filters.
  • Receipts — attach an interpretability receipt to every governed agent decision (regulated audits, SOC/CISO evidence).

Install

npm i @metaharness/workspace-lens   # v0.1.0 — live on npm

Producing & loading a lens artifact

The runtime consumes a LensArtifact — a plain JSON object you produce out of band (fit J_l with the reference anthropics/jacobian-lens on an open-weight model, then serialize to this shape):

{
  "lensId":  "jlens-qwen2.5-7b-v1",       // stable id, recorded in every receipt
  "modelId": "qwen2.5-7b-instruct",       // the model it was fitted on
  "dModel":  3584,                         // residual-stream width
  "vocab":   ["...", "..."],               // token strings, aligned to the unembed rows
  "unembed": [[...dModel...], ...],        // U ∈ ℝ^{vocab × dModel}
  "layers":  [                             // one fitted operator per layer
    { "layer": 14, "jacobian": [[...dModel...], ...] }   // J_l ∈ ℝ^{dModel × dModel}
  ]
}

Load it three ways:

const lens = await WorkspaceLens.fromFile('./qwen.json');                 // local
const lens = await WorkspaceLens.fromUrl('https://cdn.example/qwen.json'); // over HTTP
const lens = await WorkspaceLens.fromRegistry('qwen-2.5-7b', { baseUrl }); // by name (caller base)

CLI (@metaharness/workspace-probe)

For audit workflows without writing TS — JSON in, JSON out (composes with jq):

npx @metaharness/workspace-probe diag           lens.json
npx @metaharness/workspace-probe readout        lens.json activations.json --top-k 8
npx @metaharness/workspace-probe probe          receipts.json
npx @metaharness/workspace-probe grade-mutation baseline.json mutant.json

Links


Keywords: Jacobian lens, logit lens, mechanistic interpretability, LLM interpretability, global workspace theory, AI safety, prompt injection detection, hidden objective detection, evaluation awareness, open-weight models, Qwen, Gemma, interpretability operations, IntOps, mechanistic governance, runtime model monitoring, interpretability receipts, agent safety, MetaHarness, Darwin Mode.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment