Mechanistic interpretability dashboard

Towards Monosemanticity Reproduction

An evidence-backed catalogue of sparse autoencoder features learned from gelu-1l activations at blocks.0.mlp.hook_post.

4
Validated features
14/ 14
Core contextual tests correct
90.9%
Expanded contextual accuracy
32.08×
Feature 3513 control ratio
25/ 25
Registry checks passed

Interpretation warning

Feature labels are empirical summaries of observed behaviour, not complete definitions of everything represented by each feature.

FEATURE 3513

according-to continuation

Local lexical and continuation feature associated with `according → to`.

high confidence
lexicallocal continuationcausally validated

Natural activation

4.264

Mean activation on controlled according prompts

Causal specificity

32.08×

Effect relative to the mean of the unrelated control directions

Closest raw neuron

1643

SAE/control selectivity ratio: 1.29×

Probability interventions

According prompts at strength -4 18.05 percentage points lower
According prompts at strength +4 1.51 percentage points higher
Control prompts at strength +4 12.84 percentage points higher

Steering selectivity

Measurement SAE Raw neuron
Control-prompt selectivity 4.72 3.66
Control target-change rank 3.50 106.25
According-prompt selectivity 3.34 3.04

Evidence types

  • cross-document examples
  • controlled prompts
  • natural-contribution ablation
  • positive steering
  • dose-response test
  • unrelated-direction controls
  • raw-neuron steering comparison
  • vocabulary selectivity analysis

Known limitations

  • The decoder direction also changes unrelated vocabulary fragments.
  • The causal experiment tested one target token on a small prompt set.
  • Decoder-direction intervention is approximate rather than perfectly isolated.

Source result files

results/step_22b_feature_3513_steering_validation.jsonresults/step_23b_feature_neuron_selectivity.json
FEATURE 1997

raising-like to

`to` following seem, appear, and several related verb constructions.

high confidence
lexicalgrammatical context

Controlled accuracy

100.0%

8/ 8 expected examples

Exclusive activations

84.9%

Activations not shared with the other tested contextual features

Closest raw neuron

548

Pearson correlation: 0.403

Activation and separation

Active token percentage 2.18%
Exclusive activation percentage 84.90%
Mean top-100 overlap 7.50%

Separation metrics

Measurement Mean result
Active-set Jaccard 0.0598
Activation correlation 0.0500
Encoder cosine similarity 0.0349
Decoder cosine similarity 0.0391

Strong cross-document examples

  1. 4.959 — strike amid rising tensions with North Korea, seems [[ to]] have sparked an unusually rapid response on Capitol Hill
  2. 4.928 — /LEGO, the source of which seems [[ to]] be early listings on the Mexican version of Amazon
  3. 4.283 — refraction of light water in some places seems [[ to]] be dark
  4. 4.143 — A record-setting reality show appears [[ to]] be paddling in the wrong direction as its
  5. 3.741 — box at the bottom of my page which seems [[ to]] be inheriting CSS style from bootstrap.min

Evidence types

  • cross-document examples
  • controlled prompts
  • raw-neuron comparison
  • pairwise feature separation
  • controlled classification

Known limitations

  • It sometimes responds to broader infinitive constructions.
  • Its boundary is not restricted to one formal grammatical category.

Source result files

results/step_21b_contextual_feature_neuron_examples.jsonresults/step_24a_contextual_feature_separation.jsonresults/step_24b_contextual_feature_classification.json
FEATURE 1704

necessity-related to

`to` following have, had, need, and related necessity-oriented constructions.

high confidence
lexicalgrammatical context

Controlled accuracy

75.0%

6/ 8 expected examples

Exclusive activations

78.3%

Activations not shared with the other tested contextual features

Closest raw neuron

882

Pearson correlation: 0.331

Activation and separation

Active token percentage 1.33%
Exclusive activation percentage 78.26%
Mean top-100 overlap 5.00%

Separation metrics

Measurement Mean result
Active-set Jaccard 0.0579
Activation correlation 0.0418
Encoder cosine similarity 0.0297
Decoder cosine similarity 0.0311

Boundary examples

  • We decided to travel tomorrow. — Feature 1997 won by 0.273
  • They began to laugh. — Feature 1997 won by 0.100

Strong cross-document examples

  1. 4.956 — model of, but was soon overwhelmed by having [[ to]] guess so much of the flight dynamics. Is
  2. 4.914 — but uncomfortable. I felt as if I had [[ to]] go diarrhea every five minutes. If this is
  3. 4.849 — goes where, you’re going to have [[ to]] tell her you’ll help a little.
  4. 4.773 — handwritten characters. Why did they have [[ to]] use a quantum computer to do that?
  5. 4.375 — ? A: You have [[ to]] return the return value of the recursive call.

Evidence types

  • cross-document examples
  • controlled prompts
  • raw-neuron comparison
  • pairwise feature separation
  • controlled classification

Known limitations

  • It is not a general detector for every ordinary infinitive.
  • Some broader infinitives are shared with Feature 1997.

Source result files

results/step_21b_contextual_feature_neuron_examples.jsonresults/step_24a_contextual_feature_separation.jsonresults/step_24b_contextual_feature_classification.json
FEATURE 2971

capitalized To

Primarily the capitalized token `To`, with weaker sensitivity to some lowercase purpose constructions.

high confidence
lexicalcapitalizationposition-sensitive

Controlled accuracy

100.0%

6/ 6 expected examples

Exclusive activations

63.9%

Activations not shared with the other tested contextual features

Closest raw neuron

2037

Pearson correlation: 0.166

Activation and separation

Active token percentage 0.37%
Exclusive activation percentage 63.93%
Mean top-100 overlap 3.50%

Separation metrics

Measurement Mean result
Active-set Jaccard 0.0391
Activation correlation 0.0174
Encoder cosine similarity 0.0493
Decoder cosine similarity 0.0446

Strong cross-document examples

  1. 4.693 — 's exactly what it is today! [[To]] say there is a lot going on in the
  2. 4.595 — ureteric stents in vivo. [[To]] examine the relationship between encrustation and microbial
  3. 4.488 — Team was chosen by popular vote of fans. [[ To]] select the team, a panel of experts first
  4. 4.258 — of Western Rajasthan, India. [[To]] asses the impact of drought on nutritional status
  5. 4.154 — , long length, String[] hosts). [[To]] pass the starting offset you can use Configuration parameters

Evidence types

  • cross-document examples
  • controlled prompts
  • raw-neuron comparison
  • pairwise feature separation
  • controlled classification

Known limitations

  • Capitalization is a major component but may not be the only represented property.
  • Weak activations were observed for some lowercase purpose constructions.

Source result files

results/step_21b_contextual_feature_neuron_examples.jsonresults/step_24a_contextual_feature_separation.jsonresults/step_24b_contextual_feature_classification.json