CAZ Scaling Visualizations

Interactive 3D cazstellations and cross-model alignment maps from the CAZ scaling experiments.

Each cazstellation is a self-contained interactive Plotly visualization. They are large files (~4-5 MB each) and may take a moment to load.

Cazstellations by Model Family

GPT-2

• GPT-2 (124M)
• GPT-2 Medium (355M)
• GPT-2 Large (774M)
• GPT-2 XL (1.5B)

Pythia

• Pythia 70M
• Pythia 160M
• Pythia 410M
• Pythia 1B
• Pythia 1.4B
• Pythia 2.8B
• Pythia 6.9B

OPT

• OPT 125M
• OPT 350M
• OPT 1.3B
• OPT 2.7B
• OPT 6.7B

Qwen 2.5

• Qwen 2.5 0.5B
• Qwen 2.5 1.5B
• Qwen 2.5 3B
• Qwen 2.5 7B

Gemma 2

• Gemma 2 2B
• Gemma 2 9B

Dark Matter

• Dark Matter Overview
• Pythia 1.4B — Dark Matter

Labeled Dark Matter by Model

GPT-2

• GPT-2 Medium
• GPT-2 Large
• GPT-2 XL

Pythia

• Pythia 70M
• Pythia 160M
• Pythia 410M
• Pythia 1B
• Pythia 1.4B
• Pythia 2.8B
• Pythia 6.9B

OPT

• OPT 125M
• OPT 350M
• OPT 1.3B
• OPT 2.7B
• OPT 6.7B

Qwen 2.5

• Qwen 2.5 0.5B
• Qwen 2.5 1.5B
• Qwen 2.5 3B
• Qwen 2.5 7B

Gemma 2

• Gemma 2 2B

CAZ Anatomy & Growth

• CAZ Anatomy — Pythia 1.4B
• CAZ Growth Animation — Pythia 1.4B

Universal Features

• Universal Feature Atlas
• Universal Feature Ablation
• Universal Feature Provenance

Procrustes Alignment Maps

• All Concepts
• Causation
• Certainty
• Credibility
• Moral Valence
• Negation
• Sentiment
• Temporal Order