CARB Failure Observability

Research pipeline for structured failure analysis in language model reasoning.

CARB dataset → HF Inference API → failure extraction → MiniLM embeddings → KMeans → mutual information

Central question: do failure clusters align with reasoning category more than with model identity?

The MI comparison plot answers this directly — a larger MI(cluster, reasoning_type) bar relative to MI(cluster, model_identity) supports the hypothesis that failure structure is organized by reasoning difficulty, not model choice alone.

Requires HF_TOKEN set in Space secrets. See failure-geometry-demo for a fully self-contained version that needs no API key.

Research context: obversarystudios.org