Why It's Impossible to Review AIs, and Why TechCrunch Is Doing It Anyway

Traditional academic benchmarks poorly reflect real-world user experience. TechCrunch designed its own tests using practical questions on politics and healthcare to evaluate Claude, GPT-4, and other models.

Findings

• Benchmark vs reality – MMLU and similar tests don't capture how models perform on real tasks.
• Practical evaluation – Testing with domain-specific questions (politics, healthcare) reveals strengths and weaknesses that benchmarks miss.
• Model differences – Results show meaningful variation by model and by document type. Technical and legal documents remain harder to summarize accurately.

Document-type differences

• General news and articles – Most models perform well with minimal errors.
• Technical documentation – Higher hallucination rates; verify technical terms and numbers.
• Legal text – Nuance and precision matter; human review strongly recommended.