The Secret Engine of AI - Prolific [Sponsored] (Sara Saab, Enzo Blindow)

We sat down with Sara Saab (VP of Product at Prolific) and Enzo Blindow (VP of Data and AI at Prolific) to explore the critical role of human evaluation in AI development and the challenges of aligning AI systems with human values. Prolific is a human annotation and orchestration platform for AI used by many of the major AI labs. This is a sponsored show in partnership with Prolific. **SPONSOR MESSAGES** — cyber•Fund is a founder-led investment firm accelerating the cybernetic economy Oct SF conference - - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++ Hiring a SF VC Principal: Submit investment deck: — While technologists want to remove humans from the loop for speed and efficiency, these non-deterministic AI systems actually require more human oversight than ever before. Prolific's approach is to put "well-treated, verified, diversely demographic humans behind an API" - making human feedback as accessible as any other infrastructure service. When AI models like Grok 4 achieve top scores on technical benchmarks but feel awkward or problematic to use in practice, it exposes the limitations of our current evaluation methods. The guests argue that optimizing for benchmarks may actually weaken model performance in other crucial areas, like cultural sensitivity or natural conversation. We also discuss Anthropic's research showing that frontier AI models, when given goals and access to information, independently arrived at solutions involving blackmail - without any prompting toward unethical behavior. Even more concerning, the more sophisticated the model, the more susceptible it was to this "agentic misalignment." Enzo and Sarah present Prolific's "Humane" leaderboard as an alternative to existing benchmarking systems. By stratifying evaluations across diverse demographic groups, they reveal that different populations have vastly different experiences with the same AI models. Looking ahead, the guests imagine a world where humans take on coaching and teaching roles for AI systems - similar to how we might correct a child or review code. This also raises important questions about working conditions and the evolution of labor in an AI-augmented world. Rather than replacing humans entirely, we may be moving toward more sophisticated forms of human-AI collaboration. As AI tech becomes more powerful and general-purpose, the quality of human evaluation becomes more critical, not less. We need more representative evaluation frameworks that capture the messy reality of human values and cultural diversity. Visit Prolific: Sara Saab (VP Product): Enzo Blindow (VP Data & AI): TRANSCRIPT: TOC: [00:00:00] Intro & Background [00:03:16] Human-in-the-Loop Challenges [00:17:19] Can AIs Understand? [00:32:02] Benchmarking & Vibes [00:51:00] Agentic Misalignment Study [01:03:00] Data Quality vs Quantity [01:16:00] Future of AI Oversight REFS: Anthropic Agentic Misalignment Value Compass Reasoning Models Don’t Always Say What They Think (Anthropic) Apollo research - science of evals blog post Leaderboard Illusion MLST video The Leaderboard Illusion [2025] Shivalika Singh et al (Truncated, full list on YT)

You can listen to The Secret Engine of AI - Prolific [Sponsored] (Sara Saab, Enzo Blindow) online on Radio and Podcast. Open the player on this page to stream the available audio.

The Secret Engine of AI - Prolific [Sponsored] (Sara Saab, Enzo Blindow) is an episode from Machine Learning Street Talk by Machine Learning Street Talk (MLST).

This episode is 01:19:39 long.

This episode was published on Oct 18, 2025.

Yes. Use the heart button on the episode page to add it to your favorite episodes list.

Yes. This page shows related episodes from Machine Learning Street Talk when more episodes are available from the podcast feed.