VAEs Are Energy-Based Models? [Dr. Jeff Beck]

What makes something truly *intelligent?* Is a rock an agent? Could a perfect simulation of your brain actually *be* you? In this fascinating conversation, Dr. Jeff Beck takes us on a journey through the philosophical and technical foundations of agency, intelligence, and the future of AI. Jeff doesn't hold back on the big questions. He argues that from a purely mathematical perspective, there's no structural difference between an agent and a rock – both execute policies that map inputs to outputs. The real distinction lies in *sophistication* – how complex are the internal computations? Does the system engage in planning and counterfactual reasoning, or is it just a lookup table that happens to give the right answers? *Key topics explored in this conversation:* *The Black Box Problem of Agency* – How can we tell if something is truly planning versus just executing a pre-computed response? Jeff explains why this question is nearly impossible to answer from the outside, and why the best we can do is ask which model gives us the simplest explanation. *Energy-Based Models Explained* – A masterclass on how EBMs differ from standard neural networks. The key insight: traditional networks only optimize weights, while energy-based models optimize *both* weights and internal states – a subtle but profound distinction that connects to Bayesian inference. *Why Your Brain Might Have Evolved from Your Nose* – One of the most surprising moments in the conversation. Jeff proposes that the complex, non-smooth nature of olfactory space may have driven the evolution of our associative cortex and planning abilities. *The JEPA Revolution* – A deep dive into Yann LeCun's Joint Embedding Prediction Architecture and why learning in latent space (rather than predicting every pixel) might be the key to more robust AI representations. *AI Safety Without Skynet Fears* – Jeff takes a refreshingly grounded stance on AI risk. He's less worried about rogue superintelligences and more concerned about humans becoming "reward function selectors" – couch potatoes who just approve or reject AI outputs. His proposed solution? Use inverse reinforcement learning to derive AI goals from observed human behavior, then make *small* perturbations rather than naive commands like "end world hunger." Whether you're interested in the philosophy of mind, the technical details of modern machine learning, or just want to understand what makes intelligence *tick,* this conversation delivers insights you won't find anywhere else. --- TIMESTAMPS: 00:00:00 Geometric Deep Learning & Physical Symmetries 00:00:56 Defining Agency: From Rocks to Planning 00:05:25 The Black Box Problem & Counterfactuals 00:08:45 Simulated Agency vs. Physical Reality 00:12:55 Energy-Based Models & Test-Time Training 00:17:30 Bayesian Inference & Free Energy 00:20:07 JEPA, Latent Space, & Non-Contrastive Learning 00:27:07 Evolution of Intelligence & Modular Brains 00:34:00 Scientific Discovery & Automated Experimentation 00:38:04 AI Safety, Enfeeblement & The Future of Work --- REFERENCES: Concept: [00:00:58] Free Energy Principle (FEP) [00:06:00] Monte Carlo Tree Search Book: [00:09:00] The Intentional Stance Paper: [00:13:00] A Tutorial on Energy-Based Learning (LeCun 2006) [00:15:00] Auto-Encoding Variational Bayes (VAE) [00:20:15] JEPA (Joint Embedding Prediction Architecture) [00:22:30] The Wake-Sleep Algorithm <trunc, see rescript> --- RESCRIPT: PDF:

You can listen to VAEs Are Energy-Based Models? [Dr. Jeff Beck] online on Radio and Podcast. Open the player on this page to stream the available audio.

VAEs Are Energy-Based Models? [Dr. Jeff Beck] is an episode from Machine Learning Street Talk by Machine Learning Street Talk (MLST).

This episode is 00:46:56 long.

This episode was published on Jan 25, 2026.

Yes. Use the heart button on the episode page to add it to your favorite episodes list.

Yes. This page shows related episodes from Machine Learning Street Talk when more episodes are available from the podcast feed.