How 24,000 companies keep their AI from Breaking in Production Rohit Agarwal, Portkey

Over 1 Trillion AI tokens pass through Portkey every single day. Every AI product eventually runs into the same problem. The prototype works, but once it goes live the system has to manage multiple models, rising token costs, unpredictable latency, and infrastructure that was never built for AI workloads. That is the problem Rohit Agarwal is solving with Portkey, an AI gateway that sits between applications and the models, whether that’s GPT-4, Claude, or Gemini. With 24,000 companies routing their AI through Portkey, Rohit sits on ground-level data on how AI is actually being used in production. Which models enterprises are betting on. Where costs are quietly climbing. How usage patterns shift as companies move from pilots to real products. When AI spend surpasses cloud spend, and Rohit believes it will, the infrastructure running underneath it becomes one of the most important bets in tech. This episode explores what it takes to run AI systems at that scale. 00:00 – Trailer 01:05 – 500 billion AI tokens every day 04:05 – First to call an "AI gateway" 07:26 – Where did the Gateway insight come from? 12:08 – How Portkey is winning this space 13:05 – Picking the right gambles over wrong ones 14:16 – What are LLM endpoints? 15:21 – AI will 100% surpass cloud spend 19:00 – Hype is coming from people still in Q&A mode 19:33 – AI employees over humans in customer support? 23:00 – For AI startups, traffic > revenue 24:43 – The bubble is in valuations, not utility 26:05 – How Rohit built his personal automations 28:38 – Costliest model is most used now 33:21 – What's going right and wrong for AI companies 37:49 – Hiring a VP of sales after $15M is possible today 39:57 – What edge does Claude have over other models? 43:35 – Founders need a "why me vs. why Anthropic" story 52:56 – What if Anthropic or AWS builds a gateway? 55:41 – Predictions for the next 12 months 59:40 – How big is the opportunity in Agents? 01:00:48 – Startups now have to prove it's not a weekend project 01:01:54 – Is Build v/s Buy no longer a Debate? 01:03:50 – What would Rohit build if starting up today? 01:05:58 – How Portkey is different from an API gateway 01:08:26 – MCP / tool calling enables agentic workflows 01:12:00 – Portkey's Community-driven early GTM 01:13:34 – Startups have only 2 reasons for Open core ------------- India’s talent has built the world’s tech—now it’s time to lead it. This mission goes beyond startups. It’s about shifting the center of gravity in global tech to include the brilliance rising from India. What is Neon Fund? We invest in seed and early-stage founders from India and the diaspora building world-class Enterprise AI companies. We bring capital, conviction, and a community that’s done it before.

You can listen to How 24,000 companies keep their AI from Breaking in Production Rohit Agarwal, Portkey online on Radio and Podcast. Open the player on this page to stream the available audio.

How 24,000 companies keep their AI from Breaking in Production Rohit Agarwal, Portkey is an episode from 100x Entrepreneur by Siddhartha Ahluwalia.

This episode is 78:44 long.

This episode was published on Mar 19, 2026.

Yes. Use the heart button on the episode page to add it to your favorite episodes list.

Yes. This page shows related episodes from 100x Entrepreneur when more episodes are available from the podcast feed.