Analysing Robots.txt at scale with HTTP Archive and BigQuery

In this episode of Search Off the Record, Martin and Gary turn a simple robots.txt question into a data‑driven deep dive using HTTP Archive, WebPageTest, custom JavaScript metrics, and BigQuery. They explore how millions of real robots.txt files are actually written in 2025–2026, which directives and user‑agents are most common, and what that means for modern crawling and AI bots. Perfect for beginner to mid‑level developers and SEOs, you'll learn how large‑scale web measurement works (HTTP Archive, Chrome UX Report, Web Almanac), and how to turn raw crawl data into actionable SEO insights.

You can listen to Analysing Robots.txt at scale with HTTP Archive and BigQuery online on Radio and Podcast. Open the player on this page to stream the available audio.

Analysing Robots.txt at scale with HTTP Archive and BigQuery is an episode from Search Off the Record by Google Developer Studio.

This episode is 27:40 long.

This episode was published on Apr 23, 2026.

Yes. Use the heart button on the episode page to add it to your favorite episodes list.

Yes. This page shows related episodes from Search Off the Record when more episodes are available from the podcast feed.