Radio and PodcastRadio and PodcastLive Radio & Podcasts
Analysing Robots.txt at scale with HTTP Archive and BigQuery artwork
Technology

Analysing Robots.txt at scale with HTTP Archive and BigQuery

Search Off the Record by Google Developer Studio

Apr 23, 202627:40Technology

In this episode of Search Off the Record, Martin and Gary turn a simple robots.txt question into a data‑driven deep dive using HTTP Archive, WebPageTest, custom JavaScript metrics, and BigQuery. They explore how millions...

About This Episode

Analysing Robots.txt at scale with HTTP Archive and BigQuery is an episode from Search Off the Record by Google Developer Studio. In this episode of Search Off the Record, Martin and Gary turn a simple robots.txt question into a data‑driven...

Podcast

This episode belongs to Search Off the Record.

Listen Online

Use the player on this page to stream the episode online.

Episode Details

Published Apr 23, 2026, 27:40 long, audio available.

Related Episodes

Continue listening to more episodes from Search Off the Record.

How to read the Indexing Report

Jul 16, 2026 - 31:40

Should you panic when your Search Console indexing report is showing pages that aren't indexed? Is a 404...

Should I use markdown for my site?

Jun 15, 2026 - 26:12

Should you convert your website into Markdown to help Large Language Models (LLMs) understand your conten...

Vibe Coding - yay or nay?

May 7, 2026 - 33:56

In this episode of Search Off the Record, Martin Splitt and John Mueller from Google's Search Relations t...