Radio and PodcastRadio and PodcastLive Radio & Podcasts
Analysing Robots.txt at scale with HTTP Archive and BigQuery artwork
Technology

Analysing Robots.txt at scale with HTTP Archive and BigQuery

Search Off the Record by Google Developer Studio

Apr 23, 202627:40Technology

In this episode of Search Off the Record, Martin and Gary turn a simple robots.txt question into a data‑driven deep dive using HTTP Archive, WebPageTest, custom JavaScript metrics, and BigQuery. They explore how millions...

About This Episode

Analysing Robots.txt at scale with HTTP Archive and BigQuery is an episode from Search Off the Record by Google Developer Studio. In this episode of Search Off the Record, Martin and Gary turn a simple robots.txt question into a data‑driven...

Podcast

This episode belongs to Search Off the Record.

Listen Online

Use the player on this page to stream the episode online.

Episode Details

Published Apr 23, 2026, 27:40 long, audio available.