Radio and PodcastRadio and PodcastLive Radio & Podcasts
Analysing Robots.txt at scale with HTTP Archive and BigQuery artwork
Technology

Analysing Robots.txt at scale with HTTP Archive and BigQuery

Search Off the Record by Google Developer Studio

Apr 23, 202627:40Technology

In this episode of Search Off the Record, Martin and Gary turn a simple robots.txt question into a data‑driven deep dive using HTTP Archive, WebPageTest, custom JavaScript metrics, and BigQuery. They explore how millions...

Related Episodes

Continue listening to more episodes from Search Off the Record.

Do sitemaps still matter?

Oct 1, 2026 - 26:26

Sitemaps have been a core element of web development and search engine optimization for two decades, yet...

How to read the Indexing Report

Jul 16, 2026 - 31:40

Should you panic when your Search Console indexing report is showing pages that aren't indexed? Is a 404...

Should I use markdown for my site?

Jun 15, 2026 - 26:12

Should you convert your website into Markdown to help Large Language Models (LLMs) understand your conten...