User Agent

A string sent with every HTTP request that identifies the software making the request. Browsers send long, baroque strings like Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36.... AI crawlers send more honest identifiers like ClaudeBot, GPTBot, or Google-Extended. WAFs use user agent strings as one of many signals for bot detection.

Why it matters for writers: User agent strings are how site owners (and their WAFs) distinguish between human visitors, search engine crawlers, and AI systems. Tools that fetch llms.txt files face a philosophical dilemma: use a browser-like user agent and you'll probably bypass WAF blocks, but you're being dishonest. Use an honest AI-tool identifier and you'll get blocked by the very sites that published content for AI systems to read. It's the identity crisis at the heart of the AI-web relationship, expressed in a single HTTP header.

Related terms: Web Application Firewall · robots.txt · llms.txt