Web Application Firewall (WAF)

A security layer between a website's server and the internet, filtering incoming traffic to block malicious requests. WAFs detect and block threats like SQL injection, cross-site scripting, and DDoS attacks. Major providers include Cloudflare, Akamai, AWS WAF, and Fastly. They are very good at their job, and they are completely indifferent to yours.

Why it matters for writers: WAFs are the villain of the llms.txt story, or the hero, depending on your perspective. AI crawlers trip WAF bot-detection heuristics because they tick every "suspicious" box: they don't execute JavaScript, don't maintain cookies, come from data center IP ranges, and use non-browser user agents. A site owner can publish an llms.txt file specifically to help AI systems, and their WAF will block those same AI systems from reading it. This is the core of the llms.txt Access Paradox and one of the more tragicomic findings in the research.

Related terms: llms.txt · User Agent · robots.txt