llms-txt-support

Single responsibility: Detect, fetch, and utilize llms.txt files that provide LLM-optimized documentation, enabling 10x faster documentation ingestion. (BP-4)

The llms.txt standard (https://llmstxt.org/) provides a convention for websites to expose LLM-friendly documentation. Instead of scraping entire sites, check for llms.txt first.

| RELEVANT | Target base URL, llms.txt content | Full site scraping | | PERIPHERAL | llms.txt spec reference | Other sites' llms.txt | | DISTRACTOR | Previous scraping attempts | Unrelated documentation |

Rileva e utilizza i file llms.txt per la documentazione ottimizzata per LLM. Da utilizzare per verificare se un sito dispone di documenti pronti per LLM prima dello scraping. Fonte: jmagly/ai-writing-guide.