User-agent: * Allow: / Allow: /linkedin/ Allow: /llms.txt Allow: /llms-full.txt Allow: /sitemap.xml Allow: /feed.xml Allow: /site.webmanifest Allow: /.well-known/security.txt Allow: /assets/ # Thin / non-public tooling: save crawl budget Disallow: /Update/ Disallow: /forms/ User-agent: Googlebot Allow: / Allow: /linkedin/ Allow: /assets/ Disallow: /Update/ Disallow: /forms/ User-agent: Googlebot-Image Allow: /assets/img/ User-agent: Bingbot Allow: / Allow: /linkedin/ Allow: /assets/ Disallow: /Update/ Disallow: /forms/ User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: LinkedInBot Allow: / User-agent: AhrefsBot Allow: / User-agent: AhrefsSiteAudit Allow: / # AI / LLM crawlers (aligned with public llms.txt) User-agent: GPTBot Allow: / Allow: /linkedin/ Allow: /llms.txt Allow: /llms-full.txt Allow: /assets/ User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / Allow: /linkedin/ Allow: /llms.txt Allow: /llms-full.txt Allow: /assets/ User-agent: anthropic-ai Allow: / User-agent: Google-Extended Allow: / Allow: /linkedin/ Allow: /llms.txt Allow: /llms-full.txt Allow: /assets/ User-agent: Applebot-Extended Allow: / User-agent: PerplexityBot Allow: / Allow: /linkedin/ Allow: /llms.txt Allow: /llms-full.txt Allow: /assets/ User-agent: Bytespider Disallow: / # Only XML sitemaps here; do not list RSS/Atom feeds Sitemap: https://akshetpatel.com/sitemap.xml