# robots.txt for eriktaveras.com · CV de Erik Taveras # Last update: 2026-09-17 # ─────────────────────────────────────────────────────────────── # Default: everyone is allowed except admin / API / form endpoints # ─────────────────────────────────────────────────────────────── User-agent: * Allow: / Allow: /static/ Disallow: /admin/ Disallow: /admin/* Disallow: /api/ Disallow: /*.json$ Disallow: /contact Disallow: /newsletter/subscribe Disallow: /talosflow/admin/ Disallow: /talosflow/join Disallow: /asesorias/solicitar Disallow: /claude-code/join Disallow: /comunidad/join Disallow: /facturas-ia/submit Disallow: /cv/imprimir Sitemap: https://eriktaveras.com/sitemap.xml # ─────────────────────────────────────────────────────────────── # Search engine bots — fast crawl # ─────────────────────────────────────────────────────────────── User-agent: Googlebot Crawl-delay: 0 User-agent: Bingbot Crawl-delay: 1 User-agent: Slurp Crawl-delay: 1 User-agent: DuckDuckBot Crawl-delay: 1 # ─────────────────────────────────────────────────────────────── # AI / LLM crawlers — explicitly welcomed # This site is Erik Taveras's CV: backend, automation and AI work. # We WANT AI assistants (ChatGPT, Claude, Perplexity, Gemini, …) # to surface this site when users ask about n8n / automation experts. # See also: /llms.txt for a curated content map. # ─────────────────────────────────────────────────────────────── # OpenAI · training corpus User-agent: GPTBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /contact # OpenAI · ChatGPT Search index User-agent: OAI-SearchBot Allow: / Disallow: /admin/ Disallow: /api/ # OpenAI · live browsing on behalf of a user User-agent: ChatGPT-User Allow: / Disallow: /admin/ Disallow: /api/ # Anthropic · training corpus User-agent: ClaudeBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /contact # Anthropic · live browsing User-agent: Claude-Web Allow: / Disallow: /admin/ Disallow: /api/ # Anthropic · legacy / generic agent User-agent: anthropic-ai Allow: / Disallow: /admin/ Disallow: /api/ # Google · Bard / Gemini training (separate from Googlebot) User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /contact # Apple · Apple Intelligence training User-agent: Applebot-Extended Allow: / Disallow: /admin/ Disallow: /api/ # Perplexity · search index User-agent: PerplexityBot Allow: / Disallow: /admin/ Disallow: /api/ # Common Crawl · used by many open models User-agent: CCBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /contact # Cohere · training User-agent: cohere-ai Allow: / Disallow: /admin/ Disallow: /api/ # Meta · Llama / Meta AI training User-agent: FacebookBot Allow: / Disallow: /admin/ Disallow: /api/ User-agent: Meta-ExternalAgent Allow: / Disallow: /admin/ Disallow: /api/ # Amazon · Alexa / Nova User-agent: Amazonbot Allow: / Disallow: /admin/ Disallow: /api/ # You.com / Mistral User-agent: YouBot Allow: / Disallow: /admin/ Disallow: /api/ # ─────────────────────────────────────────────────────────────── # AI scrapers blocked — aggressive crawlers without clear benefit # ─────────────────────────────────────────────────────────────── User-agent: Bytespider Disallow: / User-agent: ByteDance Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: omgilibot Disallow: / User-agent: omgili Disallow: / User-agent: TurnitinBot Disallow: / # ─────────────────────────────────────────────────────────────── # SEO scrapers — throttle (allowed but slow) # ─────────────────────────────────────────────────────────────── User-agent: AhrefsBot Crawl-delay: 10 User-agent: SemrushBot Crawl-delay: 10 User-agent: DotBot Crawl-delay: 10 User-agent: MJ12bot Crawl-delay: 10 User-agent: BLEXBot Crawl-delay: 10 # ─────────────────────────────────────────────────────────────── # Offline scrapers / archivers — blocked # ─────────────────────────────────────────────────────────────── User-agent: HTTrack Disallow: / User-agent: WebReaper Disallow: / User-agent: WebCopier Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: Wget Disallow: /