# robots.txt — VICATOO.com # Updated 2026-06-06: explicit block list for harmful crawlers. # These bots are ALSO blocked at the application level (middleware), # this file is defense-in-depth + a polite request for compliant bots. # ── Content Signals ────────────────────────────────────────────────── # Default for the whole site: search=yes, ai-input=no, ai-train=no. # Owner 2026-10-21: AI search engines and AI assistants (the AI group below) read # the same public pages as Googlebot, under the same rules: hotel, room, city and # landing pages included; booking, checkout, the API and the live-price rooms # endpoint stay closed. They get the page crawlers get (no live prices, no # supplier call). That group carries its own signal: search=yes, ai-input=yes, # ai-train=no. # VICATOO does not opt into model training: the training group below is # disallowed site-wide. # Agents that want to search hotels or check a stay's current price should use # the public MCP server: no sign-in, read-only. # See /.well-known/api-catalog and /.well-known/auth.md. Content-Signal: search=yes, ai-input=no, ai-train=no # ── Machine interface ──────────────────────────────────────────────── # /api/ below is NOT a public API and stays disallowed. The supported machine # interface is the public MCP server; it is described at: # https://www.vicatoo.com/.well-known/api-catalog (RFC 9727) # https://www.vicatoo.com/.well-known/auth.md # https://mcp.vicatoo.com/mcp (MCP, Streamable HTTP) # ── SEO competitor scrapers ── User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: SERankingBacklinksBot Disallow: / User-agent: Barkrowler Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: PetalBot Disallow: / User-agent: Sogou Disallow: / User-agent: Baiduspider Disallow: / User-agent: BLEXBot Disallow: / User-agent: SeekportBot Disallow: / User-agent: SeznamBot Disallow: / User-agent: YisouSpider Disallow: / # ── AI model training ──────────────────────────────────────────────── # VICATOO does not opt into model-training crawl. These agents collect content # for AI model training (per each vendor's documentation) and are disallowed # site-wide. Search and answer-time agents are in the next group. User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Claude-Web User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: meta-externalagent User-agent: Amazonbot Content-Signal: ai-train=no Disallow: / # ── AI search and AI assistants (answer-time retrieval) ───────────── # Owner 2026-10-21: the same rules as the Googlebot group below, line for line # (the reasons are written there); change both groups together. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: YouBot Content-Signal: search=yes, ai-input=yes, ai-train=no Disallow: /admin/ Disallow: /api/ Disallow: /booking/ Disallow: /checkout Disallow: /storage/ Disallow: /hotels/etg/*/rooms Disallow: /hotels/etg/*lang=it Disallow: /hotels/etg/*lang=nl Disallow: /hotels/etg/*lang=pl Disallow: /hotels/etg/*lang=ro Disallow: /hotels/etg/*lang=sv Disallow: /hotels/etg/*lang=bg Disallow: /hotels/etg/*lang=cs Disallow: /hotels/etg/*lang=da Disallow: /hotels/etg/*lang=el Disallow: /hotels/etg/*lang=fi Disallow: /hotels/etg/*lang=hu Disallow: /hotels/etg/*lang=no Disallow: /hotels/etg/*lang=uk Disallow: /hotels/etg/*lang=pt Disallow: /cdn-cgi/ Disallow: /search-rooms Disallow: /*?*check_in= Disallow: /*?*check_out= Allow: /hotels/etg/*?*check_in= Allow: /hotels/etg/*?*check_out= Disallow: /hotels/etg/*/rooms?*check_in= Disallow: /hotels/etg/*/rooms?*check_out= Disallow: /*?*stars= Disallow: /*?*district= Disallow: /*?*adults= Disallow: /*?*children= Disallow: /*?*ages= Disallow: /*?*price_min= Disallow: /*?*price_max= Disallow: /*?*_dr= Allow: / # ── Non-AI image/index scrapers — still blocked ── User-agent: meta-webindexer Disallow: / User-agent: Omgilibot Disallow: / User-agent: FacebookBot Disallow: / User-agent: Timpibot Disallow: / User-agent: ImagesiftBot Disallow: / # ── Allowed crawlers: search engines, link previews, monitoring ── # 2026-06-16 (Round 21AO): each search engine matches ONLY its own group and # IGNORES the wildcard "*" group below — so the functional-path Disallows must be # repeated here, else Googlebot crawls /api/ (heavy ETG/reviews endpoints), # /booking/, etc. and burns crawl budget on non-indexable pages. User-agent: Googlebot Disallow: /admin/ Disallow: /api/ Disallow: /booking/ Disallow: /checkout Disallow: /storage/ Disallow: /hotels/etg/*/rooms # Owner 10-20: hotel pages in these 13 languages are noindex with the English canonical (no VICATOO hotel copy in # them, VICO-P2C) and no hreflang points at them; 18% of Googlebot's hotel requests 10-11..10-20. Room pages in # them are noindex too. City and landing pages (/hotels/{city}?lang=..) are not under /hotels/etg/ and stay open. Disallow: /hotels/etg/*lang=it Disallow: /hotels/etg/*lang=nl Disallow: /hotels/etg/*lang=pl Disallow: /hotels/etg/*lang=ro Disallow: /hotels/etg/*lang=sv Disallow: /hotels/etg/*lang=bg Disallow: /hotels/etg/*lang=cs Disallow: /hotels/etg/*lang=da Disallow: /hotels/etg/*lang=el Disallow: /hotels/etg/*lang=fi Disallow: /hotels/etg/*lang=hu Disallow: /hotels/etg/*lang=no Disallow: /hotels/etg/*lang=uk Disallow: /hotels/etg/*lang=pt Disallow: /cdn-cgi/ Disallow: /search-rooms # -- Faceted navigation, repeated here on purpose -------------------- # Google: "Only one group is valid for a particular crawler ... Other # groups are ignored. User agent specific groups and global groups (*) # are not combined." These ten rules were added 2026-09-05 under # "User-agent: *" alone, so the crawler they were written for never read # them. Measured here 2026-09-06 15:13-22:58, AFTER that change: of # 60,916 Googlebot requests, 32,122 carried one of these parameters # (stars 24,321 - district 22,458 - check_in 5,245) against 12,252 clean # URLs. Half the crawl budget spent on URLs that redirect to a page # Google already holds. Disallow: /*?*check_in= Disallow: /*?*check_out= # VICO-SEOFIX 2026-09-27 (owner): dated HOTEL urls are crawlable again. They answer 200 with # the canonical on the clean hotel URL (open_dates 301s to it) for every visitor, so crawlers # can re-read the dated URLs they indexed and consolidate them. Longest match wins: these beat # the two rules above for /hotels/etg/ only; the live-price rooms endpoint stays closed. Allow: /hotels/etg/*?*check_in= Allow: /hotels/etg/*?*check_out= Disallow: /hotels/etg/*/rooms?*check_in= Disallow: /hotels/etg/*/rooms?*check_out= Disallow: /*?*stars= Disallow: /*?*district= Disallow: /*?*adults= Disallow: /*?*children= Disallow: /*?*ages= Disallow: /*?*price_min= Disallow: /*?*price_max= Disallow: /*?*_dr= Allow: / User-agent: Bingbot Disallow: /admin/ Disallow: /api/ Disallow: /booking/ Disallow: /checkout Disallow: /storage/ Disallow: /hotels/etg/*/rooms # Owner 10-20: hotel pages in these 13 languages are noindex with the English canonical (no VICATOO hotel copy in # them, VICO-P2C) and no hreflang points at them; 18% of Googlebot's hotel requests 10-11..10-20. Room pages in # them are noindex too. City and landing pages (/hotels/{city}?lang=..) are not under /hotels/etg/ and stay open. Disallow: /hotels/etg/*lang=it Disallow: /hotels/etg/*lang=nl Disallow: /hotels/etg/*lang=pl Disallow: /hotels/etg/*lang=ro Disallow: /hotels/etg/*lang=sv Disallow: /hotels/etg/*lang=bg Disallow: /hotels/etg/*lang=cs Disallow: /hotels/etg/*lang=da Disallow: /hotels/etg/*lang=el Disallow: /hotels/etg/*lang=fi Disallow: /hotels/etg/*lang=hu Disallow: /hotels/etg/*lang=no Disallow: /hotels/etg/*lang=uk Disallow: /hotels/etg/*lang=pt Disallow: /cdn-cgi/ Disallow: /search-rooms # -- Faceted navigation, repeated here on purpose -------------------- # Google: "Only one group is valid for a particular crawler ... Other # groups are ignored. User agent specific groups and global groups (*) # are not combined." These ten rules were added 2026-09-05 under # "User-agent: *" alone, so the crawler they were written for never read # them. Measured here 2026-09-06 15:13-22:58, AFTER that change: of # 60,916 Googlebot requests, 32,122 carried one of these parameters # (stars 24,321 - district 22,458 - check_in 5,245) against 12,252 clean # URLs. Half the crawl budget spent on URLs that redirect to a page # Google already holds. Disallow: /*?*check_in= Disallow: /*?*check_out= # VICO-SEOFIX 2026-09-27 (owner): dated HOTEL urls are crawlable again. They answer 200 with # the canonical on the clean hotel URL (open_dates 301s to it) for every visitor, so crawlers # can re-read the dated URLs they indexed and consolidate them. Longest match wins: these beat # the two rules above for /hotels/etg/ only; the live-price rooms endpoint stays closed. Allow: /hotels/etg/*?*check_in= Allow: /hotels/etg/*?*check_out= Disallow: /hotels/etg/*/rooms?*check_in= Disallow: /hotels/etg/*/rooms?*check_out= Disallow: /*?*stars= Disallow: /*?*district= Disallow: /*?*adults= Disallow: /*?*children= Disallow: /*?*ages= Disallow: /*?*price_min= Disallow: /*?*price_max= Disallow: /*?*_dr= Allow: / User-agent: Applebot Disallow: /admin/ Disallow: /api/ Disallow: /booking/ Disallow: /checkout Disallow: /storage/ Disallow: /hotels/etg/*/rooms # Owner 10-20: hotel pages in these 13 languages are noindex with the English canonical (no VICATOO hotel copy in # them, VICO-P2C) and no hreflang points at them; 18% of Googlebot's hotel requests 10-11..10-20. Room pages in # them are noindex too. City and landing pages (/hotels/{city}?lang=..) are not under /hotels/etg/ and stay open. Disallow: /hotels/etg/*lang=it Disallow: /hotels/etg/*lang=nl Disallow: /hotels/etg/*lang=pl Disallow: /hotels/etg/*lang=ro Disallow: /hotels/etg/*lang=sv Disallow: /hotels/etg/*lang=bg Disallow: /hotels/etg/*lang=cs Disallow: /hotels/etg/*lang=da Disallow: /hotels/etg/*lang=el Disallow: /hotels/etg/*lang=fi Disallow: /hotels/etg/*lang=hu Disallow: /hotels/etg/*lang=no Disallow: /hotels/etg/*lang=uk Disallow: /hotels/etg/*lang=pt Disallow: /cdn-cgi/ Allow: / User-agent: YandexBot Disallow: /admin/ Disallow: /api/ Disallow: /booking/ Disallow: /checkout Disallow: /storage/ Disallow: /hotels/etg/*/rooms # Owner 10-20: hotel pages in these 13 languages are noindex with the English canonical (no VICATOO hotel copy in # them, VICO-P2C) and no hreflang points at them; 18% of Googlebot's hotel requests 10-11..10-20. Room pages in # them are noindex too. City and landing pages (/hotels/{city}?lang=..) are not under /hotels/etg/ and stay open. Disallow: /hotels/etg/*lang=it Disallow: /hotels/etg/*lang=nl Disallow: /hotels/etg/*lang=pl Disallow: /hotels/etg/*lang=ro Disallow: /hotels/etg/*lang=sv Disallow: /hotels/etg/*lang=bg Disallow: /hotels/etg/*lang=cs Disallow: /hotels/etg/*lang=da Disallow: /hotels/etg/*lang=el Disallow: /hotels/etg/*lang=fi Disallow: /hotels/etg/*lang=hu Disallow: /hotels/etg/*lang=no Disallow: /hotels/etg/*lang=uk Disallow: /hotels/etg/*lang=pt Disallow: /cdn-cgi/ # -- Faceted navigation, repeated here on purpose -------------------- # Google: "Only one group is valid for a particular crawler ... Other # groups are ignored. User agent specific groups and global groups (*) # are not combined." These ten rules were added 2026-09-05 under # "User-agent: *" alone, so the crawler they were written for never read # them. Measured here 2026-09-06 15:13-22:58, AFTER that change: of # 60,916 Googlebot requests, 32,122 carried one of these parameters # (stars 24,321 - district 22,458 - check_in 5,245) against 12,252 clean # URLs. Half the crawl budget spent on URLs that redirect to a page # Google already holds. Disallow: /*?*check_in= Disallow: /*?*check_out= # VICO-SEOFIX 2026-09-27 (owner): dated HOTEL urls are crawlable again. They answer 200 with # the canonical on the clean hotel URL (open_dates 301s to it) for every visitor, so crawlers # can re-read the dated URLs they indexed and consolidate them. Longest match wins: these beat # the two rules above for /hotels/etg/ only; the live-price rooms endpoint stays closed. Allow: /hotels/etg/*?*check_in= Allow: /hotels/etg/*?*check_out= Disallow: /hotels/etg/*/rooms?*check_in= Disallow: /hotels/etg/*/rooms?*check_out= Disallow: /*?*stars= Disallow: /*?*district= Disallow: /*?*adults= Disallow: /*?*children= Disallow: /*?*ages= Disallow: /*?*price_min= Disallow: /*?*price_max= Disallow: /*?*_dr= Allow: / User-agent: DuckDuckBot Disallow: /admin/ Disallow: /api/ Disallow: /booking/ Disallow: /checkout Disallow: /storage/ Disallow: /hotels/etg/*/rooms # Owner 10-20: hotel pages in these 13 languages are noindex with the English canonical (no VICATOO hotel copy in # them, VICO-P2C) and no hreflang points at them; 18% of Googlebot's hotel requests 10-11..10-20. Room pages in # them are noindex too. City and landing pages (/hotels/{city}?lang=..) are not under /hotels/etg/ and stay open. Disallow: /hotels/etg/*lang=it Disallow: /hotels/etg/*lang=nl Disallow: /hotels/etg/*lang=pl Disallow: /hotels/etg/*lang=ro Disallow: /hotels/etg/*lang=sv Disallow: /hotels/etg/*lang=bg Disallow: /hotels/etg/*lang=cs Disallow: /hotels/etg/*lang=da Disallow: /hotels/etg/*lang=el Disallow: /hotels/etg/*lang=fi Disallow: /hotels/etg/*lang=hu Disallow: /hotels/etg/*lang=no Disallow: /hotels/etg/*lang=uk Disallow: /hotels/etg/*lang=pt Disallow: /cdn-cgi/ # -- Faceted navigation, repeated here on purpose -------------------- # Google: "Only one group is valid for a particular crawler ... Other # groups are ignored. User agent specific groups and global groups (*) # are not combined." These ten rules were added 2026-09-05 under # "User-agent: *" alone, so the crawler they were written for never read # them. Measured here 2026-09-06 15:13-22:58, AFTER that change: of # 60,916 Googlebot requests, 32,122 carried one of these parameters # (stars 24,321 - district 22,458 - check_in 5,245) against 12,252 clean # URLs. Half the crawl budget spent on URLs that redirect to a page # Google already holds. Disallow: /*?*check_in= Disallow: /*?*check_out= # VICO-SEOFIX 2026-09-27 (owner): dated HOTEL urls are crawlable again. They answer 200 with # the canonical on the clean hotel URL (open_dates 301s to it) for every visitor, so crawlers # can re-read the dated URLs they indexed and consolidate them. Longest match wins: these beat # the two rules above for /hotels/etg/ only; the live-price rooms endpoint stays closed. Allow: /hotels/etg/*?*check_in= Allow: /hotels/etg/*?*check_out= Disallow: /hotels/etg/*/rooms?*check_in= Disallow: /hotels/etg/*/rooms?*check_out= Disallow: /*?*stars= Disallow: /*?*district= Disallow: /*?*adults= Disallow: /*?*children= Disallow: /*?*ages= Disallow: /*?*price_min= Disallow: /*?*price_max= Disallow: /*?*_dr= Allow: / User-agent: facebookexternalhit Allow: / User-agent: meta-externalads Allow: / User-agent: UptimeRobot Allow: / # Default — be reasonable User-agent: * Disallow: /admin/ Disallow: /api/ Disallow: /booking/ Disallow: /checkout Disallow: /storage/ Disallow: /hotels/etg/*/rooms # Owner 10-20: hotel pages in these 13 languages are noindex with the English canonical (no VICATOO hotel copy in # them, VICO-P2C) and no hreflang points at them; 18% of Googlebot's hotel requests 10-11..10-20. Room pages in # them are noindex too. City and landing pages (/hotels/{city}?lang=..) are not under /hotels/etg/ and stay open. Disallow: /hotels/etg/*lang=it Disallow: /hotels/etg/*lang=nl Disallow: /hotels/etg/*lang=pl Disallow: /hotels/etg/*lang=ro Disallow: /hotels/etg/*lang=sv Disallow: /hotels/etg/*lang=bg Disallow: /hotels/etg/*lang=cs Disallow: /hotels/etg/*lang=da Disallow: /hotels/etg/*lang=el Disallow: /hotels/etg/*lang=fi Disallow: /hotels/etg/*lang=hu Disallow: /hotels/etg/*lang=no Disallow: /hotels/etg/*lang=uk Disallow: /hotels/etg/*lang=pt Disallow: /cdn-cgi/ # -- Filtered listings (faceted navigation) --------------------------- # Measured from this server's own access log, 2026-09-05 15:09 to # 2026-09-06 07:04 (16 hours, 134,165 Googlebot requests): # # clean URLs, no query 46,303 x 200 9,768 x 301 # language variants only 23,165 x 200 6,851 x 301 # date / star / district filter 6,184 x 200 35,191 x 301 # # A quarter of everything Google fetched here was a filter combination # that redirects to the page it already had. Those URLs are already # canonicalised and noindexed by the application; this stops them being # fetched at all, which is what Google recommends for faceted navigation. # Language variants are deliberately NOT listed below - they are real # alternates and they are what hreflang points at. Disallow: /*?*check_in= Disallow: /*?*check_out= # VICO-SEOFIX 2026-09-27 (owner): dated HOTEL urls are crawlable again. They answer 200 with # the canonical on the clean hotel URL (open_dates 301s to it) for every visitor, so crawlers # can re-read the dated URLs they indexed and consolidate them. Longest match wins: these beat # the two rules above for /hotels/etg/ only; the live-price rooms endpoint stays closed. Allow: /hotels/etg/*?*check_in= Allow: /hotels/etg/*?*check_out= Disallow: /hotels/etg/*/rooms?*check_in= Disallow: /hotels/etg/*/rooms?*check_out= Disallow: /*?*stars= Disallow: /*?*district= Disallow: /*?*adults= Disallow: /*?*children= Disallow: /*?*ages= Disallow: /*?*price_min= Disallow: /*?*price_max= Disallow: /*?*_dr= Crawl-delay: 5 Sitemap: https://www.vicatoo.com/sitemap.xml