User-agent: * Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Allow: / # Crawl-delay for AI bots User-agent: GPTBot Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Crawl-delay: 2 Allow: / User-agent: CCBot Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Crawl-delay: 2 Allow: / User-agent: ChatGPT-User Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Crawl-delay: 2 Allow: / User-agent: anthropic-ai Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Crawl-delay: 2 Allow: / User-agent: Claude-Web Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Crawl-delay: 2 Allow: / # Google bots User-agent: Googlebot Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Allow: / Crawl-delay: 0 User-agent: Googlebot-Image Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Allow: / User-agent: Googlebot-Mobile Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Allow: / # Bing bot User-agent: Bingbot Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Allow: / Crawl-delay: 1 # Yandex User-agent: Yandex Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Allow: / Crawl-delay: 2 # Common bots User-agent: DuckDuckBot Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Allow: / User-agent: Baiduspider Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Allow: / Crawl-delay: 2 Sitemap: https://vidhyagen.com/sitemap.xml