Key ecosystem metrics across 5,000+ websites using Agent Analytics and AI Chat Referral Tracking.
↓ 3%
Compared to the previous 90 days
The percentage of bot traffic vs. human traffic
↑ 11%
Compared to the previous 90 days
The percentage of bot traffic that's AI-related
↓ 11%
Compared to the previous 90 days
The percentage of human website visits that come from AI chat
See Breakdown ↓
The amount that robots.txt reduces visits from blocked agents
See Breakdown ↓
AI Agent
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
AI Coding Agent
AI Coding Agent
Fetches documentation and other resources to help build software
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
Archiver
Archiver
Captures and stores historical website snapshots for long-term digital preservation
Automated Agent
Automated Agent
Automates browser interactions programmatically without direct human supervision
Developer Helper
Developer Helper
Assists with testing, debugging, and ensuring website functionality
Fetcher
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
Security Scanner
Security Scanner
Scans websites for security vulnerabilities, threats, and configuration weaknesses
SEO Crawler
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
Uncategorized
Uncategorized
Not yet assigned a type
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
Hover over each agent type for more information about what they do
Agent types with the most activity
bingbot
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
8.2%
Googlebot
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
7.9%
AhrefsBot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
6.0%
Known Agent
DEV
Developer Helper
Assists with testing, debugging, and ensuring website functionality
5.5%
ChatGPT-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
3.3%
PetalBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
3.2%
ClaudeBot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
3.1%
SemrushBot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
3.0%
facebookexternalhit
FTCH
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
2.6%
Amazonbot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
2.3%
meta-externalagent
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
2.3%
MJ12bot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
2.2%
DotBot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
2.2%
Amzn-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
2.1%
Applebot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
2.1%
Agents with the most activity
Operators with the most activity
These bots scrape website content to train AI models. Some belong to AI companies, while others belong to third-party services that resell the data. Automatic Robots.txt can block unwanted scraping. Included agent types include AI Data Scrapers and AI Data Providers.
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
AI scraping activity by agent type over time
Computers and Electronics
5.1%
Business and Industrial
4.9%
Internet and Telecom
4.3%
Website categories with most activity
ClaudeBot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
26.7%
Amazonbot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
19.8%
meta-externalagent
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
19.8%
GPTBot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
9.5%
Bytespider
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
7.6%
ShapBot
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
5.2%
GoogleOther
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
3.9%
CCBot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
1.9%
Timpibot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
1.9%
YouBot
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
1.1%
VelenPublicWebCrawler
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
0.8%
DeepSeekBot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
0.4%
TerraCotta
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
0.3%
Diffbot
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
0.2%
ExaBot
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
0.2%
Agents doing the most AI scraping
Operators doing the most AI scraping
These bots fetch website content in real time to power AI assistants, coding agents, and other retrieval-augmented generation (RAG) tasks. Pages inform responses on the spot, such as when an assistant summarizes an article or a coding agent references documentation. Included agent types include AI Coding Agents and AI Assistants.
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
AI Coding Agent
AI Coding Agent
Fetches documentation and other resources to help build software
AI fetching activity by agent type over time
Computers and Electronics
2.3%
Travel and Transportation
1.8%
Business and Industrial
1.7%
Internet and Telecom
1.7%
Website categories with most activity
ChatGPT-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
88.8%
DuckAssistBot
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
3.4%
Claude-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
1.9%
Perplexity-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
1.8%
Claude-Code
CODE
AI Coding Agent
Fetches documentation and other resources to help build software
1.1%
MistralAI-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
1.0%
Shap-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
1.0%
Google-NotebookLM
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.4%
Cursor
CODE
AI Coding Agent
Fetches documentation and other resources to help build software
0.4%
Gemini-Deep-Research
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.1%
meta-externalfetcher
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
GoogleAgent-URLContext
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
Code
CODE
AI Coding Agent
Fetches documentation and other resources to help build software
0.0%
Amzn-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
QualifiedBot
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
Agents doing the most AI fetching
Operators doing the most AI fetching
These bots crawl website content so it can be surfaced in AI search engines and AI-generated answers. Those answers often include citations or links back to the source pages. Included agent types include AI Search Crawlers.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
AI search indexing activity by agent type over time
Travel and Transportation
6.8%
Books and Literature
4.8%
Arts and Entertainment
4.5%
Computers and Electronics
4.5%
Website categories with most activity
PetalBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
25.9%
Amzn-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
17.0%
Applebot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
16.9%
OAI-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
11.9%
meta-webindexer
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
11.0%
Claude-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
10.0%
LinkupBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
3.7%
PerplexityBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
2.9%
Google-CloudVertexBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.3%
AzureAI-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.2%
xAI-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.2%
AddSearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
Anomura
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
amazon-kendra
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
atlassian-bot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
Agents doing the most AI search indexing
Operators doing the most AI search indexing
These bots use browsers to autonomously navigate websites, click through pages, and make decisions to complete tasks for people. Agentic UX best practices and Google PageSpeed Insights help evaluate how well websites support them. Included agent types include AI Agents.
↓ 24%
Compared to the previous 90 days
The average duration of a session
↓ 14%
Compared to the previous 90 days
The average number of pages visited per session
AI Agent
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
AI browsing activity by agent type over time
Internet and Telecom
0.0%
Travel and Transportation
0.0%
Business and Industrial
0.0%
Computers and Electronics
0.0%
Books and Literature
0.0%
Arts and Entertainment
0.0%
Website categories with most activity
Manus-User
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
55.0%
ChatGPT Agent
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
23.2%
Google-Agent
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
21.8%
NovaAct
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
GoogleAgent-Mariner
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
AmazonBuyForMe
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
TwinAgent
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
Agents doing the most AI browsing
Operators doing the most AI browsing
See which robots.txt rules are set across the web and how well agents follow them. An agent's Robots.txt Compliance Score measures the efficacy of a disallow rule for it by estimating how much the agent reduces its traffic after it's blocked.
serpstatbot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
100.0%
AhrefsSiteAudit
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
100.0%
LinkupBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
100.0%
ClarityBot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
100.0%
proximic
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
IAS crawler
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
um-IC
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
Barkrowler
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
100.0%
SiteCheckerBotCrawler
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
100.0%
SEOkicks
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
100.0%
Dragonfly
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
SiteAuditBot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
100.0%
AdsBot-Google-Mobile
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
meta-externalads
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
UptimeRobot
DEV
Developer Helper
Assists with testing, debugging, and ensuring website functionality
100.0%
Agents with the highest Robots.txt Compliance Scores
Baiduspider
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
82.4%
ShapBot
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
90.3%
CriteoBot
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
92.7%
Scrapy
SCRP
Scraper
Extracts large amounts of web data, often without explicit website permission
92.9%
Amzn-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
93.8%
archive.org_bot
ARCH
Archiver
Captures and stores historical website snapshots for long-term digital preservation
95.2%
Mediapartners-Google
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
95.7%
linkfluence
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
95.7%
OAI-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
95.8%
Amazonbot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
96.2%
meta-webindexer
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
96.7%
DotBot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
96.8%
ChatGPT-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
96.9%
SemrushBot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
97.1%
Applebot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
97.2%
Agents with the lowest Robots.txt Compliance Scores
See which AI platforms like ChatGPT, Perplexity, and Gemini cite websites and send them human referral traffic. Citations are estimated. Google's guide explains how websites can optimize their content to be more visible in AI chat responses (GEO).
Computers and Electronics
2.3%
Travel and Transportation
1.8%
Business and Industrial
1.7%
Internet and Telecom
1.6%
Website categories most frequently cited in AI chat responses
Travel and Transportation
0.3%
Computers and Electronics
0.1%
Business and Industrial
0.1%
Internet and Telecom
0.0%
Website categories receiving the most referrals from AI chat
The Index is updated daily with completed days of traffic and referral data from more than 5,000 websites using Agent Analytics and AI Chat Referral Tracking. The current partial day is excluded. Percentage-change tags compare the current period with the preceding period of the same duration. Agent names, operators, and classifications come from the Agent Directory, which is updated as new agents are discovered or existing agents change. Website categories follow the taxonomy used by Google AdSense.
Only websites meeting minimum activity and data-quality requirements are included. Internal, test, incomplete, or anomalous data is excluded. Metrics use website-level accounting, giving each website equal weight regardless of traffic volume. This prevents a small number of high-traffic websites from dominating the results. Daily charts are not smoothed, allowing normal seasonality to remain visible.
An agent's Robots.txt Compliance Score estimates how effectively robots.txt rules reduce activity from that agent across websites. It compares the agent's activity across websites where it is blocked with its activity across websites where it is not blocked, with safeguards for data quality, traffic differences, and agent identity verification. Top Blocked Bots is calculated separately using daily robots.txt scans of Similarweb's top 1,000 websites.
AI chat referral statistics are based on directly observed traffic. Citation statistics are estimates based on observed agent activity because AI platforms do not provide a complete public record of the sources included in their responses.
Absolutely. You may cite The Agentic Web Index with attribution and a link to this page. For interviews, fact-checking, background context, or a more specific breakdown for a story, contact us and include your deadline.
Absolutely. We welcome thoughtful research into how agents and bots are changing the web. Tell us about your research question, timeframe, and intended use. Depending on the scope and data constraints, we may be able to provide additional context, compare approaches, or explore a joint analysis.
Yes. If you need a breakdown by agent, operator, activity type, website category, or time period that is not shown here, contact us. When the underlying data supports it, we can examine the question and provide a focused analysis.