Meta, the company behind Facebook, Instagram, Messenger, Threads, and WhatsApp, has been crawling the web at a pace that looks a lot like the early days of a search engine. In early August, independent developer Pieter Levels posted on X that Meta staff had told him the company was allegedly building its own search index. Levels also shared screenshots of Meta crawling his sites hard enough to trigger server load alerts.
The reasoning is simple enough. When Meta AI needs live information, it relies on outside search providers, and every query it sends to Google gives Google something to learn from. Owning the web index would keep that data in-house, and so Meta may be preparing to reduce its reliance on Google and Microsoft Bing for search. Search Engine Roundtable picked up the story, noting that Facebook has chased search for more than a decade, including a Bing partnership it later dropped.
Meta’s AI Crawl Volume Increased
Meta hasn’t confirmed a search engine, but the crawl data suggests something significant is under way. Meta-WebIndexer is the crawler that feeds Meta AI’s search results, and allowing it helps Meta AI cite and link to your content. It’s separate from Meta-ExternalAgent (active since 2024), which collects content to train Meta’s AI models and improve its AI products. Both agents are crawling and indexing websites at a much higher rate right now:
- Security firm DataDome recorded 9.1 billion requests from Meta’s AI crawlers in Q2 2026, with Meta-WebIndexer’s quarterly requests rising from 1.4 billion to 3.75 billion
- In June, Meta-WebIndexer overtook Meta-ExternalAgent in monthly volume for the first time, suggesting a shift from gathering training data to building a live index
- AI visibility platform Promptwatch saw Meta-WebIndexer’s share of the AI crawler traffic it tracks jump from around 2% in mid-July to nearly 38% by 9 August.
This fits with earlier reporting. The Information said in 2024 that Meta wanted its own web index to reduce its reliance on Google and Bing.
Meta’s AI Search Index
Search has been expanding for a while. Google’s AI Overviews, ChatGPT, and Perplexity all draw on the open web to answer questions, and Meta could soon do the same at huge scale. Meta AI passed a billion monthly users in 2025, and it’s built into apps your customers open every day. If someone asks Meta AI on WhatsApp for a recommendation, the answer could come from Meta’s own index – another AI system deciding which content should be surfaced and a new route for customers to discover brands while researching products or services.
We’d keep expectations measured, though. DataDome found Meta’s crawlers currently send very little traffic back to the sites they visit, so for now this is about being present in the answer rather than winning the click. The good news is you don’t need a separate playbook. Clear structure, accurate information and well-implemented schema help you in Google, and they’re the same things AI systems lean on when deciding what to cite.
Next steps?
SEO and GEO (Generative Engine Optimisation) work hand in hand and we’re continuing to focus on the fundamentals, regardless of which AI system is doing the crawling:
- Technically accessible websites
- Genuinely useful insights and expertise
- Accurate and quotable information
- Strong authority signals
- Well-structured and correctly formatted content.
Start with your robots.txt file, the small text file that tells crawlers what they can access. Plenty of sites block AI crawlers wholesale, sometimes without realising, because a plugin or hosting setting did it for them. A blanket block on Meta’s bots could keep you out of Meta AI’s answers entirely. Treat training and indexing as two separate decisions. You might let Meta-WebIndexer in so you can be cited, while blocking Meta-ExternalAgent if you’d rather your content didn’t train models. There’s no single right answer – think about what’s right for your business.
The Final Word
Meta hasn’t launched anything official, but it’s clearly laying the groundwork. Meta AI could be yet another change in how people discover brands online. If you want to assess your AI visibility and explore where your brand is appearing across AI search, we’re always happy to chat. Get in touch and let’s start a conversation.