Crawls & Probes
How IndexMind collects data about your site and AI visibility.
IndexMind uses two complementary data collection methods to build a complete picture of your AI visibility: Crawls and Probes. Crawls analyze your website directly. Probes test how AI models respond to queries about your brand and keywords. Together, they provide both the "what's on your site" and "what AI says about you" perspectives.
Crawls
A crawl is a systematic scan of your website. When IndexMind crawls your site, it visits your pages and collects the technical and content signals that determine how AI-friendly your site is.
What a Crawl Collects
- robots.txt: Your crawler directives, including which paths are allowed or blocked for various user agents. IndexMind checks whether AI-specific crawlers (such as GPTBot, ClaudeBot, or PerplexityBot) are permitted or blocked.
- Sitemap: Your XML sitemap is parsed to discover all pages you want indexed. IndexMind checks for completeness, validity, and whether the sitemap accurately reflects your live site.
- Page content: The visible text on each page, including headings, paragraphs, lists, and other content elements. This is analyzed for quality, depth, and relevance.
- Metadata: Title tags, meta descriptions, Open Graph tags, canonical URLs, and other metadata that helps AI systems understand page purpose.
- Schema markup: All structured data present on your pages, including JSON-LD blocks, microdata, and RDFa. IndexMind validates this data and assesses its completeness.
Crawl Results
After a crawl completes, IndexMind recalculates your AI Optimization Score and updates all Four Pillars. You can view detailed crawl results on your dashboard, including page-by-page breakdowns and specific issues found.
Crawl Frequency
You have two options for when crawls run:
- Manual crawls: Trigger a crawl at any time from your dashboard. This is useful after making site changes when you want to see the impact immediately.
- Scheduled crawls: IndexMind automatically crawls your site on a recurring schedule based on your plan tier. This ensures your data stays current without manual intervention.
Probes
A probe is a query sent to one or more AI models to test whether and how they mention your brand, products, or services. Probes answer the question: "When someone asks an AI about my industry or offerings, do I show up?"
How Probes Work
- You define target keywords or questions relevant to your business (e.g., "best CRM for startups" or "top project management tools").
- IndexMind sends these queries to AI models.
- The AI responses are analyzed for mentions of your brand, citations to your website, sentiment, and competitive positioning.
- Results are surfaced in your Citations, Sentiment, and Competitor Analysis views.
What Probes Reveal
- Whether AI models mention your brand at all for a given query
- How your brand is described (positive, negative, or neutral framing)
- Whether the AI links to or cites your website
- How you compare to competitors in the same response
- What information the AI gets right or wrong about you
How Crawls and Probes Work Together
Crawls and probes serve different but complementary purposes:
| Crawls | Probes | |
|---|---|---|
| What they examine | Your website | AI model responses |
| What they measure | How AI-friendly your site is | How visible you are in AI answers |
| Primary output | AI Optimization Score, Four Pillars | Citations, Sentiment, Competitor data |
| Analogy | An audit of your storefront | A mystery shopper asking about you |
Crawl data tells you what you can improve on your site. Probe data tells you whether those improvements are translating into actual AI visibility. The most effective workflow is to review probe results to identify gaps, make site improvements based on crawl recommendations, then re-probe to verify the impact.
Probe Limits by Plan
Each plan tier includes a monthly probe allocation:
| Plan | Probes per Month |
|---|---|
| Free | 5 |
| Starter | 250 |
| Pro | 2,000 |
Probes reset at the beginning of each billing cycle. Unused probes do not roll over. You can monitor your remaining probe count on your dashboard.
Tip: Use your probes strategically. Start with your highest-priority keywords and expand from there. Each probe provides data across citations, sentiment, and competitive positioning, so even a small number of well-chosen probes can yield significant insights.