Choose How Much of Your Docs to Crawl – Page, Section, or Whole Site

When you add a documentation URL to Usedocs, you now decide how far the crawler should go:

Crawl scope	What gets indexed	Typical use‑case
Page	Only the single page you entered	Quick test, single‑article FAQ
Section	All pages that share the same path prefix (e.g., everything under `/help/` or `/api/`)	A specific product or feature area
Site	The entire site – the crawler follows the sitemap (if present) and all internal links	Full‑scale onboarding of a brand‑new docs site

The crawler also deduplicates pages automatically and skips common “app‑shell” URLs such as /login, /signup, or other non‑content pages.

How to set the crawl scope

1. Adding a new source (Dashboard → Knowledge)

Click Add source and paste the URL of the docs you want to index.
Below the URL field you’ll see a Crawl scope dropdown (it may be pre‑filled).

This page – index only the page you entered.
This section – index everything under the same path prefix.
Entire site – let the crawler discover the whole site (uses the sitemap if it exists).

Optionally hover the info icon next to the dropdown for a short hint about each option.
Click Crawl & embed.

If you leave the dropdown empty, Usedocs will infer the best scope from the URL:

URLs ending with sitemap.xml, sitemap_index.xml, or containing the word “sitemap” → Site
The root path (/) → Site
Anything else → Section

You can always change the selection before you start the crawl.

2. Onboarding wizard (first‑time setup)

During the onboarding wizard the same Crawl scope selector appears after you type the docs URL. Choose the appropriate level, then continue with the wizard as usual.

What happens behind the scenes

Scope is sent to the backend – the API now accepts an optional scope field (page, section, or site).
De‑duplication – before a page is indexed, Usedocs checks whether the exact URL (or a URL already covered by an existing source) has been indexed. If it has, the page is skipped, preventing duplicate chunks.
App‑shell skipping – URLs that look like login, signup, or other non‑content pages are automatically ignored, even if they appear in a sitemap.
Status feedback – after you start a crawl you’ll see a status line such as “Crawl started — discovering pages…” or “Indexed 12 pages → 48 chunks.”

When to use each scope

Scope	When it’s the right choice
Page	You only need a single article (e.g., a legal disclaimer or a one‑off FAQ).
Section	You have a distinct product area, version, or language that lives under its own URL prefix.
Site	You’re onboarding a brand‑new docs site or you want the most comprehensive coverage. Expect a longer crawl time for large sites.

Caveats & best‑practice tips

Scope overlap – Adding a new source that is already

_Drafted from merged PR: PR: feat(crawl): page, section, and site scope with de-dupe. Review before publish._