Knowledge sources

Crawling your website

The usual starting point. Give the project a sitemap URL — normally https://yoursite.com/sitemap.xml — and we resolve it, following sitemap index files to the sitemaps they point at.

You get a preview before anything is indexed: the list of URLs found, grouped by section, so you can exclude the parts you do not want. A single ingest is capped at 500 URLs, which is why the section filter exists — a large site’s cart, login and pagination URLs will otherwise fill that budget with pages nobody asks about.

You can also paste a plain list of URLs instead of a sitemap, if you would rather pick the pages yourself.

Pages that need JavaScript

Each page is fetched and reduced to its readable article text — navigation, footers and boilerplate are stripped out. Pages that render their content client-side are re-fetched through a real browser so the text is there to extract. This is the step most crawlers skip, and it is why a single-page app usually indexes correctly here.

Re-crawling

Re-crawl a single page, every failed page, or the whole project. We hash the extracted text, so a page that has not changed since the last crawl is skipped rather than re-embedded — re-crawling a mostly-static site is cheap.

Pasted text

For everything that is true about your business but written down nowhere public: shipping exceptions, the answer you give on the phone five times a day, policies that live in an email thread.

Pasted text is indexed immediately — there is nothing to fetch. It can be edited in place afterwards, which re-indexes it.

Uploaded documents

.txt, .md and .docx files, up to 10 MB each. The text is extracted and indexed the same way a crawled page is.

PDF is not supported yet. If your content is in a PDF, exporting it to .docx is currently the way in.

What content costs

Every source is measured in credits, and one credit is 500 words of source text. That number is the median length of a real crawled page, so one URL and an equivalent stretch of an uploaded document cost the same.

Credits measure capacity, not consumption. They count what currently exists in your account, they never reset with the billing period, and deleting a source gives its credits back. A plan’s credit allowance is a budget of how much content your account holds at once — not a number of pages, files, or questions. See Plans and limits for the figures.

Because a page’s length is unknowable until it has been fetched, a crawl that passed the pre-check can still finish slightly over your ceiling. A small overshoot is absorbed rather than thrown away; genuine excess is refused and the source is marked as over limit.

Sources too short to index

Anything under 20 words is refused rather than indexed. Below that length a source produces no chunks at all, so it could never be retrieved or cited — charging credits for it would be billing for something the bot cannot use.

This is also why a thin page — a contact form, a gallery with no prose — may crawl successfully and still never turn up in an answer.

Source status

• Pending / crawling — queued or being fetched.
• Done — text extracted and embedded. Answerable.
• Failed — the fetch or extraction did not work. Retry it, or see Troubleshooting.
• Over limit — the content exists but was not indexed, because the account is out of credit allowance. Free some by deleting sources, or upgrade, then retry indexing.