Knowledge base through webpages

The webpages data source allows you to gather and store content from various webpages. Your AI agent uses this collected information to answer customer questions effectively.

Configuration

Below are the parameters you can use to configure the ingestion process:

Full website

This knowledge base starts with a seed URL. Seed URLs are the starting points for your crawl. They act as the base addresses from which the crawler begins exploring links. The crawler visits only those URLs that match the seed URLs or belong to their subdirectories.

For example, if the seed URL is https://example.com/, the crawler explores that page and all its linked subpages such as https://example.com/blog/, https://example.com/about-us/, etc.

Excluded URLs

Excluded URLs are those you want the crawler to ignore. You can specify them using the same format as seed URLs. The crawler does not visit any URLs that match the excluded URLs or belong to their subdirectories.

For example, if you exclude https://example.com/blog/, the crawler visits neither that page nor any pages under the blog directory.

Single URLs

Single URLs are pages you want the crawler to visit without following any links from them.

Sitemap URLs

Sitemap URLs enable the crawler to fetch a list of pages to visit. For example, https://example.com/sitemap.xml can be used to locate multiple URLs for ingestion.

Excluded assets

The crawler ignores certain assets such as images, CSS, JavaScript files, and PDFs. The crawler ingests only HTML pages, so it ignores every other file type, for example:

• PNG
• JPG
• JPEG
• GIF
• PDF
• CSS
• JS

Pages that the crawler skips

The crawler fetches a page but does not add it to the knowledge base in these cases:

• The page has fewer than 40 words of text after Moveo removes the HTML.
• The page has a noindex robots meta tag.
• The page looks like an error page, for example a "page not found" page that returns HTTP 200.

The ingestion report lists the skipped pages. A crawl stops at 100 pages by default.

Scheduled crawls and drafts

Moveo crawls the website again about every 24 hours, on these conditions:

• The account is on the Enterprise plan. Other plans do not get scheduled website crawls.
• An AI agent used the knowledge base in the last 30 days.

Pages that a crawl finds for the first time arrive as Draft. Publish them before the AI agent uses them. A page that already exists keeps its status: a published page stays published, and a draft page stays a draft. See Publish.

Add a webpage

1. Navigate to Knowledge in the top navigation bar.
2. Select the knowledge base you want to add a webpage to, or create a new knowledge base as described in the Knowledge bases overview.
3. Navigate to the Webpages tab.
4. Click the Add button to toggle the Add webpages sidebar.
5. Select which method of ingestion you want to use.
6. Type the full URL of the webpage you want to add.
7. Click Add.

Ingestion report

The ingestion report shows the result of the last crawl of a website: when it started and ended, how long it took, and how many URLs Moveo uploaded, excluded, or could not fetch.

1. In the Webpages tab, select the three dots icon of the website.
2. Select View report. The Webpages Report panel opens.

Option

Report

Troubleshooting

CDN or WAF blocking the crawler

If your website is protected by a CDN or WAF (Web Application Firewall) such as Cloudflare, Akamai, or AWS CloudFront, the Moveo crawler may be blocked from accessing your pages. To resolve this, allowlist the Moveo crawler's IP addresses in your security settings.

Moveo crawler IP addresses depend on the region your account is hosted in. See Outbound IP addresses for the full list, then allowlist the addresses for your account's region (or all of them if you are unsure).

Cloudflare

1. Log in to your Cloudflare dashboard.
2. Select your domain.
3. Go to Security → WAF → Tools.
4. Under IP Access Rules, add the Moveo crawler IP addresses for your region with the action set to Allow.

Akamai

1. Log in to the Akamai Control Center.
2. Navigate to Security → IP/Geo Firewall (or your WAF configuration).
3. Add the Moveo crawler IP addresses for your region to your allowlist.
4. If using Bot Manager, create an exception rule for these IPs under Transactional Endpoints or Custom Bot Categories.

AWS CloudFront / WAF

1. Open the AWS WAF console.
2. Create or edit an IP set with the Moveo crawler IP addresses for your region.
3. Add a Rule to your Web ACL that matches the IP set and sets the action to Allow.
4. Ensure this rule has a higher priority than any blocking rules.

Other

If you use a different WAF or CDN provider (Imperva, Fastly, etc.), add the Moveo crawler IP addresses for your region to your allowlist or create an equivalent bypass rule.

The crawler indexes no content

If the ingestion report shows no documents, check these causes in order:

1. A CDN or a WAF blocks the crawler. See CDN or WAF blocking the crawler.
2. The pages have fewer than 40 words of text each, or have a noindex tag. Add the content as documents or as Q&As instead.
3. The site needs a login. The crawler cannot sign in.

Pages on another path are missing

The crawler follows only links that match a seed URL or sit under its path. For example, with the seed URL https://example.com/blog/, the crawler skips https://example.com/articles/post-1. To fix it, add the other path as a second seed URL, add the pages as Single URLs, or use a Sitemap that lists them.

Frequently asked questions

1. What file types are excluded during ingestion?
   The crawler ingests only HTML pages. It ignores other files, such as PNG, JPG, JPEG, GIF, PDF, CSS, and JS files.

2. Can I force the crawler to follow links from Single URLs?
   No. Single URLs are crawled in isolation; the crawler does not follow any further links from them.

3. Is there a way to view the status of my crawl?
   Yes. Open the Ingestion Report, which outlines the process duration, URL issues, and any potential errors.

4. How can I block the crawler from certain sections of my site?
   You can list the paths or pages to be excluded under Excluded URLs. The crawler ignores any URLs or subdirectories specified there.

5. Can I use a sitemap for only one part of my webpage?
   Absolutely. Point the crawler to any sitemap URL relevant to the sections you want to crawl.

6. How do I resolve issues with my CDN or WAF blocking the crawler? Allowlist the Moveo crawler IP addresses in your CDN or WAF settings. See the Troubleshooting section above for step-by-step instructions.

7. What happens if a URL is both in Seed URLs and Excluded URLs?
   Excluded URLs take precedence. Any URL listed in Excluded URLs (or its subdirectories) is not crawled.

8. How can I integrate this with a site protected by login credentials?
   At the moment, Moveo's crawler does not support authentication. You must provide publicly accessible URLs to be crawled.

9. Does Moveo process JavaScript on my webpage?
   Yes. The crawler renders a page in a browser when the page needs JavaScript to show its content.

Next steps

• Publish
• Test knowledge base
• Knowledge base best practices