Webcrawler external content connector
The Webcrawler external content connector retrieves pages and subdomains from a public website and makes their content and metadata searchable in AI Search applications. This connector can crawl content from predefined public web sources or your own custom web sources.
Note: This external content connector is not included in the External Content Connectors Application Suite application. To use this connector, you must install it separately. For details on installation, see Install External Content Connectors.
Connector administrators can run or schedule content crawls to retrieve updated content from pages and subdomains found on the selected website. Scheduled content crawls can run on a daily, weekly, or monthly basis. Content crawls feed their data to AI Search for indexing.
The indexed content and metadata are stored as records in a connector-specific indexed source. Search administrators can create search sources from this indexed source and link them to search profiles to make the indexed records searchable in AI Search applications.
Each Webcrawler connector can retrieve up to 50,000 items (URLs) from its source system when running content crawls.
Note: This is an exception to the general content crawl limit of one million (1,000,000) items.
By default, you can configure up to three Webcrawler connectors for custom web sources. If you need to retrieve items from more than three custom web sources, you can create a Customer Service and Support case at https://support.servicenow.com/now to request a limit increase for the Webcrawler connector.
- Webcrawler external content connector predefined web sources
Predefined public web sources are available for the Webcrawler external content connector. Search administrators can run Webcrawler connector crawls to make content from these web sources searchable. - Create a Webcrawler external content connector
Create an external content connector to retrieve searchable content from pages and subdomains in a public web source system. Select from a list of predefined web sources or specify your own web source. - Configure crawl settings for a Webcrawler external content connector
Specify the pages and subdomains you want your Webcrawler external content connector to retrieve from your specified web source.
Parent Topic:Configuring External Content Connectors
Related topics