Sitevision Crawler

Sitevision Crawler is used to populate a Custom Index with the external data, allowing users to search for the external information they need more easily. It can be an important component to complement your web search results with more information.

This function requires license feature "Search Enterprise".

Sitevision Crawler is a Nutch-based crawler that browses URLs that have been configured for indexing. It follows links on visited web pages and adds them to its internal crawling schema.

To install and start using the Sitevision Crawler, the following steps and recommendations should be followed:

  1. Preparations in Sitevision
  2. Installation of the Crawler
  3. The contents of a crawl directory
  4. Filtering in the context of indexing
  5. Indexing sources that require login