Sitevision Crawler

Sitevision Crawler is used to populate a Custom Index with the external data, allowing users to search for the external information they need more easily. It can be an important component to complement your web search results with more information.

This function requires license feature "Search Enterprise".

Sitevision Crawler is a Nutch-based crawler that browses URLs that have been configured for indexing. It follows links on visited web pages and adds them to its internal crawling schema.

Sitevision Crawler is a standalone software that is installed and managed on a Linux server. It should typically not be running on the same server as your Sitevision installation.

To install and start using the Sitevision Crawler, the following steps and recommendations should be followed:

  1. Preparations in Sitevision
  2. Installation of the Crawler
  3. The contents of a crawl directory
  4. Filtering in the context of indexing
  5. Indexing sources that require login