Indexing

The process of extracting and providing data to an index is called Indexing. There are two indexing processes in Sitevision:

Eventual consistency

When a Sitevision object is mutated, the object will be added to an Indexing Queue. The Queue is processed asynchronously by the indexing process.

First step of the processing is to extract new indexable data (the Event phase). Next step is to use the extracted data to update all affected indexes with fresh data (the Data phase).

The first step is executed on the local Sitevision cluster node where the mutation occurred. The second step is propagated and executed on all individual Sitevision cluster nodes. In other words - it will take a moment before all indexes in the Sitevision cluster are updated. Typically within a minute but when the workload is high it might take longer.

Queue overview

The indexing queues are exposed for each cluster node in the admin UI (Note! Requires System user authentication).

The Event queue section shows the local queues for the Event phase of the indexing process. The Data queue section shows the shared, cluster-propagated Data phase of the indexing process.

Indexing queues for a cluster node in admin

The Indexer User

When the Node indexing process extracts new data (e.g. renders a page that was just published), it will do so as the Indexer User.

The Indexer User is of type sv:systemUser and there are some methods in SystemUserUtil that might be helpful if you want to provide custom data to indexes in your Script module or WebApp.

var systemUserUtil = require('SystemUserUtil');
 
if (systemUserUtil.isIndexer()) {
   /* Render custom data for indexing */
} else {
   /* Render data to other users */
}

Excluding blocks of content

A Script module or WebApp that explicitly wants to exclude data from being indexed by Sitevision can use the check-if-Indexer trick (see above) but a much simpler way is by adding specific sv-no-index HTML comments to the rendered output.

  • Start excluding comment: <!--sv-no-index-->
  • Stop excluding comment: <!--/sv-no-index-->

The exclusion must include both comments to be valid.

<p>
   Some text
</p>
   
<!--sv-no-index-->
<h3>
   Legal agreement
</h3>
<p>
   Some legal stuff...
</p>   
<form action="/agreements" method="post">
	<label for="name">Name: </label>
	<input type="text" name="name">
	<button type="submit">Agree</button>
</form>   
<!--/sv-no-index-->
   
<p>
   More text
</p>