Custom Index with External data
The Search Enterprise license allows Custom Indexes to be populated with external data. External data can come from any source, as long as the function/crawler that submits the data ensures the data matches the scheme of the Custom Index. The function/crawler that populates external data is responsible to keep all external data in sync (add, update, remove). The Sitevision Indexer will only manage indexing of (internal) Sitevision data.
Search Enterprise license is required to manage external data in a Custom Index.
The function/crawler that populates a Custom Index with external data is also responsible to keep the external data in sync (add, update, remove).
Open up the Custom Index for External data
Custom Indexes are completely internal by default. The index can only be accessed from within the Sitevision runtime. This means read-only access (query access) via a Sitevision Search module or via Searcher of the Sitevision Public API.
Allowing management (create/update/delete) of external data for a Custom Index requires two steps.
First step is to set up the user account that is permissioned to do http requests that targets Custom Indexes of the site. The "Crawler user" is set up in the Search category of the website settings.
Designated Crawler user in set up in the website settings
Second step is to "open up" the designated Custom Index for external access by the "Crawler user". This is activated on the Pipeline tab in the configuration of the Custom Index.
External data management for the Custom Index is activated on the Pipeline tab
The default pipeline is also a good thing to activate. It does some background transformations to ease field matching for external data. It is also a must when using the Sitevision Crawler that relies on these default conveniences.
The Custom Index will accept http requests from the "Crawler user" when both steps above are done. External data can now be managed by the Sitevision Crawler or a custom solution.
External data must match the index scheme
Arbitrary data can be submitted to the Custom Index, but only the key/value entries that actually match the scheme of the index will stick and be persisted. The data entry key must match the name of a index field and its value must match the type of the index field.
The index fields (a.k.a "the scheme") of the index is available on the Field tab in the configuration of the Custom Index.
The index fields defines the scheme of the Custom Index
New fields can be added to hold additional external data. For instance, a field with the purpose of make indexing more precise for certain cases (e.g. "index embedded file metadata X for pdf files in field "my-pdf-x-field").
Mutation examples
The name of the Custom Index is vital since it is used in all http requests that targets the index.
Name is configured in the first tab of the Custom Index
This Custom index is named "myindex" so we will use that in all examples. We use Curl for easy understanding and simple illustration.
Add or update an external data entry
All mandatory fields must be included in the submitted data and the primary key for the data must always be included (by default this is the "id" field of the index).
A index document is added/updated via the update endpoint, using the add operation and specifying all data in the doc section:
The acl_allow and anonaccess fields
These fields determines the permission for the indexed document. The pipeline will populate the fields for "public access" state, If no such entries are included in the submitted data.
The lastindexed field [@since 2023.09.1]
The indexing process will set the timestamp data in the lastindexed field for external data (if not explicitly provided).
The svtype field
The pipeline will demarcate all external data as "external" in the svtype field.
The svtypeval field
The pipeline will calculate the value from the data in the svtype field.
The result of the submitted external data
Remove the external data entry
A index document is removed via the update endpoint, using the delete operation: