Integrating external sources into a Knowledge base (RAG)

This recipe will focus on how you can integrate data, from any accessible source into your knowledge base (RAG). This allows the Sitevision Assistant module External link. to answer questions for any subjects.

By following the steps in this guide, you will build a simple RESTApp which fetches real time data from an external source. The external data will be added to the RAG database through REST-API. The recipe covers the complete workflow.

The goal of this recipe is to create a basic integration to test the workflow which should enable you to create more advanced solutions tailored to your organisation's needs down the line.

Try to build the solution yourself when reading through the cookbook recipe. The complete RESTApp is available on github External link. if you get stuck.

Level: Intermediate
Estimated time: 45min
Outcome: A working RESTApp allowing syncing with external data for a knowledge base
Id: recipe-003-external-rag-sources
Version: 1.0

Prerequisites:

Access to a Sitevision Enviroment
Understanding of how to build and deploy apps in Sitevision
Development environment configured.
Permission to add and install modules on web site.
Access to AI, Assistants and Knowledge
Sitevision 2025.09.2
Sitevision Assistant Module

Introduction Assistant

The Sitevision Assistant module External link. have been available since around Sitevision 2025.07.1. The Assistant allows you to define a database of knowledge which will be available to the AI. This way making it possible for you to tailor the Assistant to your organisation or use case. If you are unfamiliar here is one example query.

Example query for the sitevision assistant module. User asks about the highlight of the latest release. The ai answers mcp servers

Why integrate external data?

You likely have a lot of useful information on your website. However, probably not all information you need access to. Integrating external sources to a knowledge base allows you to feed all necessary information to an Assistant. Making it possible for the user to find answers to queries or locate the correct sources in on spot.

Let us begin!

Initial setup

Let's begin with a quick setup. For this exercise we assume credentials for AI and Knowledge is already set, if you need this configured check in with your admin.

assistant and knowledge setting location

First create a Knowledge Base if you do not already have one (Site Settings / Knowledge). Check the "Allow exernal data sources" checkbox. The scope can be left empty for this exercise.

Secondly, setup an Assistant (Site Settings / Assistant). Select an AI and select the newly created Knowledge base.

If you have not already, download the Assistant module from Sitevision Marketplace External link., add it to a page (select the assistant we just created in the module configuration) and publish.

How to integrate external data?

Since we allowed external sources in the knowledge base we just created, we can start posting data. Posting and deleting external sources to/from the database is done through the website's REST API, specifically the SemanticIndex endpoint.

Which means, assuming the correct permission, you would be able to post information to the Knowledge base directly from an external system.

Today in this exercise, we will build a RESTApp which allows you to post and delete information. Both on demand and a time based sync.

If you need to locate the Knowledge base node in the API, it resides in the Semantic Index Repository.

Take a couple of minutes and familarise yourself with the SemanticIndex endpoint documentation.

Deploying the RESTApp

Start with creating an initial RESTApp External link. utilizing the create-sitevision-app-script.

create-sitevision-app

npx @sitevision/create-sitevision-app external-rag-sources

Select the RESTApp alternative. Example snippets in this article is written withput TypeScript. For addon you can set external-rag-sources.

Update the manifest.json with values you see fit. For example:

manifest.json

json

{
  "id": "external-rag-sources",
  "version": "1.0.0",
  "name": "external-rag-sources",
  "author": "Sitevision AB",
  "description": "Add external RAG sources for the 003 cookbook recipe",
  "helpUrl": "https://example.com/restapps",
  "type": "RESTApp",
  "bundled": true,
  "maxUploadSizeInMB": 0
}

As per usual when working with Sitevision apps. Create an addon with npm run create-addon, then build and deploy the RESTApp with npm run dev.

RESTApp configuration

For this RESTApp we want to permit POST and DELETE requests for authenticated users. Locate the Properties / Restrictions settings for the RESTApp-addon.

In the RESTApp itselft we need to create a basic configuration. We need to define which knowledge base we want to access. In the config folder in the newly created project add an file called index.js with the following content.

config/index.js

(() => {
  const router = require("router");
  const resourceLocatorUtil = require("ResourceLocatorUtil");

  router.get("/", (req, res) => {
    const semanticIndexes = [];
    resourceLocatorUtil
      .getSemanticIndexRepository()
      .getNodes()
      .forEachRemaining((semanticIndex) => {
        semanticIndexes.push({
          id: semanticIndex.getIdentifier(),
          name: semanticIndex.getName(),
        });
      });

    res.render({
      semanticIndexes,
    });
  });
})();

A knowledge base and it's repository is reffered to as Semantic Index in the API context. We collect and map all the available semantic indexes on the site. You can easily access these nodes through ResourceLocatorUtil.getSemanticIndexRepository. External link.

In the index.html template file for the config, we simply loop the available options and add it to a custom-selector.

config/index.html

html

<div class="panel panel-default">
  <div class="panel-heading">
    <h3 class="panel-title"><%= i18n('settings') %></h3>
  </div>

  <div class="panel-body">
    <div class="form-group">
      <label for="semanticIndex"><%= i18n('semanticIndex') %></label>
      <select
        id="semanticIndex"
        name="semanticIndex"
        data-component="custom-selector"
        data-removable
        required
      >
        <% semanticIndexes.forEach(function(semanticIndex) { %>
        <option value="<%- semanticIndex.id %>">
          <%- semanticIndex.name %>
        </option>
        <% }); %>
      </select>
    </div>
  </div>
</div>

Add translations for the labels if you want. Deploy these additions and select the knowledge base in the RESTApp settings.

POST and DELETE endpoints

In the Restrictions External link. we defined that this RESTApp permits calls to POST and DELETE endpoints. Let's implement these! Locate the index.js file in the src-folder.

Middleware

Since we want to create multiple endpoints and both requires some permissions and configurations, let's setup a middleware layer to avoid duplication. If you are unfamilliar, this code will run before the actual endpoint implementations.

Run a quick check to see if the user has permission to manage AI settings on the website (another option would be to setup restrictions for the RESTApp accordingly). We'll also ensure that a Semantic Index have been selected in the RESTApp settings. The Semantic Index node is forwarded in the request data.

index.js middlwware

import router from "@sitevision/api/common/router";
import appData from "@sitevision/api/server/appData";
import permissionUtil from "@sitevision/api/server/PermissionUtil";
import resourceLocatorUtil from "@sitevision/api/server/ResourceLocatorUtil";
import MANAGE_AI from "@sitevision/api/server/PermissionUtil.Permission.MANAGE_AI";

router.use((req, res, next) => {
  if (!permissionUtil.hasPermission(resourceLocatorUtil.getSite(), MANAGE_AI)) {
    res.status(403).json({ error: "Forbidden" });
    return;
  }

  const semanticIndex = appData.getNode("semanticIndex");
  if (!semanticIndex) {
    res.status(400).json({ error: "Semantic index is required" });
    return;
  }

  req.data = req.data || {};
  req.data.semanticIndex = semanticIndex;

  next();
});

POST

Let's pause a second and explain what data we are going to collect, and how we are going to do it. For this exercise we will collect a small set of data from Sitevision Help External link., and we are going to do that by calling the Search REST-API endpoint.

We will call a Sitevision instance in this exercise, but do keep in mind you are able to call any acessible source to fetch your data.

To put it simply. By a keyword we will grab the top search result and add this to our knowledge base as external data.

We will do a super basic implementation for this exercise. Simply define an array with the desired keywords in the RESTApp.

Define the keywords-array, and initiate a basic loop in the POST-route.

index.js POST keyWords

const keywords = [
  "assistant",
  "ai",
  "mcp",
  "target audience",
  "trashcan",
  "images",
  "colors",
  "virtual groups",
];

router.post("/semanticIndexData", (req, res) => {
  const keywordData = [];
  for (const keyword of keyWords) {
    // ...
  }

  res.json(keywordData);
});

Fetch data

The first step is to fetch information. As mentioned we will call the Search REST-API endpoint for the Sitevision Help External link. site.

Call fetchDataForKeyword during the loop with the current keyword.

index.js POST call fetchDataForKeyword

for (const keyword of keywords) {
  const keyWordData = fetchDataForKeyword({ keyword });
}

The fetchDataForKeyword code will utilize Requester to make request to the external source. Since this is a public available source we can call it without any specific authentication. If you are calling a more restricted source, consult the Requester documentation about how to define authentication.

As mentioned, we include only the top result for each keyword. We also add a filter to only collect pages with english language.

The request is sent to the Search REST-API endpoint and we collect summary, title and url from the top search hit.

index.js fetchDataForKeyword

import requester from "@sitevision/api/server/Requester";

// ...

function fetchDataForKeyword({ keyword }) {
  const url =
    "https://help.sitevision.se/rest-api/1/1/Sitevision%20Help/Index%20Repository/Online/search";
  const options = {
    data: {
      limit: "1",
      query: keyword,
      filterQuery: "+svtype:page +language:en",
    },
  };

  let keywordData = null;
  requester
    .get(url, options)
    .done((data) => {
      const { summary, title, url } = data[0] || {};
      if (summary && title && url) {
        keywordData = { keyword, summary, title, url };
      }
    })
    .fail((error) => {
      console.warn(`Error fetching data for keyword "${keyword}":`, error);
    });

  if (!keywordData) {
    console.warn(`No data found for keyword "${keyword}"`);
    return null;
  }

  return keywordData;
}

Post data

In our loop we now collect some data. Next let's post it to the knowledge base.

In the loop, make a couple of additions. Call the postKeyWordDataToSemanticIndex function which will add the data to the knowledge base, and collect the retuned value in an array (which we use as the return value after a successfull request).

index.js POST postKeyWordDataToSemanticIndex

const postData = postKeyWordDataToSemanticIndex({
  keywordData: keyWordData,
  semanticIndex: req.data.semanticIndex,
});

if (postData) {
  keywordData.push(postData);
}

Declare the postKeyWordDataToSemanticIndex function. The responsibility of this function is to add the post to the knowledge base by calling the SemanticIndex REST-API endpoint.

Re-map the data we collected from Search the correct format.

Ensure that the id is unique and has a suitable format for an identifier. In this example we add a prefix and replace any whitespace with an underscore.

index.js POST postKeyWordDataToSemanticIndex data

function postKeyWordDataToSemanticIndex({ keywordData, semanticIndex }) {
  if (!keywordData) {
    return null;
  }

  const { keyword, summary, title, url } = keywordData;
  const post = {
    id: getKeywordKey(keyword),
    source: "Sitevision Help",
    text: summary,
    name: title,
    url: url,
    acl_allow: ["anonymous"],
  };

  // ...
}

function getKeywordKey(keyword) {
  return `sitevision-help-${keyword.replace(/\s/g, "_")}`;
}

When we got the data collected in the correct format it is time to post it. Call the SemanticIndex endpoint through the RestApi External link. utility from the Public API. On a successful request, return the data.

The Rest API is accessible through the RestApi External link. utility even when it is disabled (assuming you call with the correct permissions of curse). Meaning the Rest API is always accessible for internal use.

index.js POST postKeyWordDataToSemanticIndex post

import restApi from "@sitevision/api/server/RestApi";

// ...

function postKeyWordDataToSemanticIndex({ keywordData, semanticIndex }) {
  // ...

  const result = restApi.post(semanticIndex, "semanticIndex", post);

  if (result.statusCode >= 200 && result.statusCode < 300) {
    return post;
  }

  console.warn(
    `Error posting data to semantic index (${semanticIndex.getIdentifier()}): ${JSON.stringify(
      result.body
    )}`
  );

  return null;
}

DELETE

Add a simple delete endpoint aswell for easy cleanup. This simply calls the delete endpoint for the SemanticIndex.

Remember since we ran the permission check and collected the semantic index node in the middleware, we do not need to duplicate this code for this endpoint.

index.js DELETE

router.delete("/semanticIndexData", (req, res) => {
  for (const keyword of keywords) {
    restApi.delete(req.data.semanticIndex, "semanticIndex", null, {
      id: getKeywordKey(keyword),
    });
  }
  res.status(200).json({ message: "Deleted successfully" });
});

Testing

Looks like we are ready to test it out. Ensure all changes are saved and deployed.

Since the we completed all configuration earlier we simply have to call the POST endpoint of the RESTApp to run the sync. Let's do it from the console in Sitevision, but you can call it in any way you see fit.

Navigate to the RESTApp's console and submit to the POST endoint. If successfull you should see the result in the retun data.

This call looks successful, but let's have a look a look in the knowledge base as well. If you navigate to the knowledge base, you can check indexed data in the bottom panel. We ran 8 searches, so assuming all searches had hits we should find 8 posts indexed.

We got data! if you did not create a page with an Assistant module External link. earlier, create one and select the assistant we created which uses the knowledge base we now got data in.

Run a couple of test queries aligning with the data you collected.

The AI have used the information we collected from an external source to answer the question. It also provides a handy source list which the user can use to navigate to the source if needed.

Automatic sync

We got our external data integrated. The current solution however demands a call to the endpoint to re-sync, meaning we might risk the data being outdated. You most likely need to sync this data sometimes.

As mentioned, you add and delete data through a REST-API. If your external system allows you to trigger workflows or such on changes, it might be a good idea to make a new POST to the corresponding index. If you use the same id for the specific post it will be updated.

Another simple option within Sitevision is to use timer events in the RESTApp we just created.

Daily sync

We will use timer events in this RESTApp. These events are always executed with anonymous permissions. Since the requests to the SemanticIndex endpoint we are about to call requires som elevated permissions we need the privileged SDK.

To enable use of this SDK declare it in the manifest.json file of the RESTApp.

manifest.json

json

"requirePrivileged": true

With this property in place you should be able to select a Service user External link. in the Privileged actions section of the Restrictions External link. tab of the RESTApp.

For the event we define a daily one with the key sv:every-day. Also add some precaution checks to avoid unnecessary runtime errors.

index.js events sv:every-day

import events from "@sitevision/api/common/events";

// ...

events.on("sv:every-day", () => {
  if (!privileged.isConfigured()) {
    console.warn(
      "Privileged action is not configured. Skipping the update of the semantic index."
    );
    return;
  }

  const semanticIndex = appData.getNode("semanticIndex");
  if (!semanticIndex) {
    console.warn(
      "Semantic index is not configured. Skipping the update of the semantic index."
    );
    return;
  }

  // ...
});

To run a code block as the user with the correct permissions. Wrap it in the privileged.doPrivilegedAction function.

Fetch and post data the same way as before.

index.js events sv:every-day

import privileged from "@sitevision/api/server/privileged";

// ...

events.on("sv:every-day", () => {
  // ...

  privileged.doPrivilegedAction(() => {
    if (
      !permissionUtil.hasPermission(resourceLocatorUtil.getSite(), MANAGE_AI)
    ) {
      console.warn(
        "No permission to manage AI. Skipping the update of the semantic index."
      );
      return;
    }

    for (const keyword of keywords) {
      const keyWordData = fetchDataForKeyword({ keyword });
      postKeyWordDataToSemanticIndex({
        keywordData: keyWordData,
        semanticIndex,
      });
    }
  });
});

Added another precaution permission check here. Do note this was added withing the privileged block to run the permission check with the service user, and not the anonmous user which calls the actual event.

This should run once a day to resync the data. If you wish to test it you can temporarily rename the event to sv:every-5-minutes and it will trigger a bit quicker. Just remember to switch back, a very frequent re-sync is probably unnecessary.

You can check the Last updated field in the knowledge base data to confirm if updates have been made.

If you want to check it more explicitly, run a request to the delete endpoint we created first to wipe the data completely first.

Wrapping up

In this cookbook recipe we have explored how you can integrate information from external systems to a knowledge base. This information can the be accessed with the Sitevision Assistant module allowing users a simple way to interact with the content.

This is just a basic example which fetches a few basic publically accessible sources. Do note that you should be able to fetch and add pretty much any text content you are able to access. We used Requester in this example to call Sitevision Help, this could realistically be used to call any source you have permissions to access.

While it is simple to post information to the Knowledge, do ensure that information is kept up to date. Once it is posted it will remain in the Knowledge base. If the source material changes, it needs to be updated.

Resources

create-sitevision-app External link.
Assistant module documentation External link. / marketplace External link.
Example code available on github External link.
RESTApp documentation
SemanticIndex endpoint
Search endpoint
Events
Requester
RestApi External link. utility