Collection Data Store

A collection data store is simply a collection of items. Data stored in collection data stores is indexed to make it easy to work with.

API

The data storage API is available in WebApps, RESTApps, Widgets and MCP Servers.

Example of how to get a collection datastore from any server-side JS-file

// retrieve an instance of the storage API
import storage from '@sitevision/api/server/storage';

// get a collection store
// if no collection store exists with the provided name, one will be created
// (note: the name must also be explicitly registered in the app's manifest.json)

const collectionDataStore = storage.getCollectionDataStore('myCollectionStore');

API variants [@since 5.2]

The data storage API is available in two variants. One is callback-based whereas the other is callback-free. Please note that both variants execute synchronously, which probably makes the callback-free approach easier to work with.

Switch to callback-free API documentation

add(data, callback)

Adds an item to a collection

Argument	Description
`data`	JSON data to store
`callback`	Called after execution. The first parameter of the callback will contain an error object if an error occurred, `undefined` otherwise. The second parameter will contain the stored data.

collectionStore.add with callback example

const person = {
   name: 'Foo',
   age: 33
};

collectionDataStore.add(person, (err, data) => {
   if (err) {
      // handle error
   }
   // data --> 
   {
      dsid: '00e40be0-59c7-11e9-9d84-7a4f433f610f', 
      name: 'Foo', 
      dstimestamp: 1554704685982, 
      age: 33
   }
});

addAll(data, callback)

Adds an array of items to a collection

Argument	Description
`data`	An array of objects to store
`callback`	Called after execution. The first parameter of the callback will contain an error object if an error occurred, `undefined` otherwise.

collectionStore.addAll with callback example

const people = [
   {
      name: 'Foo',
      age: 33
   },
   {
      name: 'Bar',
      age: 34
   }
];

collectionDataStore.addAll(people, err => {
   if (err) {
      // handle error
   }
});

get(dsid, callback)

Gets a specific item in a collection

Argument	Description
`dsid`	The `dsid` to identify the item to retrieve data from
`callback`	Called after execution. The first parameter of the callback will contain an error object if an error occurred, `undefined` otherwise. The second parameter will contain the stored data.

collectionStore.get with callback example

collectionDataStore.get(dsid, (err, data) => {
   if (err) {
      // handle error
   }
   // ...
});

set(dsid, data, callback)

Updates a specific item in a collection (partial update)

Argument	Description
`dsid`	The `dsid` to identify the item to update
`data`	JSON data to update the item
`callback`	Called after execution. The first parameter of the callback will contain an error object if an error occurred, `undefined` otherwise. The second parameter will contain the stored data.

collectionStore.set with callback example

collectionDataStore.set(dsid, data, (err, data) => {
   if (err) {
      // handle error
   }
   // ...
});

If a property value in the data is set to null, the property will be removed

remove(dsid, callback)

Removes a specific item in a collection

Argument	Description
`dsid`	The `dsid` to identify the item to remove
`callback`	Called after execution. The first parameter of the callback will contain an error object if an error occurred, `undefined` otherwise. The second parameter will contain the removed data.

collectionStore.remove with callback example

collectionDataStore.remove(dsid, (err, data) => {
   if (err) {
      // handle error
   }
   // ...
});

removeAll(callback)

Removes all items in a collection

Argument	Description
`callback`	Called after execution. The first parameter of the callback will contain an error object if an error occurred, `undefined` otherwise.

collectionStore.removeAll with callback example

collectionDataStore.removeAll(err => {
   if (err) {
      // handle error
   }
});

Search

First level data is asynchronously indexed in a new DataStorage Solr index whenever an entry is mutated (add/update/remove). Queries are performed using regular Solr-syntax.

Index fields (object keys) are prefixed with 'ds' and its type. Numerical values will also get a numerical field.

Values for keys that contains "sortable" (case-insensitive) somewhere in the name will also get a sortable field [@since 8.1].

Index field name	Technical description of index field
`ds.analyzed.<key>`	Analyzed, multi-valued and stored solr.TextField
`ds.double.<key>`	Numerical, single-valued and stored solr.TrieDoubleField
`ds.sortable.<sortable-named key>`	Single-valued sortable field [@since 8.1]

Value note! Only base-type values (string, boolean, number) and arrays of such values will be indexed. Hence, object values and array of object(-s) won't be indexed.

Example object where some entries will be indexed and some will not

{
  indexed_s: 'foo',
  indexed_n: 1,
  indexed_b: true,
  indexed_sa: ['foo', 'bar'],
  indexed_na: [1, 2, 3, 4],
  indexed_ba: [true, false],

  notIndexed_obj: {id:1},
  notIndexed_obja: [{id:1},{id:2}]
}

You can search multiple fields at once to create specific queries. E.g:
+ds.analyzed.name:Foo +ds.analyzed.surname:Bar

find(query, count, skip)

Returns a search result

Argument	Description	Default
`query`	The search query (Solr query syntax)
`count`	Maximum number of hits to return	`10`
`skip`	Number of leading hits to be skipped	`0`

find(query, options) [@since 8.1]

Returns a search result, potentially in a custom sort order

name Description Default

count Maximum number of hits to return 10

skip Number of leading hits to be skipped 0

orderBy

Object (or array of objects) that describes the sort order

key	value
`field`	Name of the sortable index field
`order`	"ASC" or "DESC" (ascending or descending)

modified

DESC

collectionStore.find examples

const cheeseResult = store.find('cheese');
const janeResult = store.find('ds.analyzed.name:jane');
const ageResult = store.find('ds.double.age:[30 TO 40]', {count: 50});

const allResult = store.find('*'); // Iterates the data store without querying the index

Search results are always sorted!

If no explicit orderBy is given, it will be sorted by the modified field.

Hence the search result is "last updated data first" if no explicit orderBy is given.

Custom sorting examples

Example of find with custom sorting

// Find max 10 entries that has a sortableName value that starts with "mag"
// ...sorted by the sortableName field
const magResult = store.find('ds.analyzed.sortableName:mag*', {
   count: 10,
   orderBy: {
      field: 'ds.sortable.sortableName',
      order: 'ASC'
   }
});

// Find max 100 entries that has data in the employee field
// ...sorted by the sortableName field
// ...(and if equivalent sortableName, do secondary sort by the salary field)
const employeeResult = store.find('ds.analyzed.employee:*', {
   count: 100,
   orderBy: [
   	{
      	field: 'ds.sortable.sortableName',
      	order: 'ASC'
   	},
   	{
      	field: 'ds.double.salary',
      	order: 'DESC'
   	}
   ]
});

Match-all queries ('*', '*:*') bypasses the index and iterates the data store directly if no explicit orderBy is specified. Typical use-case is when you just want to get the last updated entries.

instantIndex(dsid) [@since 6.1]

Performs instant indexing (blocking) of a collection data store post.

This function is typically useful for apps that uses query-based data that always needs to be up-to-date. Such apps would typically call this function after each "add entry" and "remove entry" operation.

Note! The blocking "index now" operation will only be performed on the index of the local cluster node! Indexes on other cluster nodes still relies on the asynchronous indexing triggered by data mutations!

Argument	Description
`dsid`	The `dsid` to identify the item to instantly index

collectionStore.instantIndex example

try {
   store.instantIndex(dsid);
} catch (e) {
   // Error handling, see https://developer.sitevision.se/docs/data-storage/error-handling
}

Search result

The find method returns a search result wrapper which exposes helpful methods to operate on the data.

toArray(callback)

Gets an array of items from the search result

Argument	Description
`callback`	Called after execution. The first parameter of the callback will contain an error object if an error occurred, `undefined` otherwise. The second parameter will contain an array of items matching the query.

findResult.toArray with callback example

result.toArray((err, data) => {
   if (err) {
      // handle error  
   }
   // data -->
   [
      {
         name: 'Bar',
         dstimestamp: 1554719693812,
         dsid: 'f2412b40-59e9-11e9-ae61-7a4f433f610f',
         age: 30
      },
      {
         name: 'Foo',
         dstimestamp: 1554719663367,
         dsid: 'e01ba170-59e9-11e9-ae61-7a4f433f610f',
         age: 33
      }
   ]
});

each(callback)

Calls the supplied callback for every item in the search result

Argument	Description
`callback`	Called after execution. The first parameter of the callback will contain an error object if an error occurred, `undefined` otherwise. The second parameter will contain an item from the result

findResult.each example

result.each((err, data) => {
   if (err) {
      // handle error  
   }
   handleItem(data);
});

hasNext()

Returns a boolean indicating whether there are available items in the search result iterator

findResult.hasNext example

if (result.hasNext()) {
   // ...
}

next(callback)

Reads one item at a time from the result iterator using a callback

Argument	Description
`callback`	Called after execution. The first parameter of the callback will contain an error object if an error occurred, `undefined` otherwise. The second parameter will contain an item from the result

findResult.naxt with callback example

if (result.hasNext()) {
   result.next((err, data) => {
      if (err) {
         // handle error  
      }
      res.json(data);
   });
}

length(callback)

Get the number of items in the search result

Argument	Description
`callback`	Called after execution. The first parameter of the callback will contain an error object if an error occurred, `undefined` otherwise. The second parameter will contain the number of hits in the search result

findResult.length example

result.length((err, length) => {
   if (err) {
      // handle error  
   }
   // ...
});

Did you find the content on this page useful?