Implementing MCM
On this page
Multi-Cluster Management (MCM) with the MultiClusters API
When your data no longer fits on one machine, you’ll need to start adding new clusters. MCM makes this easy for you by helping you distribute and manage your data over multiple machines.
The following tutorial should help you get started using MCM.
But first, you’ll need to enable MCM on your account. Once that’s done, you can start using this feature.
The tutorial
We’ve decided to use a special kind of use case that shows how to use multiple clusters with both private and public data. Our example is a music streaming application that allows users to create public and private playlists. The private playlists are only accessible to the users who create them.
We follow a Without MCM / With MCM approach, to highlight what you’ll need to change if you are already using multiple clusters without MCM. For those implementing multiple clusters for the first time, you can look only at the MCM implmentation.
Splitting data across multiple clusters
Assign user data to a cluster
Without MCM, you assign data to an APPID, because an APPID can have only one cluster assigned to it. If you have many clusters, you will need a different APPID per cluster. This requires you to maintain on your own servers a 3-part mapping (user/cluster/APPID).
With MCM, every cluster uses the same APPID. Your data need only be sent to the right cluster. MCM uses a mapping on each cluster to route requests to the correct user cluster. All you need to do is assign a cluster the user’s data, which you can do using the endpoint assign userID:
1
2
3
4
5
6
7
curl -X POST \
-H "X-Algolia-User-ID: user42" \
-H "X-Algolia-API-Key: ${API_KEY}" \
-H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
--data-binary "{ \
\"cluster\":\"d4242-eu\"}" \
"https://${APPLICATION_ID}.algolia.net/1/clusters/mapping"
Add private user data
Without MCM, every record needs a userID attribute to tag the user. Additionally, the userID attribute needs to be set as a filter at query time.
With MCM, the records will be automatically tagged by the X-Algolia-User-ID header sent at query time. The engine will then automatically add an extra attribute __userID__ inside the record to enable a user-id filter to easily identify the userID associated with the record.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
const playlists = [
{
user: 'user42',
name: 'My peaceful playlist',
songs: [],
createdAt: 1500000181
},
{
user: 'user4242',
name: 'My workout playlist',
songs: [],
createdAt: 1500040452
}
];
// Without Multi-clusters API
playlists.forEach(playlist => {
// Fetch from the database the associated appID and apiKey for this user
// using your own functions (my_*)
const appID = my_getAppIDFor(playlist.user);
const apiKey = my_getIndexingApiKeyFor(playlist.user);
const client = algoliasearch(appID, apiKey);
const index = client.initIndex('playlists');
index.setSettings({attributesForFaceting: ['filterOnly(user)']}, (err, content) => {
console.log(content);
});
index.addObject(playlist, (err, content) => {
console.log(content);
});
});
// With Multi-clusters API
const client = algoliasearch('YourApplicationID', 'YourAdminAPIKey');
const index = client.initIndex('playlists');
playlists.forEach(playlist => {
client.setExtraHeader('X-Algolia-User-ID', playlist.user)
index.addObject(playlist, (err, content) => {
console.log(content);
});
});
Add public data
Without MCM, you need to tag every public record with a special value like “public” in the userID attribute. And then you need to filter on that value to search the public records.
With MCM, you use the special userID value * to flag records as public, allowing all users to see them. Public records are automatically replicated on all the clusters of your multi-cluster setup to avoid adding network latency during the search.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
const public_playlists = [{
user: 'public',
name: 'TOP50 songs',
songs: [],
createdAt: 1500240452
}];
// Without Multi-clusters API
// Fetch the list of appID and apiKey to target every clusters
// using your own functions (my_*)
const AppIDConfigurations = my_getAllAppIDConfigurations();
// Send the record to every clusters
AppIDConfigurations.each(AppIDConfiguration => {
const appID = AppIDConfiguration.appID;
const apiKey = AppIDConfiguration.apiKey;
const client = algoliasearch(appID, apiKey);
const index = client.initIndex('playlists');
index.addObjects(public_playlists, (err, content) => {
console.log(content);
});
});
// With Multi-clusters API
const client = algoliasearch('YourApplicationID', 'YourAdminAPIKey');
const index = client.initIndex('playlists');
client.setExtraHeader("X-Algolia-User-ID", '*')
index.addObject(public_playlists, (err, content) => {
console.log(content);
});
Search inside the data
Without MCM, the search needs to filter on the specific user and public data.
With MCM, the engine just needs to know which userID (with the header X-Algolia-User-ID) is targeted by the query in order to route the request to the cluster holding the data. The engine will automatically add the right filters to retrieve both the user’s data and the public data.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// Without Multi-clusters API
// Fetch from the database the associated appID and apiKey for this user
// using your own functions (my_*)
const appID = my_getAppIDFor(record.user);
const apiKey = my_getSearchOnlyApiKeyFor(record.user);
const client = algoliasearch(appID, apiKey);
const index = client.initIndex('playlists');
index.search('peace', {
facetFilters: ["user:user42", "user:public"]
}, (err, content) => {
if (err) {
console.error(err);
return;
}
for (const h in content.hits) {
console.log(`Hit(${content.hits[h].objectID}): ${content.hits[h].toString()}`);
}
});
// With Multi-clusters API
const client = algoliasearch('YourApplicationID', 'YourSearchOnlyAPIKey');
const index = client.initIndex('playlists');
client.setExtraHeader("X-Algolia-User-ID", 'user42')
index.search('peace', (err, content) => {
if (err) {
console.error(err);
return;
}
for (const h in content.hits) {
console.log(`Hit(${content.hits[h].objectID}): ${content.hits[h].toString()}`);
}
});
Secured data access
In both cases, you will need to use API keys. Otherwise, the security of private data can be at risk because a user can change the parameters of a query to access the data of another user. To respond to this sort of security risk, API keys are used to restrict access of a user to only his or her personal data (+ the public data).
1
2
3
4
5
6
7
8
9
10
// Without Multi-clusters API
// Fetch from the database the associated apiKey for this user
// using your own functions (my_*)
const apiKey = my_getSearchOnlyApiKeyFor('user42');
client.generateSecuredApiKey(apiKey, {filters: '(user:user42,user:public)'});
// With Multi-clusters API
client.generateSecuredApiKey('YourSearchOnlyApiKey', {userID: 'user42'});
Note that with MCM, the header is always required even if the query contains the parameter userID in order to route the request efficiently.
Index configuration and API keys
For endpoints that have global impact on all clusters, like Add API and setSettings, the API Key stays the same, and you don’t need to provide the X-Algolia-User-ID header since the jobs are replicated on all the clusters in order to have the same API and index configuration.
