Migrating Elasticsearch data from Component Pack for Connections
If you previously used the Elasticsearch offering of the Component Pack for HCL Connections™ to provide Metrics and Type-ahead Search data for your Connections deployment, you can migrate that data to the stand-alone Elasticsearch instance.
About this task
Verify that the Elasticsearch server is running, and ensure that the user account used has the proper access rights to perform the steps in this task.
To avoid losing new data while migrating the existing data, run this task during a maintenance window.
Procedure
-
Back up the index in the previous Elasticsearch cluster.
Note: If you have not already completed the snapshot repository registration step as described in Backing up and restoring data for Elasticsearch-based components, do that before proceeding.
-
Get the index list, which will be migrated into the new Elasticsearch instance, by running the
following command:
./sendRequest.sh GET /_cat/indices
Record the index names for the applications that you want to migrate. These names will be used in the rest of the migration process.
Application Index name Number of indices Metrics icmetrics_a_YYYY_{1h | 2h} Example: icmetrics_a_2019_2h
Two per calendar year of data collection Type-ahead Search quickresults One -
Prepare a shell script to simplify the migration request to the Elasticsearch instance.
-
Register the snapshot repository in new Elasticsearch cluster by running the following
commands:
echo "----------------to create repo" ./ESRequest.sh PUT /_snapshot/REPONAME \ -H 'Content-Type: application/json' \ -d '{"type": "fs","settings": {"compress": true, "location": "BACKUP_PATH"}}' echo "----------------to check created repo" ./ESRequest.sh GET /_snapshot/_all?pretty
where:- REPONAME is the name of the snapshot repository, which will be used to
register and manage the Elasticsearch snapshot. The first time that you perform these steps, you
must give the repository an appropriate name, for example,
connectionsbackup
. - BACKUP_PATH is the mount path of the shared Elasticsearch backup path. By default this path is /backup.
When all commands have been run, you will see the following output:
{ "REPONAME" : { "type" : "fs", "settings" : { "compress" : "true", "location" : "BACKUP_PATH" } } }
- REPONAME is the name of the snapshot repository, which will be used to
register and manage the Elasticsearch snapshot. The first time that you perform these steps, you
must give the repository an appropriate name, for example,
-
Copy the snapshot from the old Elasticsearch cluster to the new Elasticsearch cluster.
- Ensure that you have the necessary permissions to complete this step.
- Go to the location which was configured as the back-up storage of your old Elasticsearch cluster.
-
Package all content in it by running the following command:
tar -cvf backup.tar *
- Copy the package to the location that was configured as the back-up storage for the new Elasticsearch cluster.
-
Extract the package by running the following command:
tar -xvf backup.tar
-
Verify that the new Elasticsearch indexes were already created in the new Elasticsearch cluster
by running the following command:
./ESRequest.sh GET /_cat/indices
- No indexes already existIf the output shows that there are no existing indexes that use the same name that you recorded in step 2:
- Restore the snapshot directly by running the following
command:
./ESRequest.sh POST /_snapshot/REPONAME/snapshot_migration/_restore
- Skip to step 10 to verify the migrated data in the UI.
- Restore the snapshot directly by running the following
command:
- Indexes already exist
If the output shows that there are existing indexes with the same name that you recorded in step 2, then continue to step 7 to resolve the conflict.
- No indexes already exist
-
First, restore the index names that do not have any conflict by running the following
commands:
./ESRequest.sh POST /_snapshot/REPONAME/snapshot_migration/_restore \ -H 'Content-Type: application/json' \ -d' { "indices": "comma_separated_list_of_index_names_with_no_conflict", "ignore_unavailable": true, }'
-
Next, restore the index names that have conflicts, using temporary names.
Run the following commands to restore the snapshot using a temporary name for each index that has a conflict. The Metrics application is used as an example in the following commands and steps:
./ESRequest.sh POST /_snapshot/REPONAME/snapshot_migration/_restore \ -H 'Content-Type: application/json' \ -d' { "indices": "comma_separated_list_of_index_names_WITH_conflict", "ignore_unavailable": true, "rename_pattern": "icmetrics_a_(.+)", "rename_replacement": "migrated_icmetrics_a_$1" }'
-
Re-index data from the restored temporary indexes to the target indexes:
-
Populate user attributes for events in Elasticsearch by completing the following steps.
-
Use the UI to verify that the metrics data was migrated successfully:
- Log in to Metrics as the organization admin, using the following address: https://your_Connections_server_host_name/metrics/orgapp#/
- View a global report and verify that migrated data displays.
- Verify that you can view reports by Group by in either Community Metrics or Global Metrics.
-
Use the UI to verify that the type-ahead search data was migrated successfully:
- Log in to Connections.
- Navigate to some content, such as a forum, wiki, or community. View a global report and verify that the migrated data displays.
- Open the type-ahead search sidebar by clicking the Search icon. Verify that you can see the content you viewed at the top of the list.
What to do next
When you are ready to back up the data for the new deployment, see the information on Snapshot and Restore in the Elasticsearch documentation.