Migrating Elasticsearch data from Component Pack for Connections

If you previously used the Elasticsearch offering of the Component Pack for HCL Connections™ to provide Metrics and Type-ahead Search data for your Connections deployment, you can migrate that data to the stand-alone Elasticsearch instance.

About this task

Verify that the Elasticsearch server is running, and ensure that the user account used has the proper access rights to perform the steps in this task.

To avoid losing new data while migrating the existing data, run this task during a maintenance window.

Procedure

Back up the index in the previous Elasticsearch cluster.

Note: If you have not already completed the snapshot repository registration step as described in Backing up and restoring data for Elasticsearch-based components, do that before proceeding.
1. Connect to an Elasticsearch client pod in the old Elasticsearch cluster by running the following command on a kubernetes node:
```
sudo /usr/local/bin/kubectl exec -ti -n connections $(sudo /usr/local/bin/kubectl get pods -n connections  -o wide -a |grep es-client |awk '{print $1}' |head -n 1) -- bash
```
2. Back up all Elasticsearch indexes by running the following command:
```
 cd /opt/elasticsearch-5.5.1/probe/
 ./sendRequest.sh PUT /_snapshot/REPONAME/snapshot_migration?wait_for_completion=true
```
  Where REPONAME is the name of the snapshot repository, which was previously used to register and manage the Elasticsearch snapshot; for example, connectionsbackup.
Get the index list, which will be migrated into the new Elasticsearch instance, by running the following command:
```
./sendRequest.sh GET /_cat/indices
```
Record the index names for the applications that you want to migrate. These names will be used in the rest of the migration process.

Application Index name Number of indices
Metrics icmetrics_a_YYYY_{1h | 2h}
Example: icmetrics_a_2019_2h
Two per calendar year of data collection
Type-ahead Search quickresults One

Application	Index name	Number of indices
Metrics	icmetrics_a_YYYY_{1h \| 2h} Example: icmetrics_a_2019_2h	Two per calendar year of data collection
Type-ahead Search	quickresults	One

Prepare a shell script to simplify the migration request to the Elasticsearch instance.

Create a new shell script with the file name ESRequest.sh.

Add content to the script as shown in the following example.

#!bin/bash
set -o pipefail
set -o errexit
set -o nounset

ES_URL=Elasticsearch_URL_including_port_number
ADMIN_CERT=Path_to_Elasticsearch_admin_client_cert
ADMIN_CERT_PASS=Elasticsearch_admin_client_cert_password
ADMIN_KEY=Path_to_Elasticsearch_admin_client_key
CA_CERT=Path_to_CA_cert

HTTPMethod=$1
shift 1
set -f

response=$(curl \
   --insecure \
   --cert ${ADMIN_CERT}:${ADMIN_CERT_PASS} \
   --key  ${ADMIN_KEY} \
   --cacert ${CA_CERT} \
   -X${HTTPMethod} \
   ${ES_URL}"$@")

echo ${response}

set +f
#End of script

Make the new script executable by running the following command:
```
chmod +x ESRequest.sh
```

Register the snapshot repository in new Elasticsearch cluster by running the following commands:
```
echo "----------------to create repo"
        ./ESRequest.sh PUT /_snapshot/REPONAME \
                -H 'Content-Type: application/json' \
                -d '{"type": "fs","settings": {"compress": true, "location": "BACKUP_PATH"}}'

        echo "----------------to check created repo"
        ./ESRequest.sh GET /_snapshot/_all?pretty
```
where:
- REPONAME is the name of the snapshot repository, which will be used to register and manage the Elasticsearch snapshot. The first time that you perform these steps, you must give the repository an appropriate name, for example, connectionsbackup.
- BACKUP_PATH is the mount path of the shared Elasticsearch backup path. By default this path is /backup.
When all commands have been run, you will see the following output:

{ "REPONAME" : { "type" : "fs", "settings" : { "compress" : "true", "location" : "BACKUP_PATH" } } }
Copy the snapshot from the old Elasticsearch cluster to the new Elasticsearch cluster.
1. Ensure that you have the necessary permissions to complete this step.
2. Go to the location which was configured as the back-up storage of your old Elasticsearch cluster.
3. Package all content in it by running the following command: tar -cvf backup.tar *
4. Copy the package to the location that was configured as the back-up storage for the new Elasticsearch cluster.
5. Extract the package by running the following command: tar -xvf backup.tar
Verify that the new Elasticsearch indexes were already created in the new Elasticsearch cluster by running the following command:
```
./ESRequest.sh GET /_cat/indices
```
- No indexes already exist
  If the output shows that there are no existing indexes that use the same name that you recorded in step 2:
  1. Restore the snapshot directly by running the following command:
```
./ESRequest.sh POST /_snapshot/REPONAME/snapshot_migration/_restore
```
  2. Skip to step 10 to verify the migrated data in the UI.
- Indexes already exist
  If the output shows that there are existing indexes with the same name that you recorded in step 2, then continue to step 7 to resolve the conflict.

First, restore the index names that do not have any conflict by running the following commands:

./ESRequest.sh POST /_snapshot/REPONAME/snapshot_migration/_restore \
        -H 'Content-Type: application/json' \
        -d'
        {
          "indices": "comma_separated_list_of_index_names_with_no_conflict",
          "ignore_unavailable": true,
        }'

Next, restore the index names that have conflicts, using temporary names.

Run the following commands to restore the snapshot using a temporary name for each index that has a conflict. The Metrics application is used as an example in the following commands and steps:

   ./ESRequest.sh POST /_snapshot/REPONAME/snapshot_migration/_restore \
        -H 'Content-Type: application/json' \
        -d'
        {
          "indices": "comma_separated_list_of_index_names_WITH_conflict",
          "ignore_unavailable": true,
          "rename_pattern": "icmetrics_a_(.+)",
          "rename_replacement": "migrated_icmetrics_a_$1"
        }'

Re-index data from the restored temporary indexes to the target indexes:

For every temporary index restored and renamed in step 8, run following commands to re-index the migrated data into a target index:

Run this command for one index at a time.

 ./ESRequest.sh POST "/_reindex" \
        -H 'Content-Type: application/json' \
        -d'
        {
                  "source": {
                            "index": "migrated_ONE_index_name_with_conflict"
                          },
                  "dest": {
                            "index": "Corresponding_EXISTING_index_name"
                          }
        }'

Delete each temporary index that you resolved in step 9a by running the following command:
```
 ./ESRequest.sh DELETE /index_name_with_conflict_from_step_9a
```

Populate user attributes for events in Elasticsearch by completing the following steps.
1. From the Deployment Manager system, load the Python script by entering the following commands:
```
cd /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin
sudo sh wsadmin.sh -lang jython -user wasadmin_user -password wasadmin_password
execfile('metricsEventCapture.py')
```
2. Run the following command to execute the migration task:
```
MigraterService.populateUserAttributes()
```
  The command returns immediately, but the task continues to run on the server as necessary. Monitor the progress by viewing the log at the following location:
  
  /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1/PopulateUserAttributes_YY.YY.DD_HH:MM:SS.log
Use the UI to verify that the metrics data was migrated successfully:
1. Log in to Metrics as the organization admin, using the following address: https://your_Connections_server_host_name/metrics/orgapp#/
2. View a global report and verify that migrated data displays.
3. Verify that you can view reports by Group by in either Community Metrics or Global Metrics.
Use the UI to verify that the type-ahead search data was migrated successfully:
1. Log in to Connections.
2. Navigate to some content, such as a forum, wiki, or community. View a global report and verify that the migrated data displays.
3. Open the type-ahead search sidebar by clicking the Search icon. Verify that you can see the content you viewed at the top of the list.

What to do next

When you are ready to back up the data for the new deployment, see the information on Snapshot and Restore in the Elasticsearch documentation.