Migrating DOM-based inventory to Elasticsearch

If you operate a Distributed Order Management (DOM) system, you can import its data into Elasticsearch.

About this task

When HCL Commerce is integrated with an external DOM system, considerable coordination has to take place to ensure that order fulfilment statuses, inventory counts, and pricing is consistently logged across both systems. The default Ingest service does not come with a built-in DOM pipeline, however you can easily alter one of the default pipelines to communicate with the DOM system. The Ingest service is specifically designed for such scenarios, so you can follow the standard set of steps for creating and configuring a connector, with some variation, to enable Ingest support for your DOM system.

Procedure

Create a connector. To create the connector using the Ingest service, use the following API:

http://hostname:30800/swagger-ui/index.html?url=/v3/api-docs&validatorUrl=#/Create%20Connector%20Configuration/createConnector

For more information, see Creating a NiFi service connector.

From the Swagger UI, go to the Connector Configuration > POST: /connectors API.
Click Try it out.

Use your custom code for the API body. For example:


{
   "name":"custom",
   "description":"This is the connector for the custom processing",
   "pipes":[
      {
         "name":"_Template-Schema"
      },
      {
         "name":"_Template-Groovy-DatabaseETL",
         "properties":[
            {
               "name":"Database Driver Location(s)",
               "value":"${AUTH_JDBC_DRIVER_LOCATION}",
               "scope":{
                  "name":"Database Connection Pool",
                  "type":"CONTROLLER_SERVICE"
               }
            },
            {
               "name":"Database Driver Class Name ",
               "value":"${AUTH_JDBC_DRIVER_CLASSNAME}",
               "scope":{
                  "name":"Database Connection Pool",
                  "type":"CONTROLLER_SERVICE"
               }
            },
            {
               "name":"Database Connection URL",
               "value":"${AUTH_JDBC_URL}",
               "scope":{
                  "name":"Database Connection Pool",
                  "type":"CONTROLLER_SERVICE"
               }
            },
            {
               "name":"Database User",
               "value":"${AUTH_JDBC_USER_NAME}",
               "scope":{
                  "name":"Database Connection Pool",
                  "type":"CONTROLLER_SERVICE"
               }
            },
            {
               "name":"Password",
               "value":"${AUTH_JDBC_USER_PASSWORD}",
               "scope":{
                  "name":"Database Connection Pool",
                  "type":"CONTROLLER_SERVICE"
               }
            }
         ]
      },
     {"name":"Terminal"}
   ]
}
}
}

Click Execute.

For more information about customizing default connectors, see Extending Ingest connectors.

Configure the connector to accept data from the DOM system. With the variations described below, you can follow the steps specified in the tutorial Configure the connector in NiFi.
1. Follow the procedure up until Step 1.e. In Step 1.f, set the value as inventory, and continue the procedure to Step 2.b.
2. In Step 2.b, use the following as the contents of replacement value.
```
{
   "properties":{
      "custom":{
         "properties":{
			 "customInventory":{
               "properties": {
                        "id": {
                            "type": "keyword",
                            "store": true
                        },
                        "status": {
                            "type": "keyword",
                            "store": true
                        }
                    }
            }
         }
      }
   }
}
```
3. Double click on Set index name. Select the properties tab and change the value of index.name to ${environment.name}.inventory.
4. Continue following the procedure. After you have selected the processor to modify its settings in Step ../../tutorials/tutorial/tsd_connectorconfigure_elastic.html#task_n3x_cbg_smb__modcustomsql, set index.name as inventory in Step ../../tutorials/tutorial/tsd_connectorconfigure_elastic.html#task_n3x_cbg_smb__modindexname.
5. Update the SQL code as described in Step ../../tutorials/tutorial/tsd_connectorconfigure_elastic.html#task_n3x_cbg_smb__updateingestsql with the query provided in the file query.txt.
6. Continue following the procedure. After you have selected the processor to modify its settings in Step ../../tutorials/tutorial/tsd_connectorconfigure_elastic.html#task_n3x_cbg_smb__processorproperties, replace the code in Step ../../tutorials/tutorial/tsd_connectorconfigure_elastic.html#task_n3x_cbg_smb__updateprocessorproperties with the contents of the file groovy_script.txt. Continue the procedure through Steps 5 and 6.
Instead of connecting to the Product pipeline, connect the custom pipes to the auth.inventory pipeline, as in the example screenshot.
Build the index.
Verify that your data has been imported. To assist you, a sample is provided in the file index_data.txt that shows how the data is represented in the inventory index.
You can run an Elasticsearch query to verify that the added data is retrievable. A sample query can be found in the file sample_elastic_query.txt. Replace the appropriate catentry ID in the query to see the results.
If further configuration is required, you can follow the steps in Create the customization in NiFi as needed.

Results

If the configuration is correct, your DOM-based data should be built into and retrievable in the Elasticsearch index.