Creating the custom Dataload configuration and data files

In this lesson, sample inventory data is prepared in XML format. The Data Load utility configuration files are prepared.

The sample inventory data is prepared based on the sample catalog used to test the customization. An XML sample data file and XML loader are used, and other Data Load utility configuration files are prepared.

The Data Load utility sample configurations and data files are included under the deployed inventory index core in the following directory:

WCDE_installdir\search\solr\home\MC_masterCatalogId\generic\CatalogEntry\Inventory\indexload

At a minimum, three different configuration files and a data file are needed. Only XML data formats are used in this tutorial. For information about indexing with CSV files, see Extending the Dataload indexer mediator for WebSphere Commerce search.

Procedure

wc-solrIndex-load.xml is the entry point file that is passed to the Data Load utility. Update this file to match your environment.

For example:


<?xml version="1.0" encoding="UTF-8"?>
<_config:DataLoadConfiguration
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.ibm.com/xmlns/prod/commerce/foundation/config ../xsd/wc-dataload.xsd" 
   xmlns:_config="http://www.ibm.com/xmlns/prod/commerce/foundation/config">

<_config:DataLoadEnvironment configFile="wc-solrindex-env.xml" />

<_config:LoadOrder commitCount="100000" batchSize="100000" maxError="1" dataLoadMode="Replace" > 
   <_config:LoadItem name="AuroraEsiteStoreInventoryData" loadSequence="1.0" businessObjectConfigFile="wc-solrindex-xml-loader.xml">
   <_config:property name="valueIsList" value= "true" />
   <_config:property name="coreName" value="MC_10051_CatalogEntry_Inventory_generic" />
   <_config:DataSourceLocation location="wc-solrIndex-data.xml" />     
</_config:LoadItem>
    </_config:LoadOrder>
</_config:DataLoadConfiguration>

Where:

wc-solrindex-env.xml is the environment variable file under the same directory;
Replace is the dataLoadMode
wc-solrindex-xml-loader.xml is the Business Object Config File under the same directory;
MC_10051_CatalogEntry_Inventory_generic is the Solr core name for the inventory index;
wc-solrIndex-data.xml is the Data Source file under the same directory.

Update the wc-solrIndex-env.xml file to match your environment.

For example:


<?xml version="1.0" encoding="UTF-8"?>
<_config:DataLoadEnvConfiguration
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.ibm.com/xmlns/prod/commerce/foundation/config ../../../xml/config/xsd/wc-dataload-env.xsd" 
      xmlns:_config="http://www.ibm.com/xmlns/prod/commerce/foundation/config">
       <_config:DataWriter className="com.ibm.commerce.foundation.dataimport.dataload.datawriter.SolrDataWriter">
       <_config:property name="solrServerURL" value="http://localhost/solr/" />
       <_config:DataLoadBatchService className="com.ibm.commerce.foundation.dataimport.dataload.batchservice.SolrIndexBatchService" user="<user>" password="<password>"/>
       </_config:DataWriter>
</_config:DataLoadEnvConfiguration>

Where:

http://localhost/solr/ is the Solr server URL. For example, http://wcsolr08:3737/solr/
<User> and <password> are the Solr WAS application security credentials. Specify the user name and the encrypted password if the application security is enabled, otherwise leave empty.

Copy the following to the wc-solrIndex-xml-loader.xml file:


<_config:DataloadBusinessObjectConfiguration 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.ibm.com/xmlns/prod/commerce/foundation/config ../xsd/wc-dataload-businessobject.xsd" 
  xmlns:_config="http://www.ibm.com/xmlns/prod/commerce/foundation/config">

  <_config:DataLoader className="com.ibm.commerce.foundation.dataload.BusinessObjectLoader">
    <_config:DataReader className="com.ibm.commerce.foundation.dataload.datareader.XmlReader" >
      <_config:XmlHandler className="com.ibm.commerce.foundation.dataload.xmlhandler.NVPXmlHandler" />
    </_config:DataReader>
    <_config:BusinessObjectBuilder className="com.ibm.commerce.foundation.dataload.businessobjectbuilder.MapObjectBuilder">   
      <_config:DataMapping>
    <_config:mapping xpath="productCatentryId" value="productCatentryId" />
    <_config:mapping xpath="itemCatentryId" value="itemCatentryId" />
    <_config:mapping xpath="store_location_quantity" value="str_loc_qty" />
    <_config:mapping xpath="" value="delete"  deleteValue="true"/>
      </_config:DataMapping>
      <_config:BusinessObjectMediator className="com.mycompany.commerce.foundation.dataimport.dataload.mediator.InventorySolrInputDocumentMediator">
      <!-- idFieldName value should match the index uniqueKey value -->
      <_config:property name="idFieldName" value="catentry_id"/>  
      </_config:BusinessObjectMediator>
    </_config:BusinessObjectBuilder>
  </_config:DataLoader>

</_config:DataloadBusinessObjectConfiguration>

Where:

The _config:mapping lines map the external XML fields to the internal fields, which were defined in the InventorySolrInputDocumentMediator class that was created in the previous lesson;
catentry_id is the unique field of the inventory index.

Update the wc-solrIndex-data.xml file to match your environment. See the tutorial reference file for the sample data that is used for the following test steps.
For example:
```
<Inventories>
<Product productCatentryId="10001">
<Item itemCatentryId="10040" str_loc_qty="10202_OnlineStore:100 10202_10089:50"/>
<Item itemCatentryId="10041" str_loc_qty="10202_OnlineStore:100 10202_10089:50"/>
</Product>
<Product productCatentryId="10002" >
<Item itemCatentryId="10053" str_loc_qty="10202_OnlineStore:100 10202_10089:50"/>
<Item itemCatentryId="10054" str_loc_qty ="10202_OnlineStore:100 10202_10089:50"/>
</Product>
</Inventories> 
```
Note: The <Inventories> node must be the root node of the data XML file.
This sample data is interpreted as:
- A product with a unique ID of 10001 contains two items.
- The first item has a unique ID of 10040 and the second item has a unique ID of 10041.
- The str_loc_qty of the first Unique ID contains two different store-location quantities.
- The online store does not have a Unique store location, therefore a unique qualifier OnlineStore is used instead.
- The first store-location with a store unique ID of 10202, and Online Store Unique qualifier OnlineStore has a quantity of 100.
- The first store-location with a store unique ID of 10202, and Store location Unique ID 10089 has a quantity of 50.