The indexing process

The data import handler can perform full import or delta imports. When the full-import command is run, it stores the start time of the operation in the dataimport.properties file, which is in the same directory as the solrconfig.xml file. For example:

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
   <lst name="defaults">
     <str name="config">wc-data-config.xml</str>
    <str name="update.chain">wc-conditionalCopyFieldChain</str>
   </lst>
  </requestHandler>

The wc-data-config.xml file defines the following behaviors:

• How to fetch data, such as using queries or URLs.
• What to read, such as result set columns or XML fields.
• How to process, such as modifying, adding, or removing fields.

For example: the solrhome/v3/CatalogEntry/conf/wc-data-config.xml file contains the following content:

<dataConfig>

  <dataSource name="WC database" 
          type="JdbcDataSource" 
          jndiName="jdbc/WCDB"
          readOnly="true"
          autoCommit="true"
          transactionIsolation="TRANSACTION_READ_COMMITTED"   
          holdability="CLOSE_CURSORS_AT_COMMIT"       
  />

  <dataSource basePath="${solr.core.instanceDir}/../Unstructured/temp/" 
          name="unstructuretmpfile"  
          type="com.ibm.commerce.solr.handler.RequestFileDataSource"
/>

Where two data sources are used:

• The HCL Commerce database is the data source for structured data.
• The unstructuretmpfile specifies the path to the unstructured data.

The Product entity contains the following parameters:

query

Identifies the data to populate fields of the Solr document during full imports.

deltaImportQuery

Identifies the data to populate fields during delta imports.

deltaQuery

Identifies the primary keys of the current entities that changed since the last index time.

deletedPkQuery

Identifies the documents to remove.

Transformer

Every set of fields that are fetched by the entity can be consumed either directly by the indexing process, or changed by using transformers to modify a field or create a new set of fields.

Transformers are chained and sequentially applied in the order in which they are specified. After the fields are fetched from the data source, the list of entity columns are processed one at a time in the order that is listed inside the entity tag. Then, they are scanned by the first transformer to see whether any of the transformer's attributes are present. If present, the transformer is run.

When all of the listed entity columns are scanned, the process is repeated by using the next transformer in the list. A transformer can be used to alter the value of a field fetched from the data source or to populate an undefined field. In the preceding example, the following transformers are used:

Regex Transformer

Extracts or manipulates values from fields (from the source) by using Regular Expressions.

ClobTransformer

Creates a String out of a Clob type in the database.

NameValuePairTransformer

Dynamically creates new fields that are then used as attributes.

The wc-data-config.xml file also contains column-to-field mappings that specify the relationship between index field names and database column names. For example:

<field column="CATENTRY_ID" name="catentry_id" />
<field column="MEMBER_ID" name="member_id" />
<field column="CATENTTYPE_ID" name="catenttype_id_ntk_cs" />
<field column="PARTNUMBER" name="partNumber_ntk" />

HCL Commerce uses the Droid site content crawler to crawl the web, and put the content into the file. That is, from the unstructuretmpfile path that is specified in the wc-data-source.xml file within the CatalogEntry index. Then, Tika parses this file and the information is indexed by the DIH. Unstructured data comes from two sources: the database and the crawler. The unstructured index contains two data configuration files: The wc-data-config.xml file contains product attachments, such as PDF files, while the wc-web-data-config.xml contains web content.

For example, the solrconfig.xml file contains the following content:

<!-- Solr Cell Update Request Handler

     http://wiki.apache.org/solr/ExtractingRequestHandler

   -->
  <requestHandler name="/update/extract"
            startup="lazy"
            class="solr.extraction.ExtractingRequestHandler" >
   <lst name="defaults">
    <!-- All the main content goes into "text"... if you need to return
        the extracted text or do highlighting, use a stored field. -->
    <str name="fmap.content">text</str>
    <str name="lowernames">true</str>
    <str name="uprefix">ignored_</str>

    <!-- capture link hrefs but ignore div attributes -->
    <str name="captureAttr">true</str>
    <str name="fmap.a">links</str>
    <str name="fmap.div">ignored_</str>
   </lst>
  </requestHandler>

Where:

• Tika automatically determines the input document type and produces an XHTML stream that is then fitted to a SAX ContentHandler.
• Solr then reacts to the Tika SAX events and creates the fields to index.
• Tika produces metadata information such as Title, Subject, and Author.
• All of the extracted text is added to the content field. Setting Fmap.content to text causes the content to be added to the text field.