The indexing build summary report

Each time you perform an indexing run, a runId is generated for the process. For example, consider the following indexing query.

https://ingest-server:port/connectors/auth.reindex/run?storeId=11&envType=auth

The process returns an identifying runId JSON token pair. For example:

{
    "runId": "05abc9b9-6b1c-4215-a300-b4e9adae586a"
}

You can call the indexing build report by inputting this runId value in a call to the /status endpoint of the auth.reindex process.

https://ingest.demoqa.hclcomdev4.com/connectors/auth.reindex/runs/05abc9b9-6b1c-4215-a300-b4e9adae586a/status

The report is a single document containing three sections:

• A summary of messages collected during the run.
• An overall description of the Ingest operation.
• Statistics collected at each stage,such as SQL response times, number of documents processed, and elapsed times for the different parts of the run.

The first section of the report contains the start and end time according to standard GMT, then the unique run identifier for the process. After that it contains any parameters that were passed in to this ingest operation. This is important information collected out of the run for the given Store and Catalog.

{
	"start": "2023-02-08T03:27:15.627Z",
	"end": "2023-02-08T03:50:27.339Z",
	"run": "i-1a282ccf-dc44-4844-97ba-74a9b546f912",
	"parameters": {
		"connector": "auth.reindex",
		"store": "1"
	},

The second section of the report contains an overall status of each message type (Informational, Warning, Error), a brief description of each encountered message code and the location where they were found.

	"attributes": {
		"supported.catalog": "10502, 10001, 10501",
		"supported.language": "-5, -2, -1",
		"master.catalog": "10001",
		"default.catalog": "10502",
		"default.language": "-1",
		"default.contract": "10005",
		"default.currency": "USD",
		"flow.database.listagg": "true",
		"flow.language.fallback": "true",
		"flow.version": "9112"
	},
	"severities": {
		"I": "16"
	},
	"codes": {
		"DI0002I": "14",
		"DI0003I": "2"
	},
	"messages": {
		"DI0002I": "Successfully initialized {0} index schema",
		"DI0003I": "Run {0} of connector {1} for catalog {2} in store {3} have been processed successfully in {4} seconds"
	},
	"locations": {
		"info": {
			"Others": "14",
			"Terminal Stage": "2"
		}
	},

"sql": {
		"execution": {
			"Attribute Stage 1a (Database)": "387",
			"Attribute Stage 1b (Database)": "1071",
			"Catalog Stage 1a (Database)": "238",
			"Catalog Stage 1b (Database)": "108",
			"Category Stage 1a (Database)": "1631",
			"Category Stage 1b (Database)": "2123",
			"Price Stage 1a (Database)": "3500",
			"Price Stage 1b (Database)": "586",
			"Product Stage 1a (Database)": "8720",
			"Product Stage 1b (Database)": "407",
			"Product Stage 1e (Database)": "4456",
			"Product Stage 1h (Database)": "153",
			"Product Stage 1i (Database)": "284",
			"Store Stage 1 (Database)": "29",
			"URL Stage 1a (Database)": "6543",
			"URL Stage 1b (Database)": "956",
			"URL Stage 1c (Database)": "582",
			"URL Stage 1e (Database)": "197",
			"URL Stage 1f (Database)": "460"
		},
		"fetch": {
			"Attribute Stage 1a (Database)": "28",
			"Attribute Stage 1b (Database)": "32",
			"Catalog Stage 1a (Database)": "23",
			"Catalog Stage 1b (Database)": "6",
			"Category Stage 1a (Database)": "52",
			"Category Stage 1b (Database)": "28",
			"Price Stage 1a (Database)": "140",
			"Price Stage 1b (Database)": "3",
			"Product Stage 1a (Database)": "382",
			"Product Stage 1b (Database)": "3",
			"Product Stage 1e (Database)": "967",
			"Product Stage 1h (Database)": "53",
			"Product Stage 1i (Database)": "95",
			"Store Stage 1 (Database)": "6",
			"URL Stage 1a (Database)": "24",
			"URL Stage 1b (Database)": "566",
			"URL Stage 1c (Database)": "10",
			"URL Stage 1e (Database)": "2",
			"URL Stage 1f (Database)": "7"
		}
	}
}

concurrentTasks

This parameter controls the number of tasks that should be concurrently scheduled for the main processor of a stage. If a stage needs to process a large amount of data, then increasing the number of concurrent tasks may improve performance.

"concurrentTasks": { 
            "Store Stage 1 (Store Document)": 1, 
            "Store Stage 1 (Supported Languages)": 1, 
            "Store Stage 1 (Supported Catalogs)": 1, 
            "Store Stage 1 (Configurations)": 1, 
            "Store Stage 1 (Store Locations)": 1, 
            "STA Stage 1 (STA)": 1, 
            "Catalog Stage 1a (Catalog Document)": 1, 
            "Category Stage 1a (Category Document)": 1, 
            "STA Stage 2 (External STA)": 1, 
            "Category Stage 1d (Hierarchy)": 1, 
            "Attribute Stage 1a (Attribute Document)": 1, 
            "Category Stage 1b (Facets)": 2, 
            "Attribute Stage 1b (Attribute Values)": 8, 
            "Product Stage 1a (Product Document)": 8, 
            "Product Stage 1h (Child Items)": 4, 
            "Product Stage 1i (Parent Category)": 4, 
            "Product Stage 1b (Enrich Document - Overrides)": 4, 
            "Product Stage 1b (Enrich Document - Entitlements)": 4, 
            "Product Stage 1b (Enrich Document - Associations)": 4, 
            "Product Stage 1b (Enrich Document - Attachments)": 4, 
            "Product Stage 1e (Find Attributes)": 8, 
            "Price Stage 1a (Product Prices)": 8, 
            "Price Stage 1b (Bundle Prices)": 4, 
            "Price Stage 1c (Copy Prices From ES)": 1, 
            "Inventory Stage 1 (Product Inventory)": 8, 
            "Page Stage 1 (Page Document)": 1, 
            "Category Stage 1c (Long Descriptions)": 4, 
            "Product Stage 1g (Long Descriptions)": 4 
        },

bucketSize

This section shows the maximum number of rows per flow file. If you increase the bucket size, there will be fewer flow files but each flow file will contain more data to process. If you decrease the bucket size, each flow file contains less data to process but there will be more flow files.

Note: The number of business objects contained in the flow file is half of the bucket size, because each business object uses two rows. For example, the bucket size for Product Stage 1a is 200 by default. This means that each file contains at most 100 catalog entries.

"bucketSize": { 
            "STA Stage 1 (STA)": 1500, 
            "Catalog Stage 1a (Catalog Document)": 1500, 
            "Category Stage 1a (Category Document)": 1000, 
            "Category Stage 1d (Hierarchy)": 1000, 
            "Attribute Stage 1a (Attribute Document)": 1000, 
            "Category Stage 1b (Facets)": 1000, 
            "Attribute Stage 1b (Attribute Values)": 1000, 
            "Product Stage 1a (Product Document)": 200, 
            "Product Stage 1h (Child Items)": 200, 
            "Product Stage 1i (Parent Category)": 200, 
            "Product Stage 1b (Enrich Document - Overrides)": 200, 
            "Product Stage 1b (Enrich Document - Entitlements)": 200, 
            "Product Stage 1b (Enrich Document - Associations)": 200, 
            "Product Stage 1b (Enrich Document - Attachments)": 200, 
            "Product Stage 1e (Find Attributes)": 200, 
            "Price Stage 1a (Product Prices)": 200, 
            "Price Stage 1b (Bundle Prices)": 200, 
            "Price Stage 1c (Copy Prices From ES)": 200, 
            "Inventory Stage 1 (Product Inventory)": 200, 
            "Page Stage 1 (Page Document)": 2000, 
            "Category Stage 1c (Long Descriptions)": 15000, 
            "Product Stage 1g (Long Descriptions)": 2000 
        }, 

pageSize

The Page size parameter limits the number of rows returned by an SQL query execution (FETCH NEXT n ROWS ONLY). If the number of rows returned exceeds the page size, then the SQL will be executed multiple times and each time only return the maximum number of rows specified by the page size.

Note that not all stages’ SQL use the pageSize parameter. It is useful when the stage will process a large amount of data. Some stages, such as Catalog Stage 1a, do not typically use enough data to cause pageSize to be invoked even if it is included.

"pageSize": { 
            "Category Stage 1a (Category Document)": 10000, 
            "Category Stage 1d (Hierarchy)": 10000, 
            "Attribute Stage 1a (Attribute Document)": 1000000, 
            "Category Stage 1b (Facets)": 10000, 
            "Attribute Stage 1b (Attribute Values)": 1000000, 
            "Product Stage 1a (Product Document)": 100000, 
            "Product Stage 1h (Child Items)": 100000, 
            "Product Stage 1i (Parent Category)": 100000, 
            "Product Stage 1b (Enrich Document - Overrides)": 100000, 
            "Product Stage 1b (Enrich Document - Entitlements)": 100000, 
            "Product Stage 1b (Enrich Document - Associations)": 100000, 
            "Product Stage 1b (Enrich Document - Attachments)": 100000, 
            "Product Stage 1e (Find Attributes)": 100000, 
            "Price Stage 1a (Product Prices)": 100000, 
            "Price Stage 1b (Bundle Prices)": 100000, 
            "Price Stage 1c (Copy Prices From ES)": 100000, 
            "Inventory Stage 1 (Product Inventory)": 100000, 
            "Category Stage 1c (Long Descriptions)": 1000000, 
            "Product Stage 1g (Long Descriptions)": 1000000 
        } 

counter

Each Ingest stage includes a calculation of the average dataflow payload and average bulk ingest processing time. This metric is called the Total Flowfile Count and each ingest stage has one such counter.

"counter": { 
           "Attribute Stage la (Database) ": "1", 
           "Attribute Stage 1b (Database) ": "1", 
           "Catalog Stage la (Database) ": "1", 
           "Catalog Stage 1b (Database) ": "1", 
           "Category Stage la (Database) ": "1", 
           "Category Stage 1b (Database) ": "1", 
           "Category Stage Ic (Database) ": "1", 
           "Category Stage Id (Database) ": "1", 
           "Inventory Stage la (Database) ": "1", 
           "Inventory Stage 1b (Database) ": "2", 
           "Inventory Stage 2": "2", 
           "NLP Service": "2"
           "Price Stage la (Database) ": "4",
           "Price Stage 1b (Database) ": "1", 
           "Product Stage la (Database) ": "2", 
           "Product Stage 1b (Database) ": "1", 
           "Product Stage le (Attributes) " : "20",
           "Product Stage Ig (Database) ": "2", 
           "Product Stage Ih (Database) ": "18", 
           "Product Stage Ii (Database) ": "2", 
           "STA Stage 1 (Database) ": "15", 
           "Store Stage 1 (Database) ": "1", 
           "URL Stage 1a (Database)": "1"
           "URL Stage 1b (Database)": "2"
           "URL Stage 1c (Database)": "1"
           "URL Stage 1f (Database)": "1"
}

database

You can use the database parameter to help you review catalog data in the Summary. The database data contains information about the size of store, catalog, attribute, inventory, contract in the database. Note that this applies to the entire database rather than any specific store.

"database": { 
            "total.allowed.attribute.values": "975", 
            "total.allowed.value.attributes": "137", 
            "total.assigned.attribute.values": "2146", 
            "total.assigned.value.attributes": "80", 
            "total.attribute.values": "3131", 
            "total.attributes": "217", 
            "total.b2b.storefront.asset.stores": "3", 
            "total.b2b.stores": "5", 
            "total.b2c.storefront.asset.stores": "3", 
            "total.b2c.stores": "5", 
            "total.catalog.asset.stores": "2", 
            "total.catalogs": "6", 
            "total.categories": "75", 
            "total.catentries": "4248", 
            "total.catentry.attributes": "12613", 
            "total.catentry.defining.attributes": "9883", 
            "total.catentry.descriptive.attributes": "2730", 
            "total.contracts": "46", 
            "total.facetable.attributes": "15", 
            "total.inventories": "870160", 
            "total.items": "3519", 
            "total.master.catalogs": "2", 
            "total.offer.pricelists": "12", 
            "total.offerprices": "8476", 
            "total.offers": "8476", 
            "total.other.catenties": "12", 
            "total.pricelists": "25", 
            "total.products": "717", 
            "total.stores": "23", 
            "total.supported.languages": "1" 
        }, 

These parameters are defined as follows:

total.allowed.attribute.values

Total number of allowed attribute values in the ATTRVAL table with value usage of “1” or “2”.

total.allowed.value.attributes

Total number of allowed value attributes in the ATTR table with ATTRUSAGE as 1

total.assigned.attribute.values

Total number of assigned attribute values in the ATTRVAL table with value usage as null.

total.assigned.value.attributes

Total number of assigned value attributes in the ATTR table with ATTRUSAGE as 2 or null.

total.attribute.values

Total number of attribute values in the ATTRVAL table.

total.attributes

Total number of attributes in the ATTR table.

total.b2b.storefront.asset.stores

Total number of b2b storefront asset stores with store type of “BMP”.

total.b2b.stores

Total number of b2b stores with store type of “BMH”.

total.b2c.storefront.asset.stores

Total number of b2b storefront asset stores with store type of “MPS”.

total.b2c.stores

Total number of b2b storefront asset stores with store type of “MHS”.

total.catalog.asset.stores

Total number of catalog asset stores with store type of “CPS”.

total.catalogs

Total number of catalogs in the STORECAT table.

total.categories

Total number of all categories in database.

total.catentries

Total number of all catalog entries in database.

total.catentry.attributes

Total number of catalog entry attributes (CATENTATTR table).

total.catentry.defining.attributes

Total number of catalog entry defining attributes (CATENTATTR table with usage of “1”).

total.catentry.descriptive.attributes

Total number of catalog entry descriptive attributes (CATENTATTR table with usage of “2”).

total.contracts

Total number of active contracts (CONTRACT table with state of “3”).

total.facetable.attributes

Total number of facetable attributes.

total.inventories

Total number of inventories (INVENTORY table).

total.items

Total number of items, including product items and category level items.

total.master.catalogs

Total number of master catalogs.

total.offer.pricelists

Total number of default offer price lists.

total.offerprices

Total number of prices in OFFERPRICE table.

total.offers

Total number of valid offers in OFFER table.

total.other.catenties

Total number of other catalog entries except products and product items.

total.pricelists

Total number of price lists in TRADEPOSCN table.

total.products

Total number of products.

total.stores

Total number of all stores.

total.supported.languages

Total number of supported languages in all extended site stores.