The indexing build summary report

A thorough indexing build status report is created at the end of each full re-indexing Ingest operation. You can use this report to optimize the different stages of the index process, and troubleshoot issues such as longer-than-expected processing times.

How the report is generated

Each time you perform an indexing run, a runId is generated for the process. For example, consider the following indexing query.
https://ingest-server:port/connectors/auth.reindex/run?storeId=11&envType=auth
The process returns an identifying runId JSON token pair. For example:
{
    "runId": "05abc9b9-6b1c-4215-a300-b4e9adae586a"
}
You can call the indexing build report by inputting this runId value in a call to the /status endpoint of the auth.reindex process.
https://ingest.demoqa.hclcomdev4.com/connectors/auth.reindex/runs/05abc9b9-6b1c-4215-a300-b4e9adae586a/status

The response is an extensive report that includes process information, database calls, and the duration of each subprocess. You can review the Sample index build status report. The sections, values and measures used for each are presented below.

Interpreting the indexing build summary report

The report is a single document containing three sections:
  • An overall description of the Ingest operation.
  • A summary of messages collected during the run.
  • Statistics collected at each stage,such as SQL response times, number of documents processed, and elasped times for the different parts of the run.
Summary
The first section of the report contains the start and end time according to standard GMT, then the unique run identifier for the process. After that it contains any parameters that were passed in to this ingest operation. This is important information collected out of the run for the given Store and Catalog.
{
	"start": "2023-02-08T03:27:15.627Z",
	"end": "2023-02-08T03:50:27.339Z",
	"run": "i-1a282ccf-dc44-4844-97ba-74a9b546f912",
	"parameters": {
		"connector": "auth.reindex",
		"store": "1"
	},
Messages
The second section of the report contains an overall status of each message type (Informational, Warning, Error), a brief description of each encountered message code and the location where they were found.
	"attributes": {
		"supported.catalog": "10502, 10001, 10501",
		"supported.language": "-5, -2, -1",
		"master.catalog": "10001",
		"default.catalog": "10502",
		"default.language": "-1",
		"default.contract": "10005",
		"default.currency": "USD",
		"flow.database.listagg": "true",
		"flow.language.fallback": "true",
		"flow.version": "9112"
	},
	"severities": {
		"I": "16"
	},
	"codes": {
		"DI0002I": "14",
		"DI0003I": "2"
	},
	"messages": {
		"DI0002I": "Successfully initialized {0} index schema",
		"DI0003I": "Run {0} of connector {1} for catalog {2} in store {3} have been processed successfully in {4} seconds"
	},
	"locations": {
		"info": {
			"Others": "14",
			"Terminal Stage": "2"
		}
	},
Elapsed time and processing speed
The last section contains the overall absolute ingest time (in milliseconds) and its corresponding stage level breakdown, as well as the cumulative elapsed time (in milliseconds from all processing threads) spent at each stage. It includes a rollup total (also in milliseconds) for each index type.
The bulk elapsed time is a cumulative elapsed time (in milliseconds from all processing threads) spent inside of the Bulk Service of each index type. This cumulative elasped time should not be interpreted as the absolute elapsed time spent at that stage.
The processed section provides a total of all successfully completed index document creation and update operations. It includes the total number of bytes written to Elastic from each of the listed stages.
The processing speed section provides a stage level summary of the overall average throughput in documents per second.
The final subsection reports the cumulative total SQL execution time and SQL fetch time for each of the listed stages. SQL execution time is the time taken for the database to optimize and run the SQL, while the fetch time is the elapsed time for each batch of data to be returned from the query result-set in the database.

	"elapsed": {
		"absolute": {
			"connector": "1391712",
			"stages": {
				"Attribute Stage 1a (Database)": "388719",
				"Attribute Stage 1b (Database)": "392137",
				"Catalog Stage 1a (Database)": "23977",
				"Catalog Stage 1b (Database)": "3752",
				"Category Stage 1a (Database)": "23913",
				"Category Stage 1b (Database)": "8550",
				"Category Stage 1d (Database)": "15985",
				"Price Stage 1a (Database)": "389369",
				"Price Stage 1b (Database)": "368857",
				"Price Stage 2": "363395",
				"Product Stage 1a (Database)": "398499",
				"Product Stage 1b (Database)": "359561",
				"Product Stage 1e (Database)": "367804",
				"Product Stage 1h (Database)": "360557",
				"Product Stage 1i (Database)": "362583",
				"URL Stage 1a (Database)": "375215",
				"URL Stage 1b (Database)": "372985",
				"URL Stage 1c (Database)": "357129",
				"URL Stage 1e (Database)": "357906",
				"URL Stage 1f (Database)": "373172"
			}
		},
		"cumulative": {
			"summary": {
				"Attribute Pipeline": "30354",
				"Catalog Pipeline": "56444",
				"Category Pipeline": "149355",
				"Price Pipeline": "44645",
				"Product Pipeline": "178951",
				"Store Pipeline": "2516831",
				"URL Pipeline": "215828"
			},
			"stages": {
				"Attribute Stage 1a (Database)": "14001",
				"Attribute Stage 1b (Database)": "16353",
				"Catalog Stage 1a (Database)": "51404",
				"Catalog Stage 1b (Database)": "5040",
				"Category Stage 1a (Database)": "52158",
				"Category Stage 1b (Database)": "85888",
				"Category Stage 1d (Database)": "11309",
				"Price Stage 1a (Database)": "30089",
				"Price Stage 1b (Database)": "14556",
				"Product Stage 1a (Database)": "24165",
				"Product Stage 1b (Database)": "47434",
				"Product Stage 1e (Database)": "40016",
				"Product Stage 1h (Database)": "35144",
				"Product Stage 1i (Database)": "32192",
				"Store Stage 1 (Database)": "2516831",
				"URL Stage 1a (Database)": "95107",
				"URL Stage 1b (Database)": "43613",
				"URL Stage 1c (Database)": "46775",
				"URL Stage 1e (Database)": "16549",
				"URL Stage 1f (Database)": "13784"
			},
			"bulk": {
				"Attribute Stage 1a (Database)": "5882",
				"Attribute Stage 1b (Database)": "6113",
				"Catalog Stage 1a (Database)": "26680",
				"Category Stage 1a (Database)": "18708",
				"Category Stage 1b (Database)": "16279",
				"Category Stage 1d (Database)": "5338",
				"Price Stage 1a (Database)": "19485",
				"Price Stage 1b (Database)": "5834",
				"Product Stage 1a (Database)": "11655",
				"Product Stage 1b (Database)": "16584",
				"Product Stage 1e (Database)": "13505",
				"Product Stage 1h (Database)": "18351",
				"Product Stage 1i (Database)": "16888",
				"Store Stage 1 (Database)": "2953",
				"URL Stage 1a (Database)": "31512",
				"URL Stage 1b (Database)": "25061",
				"URL Stage 1c (Database)": "15489",
				"URL Stage 1e (Database)": "3783",
				"URL Stage 1f (Database)": "5523"
			}
		}
	},
	"processed": {
		"created": {
			"Attribute Stage 1a (Database)": "194",
			"Catalog Stage 1a (Database)": "1",
			"Category Stage 1a (Database)": "56",
			"Product Stage 1a (Database)": "3921",
			"Store Stage 1 (Database)": "12",
			"URL Stage 1a (Database)": "112",
			"URL Stage 1b (Database)": "7836",
			"URL Stage 1f (Database)": "11"
		},
		"updated": {
			"Attribute Stage 1b (Database)": "143",
			"Category Stage 1b (Database)": "56",
			"Category Stage 1d (Database)": "111",
			"Price Stage 1a (Database)": "3918",
			"Price Stage 1b (Database)": "2",
			"Product Stage 1b (Database)": "2930",
			"Product Stage 1e (Database)": "3916",
			"Product Stage 1h (Database)": "3926",
			"Product Stage 1i (Database)": "3921",
			"STA Stage 1 (Database)": "0",
			"Store Stage 1 (Database)": "3",
			"URL Stage 1c (Database)": "45",
			"URL Stage 1e (Database)": "2"
		},
		"failures": {
			"Attribute Stage 1a (Database)": "0",
			"Attribute Stage 1b (Database)": "0",
			"Catalog Stage 1a (Database)": "0",
			"Category Stage 1a (Database)": "0",
			"Category Stage 1b (Database)": "0",
			"Category Stage 1d (Database)": "0",
			"Price Stage 1a (Database)": "0",
			"Price Stage 1b (Database)": "0",
			"Product Stage 1a (Database)": "0",
			"Product Stage 1b (Database)": "0",
			"Product Stage 1e (Database)": "0",
			"Product Stage 1h (Database)": "0",
			"Product Stage 1i (Database)": "0",
			"Store Stage 1 (Database)": "0",
			"URL Stage 1a (Database)": "0",
			"URL Stage 1b (Database)": "0",
			"URL Stage 1c (Database)": "0",
			"URL Stage 1e (Database)": "0",
			"URL Stage 1f (Database)": "0"
		},
		"sizes": {
			"Attribute Stage 1a (Database)": "489675",
			"Attribute Stage 1b (Database)": "598803",
			"Catalog Stage 1a (Database)": "5913",
			"Category Stage 1a (Database)": "309371",
			"Category Stage 1b (Database)": "330336",
			"Category Stage 1d (Database)": "78642",
			"Price Stage 1a (Database)": "3313032",
			"Price Stage 1b (Database)": "1695",
			"Product Stage 1a (Database)": "15419202",
			"Product Stage 1b (Database)": "2259834",
			"Product Stage 1e (Database)": "66356979",
			"Product Stage 1h (Database)": "3964626",
			"Product Stage 1i (Database)": "5908305",
			"Store Stage 1 (Database)": "9789",
			"URL Stage 1a (Database)": "432404",
			"URL Stage 1b (Database)": "17249612",
			"URL Stage 1c (Database)": "61830",
			"URL Stage 1e (Database)": "1022",
			"URL Stage 1f (Database)": "27831"
		},
		"speed": {
			"Attribute Stage 1a (Database)": "1.00",
			"Attribute Stage 1b (Database)": "1.00",
			"Catalog Stage 1a (Database)": "0.00",
			"Category Stage 1a (Database)": "12.00",
			"Category Stage 1b (Database)": "38.00",
			"Category Stage 1d (Database)": "4.00",
			"Price Stage 1a (Database)": "8.00",
			"Price Stage 1b (Database)": "0.00",
			"Product Stage 1a (Database)": "38.00",
			"Product Stage 1b (Database)": "6.00",
			"Product Stage 1e (Database)": "180.00",
			"Product Stage 1h (Database)": "10.00",
			"Product Stage 1i (Database)": "16.00",
			"URL Stage 1a (Database)": "1.00",
			"URL Stage 1b (Database)": "46.00",
			"URL Stage 1c (Database)": "0.00",
			"URL Stage 1e (Database)": "0.00",
			"URL Stage 1f (Database)": "0.00"
		}
	},
	"sql": {
		"execution": {
			"Attribute Stage 1a (Database)": "387",
			"Attribute Stage 1b (Database)": "1071",
			"Catalog Stage 1a (Database)": "238",
			"Catalog Stage 1b (Database)": "108",
			"Category Stage 1a (Database)": "1631",
			"Category Stage 1b (Database)": "2123",
			"Price Stage 1a (Database)": "3500",
			"Price Stage 1b (Database)": "586",
			"Product Stage 1a (Database)": "8720",
			"Product Stage 1b (Database)": "407",
			"Product Stage 1e (Database)": "4456",
			"Product Stage 1h (Database)": "153",
			"Product Stage 1i (Database)": "284",
			"Store Stage 1 (Database)": "29",
			"URL Stage 1a (Database)": "6543",
			"URL Stage 1b (Database)": "956",
			"URL Stage 1c (Database)": "582",
			"URL Stage 1e (Database)": "197",
			"URL Stage 1f (Database)": "460"
		},
		"fetch": {
			"Attribute Stage 1a (Database)": "28",
			"Attribute Stage 1b (Database)": "32",
			"Catalog Stage 1a (Database)": "23",
			"Catalog Stage 1b (Database)": "6",
			"Category Stage 1a (Database)": "52",
			"Category Stage 1b (Database)": "28",
			"Price Stage 1a (Database)": "140",
			"Price Stage 1b (Database)": "3",
			"Product Stage 1a (Database)": "382",
			"Product Stage 1b (Database)": "3",
			"Product Stage 1e (Database)": "967",
			"Product Stage 1h (Database)": "53",
			"Product Stage 1i (Database)": "95",
			"Store Stage 1 (Database)": "6",
			"URL Stage 1a (Database)": "24",
			"URL Stage 1b (Database)": "566",
			"URL Stage 1c (Database)": "10",
			"URL Stage 1e (Database)": "2",
			"URL Stage 1f (Database)": "7"
		}
	}
}