Enabling global tracing ID
You can use a global tracing ID as well as customized fields in your logging files. The global ID makes problem determination easy, and you can use it with all your servers, including Liberty and the WebSphere Application Server.
HCL Commerce Server uses a modernized micro-service architecture, which includes the Transaction server, Search server, Store server, Customization server, and Utility server. This design provides numerous benefits, including an independent development environment, a sophisticated build and deployment cycle, more reasonable access volume distribution, and clear separation for customization, among other things. This does, however, increase the complexity of problem determination. The servers are separated, and they need service invocation for interaction. A large volume of requests passes through the servers, and it can be difficult to get a comprehensive picture of which servers participate in resolving any given request. Finding the root cause of a problem in a distributed system can be difficult using a logging system that is designed for a monolithic architecture.
HCL Commerce Version 9.0.1.3 introduces an enhanced global tracing ID that improves problem determination. In addition to using the tracing ID, you can add customized fields into the logging file. You can set a maximum size (in megabytes) for each trace file, and set a maximum number of trace files in WebSphere Application Server and Liberty. These enhancements make it possible for you to integrate log files from different channels. By tracing the global tracing ID or customized fields, you can find the root cause of issues much more quickly.
Before you begin
- Install Kafka Server and ZooKeeper server
- Refer to the official guide of Kafka and ZooKeeper
- Connect to Kafka and ZooKeeper
- We can use the default run engine commands to configure the Kafka and ZooKeeper setting in transaction, search, and store server.
About this task
There are two ways to configure the global tracing ID: by using the default REST API, or by updating the JVM configurations.
Procedure
-
Configure global tracing ID via the default REST API.
You can configure the global tracing ID in runtime environments by using the default REST API. For more information, see Extended Logging REST API Reference. The REST API can only be run by the SPI user.
To activate the ID, call the extended logging resource API to send the message to Kafka, and broadcast the message to all the servers. The method is PUT, and basic authentication for the
spiuser
ID and corresponding password must be included in the header.PUT https://transactionServerHost:transactionServerPort/wcs/resources/extendedlogger/asyncconfig?enableFlag=true®istrationFlag=true&customFields=XEL_example
Where the REST API parameters are:- enableFlag
- Used to enable or disable the extended logging. The value true will enable logging, while false will disable it.
- registrationFlag
- Used to register or unregister the custom fields. The value true means register, while false is used to unregister. When registrationFlag is false, and customFields is not used, all custom fields will be unregistered.
- customFields
- Used to register the custom fields added into extended log. This parameter must begin with “XEL_,". If you exclude this prefix, no custom information will be printed into the extended log. If you need to add many fields, separate them with commas. When a request is launched, add the customized fields into the header, and later, the customized fields will be printed in the extended log.
- containerNames
- Used to set the corresponding servers to consume the message. If this value is not set in the message, all the servers will response to the action. If it is set, only corresponding containers will respond and take effect. The value is the container host names, separated with commas.
Note: This method updates working memory for the server containers, but does not change the Docker images for the servers. Therefore, these configurations are not persistent; they will be lost after you restart the servers. -
Make the global tracing ID persistent via JVM configurations.
You can turn on and configure Global Tracing using two JVM parameters:
- global.tracing.enabled
- Used to enable or disable extended logging. The value true is used to enable extended logging, while false disables it.
- global.tracing.custom.fields
- Used to register the custom fields. If many fields need to be added, separate them with commas.
By configuring your application docker logic, you can ensure that these parameters are automatically set during docker startup. This approach requires that you rebuild the server docker images to resolve the specific configuration requirement. For a tutorial on customizing the default docker image based on your specific requirements, Tutorial: Customizing the Docker start-up logic.
The run engine command set-system-property can be used to set the JVM parameters. Refer to the following links for the command details in different servers:
Results
- /opt/WebSphere/AppServer/profiles/default/logs/**/extended.log.*
- /opt/WebSphere/Liberty/usr/servers/default/logs/**/extended.log.*
- Enable WebSphere Application Server logging with JSON output
-
If WebSphere Application Server traditional is configured for HPEL mode, log and trace data are output in binary format instead of plain text. Using HPEL LogViewer, log data can be continuously exported in JSON format and displayed in a Kibana dashboard.
WebSphere Application Server logs are output in plain text format by default, which is known as basic mode logging. For improved log and trace capabilities, upgrade the logging mode to HPEL (High Performance Extensible Logging).
The HPEL LogViewer is a WebSphere Application Server command-line tool for working with HPEL log data and trace data repositories. Filtering and formatting features in the tool make it easier to identify key material in data repositories.
- In your WebSphere Application Server environment in VM Quickstarter, change the basic logging mode to HPEL mode.
- Run the following LogViewer command, which
outputs the log and trace data to the specified log file in JSON
format and continuously monitors
it.
<WAS_ROOT>/bin/logViewer.sh -outLog <logviewer-output-log-file> -format json -resumable -resume -monitor &
- Enable WebSphere Application Server Liberty logging with JSON output
-
When the WebSphere Application Server Liberty
messages.log
file is output in JSON format, the output can be displayed in a Kibana dashboard.A Liberty server's messageFormat logging property controls the output format of the messages.log file. By default, the messageFormat property is set to basic, resulting in plain text output. Changing the messageFormat setting to json enables the output to appear on the Kibana dashboard.
In the
server.xml
file of your Liberty server, if you want to messageFormat logging property in JSON then uselogging messageFormat="json"
and if you want your consolseFormat property in JSON then useconsoleFormat="json"
to indicate which sources to include in the output.If you want to configure the CRS go to the following directory workspace_dir\Liberty\servers\crsServer\configDropins\overrides. Edit the custom configuration file server.xml. If the file does not exist, create it.
configDropins/overrides/server.xml <logging messageFormat="json" consoleFormat="json" traceSpecification="com.ibm.commerce.component.*=all"> </logging>
The ExtendedLogging and GlobalTracingID feature can be enabled without Kafka by enabling the JSON logging provided by WebSphere Liberty and WebSphere Application Server. When you enable both of these logging options, you will get comparable traces as shown below;-
In CRS application
{"type":"liberty_message","host":"LP1-AP-51879416.PROD.HCLPNP.COM","ibm_userDir":"W:\/IBM\/WebSphere\/Liberty\/usr\/","ibm_serverName":"crsServer","message":"Cannot find class impl with specified xpath: ComponentIntegrationServices\/Marketing\/@serviceImpl, will use default one instead.","ibm_threadId":"00000035","ibm_datetime":"2022-02-01T20:23:26.708+0530","module":"com.ibm.commerce.component.util.service.ComponentMarketingServiceFactory","loglevel":"WARNING","ibm_methodName":"initialize() executionContext:","ibm_className":"com.ibm.commerce.component.util.service.ComponentMarketingServiceFactory","ibm_sequence":"1643727206708_00000000000F2","ext_xRequestId":"ADHfu68Bvcs1kiA7","ext_thread":"Default Executor-thread-4"}
{"type":"liberty_message","host":"LP1-AP-51879416.PROD.HCLPNP.COM","ibm_userDir":"W:\/IBM\/WebSphere\/Liberty\/usr\/","ibm_serverName":"crsServer","message":"Configured name binding, com.ibm.commerce.foundation.server.services.search.contextpath, is not defined in server","ibm_threadId":"00000035","ibm_datetime":"2022-02-01T20:23:27.360+0530","module":"com.ibm.commerce.component.util.BackendServersConfigurationUtil","loglevel":"WARNING","ibm_methodName":"getComponentConfigThroughJNDI(String componentId, String configGroup, String propertyName, String defaultValue) executionContext:","ibm_className":"com.ibm.commerce.component.util.BackendServersConfigurationUtil","ibm_sequence":"1643727207360_00000000000F3","ext_xRequestId":"ADHfu68Bvcs1kiA7","ext_thread":"Default Executor-thread-4"}
- In TS application
{"type":"was_message","host":"LP1-AP-51879416.PROD.HCLPNP.COM","ibm_cellName":"localhost","ibm_nodeName":"localhost","ibm_serverName":"server1","ibm_sequence":"1643727200393_0000000000C68","message":"CMN1501E: User -1002 does not have the authority to perform action \"AjaxAccountAddressForm\" on resource \"com.ibm.commerce.command.HttpForwardViewCommandImpl\" for command \"null\".","ibm_datetime":"2022-02-01T20:23:20.393+0530","ibm_methodName":"isAllowed","ibm_className":"AccManager","ibm_threadId":"000001bc","module":"com.ibm.websphere.commerce.log.CommerceSrvr","loglevel":"SEVERE","ext_appName":"WC","ext_xRequestId":"ADHfu68Bvcs1kiA7"}
-