OpenTelemetry Java agent extension

Capabilities of the Java agent include

• traceresponse: Enables a traceresponse response header.
• passthrough: Propagates selected request headers across services.
• threadMonitor: Automatic threaddump generation when conditions such as number of active spans, or slow spans are met.
• spanLogger: Configurable logger of spans.

Enabling the extension

The HCL extension is enabled by default with the OpenTelemetry agent, as described in OpenTelemetry support. The extension binary is available in containers under /SETUP/otel/opentelemetry-hcl-agent.jar. This Jar is registered using the otel.javaagent.extensions configuration. otel.javaagent.logging=application must be set for logging to occur.

When exporterOtlpEndpointEnabled is true, the extension is registered along with the other agent configurations in the configuration map.

  configMap:
    otel_config_properties: |-
      otel.exporter.otlp.endpoint=
      otel.collector.namespace=YOUR-OTEL-NAMESPACE
      otel.exporter.otlp.metrics.temporality.preference=cumulative
      otel.exporter.otlp.protocol=http/protobuf
      otel.service.name=${OTEL_SERVICE_NAME}
      otel.resource.attributes=deployment.environment=${TENANT}${ENVIRONMENT}${ENVTYPE}
      otel.javaagent.extensions=/SETUP/otel/opentelemetry-hcl-agent.jar
      otel.javaagent.logging=application

If exporterOtlpEndpointEnabled is false, the extension is enabled by default in /SETUP/otel/otel_noexporters.properties.

In Commerce+, by default, sampling is not enabled in the agent configuration. If you decide to enable sampling in the agent configuration (using the otel.traces.sampler configuration), not all spans will be reported to the HCL extension.

Customizing configuration values

The extension configuration file hcl_otel_agent_config.yaml supports configurable values. When a value is specified with format ${key_name:default_value}, the system uses the default value if the key is not declared as a configuration file entry, environment variable, or system property. The extension follows the OpenTelemetry configuration rules as the OpenTelemetry agent.

For example, threadMonitor triggers a threaddump if the current number of active spans is higher than the configured value, which defaults to 20.

    spans:
      activeLargerThan: ${otel.javaagent.extensions.hcl.threadmonitor.active-larger-than:20} 

If exporterOtlpEndpointEnabled is true, this value can be overwritten in otel_config_properties, as defined in the Helm Chart. The change will apply to all containers.

  configMap:
    otel_config_properties: |-
      ...
      otel.javaagent.extensions=/SETUP/otel/opentelemetry-hcl-agent.jar
      otel.javaagent.logging=application
      otel.javaagent.extensions.hcl.threadmonitor.active-larger-than=20

Configurations can also be defined as environment variables to apply to a single container.

- name: OTEL_JAVAAGENT_EXTENSIONS_HCL_THREADMONITOR_ACTIVE_LARGER_THAN
  value: 20

You can customize the image for extended container-specific configurations with a new otel_config_properties configuration map. For ad-hoc testing, such as tracing, changes made to the file will be reloaded by the extension in runtime.

Available features

Following are some of the features currently provided by the HCL extension.

traceresponse response header

Appends the traceresponse response header to the response, which includes the trace_id used by the request.

traceresponse: 00-3857f023970fb38eb4b098826d31fd21-d17d97455bb26a4d-00

Passthrough

Enables the automatic propagation of specified headers across pods, such as X-Log-Level.

Edit /SETUP/otel/hcl_otel_agent_config.yaml.

passthrough:
  enabled: ${otel.javaagent.extensions.hcl.passthrough.enabled:true}
  headers:
  - name: X-Log-Level
    # Optional regex validation
    regex: TRACE|DEBUG

In addition to enabling the feature in hcl_otel_agent_config.yaml, the hclpassthrough must be registered with the OpenTelemetry configuration:

otel.propagators: tracecontext,baggage,hclpassthrough

Thread Monitor

This version replaces the original implementation of threadMonitor in Commerce+. It automates thread dump generation based on criteria such as active or slow spans, and it includes OpenTelemetry details with each thread.

"Default Executor-thread-1", id: 34, state:RUNNABLE
ACTIVE: "name": "GET", "kind": "SERVER", "tracer": "io.opentelemetry.servlet-5.0:2.7.0-alpha", "traceId": "fd7870132f4c8447ab617d68fec1e6aa-d9fc60c4f8542815", "parent.traceId": "00000000000000000000000000000000-0000000000000000", "thread": "Default Executor-thread-1", "client.address": "10.128.0.132", "server.address": "www.perfcluster9qalive.svt.commerce.com, www.perfcluster9qalive.svt.commerce.com", "user_agent.original": "curl/7.81.0", "url.query": "store=Emerald&partNumber=SKU190164&fulfillmentCenter=13001&limit=1&availableToPromise=false", "url.path": "/inventory/api/v1/item-inventories", "http.request.method": "GET", "url.scheme": "https", "thread.id": "52", "startTime": "2024-12-13T14:14:29.0304Z", "elapsedMs": 202,351, "ended": "false"
ACTIVE: "name": "SELECT inventory.items", "kind": "CLIENT", "tracer": "io.opentelemetry.jdbc:2.7.0-alpha", "traceId": "fd7870132f4c8447ab617d68fec1e6aa-3b1374671c827cc7", "parent.traceId": "fd7870132f4c8447ab617d68fec1e6aa-d9fc60c4f8542815", "thread": "Default Executor-thread-1", "server.address": "perfcluster9qalivepostgres.commerce.svc.cluster.local", "db.connection_string": "postgresql://perfcluster9qalivepostgres.commerce.svc.cluster.local:5432", "db.user": "inventory", "db.statement": "select i1_0.partnumber,i1_0.backorder_limit,i1_0.hasexpectedinventory,i1_0.inventorysystem,i1_0.lagtimeoffset,i1_0.maxbooffset,i1_0.max_threshold,i1_0.min_threshold,i1_0.product_desc,i1_0.reservetimeoffset,i1_0.trackinventory,i1_0.quantitymeasure from inventory.items i1_0 where i1_0.partnumber=?", "db.system": "postgresql", "db.sql.table": "inventory.items", "server.port": "5432", "db.operation": "SELECT", "db.name": "inventorydb", "thread.id": "52", "startTime": "2024-12-13T14:14:29.0310Z", "elapsedMs": 202,345, "ended": "false"
ENDED:  "Transaction.commit": 1ms/3 , "FulfillmentCenterRepository.findActiveNonDeletedIds": 2ms/1 , "SELECT inventory.availableinventory": 1ms/3 , "inventorydb": 0ms/1 , "SELECT inventory.fulfilmentcenter": 0ms/1 , "SELECT com.hcl.inventory.entity.AvailableInventory": 2ms/3 , "AvailableInventoryRepository.findDistinctPartNumbersByStoreAndPartNumberAndFfmCenter": 2ms/1 , "AvailableInventoryRepository.findByStoreInAndPartNumberInAndFfmCenterIdIn": 1ms/1 , "SELECT com.hcl.inventory.entity.FulfillmentCenter": 0ms/1
Java callstack:
    at ...	
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:517)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:434)
    at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194)

The OpenTelemetry details may not be synchronized with the stack reported.

threaddump files are expected to be in the location of javacore files. For example, this thread dump was generated in a transaction server pod in the default location, /opt/ol/wlp/usr/servers/transactionServer/cores/container/pod_name/threaddump.20250612.153715.2077.0002.txt.

This threadMonitor configuration example will trigger thread dumps every 30 seconds:

 # Collect threaddumps at the set interval (in seconds)
    intervalSeconds: ${otel.javaagent.extensions.hcl.threadmonitor.interval:30}

The setting can be updated in the configuration file (/SETUP/otel/hcl_otel_agent_config.yaml) in real-time. An environment variable can also be defined so the configuration can be read during startup.

Use either the Kubernetes ConfigMap or the Commerce+ server components kubernetes deployment.

For example, to configure a 30 second interval between thread dumps, follow either of these two methods:

1. Edit the deployment of the desired server component:

   kubectl edit deployment perfcluster1qalivets-app -n commerce

   Add the following environment variable:

    - name: OTEL_JAVAAGENT_EXTENSIONS_HCL_THREADMONITOR_INTERVAL
       value: "30"

   The pod automatically restarts and triggers thread dumps at the specified interval.

2. Edit the Commerce OTEL Extension ConfigMap.

   kubectl edit cm svt-qa-live-tenant_name-opentelemetry-config -n commerce

   and append this directive:

    otel.javaagent.extensions.hcl.threadmonitor.interval=30

   Restart the pod to read the changes and begin generating thread dumps.

SpanLogger

Logs span onStart and onEnd events that match the configured criteria.

To enable, set enabled=true and the logger at the required level:

start logger: com.hcl.opentelemetry.agent.spanlogger.onStart=LEVEL
end logger:   com.hcl.opentelemetry.agent.spanlogger.onEnd=LEVEL
To enable both: com.hcl.opentelemetry.agent.spanlogger=LEVEL

When enabling in the query-app server, edit the logging.xml file /opt/ol/wlp/usr/servers/defaultServer/configDropins/overrides/logging.xml. For example:

<server>
  <logging suppressSensitiveTrace="true" traceFormat="BASIC" traceSpecification="*com.hcl.opentelemetry.agent.spanlogger=all" maxFileSize="20" maxFiles="10" traceFileName="trace.log" hideMessage="SRVE9967W"/>
</server>

  onStart:
  # - level: ERROR,WARN,INFO,DEBUG (maps to FINE),TRACE (maps to FINEST). Defaults to INFO
  #   kind: optional: INTERNAL,SERVER,CLIENT,PRODUCER,CONSUMER
  #   tracer:  optional e.g. io.opentelemetry.jdbc
  #   stack: [true|FALSE]

The following sample configuration logs uses ERROR level onEnd span events from the io.opentelemetry.jdbc tracer. Events are logged if the statement execution lasts one second or longer.

  onEnd:
  - level: ERROR
    tracer: io.opentelemetry.jdbc
    durationMilliseconds: 1000
    stack: false

Log Sample:

[6/5/25, 18:36:55:642 UTC] 0000009e onEnd         3 io.opentelemetry.javaagent.instrumentation.internal.logging.Slf4jApplicationLoggerBridge$Slf4jApplicationLogger log onEnd: "name": "GET /wcs/resources/", "kind": "SERVER", "tracer": "io.opentelemetry.servlet-3.0", "traceId": "348025c5398953121b9cded1d574671e-fd15cb068b534bad", "parent.traceId": "00000000000000000000000000000000-0000000000000000", "thread": "Default Executor-thread-16", "thread.id": "158", "http.response.status_code": "200", "url.scheme": "https", "network.peer.port": "35286", "url.path": "/wcs/resources/health/ping", "client.address": "10.224.4.1", "user_agent.original": "kube-probe/1.30", "network.peer.address": "10.224.4.1", "network.protocol.version": "2.0", "http.route": "/wcs/resources/", "http.request.method": "GET", "startTime": "2025-06-05T18:36:55.0638Z", "endTime": "2025-06-05T18:36:55.0642Z", "elapsedMs": 3.52, "ended": "true"

This sample configuration defines three levels of logging: ERROR, WARN and INFO, which specify user-defined limits on the response time of JDBC statements. Queries taking 10 milliseconds or more are logged as ERRORs. Those taking between 8 and 10 ms are logged as WARNings; and those taking 6 to 8 ms are logged as INFOrmational.

onEnd
- level: ERROR
  tracer: io.opentelemetry.jdbc
  durationMilliseconds: 10
  stack: false
- level: WARN
  tracer: io.opentelemetry.jdbc
  durationMilliseconds: 8
  stack: false
- level: INFO
  tracer: io.opentelemetry.jdbc
  durationMilliseconds: 6
  stack: false

Note the order of statements. Evaluation is performed top down and stops after the first condition is met.

Following are some example sample log entries using the tiered logging levels.

[6/11/25, 13:47:28:857 UTC] 00000094 onEnd         E io.opentelemetry.javaagent.instrumentation.internal.logging.Slf4jApplicationLoggerBridge$Slf4jApplicationLogger log onEnd: "name": "UPDATE mall.ADDRESS", "kind": "CLIENT", "tracer": "io.opentelemetry.jdbc", "traceId": "0e189b533d139e9b91b5579eac2aacc2-ffcf51134894b790", "parent.traceId": "0e189b533d139e9b91b5579eac2aacc2-422bfc18e4ad1fd0", "thread": "Default Executor-thread-13", "db.connection_string": "postgresql://perfcluster1qalivesvt-postgres.commerce.svc.cluster.local:5432", "server.address": "perfcluster1qalivesvt-postgres.commerce.svc.cluster.local", "db.system": "postgresql", "db.statement": "UPDATE ADDRESS SET STATUS = ?, OPTCOUNTER = ? WHERE ((ADDRESS_ID = ?) AND (OPTCOUNTER = ?))", "db.operation": "UPDATE", "server.port": "5432", "db.sql.table": "ADDRESS", "thread.id": "148", "db.name": "mall", "startTime": "2025-06-11T13:47:28.0839Z", "endTime": "2025-06-11T13:47:28.0856Z", "elapsedMs": 17.46, "ended": "true"

[6/11/25, 13:48:26:940 UTC] 00000097 onEnd         W io.opentelemetry.javaagent.instrumentation.internal.logging.Slf4jApplicationLoggerBridge$Slf4jApplicationLogger log onEnd: "name": "INSERT mall.ORDERITEMS", "kind": "CLIENT", "tracer": "io.opentelemetry.jdbc", "traceId": "f6171a304de5ce34a6f8c8ef74465eb1-abef9752bec083a1", "parent.traceId": "f6171a304de5ce34a6f8c8ef74465eb1-b212789a9edb72d1", "thread": "Default Executor-thread-16", "db.connection_string": "postgresql://perfcluster1qalivesvt-postgres.commerce.svc.cluster.local:5432", "server.address": "perfcluster1qalivesvt-postgres.commerce.svc.cluster.local", "db.system": "postgresql", "db.statement": "INSERT INTO ORDERITEMS (ORDERITEMS_ID, ADDRESS_ID, ALLOCFFMC_ID, ALLOCQUANTITY, ALLOCADDRESS_ID, ALLOCATIONGROUP, AVAILQUANTITY, BASEPRICE, BASECURRENCY, CATENTRY_ID, COMMENTS, CONFIGURATIONID, TRADING_ID, CORRELATIONGROUP, CURRENCY, DESCRIPTION, ESTAVAILTIME, FIELD1, FIELD2, FFMCENTER_ID, FULFILLMENTSTATUS, INVENTORYSTATUS, ISEXPEDITED, ITEMSPC_ID, LASTALLOCUPDATE, LASTUPDATE, LINEITEMTYPE, MEMBER_ID, NEEDEDQUANTITY, OFFER_ID, OPTCOUNTER, ORDERS_ID, ORDRELEASENUM, OUTPUTQ_ID, PARTNUM, PREPAREFLAGS, PRICE, PROMISEDAVAILTIME, QUANTITY, REQUESTEDSHIPDATE, SHIPCHARGE, SHIPMODE_ID, SHIPPINGOFFSET, SHIPTAXAMOUNT, STATUS, STOREENT_ID, SUPPLIERDATA, SUPPLIERPARTNUMBER, TAXAMOUNT, TERMCOND_ID, TIECODE, LASTCREATE, TIMERELEASED, TIMESHIPPED, TOTALADJUSTMENT, TOTALPRODUCT, TRACKDATE, TRACKNUMBER) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", "db.operation": "INSERT", "server.port": "5432", "db.sql.table": "ORDERITEMS", "thread.id": "151", "db.name": "mall", "startTime": "2025-06-11T13:48:26.0931Z", "endTime": "2025-06-11T13:48:26.0940Z", "elapsedMs": 8.75, "ended": "true"

[6/11/25, 13:48:27:350 UTC] 00000097 onEnd         I io.opentelemetry.javaagent.instrumentation.internal.logging.Slf4jApplicationLoggerBridge$Slf4jApplicationLogger log onEnd: "name": "INSERT mall.ORDPAYINFO", "kind": "CLIENT", "tracer": "io.opentelemetry.jdbc", "traceId": "f6171a304de5ce34a6f8c8ef74465eb1-362f4062f5ffffd6", "parent.traceId": "f6171a304de5ce34a6f8c8ef74465eb1-b212789a9edb72d1", "thread": "Default Executor-thread-16", "db.connection_string": "postgresql://perfcluster1qalivesvt-postgres.commerce.svc.cluster.local:5432", "server.address": "perfcluster1qalivesvt-postgres.commerce.svc.cluster.local", "db.system": "postgresql", "db.statement": "INSERT INTO ORDPAYINFO (ORDPAYINFO_ID, OPTCOUNTER, NAME, VALUE, ORDERS_ID) VALUES (?, ?, ?, ?, ?)", "db.operation": "INSERT", "server.port": "5432", "db.sql.table": "ORDPAYINFO", "thread.id": "151", "db.name": "mall", "startTime": "2025-06-11T13:48:27.0344Z", "endTime": "2025-06-11T13:48:27.0350Z", "elapsedMs": 6.02, "ended": "true"

elapsedMs: 17.46 is indicated by an "E" message in the log because it exceeds the 10ms limit.

elapsedMs: 8.75 is indicated by an "W" message in the log because it exceeds the 8ms limit.

elapsedMs: 6.02 is indicated by an "I" message in the log because it exceeds the 6ms limit.

Note: Tracers

OpenTelemetry instrumentation libraries, also referred to as instrumented tracers, are prebuilt components that automatically generate telemetry data such as traces, metrics, and logs. They work by hooking into common libraries and frameworks (e.g., HTTP clients, databases, or messaging systems). Each library identifies itself in telemetry using a standardized instrumentation scope name (e.g., io.opentelemetry.jdbc), making it easier to attribute spans to specific technologies in your application stack.

Some common instrumentation scope names include: io.opentelemetry.jdbc, io.opentelemetry.tomcat-7.0, io.opentelemetry.elasticsearch-rest-7.0.

These libraries span a wide range of technologies, and it may not always be obvious which tracer names are in use in your system. You can inspect these names directly in your OpenTelemetry dashboard under trace metadata. Alternatively, you can enable logging using LogSpanner to capture all tracers of a given type.

For example, to log all tracers of a given kind, specify a tracer kind (see the extension configuration file hcl_otel_agent_config.yaml for supported kinds). You can log all associated instrumentation scope names automatically by specifying:

level: INFO
  kind: SERVER

This configuration logs all tracers of kind SERVER, including those listed above as INFO messages.

You can also use the specific tracer names found to further refine the filters and narrow your logging scope.