Usage Monitor Trace Reporter

Normal usage data collected by the Usage Monitor is extracted from program usage events and address space sampling and is aggregated over the daily collection cycle. The ZCAT utility can then be used to further aggregate data collected in the same calendar month to reduce the resource consumption of Usage Import.

The Usage Monitor Trace facility writes a detailed event record for each traced event, which is then preserved in a separate data set without affecting the data collected for Usage Import. You may even choose to trace events that are excluded from normal data collection. Usage data collected by address space sampling is not traced.

The data structure of the Usage Monitor Trace record is described by the HLASM DSECT in member HZASTRCD, which resides in the SHZASAMP library.

Even when HZAIBRWZ (which is shipped in the SHZAEXEC library and supplies record data item column headings) is used to browse Usage Monitor Trace records, it is difficult to extract useful, detailed information by simply browsing trace data. For this reason, the Usage Monitor Trace Reporter is provided to produce a formatted report displaying both details and summaries of the contents of Usage Monitor Trace records.

Normal usage data intended for database import contains program timing totals in units of hundredths of a second. The use of the STCK format for timing values in trace records allows more precise time measurements to be recorded in trace data. The Trace Reporter will show times less than one hundredth of a second with a precision of microseconds, longer times less than one minute with a precision of hundredths of a second, and other times as hours, minutes, and seconds. Event timestamps are shown as the system’s local time with a precision of microseconds. Times and timing totals are maintained internally without truncation, but are truncated to the selected precision when formatted for output.

Running the Trace Reporter

The Trace Reporter runs as a batch program with the trace data set being allocated to the TRACEIN DD and the report being written to the SYSPRINT DD.

The Trace Reporter will read the entire trace data set before producing the report, so a region size of about half the trace data set size is required. If a sufficiently large region is not available, utilities such as IDCAMS REPRO with the COUNT and SKIP operands can be used to make several smaller data sets that can each be processed in turn. Other software, such as ICETOOL, can be used to extract records for specific jobs or programs to be processed in isolation.

Reports are produced for each job in ASID order. The Event Trace shows the detailed log of traced events, and the Program Summary shows totals for each program.

The user identifier is shown for each event instead of in the job heading to cater for multi-user single-address-space scenarios where different tasks in the same address space may be associated with different users. For most jobs, including CICS regions, the displayed user identifier will be that of the entire address space.

Table 1. Event Trace report items
Heading	Description
Date	Local date of the event – shown as yyyy-mm-dd.
Time	Local time of the event – shown as hh:mm:ss.ffffff.
Program	Program name or nominal UNIX call stub name.
Alias-of	Real name of the program if the program name is an alias.
UserID	The security user identifier of the job or task.
Caller	Name of program that invoked the subject program.
Prov	Name of service providing the program.
Event-Type	Description of the program event type.
X	An X is displayed in this column whenever the event is excluded from normal collection.
Total-TCB	TCB CPU time of all types accumulated for this invocation.
TCB-on-CP	That part of Total-TCB that was on a general-purpose CP engine.
Offldable	That part of TCB-on-CP that was eligible for offloading to a specialty engine.
Top-RB-Tm	The elapsed time that the subject program was the active program of the task. Wait state time is included.
FDesc	The file descriptor (or file handle) of a z/OS UNIX file. This column is not shown if no UNIX system calls were traced for the job.

It is generally expected that z/OS UNIX system service calls will be dispatched entirely on general-purpose processor engines, so not showing the TCB-on-CP and Off loadable columns for these events represents no loss of data. This allows those columns to be used to report the number of directory entries or bytes (as appropriate) that were read or written by an I/O service call on the post-call report line. The label DEs will be shown after a directory-entry count, while the label bytes will be shown after a byte count.

The UNIX path name will be reported on the line following open and lstat UNIX pre-call events. The UNIX file descriptor (or file handle) is reported on open post-call, read and write, and close events, allowing file I/O events to be associated with the actual UNIX paths being processed.

The report column for the file descriptor is only displayed if any were traced for the job.

Table 2. Program Summary report items
Heading	Description
Program	Program name or nominal UNIX call stub name.
Alias-of	Real name of the program if the program name is an alias.
Events	Number of events traced for this program or service.
Executes	Number of times this program or service was executed. For programs, this is the number of EXIT events traced. For UNIX system calls, this is the number of post-call events traced.
Total-TCB	TCB CPU time total recorded in trace records for this program or service.
TCB-on-CP	That part of Total-TCB that was on a general-purpose CP engine.
Offldable	That part of TCB-on-CP that was eligible for offloading to a specialty engine.
Top-RB-Tm	The elapsed time that the subject program was the active program of the task. Wait state time is included.
Volume	Volume serial of program library – blank for UNIX files.
Data-Set-or-Filename	Data set name of program library or UNIX file name of program.

Program library details will be blank for z/OS UNIX system calls, but these report columns will be used to show directory-entry and byte totals for the UNIX I/O service.

Examples

In the following examples, the Usage Monitor started task name is assumed to be HZAJMON, and the Usage Monitor is assumed to be running while the system is active.

Example 1

Trace all usage of programs with names that begin with PROG by any job or address space. The trace data is to be collected in a new data set called USER.TRACE1 and the disk space size is limited to 100 cylinders. Events that are excluded from normal usage collection are to be included in the trace. All Usage Monitor Trace settings initially have their default values.

To execute this trace, the following system commands are issued:

F HZAJMON,TRCPGM(PROG*)
F HZAJMON,TRCEXC(Y)
F HZAJMON,TRCDSN(USER.TRACE1)
F HZAJMON,TRCSTA(NEW)
F HZAJMON,TRCPRI(1500)
F HZAJMON,TRCSEC(0)
F HZAJMON,TRCON

Example 2

Trace all program management and UNIX system call events for jobs belonging to the Stock Control application. These jobs have the characters STOK as characters 2 to 5 of the job name. The trace data is to be collected in a data set called USER.TRACE2, which has already been created. JES2 automatic commands are to be used to terminate the trace at 6 pm. This trace is to occur after the trace in the previous example is complete.

Firstly, an EBCDIC text file named stublist.txt is created in the /etc directory. The contents of /etc/stublist.txt are:

BPX1ACC BPX1ACK BPX1ACP BPX1AIO BPX1ALR BPX1ANR BPX1ASP BPX1ATM
BPX1ATX BPX1BND BPX1CCA BPX1CCS BPX1CHA BPX1CHD BPX1CHM BPX1CHO
BPX1CHP BPX1CHR BPX1CID BPX1CLD BPX1CLO BPX1CON BPX1CPL BPX1CPO
BPX1CRT BPX1CSE BPX1CTW BPX1CWA BPX1DEL BPX1DSD BPX1ENV BPX1EXC
BPX1EXI BPX1EXM BPX1EXT BPX1FAI BPX1FCA BPX1FCD BPX1FCM BPX1FCO
BPX1FCR BPX1FCT BPX1FPC BPX1FRK BPX1FST BPX1FSY BPX1FTR BPX1FTV
BPX1GAI BPX1GCL BPX1GCW BPX1GEG BPX1GEP BPX1GES BPX1GET BPX1GEU
BPX1GGE BPX1GGI BPX1GGN BPX1GGR BPX1GHA BPX1GHN BPX1GID BPX1GIV
BPX1GLG BPX1GMN BPX1GNI BPX1GNM BPX1GPE BPX1GPG BPX1GPI BPX1GPN
BPX1GPP BPX1GPS BPX1GPT BPX1GPU BPX1GPY BPX1GRL BPX1GRU BPX1GTH
BPX1GTR BPX1GUG BPX1GUI BPX1GWD BPX1HST BPX1IOC BPX1IPT BPX1ITY
BPX1KIL BPX1LCO BPX1LCR BPX1LDX BPX1LNK BPX1LOD BPX1LSK BPX1LSN
BPX1LST BPX1MAT BPX1MCT BPX1MDT BPX1MGT BPX1MKD BPX1MKN BPX1MMI
BPX1MMP BPX1MMS BPX1MNT BPX1MP BPX1MPC BPX1MPI BPX1MPR BPX1MSD
BPX1MSS BPX1MSY BPX1MUN BPX1NIC BPX1OPD BPX1OPN BPX1OPT BPX1OSE
BPX1PAF BPX1PAS BPX1PCF BPX1PCT BPX1PIO BPX1PIP BPX1POE BPX1POL
BPX1PQG BPX1PSI BPX1PST BPX1PTB BPX1PTC BPX1PTD BPX1PTI BPX1PTJ
BPX1PTK BPX1PTQ BPX1PTR BPX1PTS BPX1PTT BPX1PTX BPX1PWD BPX1QCT
BPX1QDB BPX1QGT BPX1QRC BPX1QSE BPX1QSN BPX1RCV BPX1RDD BPX1RDL
BPX1RDV BPX1RDX BPX1RD2 BPX1RED BPX1REN BPX1RFM BPX1RMD BPX1RMG
BPX1RMS BPX1RPH BPX1RW BPX1RWD BPX1SA2 BPX1SCT BPX1SDD BPX1SEC
BPX1SEG BPX1SEL BPX1SEU BPX1SF BPX1SGE BPX1SGI BPX1SGQ BPX1SGR
BPX1SGT BPX1SHT BPX1SIA BPX1SIN BPX1SIP BPX1SLK BPX1SLP BPX1SMC
BPX1SMF BPX1SMS BPX1SND BPX1SOC BPX1SOP BPX1SPB BPX1SPE BPX1SPG
BPX1SPM BPX1SPN BPX1SPR BPX1SPW BPX1SPY BPX1SRG BPX1SRL BPX1SRU
BPX1SRX BPX1SSI BPX1SSU BPX1STA BPX1STE BPX1STF BPX1STL BPX1STO
BPX1STR BPX1STV BPX1STW BPX1SUI BPX1SWT BPX1SYC BPX1SYM BPX1SYN
BPX1TAF BPX1TAK BPX1TDR BPX1TFH BPX1TFW BPX1TGA BPX1TGC BPX1TGP
BPX1TGS BPX1TIM BPX1TLS BPX1TRU BPX1TSA BPX1TSB BPX1TSC BPX1TSP
BPX1TST BPX1TYN BPX1UMK BPX1UMT BPX1UNA BPX1UNL BPX1UPT BPX1UQS
BPX1UTI BPX1VAC BPX1VCL BPX1VCR BPX1VEX BPX1VGA BPX1VGT BPX1VLK
BPX1VLN BPX1VLO BPX1VMD BPX1VOP BPX1VPC BPX1VRA BPX1VRD BPX1VRE
BPX1VRG BPX1VRL BPX1VRM BPX1VRN BPX1VRP BPX1VRW BPX1VSA BPX1VSF
BPX1VSY BPX1WAT BPX1WLM BPX1WRT BPX1WRV BPX1WTE BPX2ITY BPX2MNT
BPX2OPN BPX2RMS BPX2SMS BPX2TYN

Next, member BPXPRMT2 is created in a data set in the system PARMLIB concatenation. The contents of PARMLIB member BPXPRMT2 are:

SC_EXITTABLE('/etc/stublist.txt')

To allow dynamic exits installed by the Usage Monitor to get control when the system calls listed above are executed, the following system command is issued:

T OMVS=T2

To perform the described Usage Monitor trace, the following system commands are issued:

F HZAJMON,TRCPGD(PROG*)
F HZAJMON,TRCJOB(%STOK*)
F HZAJMON,TRCEXC(N)
F HZAJMON,TRCUNX(Y)
F HZAJMON,TRCDSN(USER.TRACE2)
F HZAJMON,TRCSTA(OLD)
F HZAJMON,TRCON
$TATRC2,T=18.00, '$VS, ''F HZAJMON,TRCOFF'''