Installing and configuring HCL ZAI Abend Intelligence

About this task

Deploy the on-host HCL ZAI Abend Intelligence pipeline to z/OS UNIX System Services (USS) to enable real-time AI analysis of application abends.

For general information about this enhancement, see The AI analysis report.

To install HCL ZAI Abend Intelligence:

Procedure

  1. Download Python 3.10 from https://www.python.org/downloads/ and deploy it and the required packages to your z/OS UNIX System Services (USS) environment. The required packages are found in data set HFZ.SHFZDOC1 member HFZZREQ. The Python engine imports components from the schemas and custom_exceptions packages. You must recreate this mandatory package subdirectory structure on your USS environment. If copied to a flat directory, the startup fails. 
    src/
├── HFZZMAIN.py            # entry point
├── HFZZUTIL.py
├── HFZZCNST.py
├── HFZZCONF.py
├── HFZZREQ.txt            # dependency pins
├── schemas/
│   ├── HFZZFDIN.py
│   ├── HFZZFDEX.py
│   └── HFZZFILE.py
└── custom_exceptions/
    └── HFZZEXCP.py
    Important:
    The pyzfile required package only works on Python 3.10. If using Python 3.11, IBM Z Open Automation Utilities (ZOAU) is mandatory.
    1. Verify the log directory path in HFZZCNST.py.

      Open HFZZCNST.py in an editor and check the LOG_DIRECTORY setting:

      LOG_DIRECTORY = "/tmp"
      Important:

      Ensure that LOG_DIRECTORY specifies a z/OS UNIX directory where background batch job user IDs have write access (for example, /tmp or a dedicated directory such as /var/fai/logs). If the user ID lacks write permission to this directory, log file creation fails and no diagnostic execution logs are generated.

  2. Create an FB 80 data set containing your LLM credentials and environment definitions. For details, see The STDENV environment variables for Abend Intelligence.
  3. Register the HCL ZAI Abend Intelligence user exits via the HFZOPTS DDname in your abending test job. 
    DATASETS(HFZEXEC(HFZ.SHFZSAM1))
EXITS(
    NOTIFY(REXX(HFZSUEAN)) 
    FORMAT(REXX(HFZSSHOW))
)
    The HFZSUEAN and HFZSSHOW user exits are provided in the HFZ.SHFZSAM1 data set. Ensure these exits are available to Z Abend Investigator via the HFZEXEC DDname.
    HFZSUEAN
    This is a Z Abend Investigator Notification user exit written in REXX that captures environment variables from the active fault recording session, invokes the orchestration launcher (HFZSBFCR) to build a JSON payload in /tmp/, and submits an asynchronous job to the internal reader to analyze the failure using a large language model (LLM).
    Verify or change the AI Service Configuration definitions in this exit as appropriate for your installation:
    Table 1. HFZSUEAN REXX Exit Configuration Variables
    Variable Name Example Value Description
    LLM_PROTOCOL_TO_USE 'watsonx' or 'ollama' Specifies the target large language model backend protocol: LLAMA or WATSONX.
    LLM_SERVICE_ENDPOINT 'http://13.82.3.70:11434' Specifies the base URL of the LLM endpoint (required for Ollama).
    LLM_MODEL_TO_USE 'llama3:latest' or 'ibm/granite-3-8b-instruct' Specifies the model identifier to invoke on the AI backend.
    USE_SOURCE_CODE YES Attempts DWARF/LANGX/LISTING source extraction.
    PROGRAM_MODE BF Only BF is supported.
    PYTHON_EXEC_TO_USE '/usr/lpp/IBM/cyp/v3r10/pyz/bin/python3' Specifies the fully qualified USS path to the Python 3.10 executable.
    PYTHON_MAIN_TO_USE '/usr/local/bin/fai/src/HFZZMAIN.py' Specifies the fully qualified USS path to the HFZZMAIN.py entry point.
    STDENV_TO_USE USERID()"ZAI.ENVS(LOCALENV)" Specifies the RACF-protected FB 80 data set containing API credentials shown in step 1.
    HFZOPTS_TO_USE USERID()".ZAI.OPTIONS(ZAIOPTS)" HCL ZAI Abend Intelligence options data set used in the generated JCL. No specific options are required to enable HCL ZAI Abend Intelligence.
    USE_ZOAU_MODE NO Setting this variable to YES passes USE_ZOAU=YES into the JSON payload.
    HFZSSHOW
    This is a Z Abend Investigator Formatting user exit written in REXX that reads the Markdown report data set <userid>.ZAI.REPORT(<fault_id>) and converts Markdown elements to Z Abend Investigator Formatting user exit tags (see Formatting tags).
  4. Submit the abending test job with a sample JSON payload in /tmp/ to verify outbound TCP/IP connectivity to your LLM endpoint.
  5. Verify the installation. You can verify the installation in the following ways:
    1. JES Spool Check: Open STDOUT from the automatically submitted HCL ZAI Abend Intelligence BPXBATCH job in SDSF and verify that the step finishes with RC=0 and displays the ### FAI Job Summary ### block with Status: Success.
      Note:
      A return code of RC=253 indicates that the pipeline executed successfully, but the LLM endpoint timed out or was unreachable over TCP/IP. Verify TCP/IP firewall rules and the LLM_SERVICE_ENDPOINT setting in HFZSUEAN.
    2. AI Report Data Set Member Check: View data set <userid>.ZAI.REPORT(<fault_id>), where <fault_id> is the fault ID created by the test job in step 4, using for example ISPF 3.4 to verify that the Markdown report content was written.
    3. Z Abend Investigator Report Check: Perform reanalysis of the fault entry written in step 4 and confirm that the AI report is included in the User section of the report.
  6. If the testing in step 5 was successful, propagate the HFZOPTS DDname options in step 3 to the HFZCNFxx parmlib member to make HCL ZAI Abend Intelligence available to all jobs and users.