Opening incidents through ServiceNow

When an anomaly is detected, the faster you react the better. To have the Z controller open incidents through ServiceNow© and manage problems through a ticketing systems, you need to:

Add the EQQINCID DD statement to the HCL Workload Automation for Z JCL procedure. For detailed information about EQQINCID, see Incident data set (EQQINCID).
Define the INCOPTS initialization statement.
Set the INCIDENT parameter in the ALERTS initialization statement.

To enable the Z controller to access the resources of ServiceNow, generate an OAuth token as follows:

From your ServiceNow instance, navigate to All > SystemOAuth > Application Registry.
Create the record to be associated with the token by clicking New and selecting Create an OAuth API endpoint for external clients.
Specify a Name for the record.
In the Access Token Lifespan field set the time, in seconds, before the token expires. The maximum value allowed is 2,147,483,647 seconds, corresponding to 6 years.
In the Client Secret field, set the value you want or leave it blank to have ServiceNow generate a value for you.
Click Submit.
Navigate to the Application Registry library and select the record the you have created. Take note of the Client ID and Client Secret.
Close the instance.
Navigate to https://www.postman.com/ and sign up.
Create a workspace with visibility Personal.
Click Import.

Copy the following JSON form in the text box:

{ 
    "info": { 
        "_postman_id": "5933fa3c-bfdd-4868-9918-50a22e277d6f", 
        "name": "Token Request", 
        "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json", 
        "_exporter_id": "15523583" 
    }, 
    "item": [ 
        { 
            "name": "Token Request", 
            "request": { 
                "method": "POST", 
                "header": [], 
                "body": { 
                    "mode": "urlencoded", 
                    "urlencoded": [ 
                        { 
                            "key": "grant_type", 
                            "value": "password", 
                            "type": "text" 
                        }, 
                        { 
                            "key": "username", 
                            "value": "", 
                            "type": "text" 
                        }, 
                        { 
                            "key": "password", 
                            "value": "", 
                            "type": "text" 
                        }, 
                        { 
                            "key": "client_id", 
                            "value": "", 
                            "type": "text" 
                        }, 
                        { 
                            "key": "client_secret", 
                            "value": "", 
                            "type": "text" 
                        } 
                    ] 
                }, 
                "url": { 
                    "raw": "https://<instance_name>.service-now.com/oauth_token.do", 
                    "protocol": "https", 
                    "host": [ 
                        "<instance_name>", 
                        "service-now", 
                        "com" 
                    ], 
                    "path": [ 
                        "oauth_token.do" 
                    ] 
                } 
            }, 
            "response": [] 
        } 
    ] 
}

Click Continue.
Under Collections, click Post Token Request.
In the URL address, replace <instance_name> with your instance name. In the Body tab, fill in the form with the name of the user who will open the incidents in ServiceNow, password, and Client ID and Client Secret that you noted in step g.
Note: All the user names are stored in All >User Administration>Users. If you are the administrator, you can use admin as the user name, along with the password provided by ServiceNow.
Click Send. A response like the following example is displayed. Take notice of the access_token, because it is the value to be set in the INCTICKETTOKEN parameter of the INCOPTS statement.
```
"access_token": "                     ",
"refresh_token": "                   ",
"scope": "useraccount",
"token_type": "Bearer",
"expires_in": 1799
```

????? DA CONFERMARE DAL TEST Add the certificates that you have downloaded from your incident notifying tool to a key ring that is trusted by HCL Workload Automation for Z, as follows:
If you use a SAF ring, perform the following steps:
1. Create the sequential data sets where the downloaded certificates are to be stored. In this procedure, the certificates INCTOOL.CERT.SERVER and INCTOOL.CERT.ROOT are used as an example.
  If your certificate is chained, you must create a data set for each intermediate or root certificate and save them.
2. Create a key ring (in this example, EQQRING) by using the certificates management command RACDCERT. Skip this step if you already use a key ring for HCL Workload Automation for Z services. (For more information about the RACDCERT command, see the section RACDCERT (Manage RACF digital certificates) in the IBM z/OS Server Security RACF Command Language Reference manual).
```
RACDCERT ADDRING(EQQRING) ID(Your_RACF_userID)
```
3. Add each certificate to the RACF database:
```
RACDCERT CERTAUTH ID(Your_RACF_userID) ADD('INCTOOL.CERT.ROOT') TRUST WITHLABEL('INCTOOL ROOT')
```
```
RACDCERT CERTAUTH ID(Your_RACF_userID) ADD('INCTOOL.CERT.SERVER') TRUST WITHLABEL('INCTOOL SERVER')
```
4. Connect each certificate to the EQQRING key ring :
```
RACDCERT ID(Your_RACF_userID) CONNECT(LABEL('INCTOOL ROOT') RING(EQQRING) USAGE(CERTAUTH)
```
```
RACDCERT ID(Your_RACF_userID) CONNECT(LABEL('INCTOOL SERVER') RING(EQQRING) USAGE(PERSONAL)
```
5. Check that the certificates have been successfully added to the chain:
```
RACDCERT ID(Your_RACF_userID) LISTCHAIN(LABEL('INCTOOL SERVER')
```
6. Update the SSL parameters in the HTTPOPTS statement according to the values that you have set in this procedure.
If you use the keystore in the UNIX System Services, perform the following steps:
1. Save the downloaded certificates into a USS directory. In this procedure, /u/mycerts is used as an example.
  If your certificate is chained, you must create a file for each intermediate or root certificate and save them.
2. From /u/mycerts, create a keystore database (in this procedure, the gskkyman utility is used). Skip this step if you already use a database for HCL Workload Automation for Z services.
```
gskkyman
```
3. From the Database Menu, select option 1 - Create new database. Skip this step if you already use a database for HCL Workload Automation for Z services.On completion, the following message is issued:
```
Key database /u/mycerts/my_db_name.kdb created
```
4. To store your database password in a file, from the Key Management Menu select option 10 - Store database password. Skip this step if you already use a database for HCL Workload Automation for Z services.
  On completion, the following message is issued:
  Database password stored in /u/mycerts/my_db_name.sth
5. From the Database Menu, select option 2 - Open database and enter the key database name and database password.
6. Import each certificate to your keystore database by selecting option 7 - Import a certificate from the Key Management Menu.
7. Based to the values that you set in this procedure, update the SSL parameters in the HTTPOPTS statement.

You can set that the incidents are opened for the following alert conditions:

DURATION: An operation in the current plan is active for an unexpectedly long time.
ERROROPER: An operation in the current plan is set to ended-in-error status.
HIGHRISK: The risk level of a critical operation in the current plan has become High.
LATEOPER: An operation in the current plan becomes late, which means that it reaches its latest start time and does not have the status started, complete, or deleted.
OPCERROR: An HCL Workload Automation for Z subtask or subsystem ends unexpectedly.
POTENTRISK: The risk level of a critical operation in the current plan has become Potential.
SPECRES: The time that an operation in the current plan is waiting to allocate a given resource exceeds the time specified by the RESOPTS CONTENTIONTIME parameter.
WLMOPER: An operation in the current plan is promoted by WLM.

The EQQINCID data set includes:

The members containing the text of the incidents, which you set in ALERTS INCIDENT.
A member named RULESTCK (required).
This member contains the rules that must be met for the incidents to be opened on ServiceNow. Each rule consists of a FILTER, HEADER, and optionally a TEXTMEMBER, in the following format:
```
FILTER(expression1, expression2, ..., expressionn)
HEADER(header_text)
[TEXTMEMBER(member_name)]
```
Note: Each rule is associated with only one FILTER, HEADER, and TEXTMEMBER. If within a single rule you specify more than one FILTER, HEADER, or TEXTMEMBER, only the first occurrence is taken into account.
Where:
FILTER(expression1, expression2, ..., expressionn)
The expressions to be satisfied for the incident to be opened, separated by commas. The incident is opened when all the expressions in the filter are met; for each satisfied filter the corresponding incident is opened.
Each expression has the following format, which is not case-sensitive:
```
value=filter
```
where:

value

String of alphanumeric characters, included variables (for details about variables, see Variables allowed in the EQQINCID members ). It cannot contain blanks.

filter

String of alphanumeric characters. It cannot contain blanks. You can use the wildcard characters asterisk (*) and percent sign (%).
For example, you can set a FILTER that includes all the applications whose name begins with MY and ended with error code 16, as follows:
```
FILTER(&OADID=MY*,&OERRCODE=16) 
```
HEADER(header_text)
Information used for the incident header, separated by blanks. As the header_text you can specify the following information.
Note:
- Each piece of information (ShortDesc, Impact, and Urgency) is followed by colons (:) and can be set only once. If you specify more than one, only the first is considered.
- The sign colons (:) cannot be specified inside the header_text. If you specify it, the text that follows is not considered.
ShortDesc:

Required.

Impact:

Optional. Valid values are high, medium, low. The default is low.

Urgency:

Optional. Valid values are high, medium, low. The default is low.
For example, you can set a HEADER as follows:
```
HEADER(
ShortDescr: This is the application error
Impact: High 
Urgency: Medium 
)
```
TEXTMEMBER (member_name)

Optional. The member containing the description of the incident. If you do not specify any, the member set in ALERTS INCIDENT is used as default. For each alert condition, one member is defined.

Variables allowed in the EQQINCID members shows you the variables that you can use in the EQQINCID members. Variables are resolved only if they are meaningful for the event condition that you have set in the ALERTS INCIDENT parameter. Otherwise, the variable is ignored.

Table 1. Variables allowed in the EQQINCID members
Variable name (must be preceded by &)	Variable description	Max length	Alert condition
ALERCOND	Alert condition that generated the incident (for details, see the alert conditions listed in ALERTS INCIDENT). It can assume the following values: DURATION ERROROPER HIGHRISK LATEOPER OPCERROR POTENTRISK SPECRES WLMOPER	10	DURATION, ERROROPER, HIGHRISK, LATEOPER, OPCERROR, POTENTRISK, SPECRES, WLMOPER
OADID	Application ID	16	DURATION, ERROROPER, HIGHRISK, LATEOPER, POTENTRISK, SPECRES, WLMOPER
OADOWNER	Occurrence owner.	16	DURATION, ERROROPER, LATEOPER, SPECRES, WLMOPER
OTOKEN	Occurrence token.	8	DURATION, ERROROPER, LATEOPER, SPECRES, WLMOPER
OAUGROUP	Authority group.	8	DURATION, ERROROPER, HIGHRISK, LATEOPER, POTENTRISK, SPECRES, WLMOPER
ODMY1	Occurrence input arrival date, DDMMYY.	6	DURATION, ERROROPER, HIGHRISK, LATEOPER, POTENTRISK, SPECRES, WLMOPER
OJOBNAME	Operation job name.	8	DURATION, ERROROPER, HIGHRISK, LATEOPER, POTENTRISK, SPECRES, WLMOPER
OOPNO	Operation number within the occurrence, right-justified and padded with zeros.	3	DURATION, ERROROPER, HIGHRISK, LATEOPER, POTENTRISK, SPECRES, WLMOPER
OWSID	Workstation ID for the current operation.	4	DURATION, ERROROPER, HIGHRISK, LATEOPER, POTENTRISK, SPECRES, WLMOPER
OJOBID	Job number.	8	DURATION, ERROROPER, WLMOPER
OERRCODE	Error code.	4	ERROROPER
RESNAME	Resource name.	44	SPECRES
RESWTTM	Resource waiting time	4	SPECRES
TASKNAME	HCL Workload Automation for Z task name.	16	OPCERROR

Troubleshooting

When errors occur in detecting and notifying an incident, messages are logged in the EQQMLOG file. You can set a further level of diagnosis by adding DIAGNOSE MONFLAGS(X'00000200') to the member of the EQQPARM library.