Troubleshooting Clara

Symptom

Docker login returns: "Error response from daemon: login attempt to https://854793243809.dkr.ecr.us-east-1.amazonaws.com/v2/ failed with status: 403 Forbidden"

Cause and solution

Your credentials are not valid. Request new credentials from your vendor.

Symptom

Docker or docker-compose commands return: "Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.38/containers/json: dial unix /var/run/docker.sock: connect: permission denied"

Cause and solution

You don't have the right permissions for Docker. Use commands with root permissions ("sudo" or "sudo su") or refer to Docker documentation.

Symptom

Nginx container doesn't start returning the following error: "ERROR: for nginx  Cannot start service nginx: driver failed programming external connectivity on endpoint ngiinx".

Cause and solution

Stop and start docker compose by running the following commands:

1. docker-compose stop

2. docker-compose up

If the problem persists, run the following steps:

1. restart docker with docker restart command. See Docker command usage.

2. docker-compose up

Symptom

When uploading the Autobot License file, it returns: "Invalid Licence, Bot Framework Not Initialized".

Cause and solution

Your license file is not valid, Request a new license file from your vendor.

Symptom

During Clara installation, the command clara.sh --load-project <prj> fails with the following error: {  "status": "Exception occuredjava.net.UnknownHostException: datastore"}.

Cause and solution

The most common cause for this error is a network issue within “clara-net” (docker network ls). In this situation the containers are unable to communicate with each other.

To verify if you are in this situation, run the following steps:

1.  Connect to clara-autobot container:​

    docker exec -ti clara-autobot bash​

2. Run the command:​

    curl http://datastore:9200​

if you get something like the following output,  the connection is working:​

{​

  "name" : "Vtmbdmn",​

  "cluster_name" : "autobot",​

  "cluster_uuid" : "b3r4-1IkR6OHk49dnpd3gA",​

  "version" : {​

    "number" : "5.6.2",​

    "build_hash" : "57e20f3",​

    "build_date" : "2017-09-23T13:16:45.703Z",​

    "build_snapshot" : false,​

    "lucene_version" : "6.6.1"​

  },​

  "tagline" : "You Know, for Search"​

}​

If you don't get any answer to the curl command, run the following steps:

1. Stop all containers:      ​

  ./clara.sh  --stop

2. Stop the docker daemon:

  systemctl stop docker​ (the command depends on platform)

3. Start docker and Clara​:

 systemctl start docker​

 ./clara.sh --start​

Symptom

Accessing Clara website you get a security warning: " Your connection is not private".

Cause and solution

Add an exception to trust the website

Symptom

Clara is installed on the same machine of Workload Automation. WA is installed on docker but Clara cannot contact WA

Cause and solution

Link Clara container with the WA network, by running the following command:

docker network connect docker_wa-net clara-autobot    

and use “wa-server” as dns of WA engine.

Symptom

ElasticSearch does not start, and the log shows the following error:

"2018-09-21T14:11:16,039][INFO ][o.e.b.BootstrapChecks ] [qVAFkOU] bound or publishing to a non-loopback address, enforcing bootstrap checks

ERROR: [2] bootstrap checks failed

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]"

Cause and solution

To solve this problem, run the following procedure.

How to verify and set ulimit parameter

1. Check the maximum number of open files for the current user ( the user that started the Docker deamon) by running the command ulimit -n

2. Verify that the number of allowed open files for the current  user is at least 65536

3. Check the Hard limit for the current user, by running the command ulimit -n -H

4. Check the Soft limit for the current user, by running the command ulimit -n -S

5. In case the value of Hard or Soft limit is lower than 65636, increase its value permanently, by editing the file:

/etc/security/limits.conf

[domain] [type] [item] [value]

where:

• [domain] can be a username, a group name, or a wildcard entry

• [type] is the type of the limit and can have the following values:

• soft: a soft limit which can be changed by user

• hard: a cap on soft limit set by super user and enforced by kernel

• [item] is the resource for which you are setting the limit

For example, for a user with id hmuser run the following steps:

1. Add or modify soft and hard limits as follow:

• hmuser soft nofile 65536

• hmuser hard nofile 65536

2. Activate the new values by running the following command sysctl -p

3. Update the following files:

• /etc/systemd/user.conf

• /etc/systemd/system.conf

by adding the following line:

DefaultLimitNOFILE=65536

4. Login again with user hmuser and verify the new limits before starting any process

Symptom

The log shows the following error: "[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]".

Cause and solution

On Linux, run the following procedure.

How to verify and set the available virtual memory

1. To verify the available virtual memory, run the following command as the user that started the Docker deamon:

sysctl vm.max_map_count

2. If the command output shows a value lower than 262144, run the command:

sysctl -w vm.max_map_count=262144

3. To set this value permanently, edit the vm.max_map_count setting in /etc/sysctl.conf.

4. Add the following as last row, or edit the row if present:

vm.max_map_count=262144

5. Verify the new value after reboot.

Symptom

Keycloak container restarts, or the log shows the following error:

"Oops...I understand that you are trying to access the intent .. But I am not trained/configured to take any action on this yet. Please contact administrator to get this added.

Security-Enhanced Linux (SELinux) on the hypervisor must be disabled or the permissions must be set up correctly".

Cause and solution

To set SELinux to permissive, run the following procedure.

How to set SELinux to permissive

1. Run the following commands:

sed -i s/^SELINUX=.*$/SELINUX=permissive/ /etc/selinux/config

setenforce 0

sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config

2. Restart the system to save the changes permanently.

3. After you restart the system, you can use the getenforce command to check the SELinux status.

Symptom

On first installation, Clara returns "Server internal error".

Cause and solution

Clara needs to be reinstalled:

./clara.sh --down --vol

./clara.sh --up

./clara.sh --load-project HWA --license ./license.properties

Symptom

On first installation, the installation script returns the following error message:

"The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

Continue with the new image Y|N?"

Cause and solution

You must first load Clara images from the clara_images.tar.gz file. This file is in the same path as the installation script.

To load Clara images, run the installation script with the appropriate options:

./clara.sh --load-images

Symptom

After upgrading Clara from previous version, one or more of the following messages appear when logging into Clara:

• "Failed to initialize the authentication adapter"

• "Undefined"

Cause and solution

You might have missed a step of Clara upgrade procedure.

Run the command ./clara.sh --migrate-data to complete the upgrade procedure and then reload Clara UI page.

Symptom

At login, the following message appears on Clara's UI: "response indicates failure, status=400, body={"error":"unauthorized_client","error_description":"Invalid client secret"}"

Cause and solution

You might have have specified an invalid key (CLIENT_SECRET) in the .env configuration file. Verify that the key specified in the .env file is equal to the one specified in keycloak. If the key is not the same, run the following procedure:

1. From Keycloak console, in the Credentials tab, copy the content of the Secret field

2. In the .env file (path <BUILD_DIR>/clara/.env),  paste the valid key  into the CLIENT_SECRET parameter, overriding the current value

3. From  the <BUILD_DIR> directory, run the commands ./clara.sh --up –nc  

4. Reload Clara UI page