Troubleshooting

This guide provides a systematic approach to troubleshooting HCL Link deployments, from initial Helm chart issues to runtime and networking problems.

Helm Troubleshooting:
Before running Helm install, check your chart for common configuration issues by using lint and dry-run commands.
- helm lint: Runs a syntax and formatting check on your chart. Run this command first to catch basic errors early.
```
# Run from inside your chart directory
helm lint . -f my-values.yaml
```
- helm template: Renders all templates locally and displays the generated YAML manifests. Use this command to verify that your values.yaml logic (such as if statements and loops) works correctly before deploying to the cluster.
```
# Render templates for a release named 'hcl-link' using your values
helm template hcl-link . -f my-values.yaml -n <your-namespace>
```
- helm install --dry-run: Simulates an installation by sending the chart to the Kubernetes API server for validation. It checks the chart against the cluster’s capabilities—such as available APIs—but does not create any resources. Use this command to detect permission issues or API version mismatches before performing an actual installation.
```
helm install hcl-link . -f my-values.yaml -n <your-namespace> --dry-run
```
- helm get manifest / helm get values: After installation, use these commands to view the deployed resources and the configuration values used during deployment.
```
# See the live manifests deployed in the cluster
helm get manifest hcl-link -n <your-namespace>

# See the values that were applied to the release
helm get values hcl-link -n <your-namespace>
```

Runtime Troubleshooting (Kubernetes & Link Logs)

After installing the chart, issues usually fall into two categories: Kubernetes resource issues (for example, a pod that fails to start) or application issues (for example, a running pod that shows errors in the logs).

Key kubectl Commands

Use these commands as your main tools to check the status and health of your deployment.


Command	Purpose
kubectl get pods	Check the status of pods. Run this command first. Look for statuses such as CrashLoopBackOff, ImagePullBackOff, Pending, or OOMKilled.
kubectl describe pod <pod-name>	Identify why a pod is failing. This is the most important troubleshooting command. Scroll to the Events section at the bottom to see details such as FailedMount (storage issue), FailedScheduling (insufficient CPU or memory), or ImagePullBackOff (invalid image name or pull secret).
kubectl logs <pod-name>	View HCL Link application logs. Displays stdout and stderr output, such as JVM startup messages or map execution errors.
kubectl logs -f <pod-name>	Stream pod logs in real time. Useful for monitoring live application activity.
kubectl logs --previous <pod-name>	Retrieve logs from a crashed pod. Use this when a pod is in CrashLoopBackOff to see logs from its last run and identify the cause of the crash.
kubectl exec -it <pod-name> -- /bin/bash	Access a shell inside a running container for deeper inspection and troubleshooting.
kubectl get pvc,svc,configmap	Check the status of persistent volume claims (PVCs), services, and config maps.
kubectl describe pvc <pvc-name>	Verify storage binding. Shows whether a persistent volume claim is successfully bound to a volume.
kubectl top pods	Monitor resource usage. Displays live CPU and memory usage for all pods (requires the metrics-server).

Locating Logs Summary

Pod won’t start (for example, Pending, CrashLoopBackOff):
- Use kubectl describe pod <pod-name>.
- Check the Events section at the bottom.
Pod is running but not working:
- Use kubectl logs <pod-name>.
- Check for application errors (e.g., Java exceptions, connection refused).
Pod Crashed and Restarted:
- Use kubectl logs --previous <pod-name>.
- Check for the fatal error that caused the crash such as OutOfMemoryError.

Linux Commands (Inside the Pod)

After running kubectl exec -it <pod-name> -- /bin/bash to open a shell inside the pod, use these standard Linux commands to inspect its internal environment and diagnose issues.


Command	Purpose
ps aux	Check running processes. Confirm that the Java process or other expected services are active.
top / htop	Monitor live CPU and memory usage inside the container in real time.
env	View environment variables. Verify that all settings from your values.yaml file are correctly passed as environment variables.
df -h	Check disk space usage. Ensure that persistent volumes (for example, /data) have enough free space.
ls -l /path/to/data	Verify file permissions. Ensure the process user has the necessary read and write access to mounted volumes.
netstat -tuln or ss -tuln	Check open network ports. Confirm that your application is listening on the expected ports (for example, whether `lnk-server` is listening on port `8443`).
cat /path/to/config.xml	Review configuration files. Manually inspect files mounted from ConfigMaps to confirm correct content.
pmap -x <pid>	Display a process memory map. Provides detailed memory usage for a process, including JVM and native components.
jcmd <pid> VM.native_memory	Check JVM memory details. If Native Memory Tracking (NMT) is enabled (`-XX:NativeMemoryTracking=summary`), this command shows a detailed breakdown of heap and non-heap memory usage.

Network & API Troubleshooting
Use these tools to verify connectivity between pods and to test API functionality.
curl
A command-line tool for making HTTP requests. Use it inside a pod to test internal service discovery and connectivity through DNS.
Test Internal Connectivity
1. Shell into the client pod:
```
kubectl exec -it <lnk-client-pod> -- /bin/bash
```
2. Use curl to test connectivity to the server service (assuming port 8443 and HTTPS):
```
curl -v -k https://<server-service-name>:8443/health
```
  - -v (verbose): Displays detailed output, including SSL/TLS handshake information.
  - -k (insecure): Skips certificate validation (useful for self-signed certificates).
  A successful response confirms that the pod can resolve the service DNS and connect to the target port.
Test External Connectivity
Run curl from your local machine to test access through the external-facing Ingress or Route:
```
curl -v https://link.mycompany.com/api/rest/health
```
Swagger UI / OpenAPI
The HCL Link REST service usually provides a Swagger (OpenAPI) interface at an endpoint such as /api/docs. This web-based UI:
- Lists all available API endpoints.
- Displays parameters and request/response formats.
- Allows you to execute API calls directly in the browser, managing authentication and formatting automatically.
  
  Use it to quickly confirm that the REST API is operational.
Postman
Postman is a desktop tool for advanced API testing. It is useful for:
- Testing external REST APIs.
- Creating and saving complex POST or PUT requests with JSON payloads.
- Managing authentication (such as Bearer tokens) for repeated requests.
- Automating sequences of API calls to test complete workflows.