Emergency Filesystem Backup (Helper Pod Method)

This section describes a manual backup procedure to access and copy data directly from a PersistentVolumeClaim (PVC) when the primary application pods (Link, MongoDB, etc.) are down, crashing, or otherwise inaccessible.

Warning: Use this method only as a last resort. It involves manually manipulating Kubernetes resources and carries risk if not done correctly. Use it only when:

Application pods are in a CrashLoopBackOff, Error, or Terminating state.
You cannot use the standard kubectl cp command on the application pods.
You cannot use the Volume Snapshot procedure (for example your storage provider does not support it).

Understanding the ReadWriteOnce(RWO) Limitation

Most database and stateful application volumes (like those used by MongoDB, Kafka, and Postgres) are ReadWriteOnce (RWO). This means the volume can only be attached to one node in the cluster at a time.

To attach a PVC to a new helper pod, you must first ensure it is fully detached from its original pod. The only safe way to do this is to scale down the application that owns it.

Backup Procedures

This procedure is performed one PVC at a time.

Stop the Link Application
Scale down all Deployments and StatefulSets for Link components to 0 replicas.
Wait and verify that all pods are terminated to ensure volumes are detached.
Identify Target PVCs
List all PVCs to back up:
```
kubectl get pvc -n <namespace>
```
Note the exact PVC names (for example data-link-mongodb-0, data-link-kafka-0, link-server-pvc).
Launch a Helper Pod
1. Create a helper-pod.yaml and set the namespace and claimName to the PVC you want to back up.
2. Apply the file to create the helper pod:
```
kubectl apply -f helper-pod.yaml
```
3. Wait until the pod is in the Running state.
  If the pod is stuck in Pending, the PVC may still be attached to the original pod.
Copy Data from the Helper Pod
1. Exec into the pod and create a compressed tar.gz archive of the mounted data:
```
kubectl exec -it <helper-pod-name> -n <namespace> -- tar czf /tmp/<component>-backup.tar.gz -C <mount-path> .
```
2. Copy the archive file from the helper pod to your local machine:
```
kubectl cp <namespace>/<helper-pod-name>:/tmp/<component>-backup.tar.gz ./<component>-backup.tar.gz
```
Clean Up and Repeat
- Delete the helper pod to release the PVC:
```
kubectl delete pod <helper-pod-name> -n <namespace>
```
- Repeat steps 3–5 for every other PVC you need to back up (e.g., Kafka, Server, Rest).
  Make sure to change the claimName in helper-pod.yaml and save the local backup with a unique name (for exanple, kafka-0-backup.tar.gz, etc.)

Restore Procedures

Restore your tar.gz backups into a new, fresh Link deployment.

Deploy a Fresh Link Instance
- Deploy a clean Link instance using your Helm chart.
- Ensure it creates new, empty PVCs.
- Verify that all new application pods are in a Running state. This confirms the new environment is healthy before you restore data.
Copy and Extract Data into New Pods
For each component, copy the corresponding backup file into the new pod and then extract it:
Note: Make sure you extract the data to the correct original path inside the container.
1. Copy the backup file into the new pod:
```
kubectl cp ./<component>-backup.tar.gz <namespace>/<pod-name>:/tmp/<component>-backup.tar.gz
```
2. Exec into the pod and extract the archive:
```
kubectl exec -it <pod-name> -n <namespace> -- tar xzf /tmp/<component>-backup.tar.gz -C <target-path> --strip-components=1
```
  Important: The tar command above extracts the files to their original absolute paths (for example, /backup-data/...). If your archive was created as specified (with /backup-data as the root), you need to adjust the extraction command to strip that prefix and place it in the correct new data directory.
  
  --strip-components=1 ensures files are restored to the correct directory.
Restart Pods
- After restoring all data, perform a rolling restart on all Deployments and StatefulSets to force the applications to load the restored data.
  After the restart, your Link application should run with all data restored.