MustGather data for Catalog service

Thanks to the Serviceability Framework, extensive diagnostics can be gathered without requiring global log level changes or service restarts.

Initial triage data (The "MustGather" core)

When an issue is reported, collect the following baseline information:

Timestamp of the incident

Include the specific timezone.

The GraphQL query or mutation

The exact query or mutation body that failed.

Response payload

The complete HTTP response received by the client, particularly the errors array.

Raw service logs

Extract the structured JSON logs (stdout/stderr) covering the time of the incident. Ensure the logs include the following events:

• http.request.start
• http.request.complete
• error.classified (if applicable)

Trace component information

Use per-request trace elevation to capture verbose traces for a specific failing request without affecting global production performance.

To gather elevated traces, inject the X-Log-Level header into the API request when you reproduce an issue. This action creates a request-scoped logger with elevated verbosity for that specific transaction.

• Required header: X-Log-Level: debug (or trace)
• Valid levels: trace, debug, info, warn, error, fatal

Example request:

                curl -s -X POST http://<catalog-service-url>/graphql \
                -H "Content-Type: application/json" \
                -H "X-Log-Level: debug" \
                -H "x-context: <valid-base64-context>" \
                -d '{"query":"query getCatalog { getCatalogs(limit: 10) { nodes { identifier catalogId workspaceId } } }"}'

The system configuration enforces a minimum logging limit in production environments to prevent log flooding. The system automatically redacts personally identifiable information (PII) before writing to the log files.

Active troubleshooting per-request trace elevation

If the baseline logs do not provide enough context, you can force the Catalog Service to emit DEBUG level traces for a single request without affecting production traffic.

Reproduce the issue by sending the exact GraphQL payload using curl and append the X-Log-Level: debug header.

Required Headers for Elevated Gathering:

X-Log-Level

X-Log-Level: debug (Print detailed logs at the debug level)

x-context

x-context: <valid-base64-context> (Required to pass Auth/Governance)

traceparent

traceparent: <custom-trace-id> (Optional: Pass a specific W3C trace ID to easily find the request in the log stream)

Example replication command:

curl -s http://catalog-service-host:port/graphql \
                -H "Content-Type: application/json" \
                -H "X-Log-Level: debug" \
                -H "x-context: <valid-base64-context>" \
                -H "traceparent: 00-abcdef1234567890abcdef1234567890-1234567890abcdef-01" \
                -d '{"query":"query getCatalog { getCatalogs(limit: 10) { nodes { identifier catalogId workspaceId } } }"}'

What to collect after running this command:

Search the service logs for the specific trace_id (e.g., abcdef1234567890abcdef1234567890) and collect all matching JSON log lines. You should see _logLevelOverride: "debug" in the log bindings, revealing SQL entry or exit notices and backend resolver metrics.

Health and connectivity checks

To isolate whether the issue is a systemic crash or a database disconnect, run and capture the output of the following endpoints on the default port:

Liveness check

curl -s http://catalog-service-host:port/healthz

Expected output: {"status":"ok"} along with an x-trace-id header.

Readiness check (Validates database pool)

curl -s http://catalog-service-host:port/readyz

Expected Output: {"status":"ready","checks":{"database":"ok"}}

Key log indicators to analyze

When reviewing the gathered JSON logs, filter by service: "catalog-service" and look for the following event types to pinpoint the failure:

A. Lifecycle and HTTP events

http.request.start & http.request.complete: Verify http.status_code and http.duration_ms.

A missing complete event indicates a failure in the Fastify hooks or an unhandled crash in the pre-handler.

B. Database and pool events

• db.pool.exhausted (WARN): Indicates the connection pool has waiting clients; the service is under heavy load or leaking connections.
• db.pool.error (ERROR): Pool-level connection drops.
• db.function.error: A PostgreSQL function raised an exception.
• db.query.slow: Identifies queries exceeding latency thresholds.

C. Error events and de-duplication

Check for error.classified, error.graphql, or error.http.

Match the trace_id or request_id to the correlationId returned to the client's GraphQL error response.

Known integration issues and troubleshooting map

If the logs look malformed or events are missing entirely, verify the service integration points in the source code: