AI Buddy – AssistCX Demo

📋 Ticket Summary

📋

Click "Generate Summary" to create an AI-powered ticket summary

🕐 Event Timeline

2025-03-23 08:14 UTC

Customer submits ticket — payment gateway timing out at checkout for 3 users

2025-03-23 08:37 UTC

Agent Alex M. requests HAR file and server logs from customer

2025-03-23 09:02 UTC

Customer provides logs — 503 responses from payment-svc every ~2 min

2025-03-23 09:15 UTC

Escalated to L2 engineering team

2025-03-23 09:41 UTC — NOW

AI Buddy investigating root cause via SRE integration

📚 Relevant Knowledge Base Articles

📚

Click "Suggest TOP 10 Articles" to find relevant help articles

🏢 Organizational Insights

🏢

Fetch insights about Acme Corp's support history and common issues

✍️ Suggested Agent Response

✍️

Generate a professional response based on KB articles and ticket history

⚡

AI SRE Agent

Autonomous incident investigation · Root cause analysis · Remediation

● INVESTIGATING

MTTR Estimate

~14m

↓ 78% vs manual avg

Alerts Correlated

across 4 services

RCA Confidence

91%

↑ High confidence

🩺 Service Health MapLast refreshed 09:41 UTC

payment-svcCRITICAL

api-gatewayDEGRADED

checkout-svcDEGRADED

db-primaryHEALTHY

cdn-edgeHEALTHY

auth-svcHEALTHY

notification-svcSLOW

inventory-svcHEALTHY

🔬 Root Cause Analysis — High Confidence (91%)

The payment-svc is exhausting its database connection pool under load. A deployment at 08:02 UTC introduced a new query in payment_processor.go:L147 that holds connections open for the full timeout window (30s) rather than releasing them immediately on error. Under sustained load, this is exhausting the pool of 50 connections within ~90 seconds, causing all subsequent requests to 503.

● Commit a3f1c22 by [email protected] — deployed 08:02 UTC via CI/CD pipeline

● Metric db.connection_pool.utilization → 97% (threshold: 80%)

● Trace /api/v2/payments — P99 latency spiked from 120ms → 4.2s at 08:07 UTC

● Log payment-svc[pod-7d4f]: "connection pool exhausted" — 1,247 occurrences since 08:07

● Correlation checkout-svc degradation started 2 min after payment-svc alert (cascade confirmed)

🔧 Recommended Remediation

Rollback deployment a3f1c22 — immediate mitigation. Restores previous payment_processor.go which correctly closes connections on error. Expected recovery: ~2 min.

AUTO-AVAILABLE

Increase connection pool size to 100 (temporary) — buys time if rollback is delayed. Update DB_POOL_SIZE=100 in payment-svc ConfigMap.

MANUAL

Fix the connection leak in code — add defer conn.Close() in error path at payment_processor.go:L147. PR generated with full context.

PR READY

Add connection pool exhaustion alert — no alert was configured at 80% threshold. SRE agent recommends adding Datadog monitor on db.connection_pool.utilization.

RECOMMENDED

📡 Live Investigation Log● streaming

[09:41:02] INFO AI SRE agent initializing for ticket #48291

[09:41:03] INFO Fetching production context — payment-svc, checkout-svc

[09:41:03] WARN payment-svc connection pool utilization at 97%

[09:41:04] INFO Pulling deployment history from CI/CD pipeline...

[09:41:05] ALERT Commit a3f1c22 detected — deployed 2h before incident window

[09:41:05] INFO Querying Datadog: db.connection_pool.* metrics (last 3h)

[09:41:06] WARN P99 latency spike confirmed at 08:07 UTC — 120ms → 4.2s

[09:41:07] INFO Correlating with checkout-svc degradation (2 min lag — cascade)

[09:41:08] INFO Hypothesis 1: DB resource exhaustion — CONFIRMED (97% pool util)

[09:41:08] INFO Hypothesis 2: Network partition — RULED OUT (healthy from 3 regions)

[09:41:09] INFO Hypothesis 3: Config drift — RULED OUT (no infra changes in window)

[09:41:10] ROOT CAUSE Connection leak in payment_processor.go:L147 post-deploy a3f1c22

[09:41:11] OK RCA confidence: 91% — generating remediation steps

[09:41:12] OK Investigation complete — posting findings to ticket #48291