All Systems Operational

We are investigating delayed webhook deliveries for GitHub and GitLab integrations.

Root cause identified: SQS consumer throttling due to a spike in concurrent policy evaluations. Scaling consumer concurrency.

Consumer scaling complete. Webhook delivery times returning to normal. Monitoring backlog drain.

Fully resolved. Webhook backlog cleared and delivery latency back to baseline. Added auto-scaling thresholds to prevent recurrence. Total impact: 70 minutes.

Migrating object storage to new region for improved latency. File uploads and knowledge base processing may be temporarily unavailable.

Upgrading SQS queue configuration and dead-letter queue policies. Background job processing may be delayed during the window.

We are investigating reports of intermittent login failures on the App dashboard.

Root cause identified: WorkOS session refresh endpoint returning 503 due to upstream rate limiting. Implementing request queuing.

Request queuing deployed. Login success rate recovering. Monitoring session refresh stability.

Fully resolved. Added exponential backoff for session refresh calls. Total impact: 72 minutes.

Planned CDN cache purge and edge configuration update. Brief availability blips expected.

Routine TLS certificate rotation across API and App services. No downtime expected, but connections may briefly reset.

We are investigating elevated p99 latency on policy evaluation endpoints.

Root cause identified: a hot partition in the vector search index following a large knowledge base upload. Rebalancing shards.

Shard rebalance complete. Latency returning to normal levels. Monitoring for stability.

Fully resolved. Added automatic shard rebalancing to prevent recurrence. Total impact: 78 minutes.

Planned database migration to upgrade PostgreSQL and apply schema optimizations. Expected duration: 45 minutes.

Maintenance window started. API requests may experience elevated latency.

Migration completed successfully. All services restored to normal operation.