CLOUDP-319783: Added one entry to AKO SRE run book (#2540)

igor-karpukhin · web-flow · commit 47f56975d10f · 2025-07-28T16:29:30.000+02:00
Added one entry to AKO SRE runbook
diff --git a/docs/sre-runbook/README.md b/docs/sre-runbook/README.md
@@ -0,0 +1,13 @@
+# SRE Runbook Index
+
+Welcome to the Atlas Kubernetes Operator (AKO) SRE Runbook collection. This directory contains documentation for common incidents and anomalies that can occur in the operation of AKO. Each runbook includes:
+
+- Description of the problem and symptoms
+- Diagnostic metrics
+- Step-by-step actions to resolve
+- Links to additional documentation
+
+Use this index to quickly navigate to the desired runbook.
+
+## Available Runbooks
+- [Resources are not reconciling](resources_are_not_reconciling.md)
diff --git a/docs/sre-runbook/resource_stuck_in_reconciliation.md b/docs/sre-runbook/resource_stuck_in_reconciliation.md
@@ -0,0 +1,40 @@
+# SRE Runbook: Resource Stuck in Reconciliation
+
+## Problem: Resource stuck in reconciliation
+This problem showcases the issue with AtlasProject resource not being ready. It can be applied to every AKO resource
+
+### Symptoms:
+- The resource is not ready.
+- High error rate metric.
+  
+To monitor the error rate, you can create a query to calculate the reconciliation error rate for the `AtlasProject` controller as a percentage over the last minute. This metric helps in identifying and monitoring the health and stability of the `AtlasProject` controller. A high or rising error percentage indicates issues in the reconciliation process.
+
+#### Example Query:
+To calculate the error rate, use the following Prometheus query:
+```prometheus
+100 * rate(controller_runtime_reconcile_errors_total{controller="AtlasProject"}[1m]) / rate(controller_runtime_reconcile_total{controller="AtlasProject"}[1m])
+```
+
+### Status:
+Check the resource status condition for further details:
+```yaml
+status:
+  conditions:
+    - type: Ready
+      status: "False"
+      reason: ....
+```
+
+### Action Items:
+1. **Verify Resource Status:**
+   - Check the status condition message for more detailed information.
+   - If the `AtlasProject` is not ready, proceed with the next troubleshooting steps.
+  
+2. **Check Connection Secret:**
+   - Ensure the connection secret referenced by `spec.connectionSecretRef.name` is correctly labeled with `atlas.mongodb.com/type=credentials`.
+
+3. **Investigate Logs:**
+   - Review logs for the `AtlasProject` controller for any potential errors or failed reconciliation attempts.
+
+### Additional Resources:
+- [AtlasProject resource](https://www.mongodb.com/docs/atlas/operator/upcoming/atlasproject-custom-resource/)
