Alert Not Firing
Debug why an alert rule isn't triggering when you expect it to.
If you've created an alert rule but it never fires even when the metric exceeds the threshold, work through these checks.
Check 1: Is the rule enabled?
Go to Alerts in the dashboard and find your rule. There's a toggle on each rule, make sure it's switched on (green). Newly created rules default to enabled, but it's easy to accidentally disable one.
Check 2: Is the metric name spelled correctly?
Metric names are case-sensitive and must exactly match the metric keys that KubeWatch collects.
Correct metric names:
| Metric | Key |
|---|---|
| CPU usage percentage | cpu_percent |
| Memory usage in MB | memory_mb |
| Memory usage percentage | memory_percent |
| Restart count | restart_count |
| Network received bytes/sec | network_rx_bytes_per_sec |
| Network transmitted bytes/sec | network_tx_bytes_per_sec |
| Container running (1=yes, 0=no) | running |
Common mistakes: cpuPercent (camelCase won't work), cpu (too short), memory_usage (wrong name).
Check 3: Is the current metric value actually exceeding the threshold?
Check the current value in the dashboard:
- Go to Containers → click the container you're monitoring
- Open the Metrics tab
- Read the current CPU % or memory value
If the current value is below your threshold, the alert is working correctly, it's just not fired because the condition isn't met.
If you're trying to test an alert, you can temporarily lower the threshold to a value you know will be exceeded.
Check 4: Has the condition held for the full duration?
KubeWatch uses a "sustained duration" model: the condition must be continuously true for the entire duration before the alert fires. If the metric spikes briefly and drops, the alert won't fire.
For example, with cpu_percent > 90 for 5m:
- If CPU hits 95% for 2 minutes then drops to 80%, the alert does not fire
- If CPU stays above 90% for 5 full minutes, the alert fires
To catch brief spikes, set duration to 0s or 30s.
Check 5: Is a notification channel configured?
An alert can be in firing state even if you never received a notification, if no notification channel is configured or the channel is broken.
- Go to Alerts → find your rule → check the Channels field
- Go to Integrations → verify the channel is connected (green status indicator)
- Send a test notification from Integrations → [channel] → Test
Check 6: Check alert history
Go to Alerts → History. Filter by your rule name. If the alert fired but you didn't receive a notification, the issue is with the notification channel, not the rule itself.
If the alert doesn't appear in history at all, the condition was never met for the configured duration.
Check 7: Is the agent reporting metrics for this container?
If the container you're targeting stopped reporting (agent disconnected, container stopped), the alert won't evaluate. Check:
- Containers page, is the container listed and does it have recent metrics?
- Agents page, is the agent Connected with a recent "last seen" timestamp?