> ## Content Index
> Fetch the complete content index at: https://trendboxgeek.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Grafana Alert Rules for Proxmox: 5 Rules You Actually Need
- URL: https://trendboxgeek.com/blog/grafana-alert-rules-proxmox/
- Published: 2026-08-22T12:20:35.000Z
- Updated: 2026-08-22T12:20:35.000Z
- Author: Pankajbhai Chavda
- Tags: #blog-col

Proxmox has those RRD graphs in the web UI. That is fine. For one node and a couple of VMs they tell you what you need to know, as long as you are looking at them. They don't alert when you aren't live watching them.

I am using Proxmox on my on-premises server. I created around 8 containers, but when some errors occurred in my system, a few containers shut down at midnight. In the morning I checked my Proxmox and found some containers had shut down, but why they stopped or shut down and at what time they went down, I had no idea.

So I built five alert rules in Grafana, wired them to Discord, and then spent a few hours deliberately breaking my own homelab to make sure each one actually worked. I stopped containers, shut down VMs, and moved thresholds around. Every gotcha below is something that bit me, not something I read on a forum.

## Why Grafana Rules Instead of Prometheus YAML

Search for Proxmox alerting and you will get handed a proxmox-alerts.yml file for Prometheus. But you need to edit files on the box. In the future, if you change your mind, you then have to reload Prometheus and run Alertmanager as a whole separate service just to get a message on your phone.

Grafana managed rules live in Grafana's own database instead. You build them in the UI and you can preview them before you save. You receive messages in Discord with no extra service running.

## What You Need First

First you need to [install Grafana and Prometheus on your Proxmox](https://trendboxgeek.com/blog/proxmox-prometheus-grafana/). Install everything in an Ubuntu 24.04 container and connect Proxmox via API, so you don't need an agent on every node.

PVE Exporter feeds Prometheus, and Grafana reads Prometheus as a datasource. The chain:

Proxmox API (:8006) -> PVE Exporter (:9221) -> Prometheus (:9090) -> Grafana (:3000)

## Look at Your Metrics First

After properly connecting Grafana and Prometheus, we need to test some queries in Grafana before creating rules.

Start with the exporter. SSH into the Ubuntu 24.04 host that runs the exporter container, because localhost:9221 only works from there. The target is your Proxmox node's IP. Replace PROXMOX\_IP.

curl -s -o /dev/null -w "%{http_code}\n" "http://localhost:9221/pve?target=PROXMOX_IP"

**Expected Output:**

![Expected Output of localhost.](https://trendboxgeek.com/content/images/2026/08/Screenshot-From-2026-08-21-16-32-32.png)

If your output shows 200 then perfect. Now we look at what it's actually giving you.

curl -s "http://localhost:9221/pve?target=192.168.100.200" | head -20

**Expected Output:**

pve_up{id="node/pve"} 1.0
pve_up{id="qemu/198"} 1.0
pve_up{id="qemu/199"} 1.0
pve_up{id="storage/pve/local-lvm"} 1.0
pve_up{id="storage/pve/local"} 1.0

Now go into Grafana, open Explore, pick your Prometheus datasource, switch to Code mode and set the output to Table. You must select Table view, not graph. You want to read label values, and the graph legend makes that harder than it needs to be.

Run these four queries one by one, click Run query, and check the results:

up{job=~".*prox.*"}
pve_up{id=~"node/.*"}
pve_up{id=~"qemu/.*|lxc/.*"} * on(id) group_left(name, node) pve_guest_info
(pve_disk_usage_bytes{id=~"storage/.*"} / pve_disk_size_bytes{id=~"storage/.*"}) * 100

**Expected Output:**

![](https://trendboxgeek.com/content/images/2026/08/Screenshot-From-2026-08-21-16-44-29-1.png)

![](https://trendboxgeek.com/content/images/2026/08/Screenshot-From-2026-08-21-17-02-04.png)

![](https://trendboxgeek.com/content/images/2026/08/Screenshot-From-2026-08-21-17-02-24.png)

![](https://trendboxgeek.com/content/images/2026/08/Screenshot-From-2026-08-21-17-02-41.png)

Write down your exact job label. Mine is proxmox-api. Most guides assume it's just proxmox.

The first query shows your proxmox-host, a node\_exporter on :9100.

The third query shows your total VMs with their IDs, like qemu/198 as ubuntu and qemu/199 as ubuntuserver.

The storage query gave me 16.9% for pve/local and 28.4% for pve/local-lvm. Note your own numbers down. You'll need them later to pick a test threshold that actually fires.

## **Discord Webhook and Contact Point**

Do this before you write any rules. Grafana won't let you save a rule without a contact point, and the built-in Email option does absolutely nothing until you've set up SMTP. Discord is a two-minute job.

For webhook settings, you need a desktop or browser. The Discord mobile app hides settings. After completing this setup you can use the mobile app.

In Discord, click your server name in the top left, then Server Settings. Scroll the left sidebar down to the APPS group, then Integrations → Webhooks → New Webhook. Give it a name you will recognize later, like Grafana. Pick a channel. Then copy the Webhook URL. That URL is a password. Anyone with it can post in your server.

Now back in Grafana, go to Alerting → Contact points → Contact points → + Add contact point.

Name:        proxmox-discord
Integration: Discord
Webhook URL: your webhook URL

Now the important part. Click **Save contact point**.

## **Rule 1, ProxmoxNodeDown**

Now we create the rules one by one. First, go to the Grafana Home page, click on Main Menu in the top left, and find Alerting. Then go to Alerting → Alert rules → + New alert rule. Name the first rule ProxmoxNodeDown.

**Enter alert rule name**

Name: ProxmoxNodeDown

**Define query and alert condition**

Query, Code mode, against Prometheus:

pve_up{id=~"node/.*"}

Then click on Run queries.

WHEN QUERY IS BELOW 1

Grafana defaults that to IS ABOVE 0\. So you need to change it to `IS BELOW 1`.

Then click on Preview rule condition. It should show Normal.

**Add folder and labels**

In this section, select the folder Proxmox. Then click on `+Add labels` below.

Key: severity     Value: critical

**Set evaluation behavior**

In this block, select `proxmox-1m` for Evaluation group and interval. Then set the Pending period to 1m. After this, click on Configure no data and error handling.

Alert state if no data or all values are null from its default to Alerting.

No Data = Alerting

**Configure notifications**

In this block, select `proxmox-discord` as the contact point.

**Configure notification message**

Summary:

Proxmox node {{ $labels.id }} is unreachable

Description:

Node {{ $labels.id }} at {{ $labels.instance }} has been unreachable for more than 2 minutes.

Then click Save.

The screenshots below show the final, working setup for Rule 1 (with the threshold set to 1 and No Data set to Alerting).

![](https://trendboxgeek.com/content/images/2026/08/Screenshot-From-2026-08-22-10-24-51-1.png)

![](https://trendboxgeek.com/content/images/2026/08/Screenshot-From-2026-08-22-10-25-20.png)

![](https://trendboxgeek.com/content/images/2026/08/Screenshot-From-2026-08-22-10-25-39.png)

### Break It on Purpose

Let's test Rule 1, because you should never trust an untested alert rule. To force this one to fire without actually shutting down your Proxmox node, edit the threshold.

Click Edit on your ProxmoxNodeDown rule.

Change the threshold from 1 to 2\. (Since a healthy node reports 1, and 1 is below 2, this forces an immediate breach.)

Click Save rule and exit.

**Important:** Check that the "Last updated at" timestamp in the right sidebar actually changed. If it didn't, your edit didn't save, and you'll waste time waiting for an alert that isn't coming.

Watch the rule state. Over the next two minutes, it should shift like this:

Normal -> Pending (orange) -> Alerting (red) -> Discord message

When the Discord message arrives, check that the template variables rendered correctly. You want real values, not {{ $labels.id }} or \[no value\].

To check the message, go to the Discord home page and open #general. Then check the message.

If you don't receive any message, go to the Grafana host's terminal and run the command below.

curl -s -o /dev/null -w "%{http_code}\n" -H "Content-Type: application/json" \
  -d '{"content":"grafana webhook test"}' "YOUR_WEBHOOK_URL"

204 = webhook good, problem is Grafana. 401/404 = wrong or revoked URL. 000/timeout = Grafana's host can't reach discord.com, which is the usual culprit on a locked-down homelab box.

After this test, edit the rule threshold back to **`1`** and click **Save.**

## Rule 2, ProxmoxGuestStopped

Rule 2, name it `ProxmoxGuestStopped`.

Query:

pve_up{id=~"qemu/.*|lxc/.*"} * on(id) group_left(name, node) pve_guest_info

Then click on Run queries. Set the Alert condition as shown below.

WHEN QUERY IS BELOW 1

After this, click the blue Preview alert rule condition button. The result should show Normal.

**Add folder and labels**

In this block, select `proxmox` and set the label Severity to warning.

**Set evaluation behavior**

Select `proxmox-1m`. Set the Pending period to `5m` and, in the Configure no data and error handling section, replace No Data with `Normal`.

**Configure notifications**

In this section, select proxmox-discord.

**Configure notification message**

Summary:

Proxmox guest {{ $labels.name }} is not running

Description:

VM/LXC {{ $labels.name }} ({{ $labels.id }}) on node {{ $labels.node }} has been stopped for more than 5 minutes.

### Test Rule 2

The preview should give you one row per guest. Mine listed ubuntu and ubuntuserver, both Normal.

Test this one properly: go into Proxmox and shut down a VM you don't care about. Give it around six minutes — one evaluation interval plus the five-minute pending period. I shut down qemu/198 and got this:

To test whether this second rule works, go to Proxmox and shut down a VM you don't worry about. Wait around 6 minutes. I shut down my qemu/198 and got the message below.

Value: A=1, C=0

Labels:

- alertname = ProxmoxGuestStopped
- grafana_folder = Proxmox
- id = qemu/198
- instance = 192.168.100.200
- job = proxmox-api
- name = ubuntu
- node = pve
- severity = warning

Annotations:

- description = VM/LXC ubuntu (qemu/198) on node pve has been stopped for more than 5 minutes.

- summary = Proxmox guest ubuntu is not running

## **Rule 3, PveExporterDown**

This is the rule that catches your monitoring failing, and it's the one people leave out.

Rules 1, 2, 4 and 5 all need the exporter delivering metrics. If it dies, none of them can evaluate anything at all. Your Proxmox monitoring can be completely broken while every dashboard sits there looking perfectly calm.

Let's create Rule 3\. Name it **PveExporterDown**.

Query:

up{job="proxmox-api"}

Then click on Run queries. Set the Alert condition as shown below.

WHEN QUERY IS BELOW 1

After this, click the blue Preview alert rule condition button. The result should show Normal.

**Add folder and labels**

In this block, select proxmox and set the label Severity to warning.

**Set evaluation behavior**

Select proxmox-1m. Set the Pending period to 5m and, in the Configure no data and error handling section, replace No Data with Alerting.

**Configure notifications**

In this section, select proxmox-discord.

**Configure notification message**

Summary:

PVE Exporter is down

Description:

Prometheus cannot scrape the PVE Exporter at {{ $labels.instance }}. Proxmox metrics are stale, so all other Proxmox alerts are unreliable until this recovers.

### Test Rule 3

Now go back to the terminal that hosts Grafana. Test it by stopping the exporter. Find it first:

sudo docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Ports}}" | grep -i pve
sudo docker stop pve-exporter
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9221/pve

`000` means dead, and this is where that 500-versus-000 thing from step 1 earns its keep. Same URL both times. Alive gave 500, dead gives 000\. Cross-check at http://YOUR\_PROMETHEUS\_IP:9090/targets too, where the proxmox-api target should have gone red.

After 6 minutes you should get a message in Discord.

Value: A=0, C=1
Labels:
 - alertname = PveExporterDown
 - grafana_folder = Proxmox
 - instance = 192.168.100.200
 - job = proxmox-api
 - severity = warning
Annotations:
  
 - description = Prometheus cannot scrape the PVE Exporter at 192.168.100.200. Proxmox metrics are stale, so all other Proxmox alerts are unreliable until this recovers.
 - summary = PVE Exporter is down

Tired of typing sudo for docker:

sudo usermod -aG docker $USER

Log out and back in. Group membership only applies to new sessions.

Bring it back:

sudo docker start pve-exporter

Then wait a moment before you panic. My curl command run straight after starting the container came back 000, which looks exactly like a crash loop. It is not real. The Python app needs a few seconds to bind the port:

sleep 10
curl -s -o /dev/null -w "%{http_code}\n" "http://localhost:9221/pve?target=192.168.100.200"

**Expected Output:**

trend@boxgeek:~$ sleep 10
curl -s -o /dev/null -w "%{http_code}\n" "http://localhost:9221/pve?target=192.168.100.200"
200

`200` means it is normal. If you're still getting 000 after ten seconds, go look at `sudo docker logs --tail 30 pve-exporter`.

## Rules 4 and 5, Storage

Rules 4 and 5 use the same query. They differ only in threshold, severity, and pending period.

(pve_disk_usage_bytes{id=~"storage/.*"} / pve_disk_size_bytes{id=~"storage/.*"}) * 100

The `{id=~"storage/.*"}` filter isn't optional. PVE Exporter reuses `pve_disk_*` for storage pools, VM disks, and node roots, and the only thing separating them is that prefix:

pve_disk_size_bytes{id="qemu/198"} 8.589934592e+09
pve_disk_size_bytes{id="node/pve"} 4.3867234304e+10
pve_disk_size_bytes{id="storage/pve/local"} 4.3867234304e+10

All three are completely different values under one metric name. If you remove the filter, you end up dividing VM disk usage by pool sizes and getting nonsense.

### Rule 4:

Name:      ProxmoxStorageHigh
Condition: WHEN QUERY  IS ABOVE  85
Severity:  warning
Pending:   10m
No data:   Normal
Summary:   Proxmox storage {{ $labels.id }} is filling up
Description: Storage pool {{ $labels.id }} is at {{ humanize $values.A.Value }}% usage.

### Rule 5:

Name:      ProxmoxStorageCritical
Condition: WHEN QUERY  IS ABOVE  95
Severity:  critical
Pending:   5m
No data:   Normal
Summary:   Proxmox storage {{ $labels.id }} is critically full
Description: Storage pool {{ $labels.id }} is at {{ humanize $values.A.Value }}% usage.

**Note:** In both rules the condition is IS ABOVE.

### Test the Storage Rules

First, test Rule 4\. At the start of this article we ran a query that showed our storage usage range. My storage range was 16.9% to 28.4%. So for this rule test, set the condition to `WHEN QUERY IS ABOVE 15` and drop the pending period to `1m`. Then save and wait for the message to arrive in Discord.

After testing Rule 4, change the condition back from WHEN QUERY IS ABOVE 15 to 85 and the pending period from 1m to 10m.

Rule 5 requires the same changes for testing.

## Conclusion

This setup gives you the best experience. Once you set it up, you don't need to configure every container separately. One big advantage of this setup is that it notifies you via Discord, so you don't need to be connected to the same network. If you connect to a different network with your mobile or laptop, you will still receive notifications. These five rules send a notification when a problem is found, such as nodes dying, VMs unexpectedly stopping, monitoring failures, and storage filling up.