Skip to content

Monitoring Proxmox PVE with Grafana and InfluxDB

There comes a point where you begin to run out of time to individually babysit each of your VM’s and LXC’s. Actively and broadly monitoring your systems becomes a crucial point to maintaining your systems.

To aid you in this, you can use Grafana dashboarding to present data Proxmox sends to InfluxDB.

Understanding our Setup

Let’s just assume you have come here because you wish to monitor your existing ProxmoxVE server. Since this most likely is the case, we shall gloss over the procedure on how to install and initially configure a ProxmoxVE server.

InfluxDB is a open source time series database. Unlike MySQL or PostgreSQL, the system is specifically optimised for storing and service of data associated with a time stamp, for instance, recording ambient temperature over time.

Grafana is an open source analytics visualisation application which can provide charts, graphs and alerts based off connected data sources, such as our InfluxDB server.

InfluxDB will store our data. Against it, we will register an API key so that ProxmoxVE can write to the Influx Database. We will also register an API key so that Grafana an read from the Influx Database. ProxmoxVE writes, Grafana reads.

InfluxDB

Installing InfluxDB

I’m installing InfluxDB v2.1.1 on an Ubuntu 21.04 server, with 512M of memory, 8G of disk space, and 1vCPU. Influx is pretty efficient, you can always increase if you need to.

Head over to the InfluxDB download portal, and under the InfluxDB section, select the latest version of InfluxDB and “Ubuntu & Debian” under the platform. Influx will give you a couple handy-dandy instructions to add Influx to your Apt Repos list.

P.S. I’ve found that sometimes Ubuntu doesn’t come with gpg, so you’ll need to install that with a sneaky apt install gpg -y.

wget -qO- https://repos.influxdata.com/influxdb.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/influxdb.gpg > /dev/null
export DISTRIB_ID=$(lsb_release -si); export DISTRIB_CODENAME=$(lsb_release -sc)
echo "deb [signed-by=/etc/apt/trusted.gpg.d/influxdb.gpg] https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list > /dev/null

Once you’ve got that squared away, you can apt update, and then apt install influxdb2.

Once installed, we can enable and start the service with systemctl.

[email protected]:~# systemctl enable --now influxdb
[email protected]:~# systemctl status influxdb
* influxdb.service - InfluxDB is an open-source, distributed, time series database
     Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-03-15 10:17:48 UTC; 50s ago

Configuring InfluxDB

Let’s get started:

  1. InfluxDB can be quickly setup by navigating to the HTTP page served on the InfluxDB server at port 8086, http://yourserver.localdomain:8086
  2. Provide some details for an admin account. Give your organisation a name, and create an initial bucket name. In this case, we can call our bucket Proxmox.
  3. On the next page (after hitting “Configure”), there will be 3 options to initially setup the server. Just hit “Quick Start”.

Now, let’s get our Proxmox Bucket (aka database) ready to collect data from ProxmoxVE:

  1. Select the “Data” tab, and then the “API Tokens”.
  2. Create a Proxmox Bucket Write API Token, click “Generate API Token”, select “Read/Write API Token”. Name it something like “Proxmox Bucket Write”, and select Proxmox under the “Write” table, save.
  3. Create a Proxmox Bucket Read API Token, click “Generate API Token”, select “Read/Write API Token”, select Proxmox under the “Read” table, save.

When we configure ProxmoxVE to send metrics to InfluxDB, we will reference the API Token to write data.

Configuring ProxmoxVE

Now, let’s get ProxmoxVE to upload metrics to the Proxmox bucket on InfluxDB:

  1. At Datacenter level in ProxmoxVE, go the the “Metric Server” tab.
  2. Add a new InfluxDB server, and provide the details to our InfluxDB server:
    Name: InfluxDB
    Server: IP/Name/FQDN to our InfluxDB Server
    Port: 8086
    Protocol: HTTP
    Enabled: Yep!
    Organisation: Your Organisation Name
    Bucket: Proxmox
    Token: WRITE API Token generated earlier

Once you’ve done this, if you give it a couple minutes you should start to see data flow into InfluxDB. If you go to the Explore tab, you can start to create queries using the query builder.

Grafana

Installing Grafana

Installing Grafana is just as wonderfully straight forward as installing InfluxDB. Grafana provides an APT repository for installing Grafana Enterprise. You can add this to your repos list by executing the following.

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/enterprise/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Once that’s added, you can update your repository list and install Grafana Enterprise.

sudo apt-get update
sudo apt-get install grafana-enterprise

Once that’s installed, you can enable and start the service by executing the following.

[email protected]:~# systemctl enable --now grafana-server
[email protected]:~# systemctl status grafana-server 
* grafana-server.service - Grafana instance
     Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-03-16 07:13:32 UTC; 7s ago

Configuring Grafana

By default, Grafana is served on port 3000, HTTP. The default login is admin for the username and password, though once you first login you’ll be required to reset it.

After this, feel free to create a user for yourself – go to the Configuration tab (gear icon on the left), then Users. Add a user there.

Because you’ll be accessing this frequently, there’s a couple things I like to do, such as serving Grafana on port 443 for HTTPS, and using LDAP authentication. While it’s not completely required, you can find that at the bottom of this post in section “Optionals”.

Adding the InfluxDB Data Source to Grafana

  1. In Grafana, click the “Configuration” gear icon on the tabs on the left and select “Data Sources”.
  2. Click “Add data source” at the top right of the page, and select InfluxDB from that list, should be near the top with a “Core” tag.
  3. Referring back to us configuring InfluxDB, you’ll need to grab that API key we generated to READ from the InfluxDB database. Set the following settings, all others can be left as default;
    Name: Something descriptive
    URL: http://your-influxdb.localdomain:8086
    Organisation: Your Organisation
    Token: The API token we created earlier
  4. Click “Save & Test”, you should get a message saying at least one bucket was found.

You should be able to now explore your data. Let’s start creating a couple queries!

Creating a Grafana Dashboard

Now, as of writing this Grafana does not have some native query builder. So you either have to know the Flux Query Langauge yourself, you use the query builder on the InfluxDB server. This is the way I do it.

Let’s just start by creating a simple dashboard, for the uninitiated; Click the “Create” button (+) on the left tabs and select “Dasboard”. Then click “Add a new panel”.

Now, let’s create a query;

  1. Jump over to the InfluxDB server and select the “Explore” button on the left.
  2. Create a query by enabling certain search filters, so what I do (for example) is
    1. “FROM” = “Proxmox” (or whatever your bucket is called),
    2. Filter “_measurement” for “system”
    3. Filter “_field” for “cpu”
  3. Hit the blue “Submit” button.

Now you have a historical graph of CPU usage per VM/LXC! You can refine this even further by adding another filter select VM/LXC’s by clicking the “+” icon, selecting “Filter”, and “host” from the two drop downs at the top, and then selecting specific hosts.

If we have a look at the Flux Query Language for this, yours should look similar. Next to the “Submit” button, click “Query Builder”.

from(bucket: "Proxmox")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "system")
  |> filter(fn: (r) => r["_field"] == "cpu")
  |> filter(fn: (r) => r["host"] == "pcbrixdbd1")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")

You can fiddle with this query if you want. Like if you want to use Regular Expression on the host filter you can change it to something like r["host"] =~ /pcbr.*/. This limits to hosts starting with “pcbr”. Could be useful! Let’s copy this query now, and paste it into Grafana.

Based on what you entered into that query builder, you should get some handy-dandy graph! You can change way this data is represented in Grafana by clicking the drop-down at the top right that might say “Time series” and selecting something else, like Gauge.

From here, the sky is the limit. There are quite a handful of tutorials online on how to use Grafana.

Optionals

None of this is really necessary, but if you’re interested in improving the accessibility of your Grafana service to other users then this may be of use.

Optional: Serving Grafana on Port 443

If it doesn’t exist, let’s make a directory to store our certificate and key.

[[ -d /etc/pki/tls/private ]] || sudo mkdir -p /etc/pki/tls/private
[[ -d /etc/pki/tls/certs ]] || sudo mkdir -p /etc/pki/tls/certs

For simplicity let’s create a self-signed certificate and private key pair, unless if you have something else from some other service, like Let’s Encrypt.

sudo openssl req -newkey rsa:2048 -nodes -keyout /etc/pki/tls/private/20220316-pcbrstatd1.key -x509 -days 365 -out /etc/pki/tls/certs/20220316-pcbrstatd1.cert

After this is created, I like to create symbolic links to the certificate, so I don’t have to mess around with config files later when I have to renew the certificate. We’ll also need to adjust the read permissions so that Grafana can read the private key.

[email protected]:~# cd /etc/pki/tls/certs/
[email protected]:/etc/pki/tls/certs# ln -s 20220316-pcbrstatd1.cert grafana.cert
[email protected]:~# cd /etc/pki/tls/private/
[email protected]:/etc/pki/tls/private# ln -s 20220316-pcbrstatd1.key grafana.key
[email protected]:/etc/pki/tls/private# chown grafana:grafana grafana.key 

Open /etc/grafana/grafana.ini in your preferred editor, under [server], remove the comment tags or apply the following settings.

[server]
protocol = https
http_port = 443
cert_file = /etc/pki/tls/certs/grafana.cert
cert_key = /etc/pki/tls/private/grafana.key

If you were to restart the server now, you’ll likely receive the following error in journalctl.

failed to open listener on address 0.0.0.0:443: listen tcp 0.0.0.0:443: bind: permission denied

Ports like 80 and 443 are restricted so that only root, or a process with appropriate permissions to do so, can listen on them. You need to allow the Grafana server to listen on these ports.

Modify /lib/systemd/system/grafana-server.service with your preferred editor, and a add the following options to the [Service] section.

CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
PrivateUsers=false

Do systemctl daemon-reload, then restart the service now and you should be good.

Optional: Authenticating Grafana with LDAP

If you happen to have an LDAP server, you can use it to authenticate your users. The config for this is located in /etc/grafana/ldap.toml. Configure as such, you need to adjust the LDAP server or this file, particularly in regards to group DN’s so that users can automatically be assigned roles based on what groups they are a member of in LDAP.

[log]
filters = ldap:debug

[[servers]]
host = "ldap.server"
port = 389
use_ssl = false
start_tls = false
ssl_ski_verify = false

bind_dn = "cn=nsspam,dc=example,dc=net"
bind_password = "nsspam'sPassword"

search_filter = "(uid=%s)"
search_base_dns = ["ou=Users,dc=example,dc=net"]

group_search_filter = "(&(objectClass=posixGroup)(memberUid=%s))"
group_search_base_dns = ["ou=grafana,ou=Groups,dc=example,dc=net"]
group_search_filter_user_attribute = "uid"

[servers.attributes]
name = "givenName"
surname = "sn"
username = "uid"
member_of = "memberOf"
email =  "mail"

[[servers.group_mappings]]
group_dn = "cn=admins,ou=grafana,ou=Groups,dc=example,dc=net"
org_role = "Admin"

[[servers.group_mappings]]
group_dn = "cn=editors,ou=grafana,ou=Groups,dc=example,dc=net"
org_role = "Editor"

[[servers.group_mappings]]
group_dn = "*"
org_role = "Viewer"

You’ll also need to enable LDAP authentication in grafana.ini.

[auth.ldap]
enabled = true
config_file = /etc/grafana/ldap.toml
allow_sign_up = true

Restart the Grafana server and you should be good to go. You can give LDAP authentication a test run from the Admin page, under the LDAP tab.

Leave a Reply

Your email address will not be published. Required fields are marked *