Digitalocean Prometheus | Installation guide

Let’s take a deeper look at how we can install Digitalocean Prometheus. Bobcares, as part of our DigitalOcean Managed Service, guides and offers solutions to every query that comes our way.

Digitalocean Prometheus Overview

Prometheus is one of the powerful open-source monitoring systems that collect or record real-time metrics from your service and further store them in a time-series database. This also offers a multi-dimensional data model, adaptable query language, and diverse visualization possibilities through tools like Grafana.

Tools that make Digitalocean Prometheus complete monitoring tool are:

• Exporters: These are libraries that help with exporting metrics from third-party systems such as Prometheus metrics.

• PromQL: Prometheus query language allows you to filter multi-dimensional time series data.

Step 1: Create Prometheus system user group

Initially create the Prometheus system with a user and group.

sudo groupadd --system prometheus

The group ID having a lesser than 1000 value is a system group. Once the system group is created, add a Prometheus system user and assign the primary group created to it.

sudo useradd -s /sbin/nologin --system -g prometheus prometheus

Step 2: Create data & configure directories for Prometheus

Prometheus needs a directory allocated so that it can store its data value. We will set it up under /var/lib/prometheus directory.

sudo mkdir /var/lib/prometheus

The primary config files directory is “/etc/prometheus/”. It will have some sub-directories:

for i in rules rules.d files_sd; do sudo mkdir -p /etc/prometheus/${i}; done

Step 3: Download Prometheus on Ubuntu 22.04/20.04/18.04

Further, we need to install the latest version release of Prometheus and extract it to get its binary files.

sudo apt update
sudo apt -y install wget curl vim

Next, install the latest binary archive for Prometheus.

mkdir -p /tmp/prometheus && cd /tmp/prometheus
curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -

Equally important to extract the file:

tar xvf prometheus*.tar.gz
cd prometheus*/

Move the binary files to /usr/local/bin/ directory.

sudo mv prometheus promtool /usr/local/bin/

Now you can check for the version:

$ prometheus --version
prometheus, version 2.35.0 (branch: HEAD, revision: 6656cd29fe6ac92bab91ecec0fe162ef0f187654)

$ promtool --version
promtool, version 2.35.0 (branch: HEAD, revision: 6656cd29fe6ac92bab91ecec0fe162ef0f187654)

Eventually, Move the Prometheus template to “/etc” directory.

sudo mv prometheus.yml /etc/prometheus/prometheus.yml

Also, move consoles and console_libraries to “/etc/prometheus”  as shown:

sudo mv consoles/ console_libraries/ /etc/prometheus/
cd $HOME

Step 4: Configure Prometheus on Ubuntu

At this point create the config file Prometheus – “/etc/prometheus/prometheus.yml”.

sudo vim /etc/prometheus/prometheus.yml

Currently, the template configurations will look similar as below:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

You can further edit the file to your default liking and save file.

Create a Prometheus systemd Service unit file

Additionally, to be able to manage Prometheus service with systemd, you need to define its unit file.

sudo tee /etc/systemd/system/prometheus.service<<EOF
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP \$MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url=

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target
EOF

Now Update the permissions, for this you change the ownership of the directories to Prometheus user and group.

for i in rules rules.d files_sd; do sudo chown -R prometheus:prometheus /etc/prometheus/${i}; done
for i in rules rules.d files_sd; do sudo chmod -R 775 /etc/prometheus/${i}; done
sudo chown -R prometheus:prometheus /var/lib/prometheus/

Then Reload systemd daemon and start the service:

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus

Check the status of Prometheus by running command:

$ systemctl status prometheus
● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2020-01-19 14:36:08 UTC; 14s ago
     Docs: https://prometheus.io/docs/introduction/overview/
 Main PID: 1397 (prometheus)
    Tasks: 7 (limit: 2377)
   Memory: 21.7M
   CGroup: /system.slice/prometheus.service
           └─1397 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.templates

Jan 19 14:36:08 deb10 prometheus[1397]: level=info ts=2020-01-19T14:36:08.964Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhil
Jan 19 14:36:08 deb10 prometheus[1397]: level=info ts=2020-01-19T14:36:08.964Z caller=web.go:506 component=web msg="Start listening for connections" add
Jan 19 14:36:08 deb10 prometheus[1397]: level=info ts=2020-01-19T14:36:08.965Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxS
Jan 19 14:36:08 deb10 prometheus[1397]: level=info ts=2020-01-19T14:36:08.966Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC
Jan 19 14:36:08 deb10 prometheus[1397]: level=info ts=2020-01-19T14:36:08.966Z caller=main.go:664 msg="TSDB started"
Jan 19 14:36:08 deb10 prometheus[1397]: level=info ts=2020-01-19T14:36:08.966Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometh
Jan 19 14:36:08 deb10 prometheus[1397]: level=info ts=2020-01-19T14:36:08.967Z caller=main.go:762 msg="Completed loading of configuration file" filename
Jan 19 14:36:08 deb10 prometheus[1397]: level=info ts=2020-01-19T14:36:08.967Z caller=main.go:617 msg="Server is ready to receive web requests."

In addition, if your server has a firewall service running then you will have to open port 9090:

sudo ufw allow 9090/tcp

Confirm if you can connect to port 9090 by accessing the Prometheus server IP address / DNS name in your web browser.

Finally, Secure the Web console with a Password.

Conclusion

The Digitalocean Prometheus metrics are collected using HTTP pulls, this only exports metrics about itself such as the number of requests received and its memory use. However, you can greatly expand Prometheus by exporters, and optional programs that generate additional metrics.