📊 Monitoring Project with Prometheus

Prometheus is an open-source monitoring and alerting tool originally developed by SoundCloud.
It is used to collect, store, and query metrics (CPU, memory, requests, errors, etc.) in order to monitor the health and performance of systems and applications.

It operates using a pull-based model:

Prometheus regularly scrapes HTTP endpoints (/metrics)
Data is stored as time series
Metrics are queried using the PromQL language
Alerts are triggered via Alertmanager

🔌 Exporters

An exporter is an agent or service that exposes metrics in a format readable by Prometheus.

👉 Role:

Collect metrics from a system or application
Expose them through an HTTP endpoint (e.g. http://zeus:9100/metrics)

📌 Prometheus does not collect anything by itself:
It requires exporters or instrumented applications installed on the monitored machine.

🧩 Common exporters examples

Exporter	Purpose
node_exporter	System metrics (CPU, RAM, disk, network)
blackbox_exporter	HTTP / TCP / ICMP availability
mysql_exporter	MySQL database
postgres_exporter	PostgreSQL
redis_exporter	Redis
nginx_exporter	Nginx web server

In this project, we will be focusing on blackbox_exporter and node_exporter.

🧱 Architecture

[ zeus (node & blackbox) ]
|
v
[ prometheus (alermanager) ] ---> [ Email / Slack / Webhook ]

prometheus is the monitoring server zeus is the server we are monitoring

🛠️ Installations

Prometheus & alermanager

Go to the official prometheus website

Download the archives and copy the extracted folders to the destination of your choice.

Prometheus tree default install

.
├── LICENSE
├── NOTICE
├── prometheus
├── prometheus.yml
└── promtool

Alermanager tree default install

.
├── alertmanager
├── alertmanager.yml
├── amtool
├── data
│   ├── nflog
│   └── silences
├── LICENSE
└── NOTICE

Zeus (Node & Blackbox)

Download the archives and copy the extracted folders to the destination of your choice.

Node tree default install

├── LICENSE
├── node_exporter
└── NOTICE

Blackbox tree default install

.
├── blackbox_exporter
├── blackbox.yml
├── LICENSE
└── NOTICE

⚙️ Configurations

Prometheus

The configuration files are fully commented and explained in detail:

prometheus.yml

alertmanager.yml stays in the root directory of prometheus

alert.rules.yml stays in the root directory of prometheus

Zeus

By default, exporters do not start automatically.
On zeus, systemctl services were configured to ensure they start at boot.

Node

$ cat /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=ir0nx
ExecStart=/usr/bin/node_exporter
Restart=always
NoNewPrivileges=yes

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable node_exporter.service
sudo systemctl start node_exporter.service
sudo systemctl status node_exporter.service

Black box

$ cat /etc/systemd/system/blackbox_exporter.service
[Unit]
Description=Prometheus Blackbox Exporter
After=network.target

[Service]
User=ir0nx
ExecStart=/usr/bin/black_box/blackbox_exporter
Restart=always
NoNewPrivileges=yes

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable blackbox_exporter.service
sudo systemctl start blackbox_exporter.service
sudo systemctl status blackbox_exporter.service

Check the ports

$ ss -lntp | grep -E "node|black"
LISTEN 0      4096               *:9115             *:*    users:(("blackbox_export",pid=1444,fd=3))
LISTEN 0      4096               *:9100             *:*    users:(("node_exporter",pid=1450,fd=3))

Simple login page

We created a simple login page on zeus to detect a bruteforce attack on login. The code base of the login page is in zeus/simple_login_page. The login page is exposed on port 5000. To run the app.py you need "prometheus_client" python package

pip3 install prometheus_client

"prometheus_client" is not a default python package. You might need to set a venv path with python

python3 -m venv /path/to/your/virtual/environment

🧠 Testing the architecture

You will need to run "./prometheus" and "./alertmanger" binaries from their respective directories in order to start testing. You can configure them as systemctl services to set them to launch at startup.

All the testing scripts are in zeus/scripts (You might need to change some values of the scripts) Here is the prometheus dashboard /targets and /alerts