📊 Grafana / Prometheus / Alerting

🧭 Overview

Monitoring is an essential part of validator operations. We use Prometheus for metrics collection, Grafana for visualization, and Alertmanager (or simple alert scripts) for real-time notifications about node health, performance, and uptime.

---

Prometheus Setup

1. Install Prometheus

sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus /var/lib/prometheus
sudo apt update
sudo apt install prometheus -y

On some systems, you may prefer to install manually for full control:

wget https://github.com/prometheus/prometheus/releases/latest/download/prometheus-*.linux-amd64.tar.gz
tar xvf prometheus-*.tar.gz
sudo mv prometheus*/prometheus prometheus*/promtool /usr/local/bin/
sudo mv prometheus*/consoles prometheus*/console_libraries /etc/prometheus/

2. Configure Prometheus

Edit the configuration file:

sudo nano /etc/prometheus/prometheus.yml

Example configuration:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "node_exporter"
    static_configs:
      - targets: ["localhost:9100"]

  - job_name: "validator_metrics"
    static_configs:
      - targets: ["127.0.0.1:26660"]

3. Start Prometheus

sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Access via browser: 👉 http://your-server-ip:9090

---

Node Exporter (System Metrics)

Install Node Exporter — collects CPU, memory, and disk usage metrics.

cd /opt
wget https://github.com/prometheus/node_exporter/releases/latest/download/node_exporter-*.linux-amd64.tar.gz
tar xvf node_exporter-*.tar.gz
sudo mv node_exporter*/node_exporter /usr/local/bin/

Create a systemd service:

sudo nano /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Check in browser: 👉 http://your-server-ip:9100/metrics

---

Grafana Setup

1. Install Grafana

sudo apt-get install -y apt-transport-https software-properties-common
sudo mkdir -p /etc/apt/keyrings/
sudo wget -q -O /etc/apt/keyrings/grafana.gpg https://apt.grafana.com/gpg.key
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

sudo apt update
sudo apt install grafana -y

2. Start and enable Grafana

sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Access Grafana via browser: 👉 http://your-server-ip:3000 Default credentials: admin / admin (you’ll be prompted to change it after first login)

3. Add Prometheus as a data source

• Open Grafana → ⚙️ Configuration → Data Sources → Add data source

• Choose Prometheus

• Set URL: http://localhost:9090

• Click Save & Test

---

Alerting Configuration

Option 1 — Prometheus Alertmanager

Install Alertmanager:

wget https://github.com/prometheus/alertmanager/releases/latest/download/alertmanager-*.linux-amd64.tar.gz
tar xvf alertmanager-*.tar.gz
sudo mv alertmanager*/alertmanager /usr/local/bin/

Create configuration:

sudo nano /etc/prometheus/alertmanager.yml

Example (email alert):

route:
  receiver: 'email-alert'

receivers:
  - name: 'email-alert'
    email_configs:
      - to: '[e-mail]'
        from: 'alert@server'
        smarthost: 'smtp.gmail.com:587'
        auth_username: 'alert@server'
        auth_identity: 'alert@server'
        auth_password: 'your_password'

Add rule to Prometheus:

rule_files:
  - "alerts.yml"

Example alert rule (/etc/prometheus/alerts.yml):

groups:
  - name: node_alerts
    rules:
      - alert: HighMemoryUsage
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
        for: 1m
        labels:
          severity: critical
        annotations:
          description: "Less than 10% memory available"

Restart Prometheus to apply changes:

sudo systemctl restart prometheus

Option 2 — Simple Script Alerts (Lightweight)

For validators without full Alertmanager setup:

#!/bin/bash
MEM=$(free | awk '/Mem/{printf("%.0f"), $3/$2 * 100}')
if [ "$MEM" -gt 90 ]; then
  echo "⚠️ High memory usage detected: $MEM%" | mail -s "Server Alert" [e-mail]
fi

You can schedule this via cron:

crontab -e
# Add line:
*/5 * * * * /usr/local/bin/mem_alert.sh

---

🧠 Summary

• Prometheus — collects metrics

• Node Exporter — system resource monitoring

• Grafana — visualization dashboards

• Alertmanager / Scripts — notifications on performance issues

This stack provides full observability for your validator infrastructure — from performance and uptime to alerts and trends.