Validator Security (Cosmos SDK)

4 min readDec 21, 2022

Reference this page for recommend security measures that can be taken as a node validator.

Each validator candidate is encouraged to run its operations independently, as diverse setups increase the resilience of the network.

Key Management — HSM

It is mission-critical that an attacker cannot steal a validator’s key. If this is possible, it puts the entire stake delegated to the compromised validator at risk. Hardware security modules are an important strategy for mitigating this risk.

The recommended hardware for storing private keys is Yubikey and Ledger Nano.
For the moment, you can get further information by looking at tmkms.

Firewalls

A firewall is a network security system that uses predetermined rules to filter incoming and outgoing traffic, blocking suspicious requests. A properly configured VPS firewall will block all connections to ports that are not used by any legitimate services. Firewalls help to stop attacks and threats that can lead to outages or server takeovers. For example, you can set up your firewall to allow access from only specific IPs, and have the necessary ports only open to sentry nodes.

SSH keys

SSH keys are more difficult to hack than passwords, as SSH keys can be up to 4096 bits in length and are typically at least 1024 bits long ( which is the security equivalent of a 12 characters password).

Also, SSH keys are more robust than passwords against compromises on the server-side, as even if the server is compromised, your SSH key remains safe. The SSH key remains on the client-side and no secret value is ever sent to the server.

As the SSH key is stored on a device, keep in mind that you should protect the key with a passphrase (to defend your SSH key in case the device gets compromised).

Dedicated Server and Auditing

Try using a dedicated server, without unnecessary services. This will minimize the attack surface. Keep in mind using auditing tools to perform in-depth security scans. This will help test security defenses and enhance system hardening, keep your operating system updated and dependencies updated, as his will help protect against vulnerabilities.

Backups

Backups of chain state are possible using the commands specified here. If you are using a recent version of Cosmovisor, then the default configuration is that a state backup will be created before upgrades are applied. This can be turned off using environment flags.

Alerting and monitoring

Alerting and monitoring is desirable as well — you are encouraged to explore solutions and find one that works for your setup. Prometheus is available out-of-the box, and there are a variety of open-source tools. Recommended reading:

Alerting:

Tenderduty: https://github.com/blockpane/tenderduty
PANIC: https://github.com/SimplyVC/panic_cosmos

Monitoring:

Simple setup using Grafana Cloud

Using only the raw metrics endpoint provided by okp4d you can get a working dashboard and alerting setup using Grafana Cloud. This means you don't have to run Grafana on the instance.

First, in config.toml enable Prometheus. The default metrics port will be 26660
Download Prometheus — this is needed to ship logs to Grafana Cloud.
Create a prometheus.yml file with your Grafana Cloud credentials in the Prometheus folder. You can get these via the Grafana UI. Click 'details' on the Prometheus card:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: cosmops
    static_configs:
    - targets: ['localhost:26660']
      labels:
        group: 'cosmops'

remote_write:
  - url: https://your-grafana-cloud-endpoint/api/prom/push
    basic_auth:
      username: ID_HERE
      password: "API KEY HERE"

3. Set up a service file, with sudo nano /etc/systemd/system/prometheus.service, replacing <your-user> and <prometheus-folder> with the location of Prometheus. This sets the Prometheus port to 6666

[Unit]
Description=prometheus
After=network-online.target

[Service]
User=<your-user>
ExecStart=/home/<your-user>/<prometheus-folder>/prometheus --config.file=/home/<your-user>/<prometheus-folder>/prometheus.yml --web.listen-address=:6666 --storage.tsdb.path=/home/<your-user>/<prometheus-folder>/data
Restart=always
RestartSec=3
LimitNOFILE=4096

[Install]
WantedBy=multi-user.target
/etc/systemd/system/prometheus.service

4. Enable and start the service.

sudo -S systemctl daemon-reload
sudo -S systemctl enable prometheus
sudo systemctl start prometheus

5. Import a dashboard to your Grafana. Search for ‘Cosmos Validator’ to find several options. You should see logs arriving in the dashboard after a couple of minutes.

For more info:

Avoiding DDOS attacks

If you are comfortable with server ops, you might want to build out a Sentry Node Architecture validator to protect against DDOS attacks.

The current best practice for running mainnet nodes is a Sentry Node Architecture. There are various approaches, as detailed here. Some validators advocate co-locating all three nodes in virtual partitions on a single box, using Docker or other virtualisation tools. However, if in doubt, just run each node on a different server.

Bear in mind that Sentries can have pruning turned on, as outlined here. It is desirable, but not essential, to have pruning disabled on the validator node itself.

It’s time to take the network live!