Protocol of installation HashiCorp Vault in HA mode

The installation will be performed on three virtual machines, preferably distributed across different physical hosts. I will be using Keepalived to support the cluster's virtual IP instead of an external hardware load balancer. I will install the cluster on SUSE 15 SP4 just only because I have a ready-made template for this operating system.

DNS and certificate

I created three virtual machines and added four IP addresses for hcv{1,2,3}.domain.com and hcv.domain.com to the DNS. The last name is for the cluster's VIP to be an entry point.

Then create a certificate for this hcv.domain.com name, including all hcv{1,2,3}.domain.coms as DNS alternate names. It is important to enable them because some calls between nodes will be made using hostnames. It is also useful to add the DNS name localhost and IP 127.0.0.1. Adding this will save you a lot of keystrokes when using the CLI.

If not already done, add your CA to be trusted by creating a file /etc/pki/trust/anchors/ca.crt and run update-ca-certificates.

Install HashiCorp Vault

Download RPM for RedHat. It is also suitable for SUSE. Install it with the rpm command.

Place the certificate, you had created in step above, and its key in the /etc/vault.d/ directory, then edit /etc/vault.d/vault.hcl. Here is an example of this file for hcv1 node:

# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1
# Full configuration options can be found at https://developer.hashicorp.com/vault/docs/configuration

cluster_name = "HCV"
ui = true
disable_mlock = true
storage "raft" {
  path = "/opt/vault/data"
  node_id = "hcv1"
  retry_join {
    leader_api_addr = "https://hcv2.domain.com:8200"
  }
  retry_join {
    leader_api_addr = "https://hcv3.domain.com:8200"
  }
}

# HTTPS listener
listener "tcp" {
  address       = "127.0.0.1:8200"
  tls_cert_file = "/etc/vault.d/hcv.domain.com.crt"
  tls_key_file  = "/etc/vault.d/hcv.domain.com.key"
}
listener "tcp" {
  address       = "10.0.0.11:8200"
  tls_cert_file = "/etc/vault.d/hcv.domain.com.crt"
  tls_key_file  = "/etc/vault.d/hcv.domain.com.key"
}

api_addr = "https://hcv.domain.com:8200"
cluster_addr = "https://hcv1.domain.com:8201"

An api_addr points to our VIP DNS. Copy this file to the second node and replace the red node name with hcv2 and the blue node names with the names of neighboring nodes. Fix listener address to match hosts IP. Two listener lines required instead of common recommended "0.0.0.0:8200" because of keepalived load balancer. Copy the file to the third node and make similar changes.

Start vault service on every node by commands

# systemctl daemon-reload
# systemctl enable --now vault.service

Initialize internal database

Initialization should only be done on one node (say hcv1). Open a shell on the node and run the following command:

# vault operator init | tee vault.credentials
Unseal Key 1: l4/biMcRLPDG53ukLG/MSgD0BVnrVlJGtw95cycqR0Vb
Unseal Key 2: CN089ZHSTIVEivxk03XGh48LD2GgL8dJ6ssiuLXZhda1
Unseal Key 3: f7oWJN7aWSxv8tSCI1+g5w8A40WofGFvf7c/cDrFvVXu
Unseal Key 4: vGNyIRDVSUJ/o93N7FiXdOveAjStu+eV+fCI0VWQpMZw
Unseal Key 5: tlWZh2Xmeqii4y6OKDTIxt8V5uIH5JyokQxv7Xfw8i7P

Initial Root Token: hvs.3mr6C5BG4uEAg33Jw2Duti5j

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

This output is shown only once. Now. It shows the initial root token usefull for further configuration. Five Unseal Keys are meant to be stored with five different keepers, any three of which must be available at database startup. Because of the importance of this data, I have saved it now in the vault.credentials file. Don't forget to take care of it after installation.

Initialization does not mean that the database has become open. You need to "unseal" it using any three keys printed above. Here is an example of this process

# vault operator unseal l4/biMcRLPDG53ukLG/MSgD0BVnrVlJGtw95cycqR0Vb
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       2cbde089-d2eb-907d-a41c-7a9cff956524
Version            1.15.2
Build Date         2023-11-06T11:33:28Z
Storage Type       raft
HA Enabled         true
# vault operator unseal CN089ZHSTIVEivxk03XGh48LD2GgL8dJ6ssiuLXZhda1
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       2cbde089-d2eb-907d-a41c-7a9cff956524
Version            1.15.2
Build Date         2023-11-06T11:33:28Z
Storage Type       raft
HA Enabled         true
# vault operator unseal f7oWJN7aWSxv8tSCI1+g5w8A40WofGFvf7c/cDrFvVXu
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.15.2
Build Date              2023-11-06T11:33:28Z
Storage Type            raft
Cluster Name            HCV
Cluster ID              2f22afcc-d95e-b788-0526-41d97bd5653a
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     
Raft Committed Index    32
Raft Applied Index      32

Each command prints the output of the vault status command at the end of execution. You can see that the third output is very different and indicates that the storage becomes usable.

You are not required to unseal the database using the CLI. It is possible to open browser and point it to the address https://hcv1.domain.com:8200, and then paste the required key. This method is useful if you actually use three different key holders.

Once unsealed, you can log into the vault using the saved root token and make changes or view the cluster status.

# vault login
Token (will be hidden): 
# vault operator raft list-peers
Node    Address                  State       Voter
----    -------                  -----       -----
hcv1    hcv1.domain.com:8201    leader      true

Add other nodes to cluster

Actually, our cluster is almost ready, because all three nodes are already trying to communicate with each other. However, they still have nothing in common.

To add another node, open a shell (or browser) on it and run the same unseal commands. If successful, repeat on the third node. The resulting status will be

# vault operator raft list-peers
Node    Address                  State       Voter
----    -------                  -----       -----
hcv1    hcv1.domain.com:8201    leader      true
hcv2    hcv2.domain.com:8201    follower    true
hcv3    hcv3.domain.com:8201    follower    true

Maintain cluster VIP using Keepalived

Install Keepalived. It usually comes with the distribution.

# zypper in -y iptables keepalived ipvsadm

Create a configuration file /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type AH
        auth_pass fe75c3ac
    }
    virtual_ipaddress {
        10.0.0.14/24
    }
}

virtual_server 10.0.0.14 8200 {
    delay_loop 6
    lb_algo sh
    lb_kind NAT
    persistence_timeout 50
    protocol TCP

    real_server 10.0.0.11 8200 {
        weight 10
        SSL_GET {
          connect_timeout 2
          url { 
            path /
            status_code 307
          }
        }
    }

    real_server 10.0.0.12 8200 {
        weight 10
        SSL_GET {
          connect_timeout 2
          url { 
            path /
            status_code 307
          }
        }
    }

    real_server 10.0.0.13 8200 {
        weight 10
        SSL_GET {
          connect_timeout 2
          url { 
            path /
            status_code 307
          }
        }
    }

}

An important information about configuration options. The brown parameters must be unique for this cluster in the network, otherwise this cluster will interfere with other clusters. The preferences goes to localhost to save unnesesary traffic. Neigbour nodes used only if localhost is not serving. Because of lb_kind NAT, IP forwarding should be enabled on each node:

# echo 1 > /proc/sys/net/ipv4/ip_forward
# echo "net.ipv4.ip_forward=1" > /etc/sysctl.d/50-ip_forward.conf

Copy this file to other nodes, replacing MASTER to BACKUP.

You can start keepalived now.

Roll over Vault cluster

The only thing you should be aware of when restarting a storage cluster node is that it starts up sealed (its database is still locked with the encryption key). The node will try to participate in the cluster, but will be unsuccessful. You can check its status using the vault status command. You should run the command vault operator unseal XXX until the status changes and the node becomes opened. Then you can move on to another node.


Updated on Mon Dec 18 12:16:14 IST 2023 More documentations here