Protocol of installation HashiCorp Vault in HA mode

The installation will be performed on three virtual machines, preferably distributed across different physical hosts. I will be using Keepalived to support the cluster's virtual IP instead of an external hardware load balancer. I will install the cluster on SUSE 15 SP4 just only because I have a ready-made template for this operating system.

DNS and certificate

I created three virtual machines and added four IP addresses for hcv{1,2,3} and to the DNS. The last name is for the cluster's VIP to be an entry point.

Then create a certificate for this name, including all hcv{1,2,3}.domain.coms as DNS alternate names. It is important to enable them because some calls between nodes will be made using hostnames. It is also useful to add the DNS name localhost and IP Adding this will save you a lot of keystrokes when using the CLI.

If not already done, add your CA to be trusted by creating a file /etc/pki/trust/anchors/ca.crt and run update-ca-certificates.

Install HashiCorp Vault

Download RPM for RedHat. It is also suitable for SUSE. Install it with the rpm command.

Place the certificate, you had created in step above, and its key in the /etc/vault.d/ directory, then edit /etc/vault.d/vault.hcl. Here is an example of this file for hcv1 node:

# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1
# Full configuration options can be found at

cluster_name = "HCV"
ui = true
disable_mlock = true
storage "raft" {
  path = "/opt/vault/data"
  node_id = "hcv1"
  retry_join {
    leader_api_addr = ""
  retry_join {
    leader_api_addr = ""

# HTTPS listener
listener "tcp" {
  address       = ""
  tls_cert_file = "/etc/vault.d/"
  tls_key_file  = "/etc/vault.d/"

api_addr = ""
cluster_addr = ""

Copy this file to the second node and replace the red node name with hcv2 and the blue node names with the names of neighboring nodes. Copy the file to the third node and make similar changes.

Start vault service on every node by commands

# systemctl daemon-reload
# systemctl enable --now vault.service

Initialize internal database

Initialization should only be done on one node (say hcv1). Open a shell on the node and run the following command:

# vault operator init | tee vault.credentials
Unseal Key 1: l4/biMcRLPDG53ukLG/MSgD0BVnrVlJGtw95cycqR0Vb
Unseal Key 2: CN089ZHSTIVEivxk03XGh48LD2GgL8dJ6ssiuLXZhda1
Unseal Key 3: f7oWJN7aWSxv8tSCI1+g5w8A40WofGFvf7c/cDrFvVXu
Unseal Key 4: vGNyIRDVSUJ/o93N7FiXdOveAjStu+eV+fCI0VWQpMZw
Unseal Key 5: tlWZh2Xmeqii4y6OKDTIxt8V5uIH5JyokQxv7Xfw8i7P

Initial Root Token: hvs.3mr6C5BG4uEAg33Jw2Duti5j

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

This output is shown only once. Now. It shows the initial root token usefull for further configuration. Five Unseal Keys are meant to be stored with five different keepers, any three of which must be available at database startup. Because of the importance of this data, I have saved it now in the vault.credentials file. Don't forget to take care of it after installation.

Initialization does not mean that the database has become open. You need to "unseal" it using any three keys printed above. Here is an example of this process

# vault operator unseal l4/biMcRLPDG53ukLG/MSgD0BVnrVlJGtw95cycqR0Vb
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       2cbde089-d2eb-907d-a41c-7a9cff956524
Version            1.15.2
Build Date         2023-11-06T11:33:28Z
Storage Type       raft
HA Enabled         true
# vault operator unseal CN089ZHSTIVEivxk03XGh48LD2GgL8dJ6ssiuLXZhda1
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       2cbde089-d2eb-907d-a41c-7a9cff956524
Version            1.15.2
Build Date         2023-11-06T11:33:28Z
Storage Type       raft
HA Enabled         true
# vault operator unseal f7oWJN7aWSxv8tSCI1+g5w8A40WofGFvf7c/cDrFvVXu
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.15.2
Build Date              2023-11-06T11:33:28Z
Storage Type            raft
Cluster Name            HCV
Cluster ID              2f22afcc-d95e-b788-0526-41d97bd5653a
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     
Raft Committed Index    32
Raft Applied Index      32

Each command prints the output of the vault status command at the end of execution. You can see that the third output is very different and indicates that the storage becomes usable.

You are not required to unseal the database using the CLI. It is possible to open browser and point it to the address, and then paste the required key. This method is useful if you actually use three different key holders.

Once unsealed, you can log into the vault using the saved root token and make changes or view the cluster status.

# vault login
Token (will be hidden): 
# vault operator raft list-peers
Node    Address                  State       Voter
----    -------                  -----       -----
hcv1    leader      true

Add other nodes to cluster

Actually, our cluster is almost ready, because all three nodes are already trying to communicate with each other. However, they still have nothing in common.

To add another node, open a shell (or browser) on it and run the same unseal commands. If successful, repeat on the third node. The resulting status will be

# vault operator raft list-peers
Node    Address                  State       Voter
----    -------                  -----       -----
hcv1    leader      true
hcv2    follower    true
hcv3    follower    true

Maintain cluster VIP using Keepalived

Install Keepalived. It usually comes with the distribution.

# zypper in -y iptables keepalived ipvsadm

Create a configuration file /etc/keepalived/keepalived.conf

global_defs {
   router_id HCV_POC

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type AH
        auth_pass fe75c3ac-1962-4124-912a-2b4ca36b4720
    virtual_ipaddress {

An important information about configuration options. The brown parameters must be unique for this cluster in the network, otherwise this cluster will interfere with other clusters. This is the simplest configuration file. It will just support VIP without checking serviceability or load balancing between nodes.

Copy this file to other nodes, replacing MASTER to BACKUP.

You can start keepalived now.

Roll over Vault cluster

The only thing you should be aware of when restarting a storage cluster node is that it starts up sealed (its database is still locked with the encryption key). The node will try to participate in the cluster, but will be unsuccessful. You can check its status using the vault status command. You should run the command vault operator unseal XXX until the status changes and the node becomes opened. Then you can move on to another node.

Updated on Mon Dec 18 12:16:14 IST 2023 More documentations here