Tutorial: persistent vault

Goal

At the end you will see an OpenBao server that keeps its data on disk and speaks TLS: initialized once, unsealed with 2 of 3 keys, set up without keeping the root token, used by Keyholder over TLS, sealed again by a restart, unsealed, and backed up.

Every command and output below comes from a real run, recorded in demo/tutorials/server/expected/ and checked by run.sh --check. The steps, and why each one is there, are in the persistent vault guide.

What is different from a real install. The server runs deploy/server.hcl itself, with its paths moved to a temporary directory and its port moved to 18200. The certificate authority is created for the run. The unseal keys, the root token and the administrator's password stay in that temporary directory and are deleted with it.

Run it

bash demo/tutorials/server/run.sh all       # needs bao, openssl, python3, curl and go
bash demo/tutorials/server/run.sh --check

Script: demo/tutorials/server/run.sh. Policies: policy-project-read.hcl and policy-admin.hcl.

1. Initialize once, unseal with 2 of 3 keys

A new server is sealed and not initialized. bao operator init runs once in the vault's life and prints the unseal keys and the root token; here they go to a file that never enters the repository. Each unseal key moves the progress by one; the second one opens the vault.

# Start the server, initialize it once, unseal it with 2 of 3 keys

$ bao status
Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            2.7.1
Commit Date        <time>
Storage Type       raft
HA Enabled         true
exit_code=2

$ bao operator init -key-shares=3 -key-threshold=2 -format=json > <file kept offline>
exit_code=0

The file now holds 3 unseal keys and the root token. It never goes into a repository; give each key to a different place or person.

$ bao operator unseal <unseal key 1> >/dev/null
exit_code=0

Vault: initialized=true sealed=true unseal_progress=1/2

$ bao operator unseal <unseal key 2> >/dev/null
exit_code=0

Vault: initialized=true sealed=false unseal_progress=0/2

2. Set up, then revoke the root token

The root token creates the KV mount, the policies and an administrator login, and is then revoked. From then on the administrator logs in with a password and gets an 8-hour token that can store secrets and issue the agents' wrapped tokens, but cannot change the vault's configuration.

# With the root token: KV, policies and an administrator login; then the root token is revoked

$ bao secrets enable -path=secret kv-v2
Success! Enabled the kv-v2 secrets engine at: secret/
exit_code=0

$ bao kv put -mount=secret my-project/dev API_TOKEN=tok-fake-not-a-real-secret
======= Secret Path =======
secret/data/my-project/dev

======= Metadata =======
Key                Value
---                -----
created_time       <time>
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
exit_code=0

$ bao policy write my-project-read deploy/policy-project-read.hcl
Success! Uploaded policy: my-project-read
exit_code=0

$ bao policy write keyholder-admin deploy/policy-admin.hcl
Success! Uploaded policy: keyholder-admin
exit_code=0

$ bao auth enable userpass
Success! Enabled userpass auth method at: userpass/
exit_code=0

$ bao write auth/userpass/users/admin password=<password> token_policies=keyholder-admin token_ttl=8h token_max_ttl=8h
Success! Data written to: auth/userpass/users/admin
exit_code=0

$ bao token revoke -self
Success! Revoked token (if it existed)
exit_code=0

The root token no longer exists. Each day the administrator logs in instead:

$ bao login -method=userpass -field=token username=admin password=<password> > <administrator token>
exit_code=0

Policies of the administrator's token: default, keyholder-admin

3. Keyholder over TLS

With BAO_CACERT pointing at the certificate authority, the session opens and the API receives the token. Without it, Keyholder refuses the connection and says which variable to set.

# Keyholder reaches the vault over TLS; without the CA it refuses to connect

$ BAO_WRAPPING_TOKEN=<wrapped token> khcall keyholder_status keyholder_run '{"command":"curl","args":["-sS","-H","Authorization: Bearer {{vault://my-project/dev/API_TOKEN}}","http://127.0.0.1:18080/"]}'
### keyholder_status
{
  "addr": "https://127.0.0.1:18200",
  "initialized": true,
  "kv_mount": "secret",
  "scope": "my-project",
  "sealed": false,
  "server_version": "2.7.1",
  "token_expire_time": "<time>",
  "token_policies": [
    "default",
    "my-project-read"
  ],
  "token_renewable": true,
  "token_ttl_seconds": <ttl>
}
### keyholder_run
{
  "exit_code": 0,
  "refs_used": [
    "vault://my-project/dev/API_TOKEN"
  ],
  "stderr": "",
  "stdout": "api says: hello, caller with header Authorization: Bearer [hidden:vault://my-project/dev/API_TOKEN]\n"
}
--- stderr
<timestamp> audit cmd="curl" refs=[vault://my-project/dev/API_TOKEN] exit=0 dur=<duration> status=ok
exit_code=0

Did the API receive the stored token? yes

$ BAO_CACERT= BAO_WRAPPING_TOKEN=<wrapped token> khcall keyholder_status
{
  "addr": "https://127.0.0.1:18200",
  "initialized": false,
  "kv_mount": "secret",
  "scope": "my-project",
  "sealed": false,
  "server_version": "",
  "session_error": "unwrap token: cannot reach the vault at https://127.0.0.1:18200: tls: failed to verify certificate: x509: certificate signed by unknown authority; set BAO_CACERT to the CA certificate that signed the vault's certificate",
  "token_policies": null,
  "token_renewable": false,
  "token_ttl_seconds": 0
}
--- stderr
<timestamp> session not opened: unwrap token: cannot reach the vault at https://127.0.0.1:18200: tls: failed to verify certificate: x509: certificate signed by unknown authority; set BAO_CACERT to the CA certificate that signed the vault's certificate
exit_code=0

4. A restart seals the vault

A wrapped token is issued, then the server restarts. It comes back sealed: Keyholder reports it and does not consume the token. After two unseal keys a new Keyholder start opens the session with that same token, and the secret stored before the restart is still there.

# After a restart the vault is sealed: Keyholder fails cleanly, then works once it is unsealed

A wrapped token was issued before the restart.

Vault after the restart: initialized=true sealed=true unseal_progress=0/2

$ BAO_WRAPPING_TOKEN=<same wrapped token> khcall keyholder_status
{
  "addr": "https://127.0.0.1:18200",
  "initialized": true,
  "kv_mount": "secret",
  "scope": "my-project",
  "sealed": true,
  "server_version": "2.7.1",
  "session_error": "unwrap token: the vault is sealed: an administrator must unseal it (bao operator unseal), then restart this server; the wrapped token was not used",
  "token_policies": null,
  "token_renewable": false,
  "token_ttl_seconds": 0
}
--- stderr
<timestamp> session not opened: unwrap token: the vault is sealed: an administrator must unseal it (bao operator unseal), then restart this server; the wrapped token was not used
exit_code=0

$ bao operator unseal <unseal key 2> >/dev/null
exit_code=0

Vault: initialized=true sealed=true unseal_progress=1/2

$ bao operator unseal <unseal key 3> >/dev/null
exit_code=0

Vault: initialized=true sealed=false unseal_progress=0/2

$ BAO_WRAPPING_TOKEN=<same wrapped token> khcall keyholder_list
{
  "entries": [
    {
      "ref": "vault://my-project/dev/API_TOKEN",
      "version": 1
    }
  ],
  "note": "Use {{vault://path/KEY}} in the args, env or files of keyholder_run.",
  "scope": "my-project"
}
exit_code=0

5. Back up

The administrator saves a snapshot of the whole vault. It is encrypted with the vault's keys, so it is restored and unsealed with the same unseal keys.

# A snapshot of the whole vault, taken by the administrator

$ bao operator raft snapshot save <backup file>
exit_code=0

Snapshot written: yes. It is encrypted with the vault's keys; it is restored with `bao operator raft snapshot restore` and unsealed with the same keys.