In this article, we'll show you how to rotate your encryption key on your Virtru Private Keystore (VPK) server for Google Client Side Encryption (CSE). We'll also show you how to keep every key you need to decrypt existing content. This article is for customers whose VPK server doesn't use a Customer Key Server (CKS).
Jump to:
- Before You Begin
- Choose Your Scenario
- Back Up Your Configuration
- Import Your Current Key
- Generate a New Key and Rotate
- Rotate Again on a Server That Uses secrets.json
- Sync Keys Across Multiple Servers
- Add Keys from Another CSE Server
- Retire a CSE Server
- Roll Back a Change
- Troubleshooting
- Additional Resources
Before You Begin
- Upgrade your CSE container to v5.4.0 or later.
- You need shell access to each CSE server, plus your
cse.envandrun.shfiles.
Important
Rotation applies to new content only. Content you've already encrypted stays wrapped with the key that encrypted it. Keep every previous key in secrets.json, or you can't decrypt that content.
Please Note:
The examples use v5.4.0 and the /var/virtru/cse working directory. Substitute your own version and directory, and replace each configuration file with your own. Sample key values are dummies.
Choose Your Scenario
| Your situation | Follow these sections |
|---|---|
First rotation. Your cse.env file still uses SECRET_KEY. |
Back Up Your Configuration, then Import Your Current Key, then Generate a New Key and Rotate |
You've rotated before. Your server already uses secrets.json. |
Back Up Your Configuration, then Rotate Again on a Server That Uses secrets.json |
| You run more than one CSE server for the same key service. | Your rotation steps above, then Sync Keys Across Multiple Servers |
| You're moving content from another CSE server (for example, another domain) onto this server. | Back Up Your Configuration, then Add Keys from Another CSE Server, then Retire a CSE Server |
| A change didn't work. | Roll Back a Change |
Back Up Your Configuration
Before you make any change, copy your configuration files on each server you'll touch:
cp /var/virtru/cse/cse.env /var/virtru/cse/cse.env.bak
cp /var/virtru/cse/run.sh /var/virtru/cse/run.sh.bak
cp /var/virtru/cse/secrets.json /var/virtru/cse/secrets.json.bakIf secrets.json doesn't exist yet, skip the third command.
Important
These files contain your key values. Store your backups in a secure location with restricted access.
Import Your Current Key
Complete this section once, the first time you move from SECRET_KEY to secrets.json.
1. In cse.env, comment out SECRET_KEY and add SECRET_KEYS_PATH:
#SECRET_KEY=secret-key:thisisadummyvalue+VgE2TZuJwWtest7fi8=
SECRET_KEYS_PATH=/app/cse/secrets.jsonNote the key name (before the colon) and the key value (after the colon). You need both in step 5.
2. In your CSE working directory, create a file named secrets.json that contains an empty JavaScript Object Notation (JSON) object:
vi secrets.json{}3. Update the file permissions:
chmod 646 secrets.json4. Store your current key and set it as active:
docker run --env-file /var/virtru/cse/cse.env \
--mount type=bind,source=/var/virtru/cse/secrets.json,target=/app/cse/secrets.json \
-p 443:9000 virtru/cse:v5.4.0 \
store-secret -n secret-key -v thisisadummyvalue+VgE2TZuJwWtest7fi8= --active5. Stop and remove the running CSE container:
docker stop <containerID>
docker rm <containerID>| Option | Description |
|---|---|
-n |
The key name. Use the name from your SECRET_KEY line. |
-v |
The key value from your SECRET_KEY line. |
--active |
Sets this key as the one CSE uses to encrypt new content. |
You should see this output:
RUNNING COMMAND: store-secret
A new secret has been successfully generated and stored in the secrets file.
New Secret:
secret-key6. Confirm that your key is stored and active:
cat secrets.json{"active":"secret-key","secrets":[{"name":"secret-key","value":"thisisadummyvalue+VgE2TZuJwWtest7fi8="}]}7. Remove the temporary container that the command created:
docker ps -a
docker rm <containerID>Generate a New Key and Rotate
1. Generate a new key. The command sets the new key as active:
docker run --env-file /var/virtru/cse/cse.env \
--mount type=bind,source=/var/virtru/cse/secrets.json,target=/app/cse/secrets.json \
-p 443:9000 virtru/cse:v5.4.0 \
generate-secretYou should see this output:
RUNNING COMMAND: generate-secret
A new secret has been successfully generated and stored in the secrets file.
New Secret:
0123456789abcdef
Existing Secret:
secret-key2. Confirm that both keys are listed and the new key is active:
cat secrets.json{"active":"0123456789abcdef","secrets":[{"name":"secret-key","value":"thisisadummyvalue+VgE2TZuJwWtest7fi8="},{"name":"0123456789abcdef","value":"anotherdummyvalue+Xk9QmPz3rtest2d5="}]}3. Remove the temporary container:
docker ps -a
docker rm <containerID>4. In run.sh, add the line that mounts secrets.json (the -v /var/virtru/cse/secrets.json line):
docker run --detach \
--env-file ./cse.env \
-p 443:9000 \
-v /var/virtru/cse/server.cert:/run/secrets/server.cert \
-v /var/virtru/cse/server.key:/run/secrets/server.key \
-v /var/virtru/cse/secrets.json:/app/cse/secrets.json \
--restart unless-stopped \
--name cse-5.4.0 \
virtru/cse:v5.4.0If you run multiple CSE servers, read Sync Keys Across Multiple Servers before you start the container.
5. Start your CSE container:
sh run.sh6. Confirm that the container is running:
docker psA healthy container shows a status of Up:
CONTAINER ID IMAGE STATUS PORTS NAMES
<containerID> virtru/cse:v5.4.0 Up 2 seconds 0.0.0.0:443->9000/tcp, :::443->9000/tcp cse-5.4.07. Verify the server:
- In Google Workspace, create a new piece of encrypted content and confirm that it opens.
- Open a file you encrypted before the rotation to confirm that your previous key still works.
8. Check the container logs for the wrap and policy messages:
docker logs <containerID> -fWrap requested by "<user-email>" on resource "<resource-path>" with reason ""
Policy created for "<resource-path>" with policy id "<policy-id>"Rotate Again on a Server That Uses secrets.json
If your server already uses secrets.json, you don't need to import your key again.
1. Stop and remove the running CSE container:
docker stop <containerID>
docker rm <containerID>2. Run generate-secret, as shown in step 1 of Generate a New Key and Rotate.
3. Confirm that secrets.json lists all previous keys plus the new active key:
cat secrets.json4. Remove the temporary container, then start your CSE container with sh run.sh.
5. Verify new and previously encrypted content, as shown in step 7 of Generate a New Key and Rotate.
Do Not Delete Keys You Still Need
Never delete a key from secrets.json while content encrypted with that key still exists. You can't recover that content without the key.
Sync Keys Across Multiple Servers
Every server that serves the same key service URL must use the same secrets.json. Otherwise, a server can't decrypt content that another server encrypted with a newer key.
To keep that gap short, update every server's file before any server starts using the new key.
1. On your primary server, complete your rotation steps, but stop before you start the container.
2. Copy the new secrets.json to the working directory on each additional server. Use a secure method, such as scp.
3. On each additional server, back up your configuration. Then comment out SECRET_KEY and add SECRET_KEYS_PATH in cse.env:
#SECRET_KEY=secret-key:thisisadummyvalue+VgE2TZuJwWtest7fi8=
SECRET_KEYS_PATH=/app/cse/secrets.jsonSkip this step if the server already uses secrets.json.
4. On each additional server, update run.sh to match the primary server: the same image version and the same secrets.json mount.
5. Restart the servers one after another, starting with the primary server:
docker stop <containerID>
docker rm <containerID>
sh run.sh6. On each server, confirm that the container status is Up and that both new and previously encrypted content open.
Add Keys from Another CSE Server
Use this section to consolidate onto one server. For example, you run separate CSE servers for domain A and domain B, and you want to keep only domain A's server without losing access to domain B's content.
Two separate tasks are involved:
- Keys: Domain B's keys must be available to the server that handles domain B's content.
- Routing: Google sends each decrypt request to the key service that encrypted the content. Adding domain B's keys to domain A's server doesn't redirect those requests. You also need to change where Google sends them (step 3).
Step 1: Check your setup
| Question | Why it matters |
|---|---|
| Are both domains in the same Google Workspace tenant? | Google's key service migration works within one Admin console. If your domains are in separate tenants, contact Virtru Support before you continue. |
Do both servers use the same HMAC_TOKEN_ID (the same Virtru organization)? |
If the values differ, contact Virtru Support before you continue. |
Is domain B's server on v5.4.0 or later with a secrets.json file? |
If so, it may hold several keys. You need every one of them. |
| Do any key names on server B match key names on server A? | Duplicate names in one file may conflict. If any match, contact Virtru Support before you continue. |
Step 2: Add domain B's keys to domain A's server
1. On both servers, back up your configuration (see Back Up Your Configuration).
2. Collect domain B's keys:
- If server B uses
SECRET_KEY, the line isSECRET_KEY=<name>:<value>. - If server B uses
secrets.json, collect every entry in thesecretslist, not only the active one.
3. On server A, stop and remove the running container.
4. For each domain B key, run store-secret without --active, and use the key's original name:
docker run --env-file /var/virtru/cse/cse.env \
--mount type=bind,source=/var/virtru/cse/secrets.json,target=/app/cse/secrets.json \
-p 443:9000 virtru/cse:v5.4.0 \
store-secret -n <domain-b-key-name> -v <domain-b-key-value>Remove the temporary container after each command.
5. Confirm the result:
cat secrets.jsonDomain A's key stays active, and domain B's keys are listed:
{"active":"secret-key","secrets":[{"name":"secret-key","value":"thisisadummyvalue+VgE2TZuJwWtest7fi8="},{"name":"<domain-b-key-name>","value":"<domain-b-key-value>"}]}If active changed to a domain B key, restore your secrets.json backup and contact Virtru Support.
6. Start server A with sh run.sh, then confirm that domain A content still encrypts and decrypts.
7. If other servers serve the same key service as server A, sync the updated secrets.json to them. See Sync Keys Across Multiple Servers.
Step 3: Route domain B's content to domain A's server
Choose one option.
Option A: Migrate with Google's key service migration (recommended)
Google moves the content's encryption layer from server B to server A. Users keep working throughout. Keep server B online until the migration finishes.
1. In the Google Admin console, assign domain A's key service to domain B's users, and select domain B's key service as the backup key service. [TBD — exact Admin console menu path and link to Google's documentation]
2. Start the migration for a small group of users first, and confirm that their existing files open.
3. Run the migration for the rest of domain B's users.
4. Confirm that the migration has finished before you retire server B.
Please Note:
With this option, server A may not need domain B's keys, because Google re-wraps the content with server A's key service. We still recommend that you keep a secure backup of domain B's keys.
Option B: Point domain B's key service hostname at domain A's server
1. Add domain B's hostname to server A's Transport Layer Security (TLS) certificate, then update server.cert and server.key.
2. Complete step 2 so that server A holds domain B's keys.
3. Update the Domain Name System (DNS) record for domain B's key service hostname to point to server A.
4. Confirm that domain B's existing files open before you retire server B.
Retire a CSE Server
Retire a server only after you confirm that all its content opens through another server.
1. Open a sample of files that the server encrypted, including the oldest content you can find.
2. Keep a secure, offline backup of the server's cse.env and secrets.json files.
3. Stop and remove the container:
docker stop <containerID>
docker rm <containerID>Do Not Destroy Key Backups
If you destroy a key that encrypted content, Google Workspace can't decrypt that content.
Roll Back a Change
If you haven't started the container with the new configuration
Restore your backups, then start the container:
cp /var/virtru/cse/cse.env.bak /var/virtru/cse/cse.env
cp /var/virtru/cse/run.sh.bak /var/virtru/cse/run.sh
cp /var/virtru/cse/secrets.json.bak /var/virtru/cse/secrets.json
sh run.shIf the server has already encrypted content with a new key
Don't restore an older secrets.json, because it doesn't contain the new key. Keep the current file and contact Virtru Support to change the active key.
Troubleshooting
| Symptom | What to check |
|---|---|
| Older files won't open after rotation. | The key that encrypted them is missing from secrets.json, or run.sh doesn't mount secrets.json. |
| New files open on one server but not another. | The servers have different secrets.json files. See Sync Keys Across Multiple Servers. |
active shows the wrong key. |
You may have run store-secret with --active by mistake. See Roll Back a Change. |
| The container won't start or exits right away. | Run docker logs <containerID>. Check the secrets.json path and permissions, and confirm that SECRET_KEYS_PATH is set in cse.env. |
| Domain B files still go to server B. | Adding keys doesn't change routing. Complete step 3 of Add Keys from Another CSE Server. |
Additional Resources
- [TBD — link text and article ID for the Virtru Private Keystore (for Google Workspace CSE) version upgrade article]
- Virtru Private Keystore for Google Workspace CSE release notes
- [TBD — link text and URL for Google's Help article on managing client-side encryption key services]