Virtru Private Keystore (VPK) Log & Error Reference
Table of Contents
- Viewing Container Logs
- Log File Paths
- Downloading & Compressing Logs
- Log Rotation Configuration
- Exporting Logs to a SIEM
- Filtering for Errors
- Enable Debug Logging
- HTTP API Error Codes
- ACM → VPK Errors
- Key Provider Errors
- Rewrap Errors
- HSM Errors
- Authentication Errors
- Startup/Boot Errors
- TLS/Certificate Errors
- Getting Help
Viewing Container Logs
VPK outputs logs to stdout in JSON format. Both Docker and Podman provide commands to view these logs, displaying the stdout (standard output) and stderr (standard error) streams from the running container.
Basic Commands
| Action | Docker | Podman | Kubernetes |
|---|---|---|---|
| View logs | docker logs <container_id> |
podman logs <container_id> |
kubectl logs <pod> -n <namespace> |
| Last 100 lines | docker logs --tail 100 <container_id> |
podman logs --tail 100 <container_id> |
kubectl logs --tail=100 <pod> -n <namespace> |
| Follow in real-time | docker logs -f <container_id> |
podman logs -f <container_id> |
kubectl logs -f <pod> -n <namespace> |
| With timestamps | docker logs --timestamps <container_id> |
podman logs --timestamps <container_id> |
(timestamps included by default) |
| Last 1 hour | docker logs --since=1h <container_id> |
podman logs --since=1h <container_id> |
kubectl logs --since=1h <pod> -n <namespace> |
| Previous instance | (use log file path) | (use log file path) | kubectl logs --previous <pod> -n <namespace> |
Log File Paths
Docker
Docker stores logs in a specific path on the host machine:
/var/lib/docker/containers/<containerID>/<containerID>-json.log
Each container has its own directory, and log files are saved as JSON.
Podman
Podman allows you to configure a custom log path using the --log-opt parameter:
podman run \ --log-opt path=/var/log/Virtru_CKS.log \ ...
This allows you to specify a custom log location, which is useful for centralized log storage and management.
Kubernetes
Logs are managed by the container runtime and accessible via kubectl logs. For persistent storage, configure a logging sidecar or external log aggregation.
Downloading & Compressing Logs
Export Logs to a File
Docker:
docker logs <container_id> > cks-logs.txt 2>&1
Podman:
podman logs <container_id> > cks-logs.txt 2>&1
Kubernetes:
kubectl logs <pod> -n <namespace> > cks-logs.txt
Transfer Logs from Remote Host
Use scp or other file transfer tools to download log files:
scp user@remote_host:/path/to/cks-logs.txt /local/path/
Compress Logs
Logs can grow quite large. Compressing them makes it easier to store and transfer:
Docker:
docker logs <container_id> 2>&1 | gzip > cks-logs.gz
Podman:
podman logs <container_id> 2>&1 | gzip > cks-logs.gz
Extract compressed logs:
gunzip cks-logs.gz
Log Rotation Configuration
Log rotation prevents container logs from consuming too much disk space.
Docker Log Rotation
Configure log rotation as part of the container setup:
docker run \ --name Virtru_CKS \ --interactive --tty --detach \ --env-file /var/virtru/cks/env/cks.env \ -v /var/virtru/cks/keys/:/app/keys \ -v /var/virtru/cks/ssl/:/app/ssl \ -p 443:9000 \ --log-opt max-size=100m \ --log-opt max-file=10 \ --restart unless-stopped \ containers.virtru.com/cks:<version>
| Option | Description |
|---|---|
--log-opt max-size=100m |
Limits each log file to 100MB |
--log-opt max-file=10 |
Keeps a maximum of 10 log files, discarding oldest |
Podman Log Rotation
Configure log rotation with custom log path:
podman run \ --name Virtru_CKS \ --interactive --tty --detach \ --env-file /var/virtru/cks/env/cks.env \ -v /var/virtru/cks/keys/:/app/keys \ -v /var/virtru/cks/ssl/:/app/ssl \ -p 443:9000 \ --restart unless-stopped \ --log-opt path=/var/log/Virtru_CKS.log \ containers.virtru.com/cks:<version>
For more information on Docker logging drivers, see: Docker Logging Configuration
Exporting Logs to a SIEM
Logs can be forwarded to a remote syslog server for centralized monitoring and analysis.
Docker Syslog Driver
Configure the container to send logs directly to a remote syslog server:
docker run \ --name Virtru_CKS \ --interactive --tty --detach \ --env-file /var/virtru/cks/env/cks.env \ -v /var/virtru/cks/keys/:/app/keys \ -v /var/virtru/cks/ssl/:/app/ssl \ -p 443:9000 \ --log-driver syslog \ --log-opt syslog-address=tcp://192.168.10.15:10514 \ --restart unless-stopped \ containers.virtru.com/cks:<version>
| Option | Description |
|---|---|
--log-driver syslog |
Use syslog logging driver |
--log-opt syslog-address=tcp://192.168.10.15:10514 |
Remote syslog server address and port |
Host-Level Syslog Forwarding
Alternatively, forward all host syslog entries (including container logs) to a remote server.
Note: This will include additional host events beyond container logs.
Edit /etc/rsyslog.d/50-default.conf:
nano /etc/rsyslog.d/50-default.conf
Add at the top:
*.* action(type="omfwd" target="192.168.10.15" port="10514" protocol="tcp" action.resumeRetryCount="100" queue.type="linkedList" queue.size="10000")
| Parameter | Description |
|---|---|
target |
Remote syslog server IP |
port |
Remote syslog server port |
protocol |
Transport protocol (tcp/udp) |
action.resumeRetryCount |
Number of retry attempts |
queue.size |
Number of entries to queue before discarding |
Important: Due to the nature of TCP, if the remote syslog server is unavailable, entries will be blocked and discarded if a retry queue is not configured.
Restart rsyslog after making changes:
sudo systemctl restart rsyslog
Filtering for Errors
Use these patterns to filter logs for specific error types.
Useful Grep Patterns
| Pattern | What It Finds |
|---|---|
grep -i "error" |
All errors |
grep -i "fatal" |
Fatal startup errors |
grep "NoKeyError" |
Key not found errors |
grep "RewrapError" |
Rewrap operation failures |
grep "HsmAgentError" |
HSM communication errors |
grep "401\|Unauthorized" |
Authentication failures |
grep "rewrap" |
Rewrap operations |
grep -i "certificate\|ssl\|tls" |
TLS/certificate errors |
Example: Filter Errors from Docker
docker logs Virtru_CKS 2>&1 | grep -iE "(error|fatal|failed)"
Example: Parse JSON Logs with jq
docker logs Virtru_CKS 2>&1 | jq 'select(.level == "ERROR")'
Enable Debug Logging
For more verbose output when troubleshooting, set these environment variables:
| Variable | Value | Description |
|---|---|---|
LOG_LEVEL |
DEBUG |
Detailed application logs |
LOG_CONSOLE_ENABLED |
true |
Enable console output |
Available levels: ALL, TRACE, DEBUG, INFO (default), WARN, ERROR, FATAL, OFF
Note: Debug logging increases log volume significantly. Disable after troubleshooting is complete.
HTTP API Error Codes
API Endpoints
| Endpoint | Method | Success | Error Codes |
|---|---|---|---|
/status |
GET | 200 | 500 |
/healthz |
GET | 200 | 500 |
/public-keys |
GET | 200 | 401, 500 |
/rewrap |
POST | 200 | 400, 401, 409, 500 |
/bulk-rewrap |
POST | 200 | 400, 401, 409, 500 |
/key-pairs |
PUT | 201 | 400, 401, 500 |
/key-pairs/:fingerprint |
DELETE | 200 | 400, 401, 500 |
HTTP Status Codes
| Status | Meaning | Common Cause |
|---|---|---|
200 |
Success | Request completed successfully |
201 |
Created | Key pair created successfully |
400 |
Bad Request | Invalid request body, malformed JSON, invalid public key |
401 |
Unauthorized | Invalid or missing authentication token |
409 |
Conflict | Key not found or key pair mismatch |
500 |
Internal Server Error | Unexpected server error |
ACM → VPK Errors
These errors originate from Virtru's Access Control Manager (ACM) when communicating with VPK. You may see these as API responses or in application logs.
| Status | Error | Meaning | Resolution |
|---|---|---|---|
412 |
PreconditionsUnmetError |
User public key required but missing | User must generate key pair via Secure Reader |
422 |
CKSRequestError |
VPK request failed | Check VPK connectivity, authentication, logs |
400/500 |
NoCustomerKeyServer |
Organization doesn't have VPK configured | Verify VPK URL registered with Virtru |
Key Type Mismatch Errors (412)
ACM validates that user public keys match your organization's VPK key type:
| User Key | Org Key | Result |
|---|---|---|
| RSA | RSA | ✅ Success |
| ECC | ECC (same or weaker curve) | ✅ Success |
| RSA | ECC | ❌ 412 Error |
| ECC | RSA | ❌ 412 Error |
| ECC (weaker curve) | ECC (stronger curve) | ❌ 412 Error |
ECC Curve Weights
| Curve | Weight | User key must be ≥ this to decrypt |
|---|---|---|
| P-256 (secp256r1) | 1 | P-256, P-384, or P-521 |
| P-384 (secp384r1) | 2 | P-384 or P-521 |
| P-521 (secp521r1) | 3 | P-521 only |
Key Provider Errors
NoKeyError (HTTP 409)
| Error Message | Meaning | Resolution |
|---|---|---|
No suitable public key found |
No public keys loaded | Check key file paths, reimport keys |
No suitable private key found |
No private keys loaded | Check key file paths, reimport keys |
Unable to locate the public key with '<fingerprint>' fingerprint |
Specific public key not found | Verify fingerprint, reload keys |
Unable to locate the private key with '<fingerprint>' fingerprint |
Specific private key not found | Verify fingerprint, reload keys |
No default keys were defined |
No keys available for validation | Import or generate key pairs |
UnpairedPublicKeyError (HTTP 409)
| Error Message | Meaning | Resolution |
|---|---|---|
One or more public keys has no private key related |
Public key exists without matching private key | Ensure both keys are present |
KeyError (HTTP 400)
| Error Message | Meaning | Resolution |
|---|---|---|
The keys with '<fingerprint>' fingerprint is not used by CKS currently |
Attempting to delete non-existent key | Verify fingerprint |
The keys with '<fingerprint>' fingerprint is only public key used and cannot be removed |
Cannot delete last remaining key pair | Add another key pair before deleting |
Supported Key Types
| Type | File Naming | Key Provider Path |
|---|---|---|
| RSA (2048-bit, 4096-bit) |
rsa_001.pem, rsa_001.pub
|
/app/keys (Docker) or /run/secrets (K8s) |
| ECC (P-256, P-384, P-521) |
ecc_p256_001.pem, ecc_p256_001.pub
|
/app/keys (Docker) or /run/secrets (K8s) |
Rewrap Errors
RewrapError Types (HTTP 400)
| Error Name | Meaning | Resolution |
|---|---|---|
RewrapErrorInvalidPublicKey |
Public key is malformed or invalid | Verify public key format (PEM) |
RewrapErrorInvalidWrappedPayload |
Encrypted payload cannot be decrypted | Verify payload is properly encrypted |
OpenSSL Error Mappings
| OpenSSL Error Code | VPK Error | Meaning |
|---|---|---|
ERR_OSSL_ASN1_TOO_LONG |
RewrapErrorInvalidPublicKey |
Invalid public key format |
ERR_OSSL_UNSUPPORTED |
RewrapErrorInvalidPublicKey |
Unsupported key type |
ERR_OSSL_PEM_NO_START_LINE |
RewrapErrorInvalidPublicKey |
Invalid PEM format |
ERR_OSSL_EVP_OPERATION_NOT_SUPPORTED_FOR_THIS_KEYTYPE |
RewrapErrorInvalidPublicKey |
Key type mismatch |
ERR_OSSL_RSA_OAEP_DECODING_ERROR |
RewrapErrorInvalidWrappedPayload |
Decryption failed |
Common Rewrap Issues
| Log Pattern | Meaning | Resolution |
|---|---|---|
Unsupported algorithm type. Please use RSA or ECC. |
Unsupported key algorithm | Use RSA or ECC keys only |
Unsupported public key curve: <curve> |
ECC curve not supported | Use secp256r1, secp384r1, or secp521r1 |
Mismatched publicKey payload and type |
Key type doesn't match payload | Verify key type matches actual key |
HSM Errors
PKCS#11 Errors
| Log Pattern | Meaning | Resolution |
|---|---|---|
Failed to load PKCS#11 library |
Library not found | Verify PKCS11_LIB_PATH
|
Slot not found |
HSM slot unavailable | Verify PKCS11_SLOT_LBL
|
PIN verification failed |
Invalid HSM PIN | Check PKCS11_PIN (format: user:password) |
Key not found on HSM |
Key label mismatch | Verify PKCS11_KEY_LBL
|
HsmAgentError |
General HSM communication failure | Check HSM connectivity, credentials |
HsmAgentDecryptError |
HSM failed to decrypt payload | Check HSM status, key availability |
Validate HSM Setup
Use list-keys to verify HSM connectivity before starting VPK:
docker run \ -e HSM_IP="<hsm_ip>" \ -e PKCS11_VENDOR="custom" \ -e PKCS11_LIB_NAME="CloudHSM" \ -e PKCS11_LIB_PATH="/opt/cloudhsm/lib/libcloudhsm_pkcs11.so" \ -e PKCS11_SLOT_LBL="<slot_label>" \ -e PKCS11_KEY_LBL="<key_label>" \ -e PKCS11_PIN="<user>:<password>" \ -e KEY_PROVIDER_TYPE="hsm" \ -e CRYPTO_OPERATIONS_TYPE="hsm" \ --mount type=bind,source=/path/to/customerCA.crt,target=/opt/cloudhsm/etc/customerCA.crt \ containers.virtru.com/cks:<version> list-keys
Authentication Errors
Authentication Modes
| Mode | Variable | Use Case |
|---|---|---|
| JWT | JWT_AUTH_ENABLED=true |
Virtru ACM → VPK (default) |
| HMAC | HMAC_AUTH_ENABLED=true |
Google CSE → VPK |
Note: At least one authentication method must be enabled.
JWT Errors
| Error Pattern | Meaning | Resolution |
|---|---|---|
401 Unauthorized |
JWT validation failed | Check token expiration, issuer, audience |
Invalid JWT signature |
Token signature doesn't match | Verify JWKS endpoint accessible |
Token expired |
JWT has expired | Request new token |
Invalid audience |
Token audience mismatch | Check JWT_AUTH_AUDIENCE matches org ID |
Invalid issuer |
Token issuer mismatch | Check JWT_AUTH_ISSUER
|
HMAC Errors
| Error Pattern | Meaning | Resolution |
|---|---|---|
401 Unauthorized |
HMAC validation failed | Verify API token and secret. Also compare the request date header to the VPK server timestamp to rule out clock drift before rotating the HMAC token. |
Invalid HMAC signature |
Signature mismatch | Check token secret, request signing, and system time on the host generating the HMAC-signed request. |
Token not found |
API token doesn't exist | Verify token in storage |
InvalidAuthenticationError or repeated 401 Unauthorized
|
Clock drift may be causing the request timestamp to fall outside the allowed HMAC time window | Compare the request date header to the VPK server timestamp. If the times differ by more than a few minutes, restore time synchronization on the system generating the HMAC-signed requests before rotating the HMAC token. |
Tip: Before rotating an HMAC token, compare the request date header to the VPK server timestamp. If the system generating the HMAC-signed request has clock drift, restoring time synchronization may resolve the issue more quickly than rotating the token.
Configuration Errors
| Error Message | Meaning | Resolution |
|---|---|---|
No Authentication Middleware. JWT or HMAC Auth must be enabled to start CKS. |
Neither auth method enabled | Enable JWT or HMAC auth |
Invalid AuthTokenStorage Type <type> |
Invalid token storage config | Use file or in-memory
|
Startup/Boot Errors
Boot Check Failures
| Log Pattern | Meaning | Resolution |
|---|---|---|
Unable to initialize the service |
Boot check failed | Check full error, verify configuration |
There's no public key found |
No keys at startup | See BOOT_CHECKER_NOKEYS_RULE options |
Failed to load the Private Key |
HTTPS key not found | Verify HTTPS_KEY_PATH
|
Failed to load the Certificate |
HTTPS cert not found | Verify HTTPS_CERT_PATH
|
No-Keys Rule Options
| Value | Behavior |
|---|---|
fail |
VPK fails to start if no keys |
generate |
Generate new key pair automatically |
importPEM |
Import from PEM files (default) |
Common Setup Issues
| Issue | Symptom | Resolution |
|---|---|---|
| Invalid Org ID | Setup script rejects org ID | Org ID must be 36 characters (UUID format) |
| Volume mount fails | Container can't find keys | Verify source paths exist on host |
| Missing HSM CA cert | HSM connection fails | Place customerCA.crt in hsm-config/
|
| Wrong PIN format | HSM auth fails | Use format <CU_user_name>:<password>
|
TLS/Certificate Errors
| Error Message | Meaning | Resolution |
|---|---|---|
Failed to load the Private Key |
HTTPS key not found or invalid | Verify HTTPS_KEY_PATH and permissions |
Failed to load the Certificate |
HTTPS cert not found or invalid | Verify HTTPS_CERT_PATH and permissions |
Failed to load the Certificate Authority |
CA file not found or invalid | Verify HTTPS_CA_PATH and permissions |
ENOENT: no such file or directory |
File path doesn't exist | Check mounted volumes and paths |
EACCES: permission denied |
File permission issue | Fix permissions (readable by container user) |
Getting Help
If you're unable to resolve an issue using this guide, please contact Virtru Support:
Virtru Support Portal: support.virtru.com
When contacting support, please include:
- VPK version
- Deployment type (Docker, Podman, Kubernetes, HSM)
- Relevant log output (filtered for errors)
- Steps to reproduce the issue
- Any recent configuration changes