In this proof of concept (POC), we will explore the installation and management of a stateful Key Encryption Service (KES) within a Kubernetes (k8s) ecosystem. This guide facilitates cryptographic operations seamlessly, without exposing sensitive key material to consuming applications.
Implementing KES within Kubernetes in a stateful configuration ensures the persistence of encryption keys through pod lifecycle events and restarts. This setup offers an architectural resilience that is especially crucial in environments where relying on external Key Management Systems (KMS) is not an option or preferred.
When you have several million billion objects generated from AI/ML workloads such as LLMs, as several hundreds of applications access them simultaneously, you need to ensure the data encryption/decryption layer does not become the bottleneck and is as fast as possible since each object requires its own unique key. By storing these potentially several billions of keys in a stateful KES backend, even if the KMS goes offline for an extended period, keys can be saved to the KES until the KMS comes back online.
Prerequisites Before proceeding, ensure you have:
Kubernetes CLI (kubectl) installed and configured. Access to a Kubernetes cluster with appropriate permissions. MinIO Operator and CLI (mc) ready for deployment. Basic knowledge of Kubernetes and encryption concepts. Deploy MinIO Operator Setup Kubernetes Environment Delete Previous Cluster To avoid conflicts, remove any existing clusters:
kind delete clusters kind
Create New Cluster Set up a fresh Kubernetes cluster with the desired configuration using this kind-config.yaml
kind create cluster --config ~/operator/testing/kind-config.yaml
Deploy MinIO Operator Deploy the MinIO Operator using kubectl for managing MinIO instances in the cluster:
kubectl apply -k github.com/minio/operator/
Deploy MinIO Tenant Deploy the MinIO Tenant with a lightweight configuration suitable for development and testing purposes:
kubectl apply -k github.com/minio/operator/examples/kustomization/tenant-lite
Create Ubuntu Pod Deploy an Ubuntu pod to host the KES service within the MinIO tenant namespace:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: ubuntu
namespace: tenant-lite
labels:
app: ubuntu
spec:
containers:
- image: ubuntu
command:
- "sleep"
- "604800"
imagePullPolicy: IfNotPresent
name: ubuntu
restartPolicy: Always
EOF
Install KES and mc Execute these commands within the Ubuntu pod to install KES and MinIO Client (mc):
apt update
apt install wget
wget https://github.com/minio/kes/releases/latest/download/kes-linux-amd64
mv kes-linux-amd64 kes
chmod +x kes
mv kes /usr/local/bin/kes
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
mv mc /usr/local/bin/mc
Persistent Directory Creation Create a directory within the Ubuntu pod for KES configuration and data files where KES is located:
rm -rf ~/kes
mkdir ~/kes
cd ~/kes
touch init.yml
Identity Management Generate the necessary identities required for KES and MinIO authentication:
cd ~/kes
kes identity new --key sys-admin.key --cert sys-admin.crt kes-sys-admin
kes identity new --key minio-admin.key --cert minio-admin.crt minio-admin
kes identity new --key minio.key --cert minio.crt minio
kes identity new --ip "10.244.2.7" localhost # will generate private.key and public.crt
|
|____ IP Address of the Ubuntu Pod.
Expected 4 identities.
root@ubuntu:/# cd ~/kes
root@ubuntu:~/kes# ls
data init.yml minio-admin.crt minio-admin.key minio.crt minio.key private.key public.crt sys-admin.crt sys-admin.key
| | | | | |
| | |__________|___ minio |________________|___ kes-sys-admin
| |
|_______________|___ minio-admin
root@ubuntu:~/kes#
Create a KES unseal key.
cat /dev/urandom | head -c 32 | base64 # put the result in the .bashrc
vi ~/.bashrc
export KES_UNSEAL_KEY=<VALUE-FROM-ABOVE-COMMAND>
source ~/.bashrc
echo $KES_UNSEAL_KEY # it should print the value
Initialize KES Deployment
KES Configuration - Edit/Create KES config file Configure the KES service by setting up the necessary configuration parameters:
cd ~/kes
echo "version: v1" > ~/kes/init.yml
echo "address: 0.0.0.0:7373" >> ~/kes/init.yml
echo "" >> ~/kes/init.yml
echo "tls:" >> ~/kes/init.yml
echo " key: private.key" >> ~/kes/init.yml
echo " cert: public.crt" >> ~/kes/init.yml
echo " client:" >> ~/kes/init.yml
echo " verify_cert: false" >> ~/kes/init.yml
echo "" >> ~/kes/init.yml
echo "system:" >> ~/kes/init.yml
echo " admin:" >> ~/kes/init.yml
echo " identity: $(kes identity of sys-admin.crt)" >> ~/kes/init.yml
echo "" >> ~/kes/init.yml
echo "unseal:" >> ~/kes/init.yml
echo " environment:" >> ~/kes/init.yml
echo " name: KES_UNSEAL_KEY" >> ~/kes/init.yml
echo "" >> ~/kes/init.yml
echo "enclave:" >> ~/kes/init.yml
echo " default:" >> ~/kes/init.yml
echo " admin:" >> ~/kes/init.yml
echo " identity: $(kes identity of minio-admin.crt)" >> ~/kes/init.yml
echo " policy:" >> ~/kes/init.yml
echo " minio:" >> ~/kes/init.yml
echo " allow:" >> ~/kes/init.yml
echo " - /v1/api" >> ~/kes/init.yml
echo " - /v1/log/audit" >> ~/kes/init.yml
echo " - /v1/log/error" >> ~/kes/init.yml
echo " - /v1/key/create/*" >> ~/kes/init.yml
echo " - /v1/key/generate/*" >> ~/kes/init.yml
echo " - /v1/key/decrypt/*" >> ~/kes/init.yml
echo " - /v1/key/bulk/decrypt/*" >> ~/kes/init.yml
Initialization Initialize the KES server with the newly created configuration file:
cd ~/kes # where init.yml is saved
kes init --config init.yml ~/kes/data
Expected is
root@ubuntu:~/kes# cd ~/kes # where init.yml is saved
kes init --config init.yml ~/kes/data
TLS:
· Private Key: private.key
· Certificate: public.crt
System:
· Identity: 1a65f6f86c3268b30528fe4aab88fc6994730346e1c1863052fa3fa192d77d3e
Unseal:
· Environment: KES_UNSEAL_KEY
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Initialized KES v0.22.3 in /root/kes/data │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@ubuntu:~/kes#
Start KES Server Start the KES server and verify it's running correctly:
Expected is:
root@ubuntu:~/kes# kes server ~/kes/data
Copyright MinIO, Inc. https://min.io
License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html
Version v0.22.3 linux/amd64
Endpoints https://127.0.0.1:7373
https://10.244.4.2:7373
mTLS skip verify Client certificates are not verified
Mem Lock off Failed to lock RAM pages. Consider granting CAP_IPC_LOCK
Assign Policies Link MinIO and KES together by assigning the appropriate policies and identities.
In Ubuntu Pod Terminal where KES is located: Assign MinIO identity to MinIO policy.
cd ~/kes
export KES_SERVER=https://127.0.0.1:7373
export KES_CLIENT_KEY=minio-admin.key
export KES_CLIENT_CERT=minio-admin.crt
kes policy assign -k minio $(kes identity of minio.crt)
Expected:
root@ubuntu:/# cd ~/kes
export KES_SERVER=https://127.0.0.1:7373
export KES_CLIENT_KEY=minio-admin.key
export KES_CLIENT_CERT=minio-admin.crt
kes policy assign -k minio $(kes identity of minio.crt)
root@ubuntu:~/kes#
root@ubuntu:~/kes#
root@ubuntu:~/kes#
root@ubuntu:~/kes#
root@ubuntu:~/kes#
root@ubuntu:~/kes# printenv | grep KES_SERVER
KES_SERVER=https://127.0.0.1:7373
root@ubuntu:~/kes#
MinIO Server Setup Establish the connection between MinIO and KES, defining the encryption endpoint and credentials:
Create kes-minio secret Copy ~/kes/minio.key and ~/kes/minio.crt to your laptop from the ubuntu pod These two files below come from this line: kes identity new --key minio.key --cert minio.crt minio
# /Users/aj/minio/private.key is ~/kes/minio.key
# /Users/aj/minio/public.crt is ~/kes/minio.crt
kubectl create secret generic kes-minio -n tenant-lite --from-file=/Users/aj/minio/private.key --from-file=/Users/aj/minio/public.crt
Create kes-minio-public secret:Copy ~/kes/private.key and ~/kes/public.crt to your laptop from the ubuntu pod These two files below comes from this line: kes identity new --ip "10.244.2.7" localhost: # /Users/aj/minio/private.key is ~/kes/private.key
# /Users/aj/minio/public.crt is ~/kes/public.crt
kubectl create secret generic kes-minio-public -n tenant-lite --from-file=/Users/aj/minio/private.key --from-file=/Users/aj/minio/public.crt
k edit tenant -n tenant-lite
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
name: storage
namespace: minio-tenant
spec:
# externalClientCertSecrets is to share the secret with the MinIO Pods:
# Under: /tmp/certs/client-0 You will find:
# client.crt and client.key
# And we can use these files to setup KES in k8s
externalClientCertSecrets:
- name: kes-minio
type: Opaque
- name: kes-minio-public
type: Opaque
env:
# Set MINIO_KMS_KES_ENDPOINT
# It is the IP of the Ubuntu Pod.
- name: MINIO_KMS_KES_ENDPOINT
value: "https://<IP-ADDRESS-OF-UBUNTU-POD>:7373"
# Set MinIO Client Credentials, it comes from kes-minio secret
- name: MINIO_KMS_KES_CERT_FILE
value: "/tmp/certs/client-0/client.crt"
# Set MinIO Client Credentials, it comes from kes-minio secret
- name: MINIO_KMS_KES_KEY_FILE
value: "/tmp/certs/client-0/client.key"
# Set MinIO Default Key
- name: MINIO_KMS_KES_KEY_NAME
value: "minio-default-key"
# Trust the KES Server Certificate, it comes from kes-minio-public secret
- name: MINIO_KMS_KES_CAPATH
value: "/tmp/certs/client-1/client.crt"
# Root User
- name: MINIO_ROOT_USER
value: minio
# ROOT Password:
- name: MINIO_ROOT_PASSWORD
value: minio123
Encryption Operations Conduct encryption operations by creating an encrypted bucket and configuring server-side encryption:
mc alias set myminio https://minio.tenant-lite.svc.cluster.local:443 minio minio123
mc rb myminio/my-bucket --force # remove previous bucket to start fresh
mc mb myminio/my-bucket # create new bucket
mc admin kms key create myminio minio-my-bucket # create key
mc encrypt set sse-kms minio-my-bucket myminio/my-bucket # encrypt bucket
Verification and Testing of the Results Ensure that your setup is correct by verifying that keys can be created and accessed through MinIO:
root@ubuntu:/# mc alias set myminio https://minio.tenant-lite.svc.cluster.local:443 minio minio123
Added `myminio` successfully.
root@ubuntu:/# mc rb myminio/my-bucket --force # remove previous bucket to start fresh
mc: <ERROR> Unable to validate target `myminio/my-bucket`. Bucket `my-bucket` does not exist.
root@ubuntu:/# mc mb myminio/my-bucket # create new bucket
Bucket created successfully `myminio/my-bucket`.
root@ubuntu:/# mc admin kms key create myminio minio-my-bucket # create key
Created master key `minio-my-bucket` successfully
root@ubuntu:/# mc encrypt set sse-kms minio-my-bucket myminio/my-bucket # encrypt bucket
Auto encryption configuration has been set successfully for myminio/my-bucket
root@ubuntu:/#
Final Thoughts
After following the outlined steps, you have successfully deployed a stateful KES in your Kubernetes cluster. This configuration allows you to manage encryption keys within your cluster, simplifying your deployment and reducing dependencies on external KMS solutions. This ensures your AI/ML workloads perform smoothly without interruption that depend on this critical feature in the infrastructure.
The integration of stateful KES in Kubernetes with this hands-on experience reinforces your skills as a DevOps engineer as you continue to explore. If you have any questions on deploying stateful KES in your Kubernetes cluster be sure to reach out to us on Slack
Erasure Code Calculator
Reference Hardware
