TLS Authentication#

Ray can be configured to use TLS on its gRPC channels. This means that connecting to the Ray head will require an appropriate set of credentials and also that data exchanged between various processes (client, head, workers) will be encrypted (Ray’s document).

This document provides detailed instructions for generating a public-private key pair and CA certificate for configuring KubeRay.

Warning: Enabling TLS will cause a performance hit due to the extra overhead of mutual authentication and encryption. Testing has shown that this overhead is large for small workloads and becomes relatively smaller for large workloads. The exact overhead will depend on the nature of your workload.

Prerequisites#

To fully understand this document, it’s highly recommended that you have a solid understanding of the following concepts:

  • private/public key

  • CA (certificate authority)

  • CSR (certificate signing request)

  • self-signed certificate

This YouTube video is a good start.

TL;DR#

Please note that this document is designed to support KubeRay version 0.5.0 or later. If you are using an older version of KubeRay, some of the instructions or configurations may not apply or may require additional modifications.

Warning: Please note that the ray-cluster.tls.yaml file is intended for demo purposes only. It is crucial that you do not store your CA private key in a Kubernetes Secret in your production environment.

# Install v1.0.0 KubeRay operator
# `ray-cluster.tls.yaml` will cover from Step 1 to Step 3

# Download `ray-cluster.tls.yaml`
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/v1.0.0/ray-operator/config/samples/ray-cluster.tls.yaml

# Create a RayCluster
kubectl apply -f ray-cluster.tls.yaml

# Jump to Step 4 "Verify TLS authentication" to verify the connection.

ray-cluster.tls.yaml will create:

  • A Kubernetes Secret containing the CA’s private key (ca.key) and self-signed certificate (ca.crt) (Step 1)

  • A Kubernetes ConfigMap containing the scripts gencert_head.sh and gencert_worker.sh, which allow Ray Pods to generate private keys (tls.key) and self-signed certificates (tls.crt) (Step 2)

  • A RayCluster with proper TLS environment variables configurations (Step 3)

The certificate (tls.crt) for a Ray Pod is encrypted using the CA’s private key (ca.key). Additionally, all Ray Pods have the CA’s public key included in ca.crt, which allows them to decrypt certificates from other Ray Pods.

Step 1: Generate a private key and self-signed certificate for CA#

In this document, a self-signed certificate is used, but users also have the option to choose a publicly trusted certificate authority (CA) for their TLS authentication.

# Step 1-1: Generate a self-signed certificate and a new private key file for CA.
openssl req -x509 \
            -sha256 -days 3650 \
            -nodes \
            -newkey rsa:2048 \
            -subj "/CN=*.kuberay.com/C=US/L=San Francisco" \
            -keyout ca.key -out ca.crt

# Step 1-2: Check the CA's public key from the self-signed certificate.
openssl x509 -in ca.crt -noout -text

# Step 1-3
# Method 1: Use `cat $FILENAME | base64` to encode `ca.key` and `ca.crt`.
#           Then, paste the encoding strings to the Kubernetes Secret in `ray-cluster.tls.yaml`.

# Method 2: Use kubectl to encode the certifcate as Kubernetes Secret automatically.
#           (Note: You should comment out the Kubernetes Secret in `ray-cluster.tls.yaml`.)
kubectl create secret generic ca-tls --from-file=ca.key --from-file=ca.crt

  • ca.key: CA’s private key

  • ca.crt: CA’s self-signed certificate

This step is optional because the ca.key and ca.crt files have already been included in the Kubernetes Secret specified in ray-cluster.tls.yaml.

Step 2: Create separate private key and self-signed certificate for Ray Pods#

In ray-cluster.tls.yaml, each Ray Pod (both head and workers) generates its own private key file (tls.key) and self-signed certificate file (tls.crt) in its init container. We generate separate files for each Pod because worker Pods do not have deterministic DNS names, and we cannot use the same certificate across different Pods.

In the YAML file, you’ll find a ConfigMap named tls that contains two shell scripts: gencert_head.sh and gencert_worker.sh. These scripts are used to generate the private key and self-signed certificate files (tls.key and tls.crt) for the Ray head and worker Pods. An alternative approach for users is to prebake the shell scripts directly into the docker image that’s utilized by the init containers, rather than relying on a ConfigMap.

Please find below a brief explanation of what happens in each of these scripts:

  1. A 2048-bit RSA private key is generated and saved as /etc/ray/tls/tls.key.

  2. A Certificate Signing Request (CSR) is generated using the private key file (tls.key) and the csr.conf configuration file.

  3. A self-signed certificate (tls.crt) is generated using the private key of the Certificate Authority (ca.key) and the previously generated CSR.

The only difference between gencert_head.sh and gencert_worker.sh is the [ alt_names ] section in csr.conf and cert.conf. The worker Pods use the fully qualified domain name (FQDN) of the head Kubernetes Service to establish a connection with the head Pod. Therefore, the [alt_names] section for the head Pod needs to include the FQDN of the head Kubernetes Service. By the way, the head Pod uses $POD_IP to communicate with worker Pods.

# gencert_head.sh
[alt_names]
DNS.1 = localhost
DNS.2 = $FQ_RAY_IP
IP.1 = 127.0.0.1
IP.2 = $POD_IP

# gencert_worker.sh
[alt_names]
DNS.1 = localhost
IP.1 = 127.0.0.1
IP.2 = $POD_IP

In Kubernetes networking model, the IP that a Pod sees itself as is the same IP that others see it as. That’s why Ray Pods can self-register for the certificates.

Step 3: Configure environment variables for Ray TLS authentication#

To enable TLS authentication in your Ray cluster, set the following environment variables:

  • RAY_USE_TLS: Either 1 or 0 to use/not-use TLS. If this is set to 1 then all of the environment variables below must be set. Default: 0.

  • RAY_TLS_SERVER_CERT: Location of a certificate file which is presented to other endpoints so as to achieve mutual authentication (i.e. tls.crt).

  • RAY_TLS_SERVER_KEY: Location of a private key file which is the cryptographic means to prove to other endpoints that you are the authorized user of a given certificate (i.e. tls.key).

  • RAY_TLS_CA_CERT: Location of a CA certificate file which allows TLS to decide whether an endpoint’s certificate has been signed by the correct authority (i.e. ca.crt).

For more information on how to configure Ray with TLS authentication, please refer to Ray’s document.

Step 4: Verify TLS authentication#

# Log in to the worker Pod
kubectl exec -it ${WORKER_POD} -- bash

# Since the head Pod has the certificate of $FQ_RAY_IP, the connection to the worker Pods
# will be established successfully, and the exit code of the ray health-check command
# should be 0.
ray health-check --address $FQ_RAY_IP:6379
echo $? # 0

# Since the head Pod has the certificate of $RAY_IP, the connection will fail and an error
# message similar to the following will be displayed: "Peer name raycluster-tls-head-svc is
# not in peer certificate".
ray health-check --address $RAY_IP:6379

# If you add `DNS.3 = $RAY_IP` to the [alt_names] section in `gencert_head.sh`,
# the head Pod will generate the certificate of $RAY_IP.
#
# For KubeRay versions prior to 0.5.0, this step is necessary because Ray workers in earlier
# versions use $RAY_IP to connect with Ray head.