Private registries

Starting in the 1.3.0 release, K8ssandra supports the use of private registries. This feature is designed for those users who have limited or restricted access to public registries such as Docker Hub. Site Reliability Engineers and developers can use the examples in this topic (and linked related topics) to provide access to all the images required for their K8ssandra environment.

Introduction

All images used by K8ssandra are pulled from Docker Hub, a publicly accessible registry. There are situations where some users need the ability to pull images from a different registry. Examples:

• Consider “air gapped” deployments, where there is no access to the public Internet. This scenario means images cannot be pulled from Docker Hub or any other external registry. Images must be pulled from a registry on the user’s internal network.

• A third-party organization may provide its own registry for its users where images are built from source, and go through additional security audits. Again, relying on the default registry used by K8ssandra – Docker Hub – is insufficient.

Image coordinates

K8ssandra 1.3.0 adds the ability to fully specify coordinates for every image that it deploys. To better understand what we mean by image coordinates, let’s look closer.

Format of image coordinates

Consider this example:

docker.io/k8ssandra/cass-management-api:3.11.10-v0.1.26

• docker.io is the registry.

• k8ssandra is the repository.

• cass-management-api is the image.

• 3.11.10-v0.1.26 is the image tag.

Given this, the complete format of coordinates looks like this:

registry/repository/image:tag

K8ssandra allows you to configure each of these parts for each image that it deploys.

Chart properties YAML

For each container that K8ssandra deploys, there are chart properties like this:

<container-name>:
  image:
    registry:
    repository:
    tag:
    pullPolicy: 

Example YAML:

cassandra:
  serviceAccount: cassandra
  image:
    registry: myregistry
    repository: myrepo/cass-management-api
    tag: 3.11.10-v0.1.26
    pullPolicy: IfNotPresent

List of chart properties

Here is a list of chart properties that correspond to K8ssandra deployed containers having an image property:

Cassandra container defaults

By default, K8ssandra uses a version mapping to decide on the image to use for Cassandra. K8ssandra looks up the image coordinates based on the value of the cassandra.version property. (Starting in K8ssandra 1.3.0, the default cassandra.version is 4.0.0.)

If you choose to set the cassandra.image property, the version mapping won’t be used. You will be responsible for ensuring that a valid image is used.

Image Pull Secrets and Service Accounts

Private registries require authentication to pull images. Pods need credentials in order to pull images. This is accomplished with service accounts and image pull secrets. Every pod has a service account, and image pull secrets can be assigned to a service account.

For related details, see these topics in the Kubernetes documentation:

• Pull an Image from a Private Registry.

• Add ImagePullSecrets to a service account for details on configuring and creating a service account with image pull secrets.

Properties for configuring service accounts

The following table lists the relevant properties for configuring service accounts:

Configure Image Pull Secrets

There are some differences in the way K8ssandra handles the configuration of service accounts, depending on the component. Let’s look at each component to see how to configure image pull secrets.

Cassandra, Stargate, Reaper

For each of these components – Cassandra, Stargate, Reaper – K8ssandra does not create or configure the service account. For details on configuring and creating a service account with image pull secrets for these components, see Configure Service Accounts for Pods in the Kubernetes documentation.

Prometheus Operator

Configuration for its service account:

global:
  imagePullSecrets:
    - myregistrykey
kube-prometheus-stack:
  prometheusOperator:
      serviceAccount:
     name: prometheus-operator

Prometheus

The configuration for the Prometheus service account is the same as for Prometheus Operator, but each requires its own service account:

global:
  imagePullSecrets:
    - myregistrykey
kube-prometheus-stack:
  prometheus:
    serviceAccount:
      name: prometheus-operator

Reaper Operator

Configuration for its service account:

reaper-operator:
  imagePullSecrets:
  - myregistrykey
  serviceAccount:
    name: reaper-operator

Medusa Operator

Configuration for its service account:

medusa-operator:
  imagePullSecrets:
  - myregistrykey
  serviceAccount:
    name: medusa-operator

Cass Operator

Configuration for its service account:

cass-operator:
  imagePullSecrets:
  - myregistrykey
  serviceAccount:
    name: cass-operator

cleaner

Configuration for its service account:

cleaner:
  serviceAccount: cleaner

CRD Updater

Configuration for its service account:

client:
  serviceAccount: cleaner

Image property examples

Let’s look at two examples that demonstrates how to work with the image properties.

Custom stargate example with the default registry

Suppose we are working on a fork of Stargate and we want to deploy it in K8ssandra. We also want to use a custom image for the waitForCassandra init container. We could create and apply an overrides values YAML file with properties like this:

stargate:
  image:    
    repository: example-user/stargate
    tag: 1.0.29-dev
  waitForCassandra:
    image:
      repository: example-user/stargate-init
      tag: 0.1.0

Notice that the property names (stargate and waitForCassandra) in the example correspond directly to properties in the list of chart properties table above.

Because the registry property was not specified, the default (docker.io) will be used.

Complete example with a private registry

Here is a complete example of an overrides file that is configured to use a private registry. See the comments embedded in the YAML.

# We have to specify global image pull secrets for kube-prometheus-stack
global:
  imagePullSecrets:
    - myregistrykey

cassandra:
  # This service account needs to be configured and created prior to installing
  # the chart.
  serviceAccount: cassandra
  image:
    registry: myregistry
    repository: myrepo/cass-management-api
    # We have to specify the tag since we are not using the version/image mapping
    tag: 3.11.10-v0.1.26

  configBuilder:
    image:
      registry: myregistry
      repository: myrepo/cass-config-builder

  jmxCredentialsConfig:
    image:
      registry: myregistry
      repository: myrepo/busybox

  loggingSidecar:
    image:
      registry: myregistry
      repository: myrepo/system-logger

stargate:
  # This service account needs to be configured and created prior to installing
  # the chart.
  serviceAccount: stargate
  image:
    registry: myregistry
    repository: myrepo/stargate-3_11
    # We have to specify the tag since we are not using the image mapping
    tag: 1.0.29

  waitForCassandra:
    image:
      registry: myregistry
      repository: myrepo/alpine

reaper:
  # This service account needs to be configured and created prior to installing
  # the chart.
  serviceAccount: reaper
  image:
    registry: myregistry
    repository: myrepo/cassandra-reaper

medusa:
  # We don't configure the service account here because Medusa
  # runs in the Cassandra pod.
  enabled: true
  image:
    registry: myregistry
    repository: myrepo/medusa

cleaner:
  serviceAccount: cleaner
  image:
    registry: myregistry
    repository: myrepo/k8ssandra-tools

client:
  serviceAccount: client
  image:
    registry: myregistry
    repository: myrepo/k8ssandra-tools

cass-operator:
  imagePullSecrets:
  - myregistrykey
  serviceAccount:
    name: cass-operator
  image:
    registry: myregistry
    repository: myrepo/cass-operator

reaper-operator:
  imagePullSecrets:
  - myregistrykey
  serviceAccount:
    name: reaper-operator
  image:
    registry: myregistry
    repository: myrepo/reaper-operator

medusa-operator:
  imagePullSecrets:
  - myregistrykey
  serviceAccount:
    name: medusa-operator
  image:
    registry: myregistry
    repository: myrepo/medusa-operator

kube-prometheus-stack:
  prometheusOperator:
    serviceAccount:
      name: prometheus-operator
    image:
      repository: myregistry/myrepo/prometheus-operator
  prometheus:
    serviceAccount:
      name: prometheus-operator
    image:
      repository: myregistry/myrepo/prometheus

Related issues and PR

From the K8ssandra GitHub repo, here are the related issues and pull request for your reference:

• https://github.com/k8ssandra/k8ssandra/pull/901
• https://github.com/k8ssandra/k8ssandra/issues/420
• https://github.com/k8ssandra/k8ssandra/issues/839
• https://github.com/k8ssandra/k8ssandra/issues/840

Next steps

Explore other K8ssandra tasks.

See the Helm Chart reference topics.