Building a Customized Kafka Connect Image

But why?

Is building a customized Kafka Connect image really necessary?

If you are familiar with Kafka Connect (if you’re not, I highly recommend reading my previous post on Kafka Connect), you would know that we just need to include the plugins that we would like to use in the custom resource, and the Kafka Connect cluster would download these plugins and build an image out of it — and spin up the worker pods using this image. This would work perfectly if you are using publicly available connector plugins, for example those that are hosted on the Camel Maven repository.

But what if you have built a customized connector plugin and you want to host it in a private repository? A private repository would required one to put in the credentials in order to gain access to it. As of Strimzi v0.29, there is no support for any form of authentication for artifacts repository.

Hence, building a customized Kafka Connect image would come in handy.

While we cannot authenticate against private artifact repository, we do have an option to put in the authentication for private image registry — a way to use our own built plugin securely in Kafka Connect.

Referring to this snippet, we can create a Kubernetes Secret — my-secret, which contains the credentials for accessing the image registry, and use it in the Kafka Connect as a imagePullSecrets. As its name suggests, the Kafka Connect cluster will use imagePullSecrets to authenticate itself against the image registry and pull the image.

The entire .spect.buildYAML block should be removed if you plan to use customized image in .spec.image. Otherwise Kafka Connect cluster will still attempt to build the image

Let’s build it!

Following the documentation from Strimzi, we will build the customized Docker image based on the original Kafka Connect image, and all connector plugins must be added into /opt/kafka/plugins.

With this Dockerfile, we will build and push the Docker image to a private image registry (e.g. Docker Hub, GitHub Package and etc.)

docker build . — file Dockerfile — tag ghcr.io/username/repo:latest
docker push ghcr.io/username/repo:latest

Once that is done, as shown in the Kafka Connect snippet earlier, we can specify the customized image in spec.image , and the Kafka Connect cluster will spin up the worker pod using this image, together with the plugins that we have added earlier to the image. In the status section of the Kafka Connect custom resource, we shall see that our plugins are loaded and ready to be used as a Kafka Connector.