Kafka Authentication with SCRAM

SCRAM stands for Salted Challenge Response Authentication Mechanism. In the diagram below, it illustrates a simple flow of SCRAM.

The server would sends a random text (Star shape) over to the client, and the client would use this together with the password (Diamond shape) to compute the answer (Square shape). At the same time, Server would also use the same password and same random text to compute the answer, which the Server will use it to validate against the answer sends by the Client. If the answer is the same, the client will be authenticated.

SCRAM

We all know that this is a super diluted diagram for the sake of simplicity and explanation. If you are really interested on the specifics and details, head over to have a go at the good twenty eight pages long RFC 5802 document.

SCRAM authentication is in fact being used by many software, like Kafka, MongoDB, PostgreSQL database and etc. And it is also being used in some protocols like IMAP and SMTP. In SCRAM, the password will never be sent on the transaction itself, and the password can be kept as a one directional hash. Besides that, SCRAM can be used in conjunction with TLS to prevent ones from the man-in-the-middle attacks.

Next, we will look into Kafka authentication with SCRAM-SHA-512. SCRAM is one of the authentication methods supported by Kafka. While you can use Kafka cluster without having any authentication, but if you were to expose Kafka to the external network, or in a large corporate internal network, it only make sense if have one. If you are new to Kafka and would like to know how to setup a cluster, I have a blogpost that will be able to guide you to do so.

On Strimzi Operator, we can configure the 1 of the Kafka cluster listener to use SCRAM-SHA-512 as the authentication method. And in order to use this effectively we will need to put in a authorization type, which we will use simple ACL for now. Also just to mention that since I am using OpenShift, the listener would be of type Route, and TLS is mandatory.

Secondly, we will need to create a Kafka User, using SCRAM-SHA-512 as its authentication method. On authorization side, we may grant all access to “testtopic” for testing purposes.

Now, we will need to prepare a properties file for the testing later. This file essentially contains the credential information about the truststore and the SCRAM. Assuming the kafka cluster name is “demo”

1. Download .p12 truststore file from demo-cluster-ca-cert secret
2. Copy truststore password from demo-cluster-ca-cert secret
3. Copy JAAS config string from “my-user” secret

ssl.truststore.location=/path/to/ca.p12 
ssl.truststore.type=PKCS12 
ssl.truststore.password=<truststore_password>
security.protocol=SASL_SSL 
sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="my-user" password="<password>";  
# consumer only properties 
group.id=my-group

Next, to facilitate the testing, we can use the ready made tools by Kafka. Go to this Apache Kafka site to download the tar ball.

tar zxvf kafka_<version>.tgz
cd kafka_<version>/bin

Now, start a Producer

# PRODUCER
bin/kafka-console-producer.sh --topic testtopic 
--bootstrap-server demo-kafka-bootstrap-kafka.apps.your.cluster.domain:443 --producer.config /path/to/anyname.properties

and a Consumer

# CONSUMER
bin/kafka-console-consumer.sh --topic testtopic 
--bootstrap-server demo-kafka-bootstrap-kafka.apps.your.cluster.domain:443 --producer.config /path/to/anyname.properties

Now, we have successfully authenticated both Producer and Consumer clients against Kafka brokers using SCRAM-SHA-512. Here we can go ahead and enjoy talking to ourselves using the clients.

Hope it helps your journey in learning Kafka.