日本語

How to use a Kafka broker with SSL/TLS authentication (client authentication)

Overview

This page describes how to connect from SINETStream to a Kafka broker that requires SSL/TLS two-way authentication.

The description will be made in the following order.

  1. Prerequisites
  2. Configurations on the Kafka broker (server side)
  3. Configurations on SINETStream (client side)
  4. Behavior on authentication errors

Prerequisites

Though the configuration and setting of a Kafka broker may vary, the following conditions are assumed for simplicity in this document.

(*1) Refer to How to create a certificate with a private certificate authority for details.

The following values are used in the examples.

In practice, use appropriate values for your environment.

Configurations on the Kafka broker (server side)

The following procedure is needed for a Kafka broker to perform SSL/TLS two-way authentication.

  1. Convert the file format
  2. Edit the Kafka broker’s properties file

Convert the file format

Convert the certificate and the private key files from PEM format to PKCS#12 format so that the Kafka broker can read them. The CA certificate and its private key are converted and stored in a trust store, while the server certificate and its private key are converted and stored in a key store.

First, create a trust store using the follwing command. Specify the CA certificate filename after -in, its private key filename after -inkey, the output filename of the trust store after -out, and the password to be set for the trust store after -passout.

$ sudo mkdir -p /srv/kafka/config/cert
$ sudo openssl pkcs12 -export -in /etc/pki/CA/cacert.pem \
         -inkey /etc/pki/CA/private/cakey.pem -name private-ca \
         -CAfile /etc/pki/CA/cacert.pem -caname private-ca \
         -out /srv/kafka/config/cert/truststore.p12 \
         -passout pass:trust-pass-00

Next, create a key store using the following command. Specify the server certificate filename after -in, its private key filename after -inkey, the output filename of the key store after -out, and the password to be set for the key store after -passout.

$ sudo openssl pkcs12 -export -in /etc/pki/CA/certs/broker.crt \
         -inkey /etc/pki/CA/private/broker.key -name broker \
         -CAfile /etc/pki/CA/cacert.pem -caname private-ca \
         -out /srv/kafka/config/cert/keystore.p12 \
         -passout pass:key-pass-00

Edit the Kafka broker’s properties file

Add the following lines to the Kafka broker’s properties file /srv/kafka/config/server.properties.

listeners=SSL://:9093
advertised.listeners=SSL://broker.example.org:9093
ssl.truststore.location=/srv/kafka/config/cert/truststore.p12
ssl.truststore.password=trust-pass-00
ssl.truststore.type=pkcs12
ssl.keystore.location=/srv/kafka/config/cert/keystore.p12
ssl.keystore.password=key-pass-00
ssl.keystore.type=pkcs12
ssl.client.auth=required

The meanings of the above settings are:

Restart the Kafka broker to apply the changes in the properties file.

$ sudo /srv/kafka/bin/kafka-server-stop.sh
$ sudo /srv/kafka/bin/kafka-server-start.sh /srv/kafka/config/server.properties

In order to change the settings without interrupting the service, configure multiple Kafka brokers and reflect the changes by rolling restart.

Configurations on SINETStream (client side)

The following procedure is needed for SINETStream to connect to the Kafka broker with authentication.

  1. Prepare certificate
  2. Edit the SINETStream’s configuration file
  3. Create a program that uses SINETStream

Prepare certificate

The following certificate is required on the client side to use SSL/TLS connection.

Deploy the certificate created by a private CA etc. to your convenient location. SINETStream reads the certificate from the path specified in the configuration file.

Edit the SINETStream’s configuration file

An example of SINETStream’s configuration file is shown below.

service-kafka-ssl:
  brokers: broker.example.org:9093
  type: kafka
  topic: topic-001
  tls:
    ca_certs: /opt/certs/cacert.pem
    certfile: /home/user01/certs/client0.crt
    keyfile: /home/user01/certs/client0.key

The settings for brokers, type, topic, consistency, tls are identical to those without authentication. Settings related to SSL/TLS authentication are under tls:.

Create a program that uses SINETStream

Your program will be identical with or without SSL/TLS authentication. For example, a program that uses MessageWriter of the SINETStream’s Python API is shown below.

with sinetstream.MessageWriter(service='service-kafka-ssl') as writer:
    writer.publish(b'message 001')

As you see, no code is written for authentication.

If you want to configure the authentication within your program, add parameters to the constructor arguments.

tls = {
    'ca_certs': '/opt/certs/cacert.pem',
    'certfile': '/home/user01/certs/client0.crt',
    'keyfile': '/home/user01/certs/client0.key',
}
with sinetstream.MessageWriter(service='service-kafka', tls=tls) as writer:
    writer.publish(b'message 001')

Behavior on authentication errors

Python API

The methods listed below raises the sinetstream.error.ConnectionError exception when an authentication error occurs.

Java API

The methods listed below throws the jp.ad.sinet.stream.api.AuthenticationException exception when an authentication error occurs.