Striim 3.9.4 / 3.9.5 documentation

Configuring Kafka

Kafka streamsKafkaReader, and KafkaWriter all use the properties described in this section. In Kafka streams, the properties are specified in a property set and are separated by commas. In KafkaReader and KafkaWriter, the properties are specified in the KafkaConfig property and are separated by semicolons.

Using Striim's internal Kafka server

Warning

On Windows, Zookeeper and Kafka do not shut down cleanly. (This is a well-known problem.) Before you restart Kafka, you must delete the files they leave in c:\tmp. Alternatively, look on stackoverflow.com for instructions on running Zookeeper and Kafka as services on Windows, or run an external Kafka server in a Linux virtual machine.

Kafka streams may be persisted to Striim's internal Kafka server or to an external Kafka server.  The default property set for the internal Kafka server that is installed with Striim at Striim/Kafka  is Global.DefaultKafkaProperties:

jmx.broker=localhost:9998, 
bootstrap.brokers=localhost:9092, 
zk.address=localhost:2181

If you installed Striim with the JAR installer as discussed in Installing Striim for evaluation purposes and enabled Kafka in the setup wizard, it will start automatically. If you did not enable Kafka during installation, you may do so by re-running the setup wizard in the Strim/bin directory (WebConfig.exe for Windows, WebConfig for Mac, or WebConfig.sh for Linux).

If you installed Striim from a DEB, RPM, or TGZ package as discussed in Running Striim in UbuntuRunning Striim in CentOS, or Running Striim as a process, start Kafka as follows:

  1. Open a terminal.

  2. Change to Striim/Kafka, and enter bin/zookeeper-server-start.sh config/zookeeper.properties (this will start Zookeeper).

  3. Open another terminal.

  4. Change to Striim/Kafka and enter JMX_PORT=9998 bin/kafka-server-start.sh config/server.properties (this will start Kafka).

You can then persist Kafka streams using the default property set.

Using an external Kafka server

When using an external Kafka server, to handle Striim's maximum batch size the following entries in config/server.properties must have at least these minimum values:

message.max.bytes = 43264200
replica.fetch.max.bytes = 43264200
socket.request.max.bytes=104857600 

To support persisting streams to an external server, use the Tungsten console to create a custom Striim property set using the following syntax:

CREATE PROPERTYSET <name> (
  bootstrap.brokers:'<bootstrap IP address>:<port>',
  jmx.broker:'<jmx IP address>:<port>'),
  zk.address:'<zookeeper IP address>:<port>',
  partitions:'<number of partitions to use>'
  kafkaversion:'{0.8|0.9|0.10|0.11}';

If not specified, partitions defaults to 200.

Using Kafka SASL (Kerberos) authentication with SSL encryption

To use SASL authentication with SSL encryption, do the following:

  1. Get the files krb5.conf , principal.keytab , server.keystore.jks, and server.truststore.jks from your Kafka administrator and copy them to the Striim server's file system outside of the Striim program directory, for example, to /etc/striim/kafkaconf.

  2. In the same directory, create the file jaas.conf, including the following lines, adjusting the keyTab path and principal to match your environment:

    KafkaClient {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      doNotPrompt=true
      client=true
      keyTab="/etc/striim/kafkaconf/principal.keytab"
      principal="principal@example.com";
    };
    
  3. Add the following to Striim's Java environment:

    JAVA_SYSTEM_PROPERTIES=" \
    -Djava.security.krb5.conf='/etc/striim/kafkaconf/krb5.conf' \
    -Djava.security.auth.login.config='/etc/striim/kafkaconf/kafka_server_jaas.conf' "
  4. Include the following properties in your Kafka stream's property set or KafkaReader or KafkaWriter KafkaConfig, adjusting the paths to match your environment and using the passwords provided by your Kafka administrator. For KafkaConfig, replace the commas with semicolons.

    security.protocol=SASL_SSL,
    sasl.kerberos.service.name=kafka,
    security.protocol=SSL,
    ssl.truststore.location=/etc/striim/kafkaconf/server.truststore.jks,
    ssl.truststore.password=password,
    ssl.keystore.location=/etc/striim/kafkaconf/server.keystore.jks,
    ssl.keystore.password=password,
    ssl.key.password=password

Using Kafka SASL (Kerberos) authentication without SSL encryption

To use SASL authentication without SSL encryption, do the following:

  1. Get the files krb5.conf and principal.keytab from your Kafka administrator and copy them to the Striim server's file system outside of the Striim program directory, for example, to /etc/striim/kafkaconf.

  2. In the same directory, create the file jaas.conf, including the following lines, adjusting the keyTab path and principal to match your environment:

    KafkaClient {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      doNotPrompt=true
      client=true
      keyTab="/etc/striim/kafkaconf/principal.keytab"
      principal="principal@example.com";
    };
    
  3. Add the following to Striim's Java environment:

    JAVA_SYSTEM_PROPERTIES=" \
    -Djava.security.krb5.conf='/etc/striim/kafkaconf/krb5.conf' \
    -Djava.security.auth.login.config='/etc/striim/kafkaconf/kafka_server_jaas.conf' "
  4. Include the following properties in your Kafka stream's property set or KafkaReader or KafkaWriter KafkaConfig . For KafkaConfig, replace the comma with a semicolon.

    security.protocol=SASL_PLAINTEXT,
    sasl.kerberos.service.name=kafka

Using Kafka SSL encryption without SASL (Kerberos)  authentication

To use SSL encryption without SASL authentication, do the following:

  1. Get the files server.truststore.jks and server.keystore.jks from your Kafka administrator and copy them to the Striim server's file system outside of the Striim program directory, for example, to /etc/striim/kafkaconf.

  2. Include the following properties in your Kafka stream property set or KafkaReader or KafkaWriter KafkaConfig, adjusting the paths to match your environment and using the passwords provided by your Kafka administrator. For KafkaConfig, replace the commas with semicolons.

    security.protocol=SSL,
    ssl.truststore.location=/etc/striim/kafkaconf/server.truststore.jks,
    ssl.truststore.password=password,
    ssl.keystore.location=/etc/striim/kafkaconf/server.keystore.jks,
    ssl.keystore.password=password,
    ssl.key.password=password

Using Kafka without SASL (Kerberos) authentication or SSL encryption

To use neither SASL authentication nor SSL encryption, do not specify security.protocol in the KafkaReader or KafkaWriter KafkaConfig or in your Kafka stream's property set.

Additional properties for Kafka streams

Use these properties only in Kafka stream property sets, not with KafkaReader or KafkaWriter. Use single quotes around the values.

Kafka property

default value

notes

partitions

 '200'

the maximum number of Kafka partitions to be used if the stream is partitioned; if the stream is not partitioned, only one partition is used and this value is ignored

replication.factor

'1'

the number of replicas to keep for each event; if this is greater than the number of brokers, creation of a topic will fail