Striim 3.9.4 / 3.9.5 documentation

Creating multiple writer instances

Some writers have a Parallel Threads property, which allows you to create multiple instances for better performance. For example:

CREATE TARGET KWSample USING KafkaWriter VERSION '0.9.0' (
  brokeraddress:'localhost:9092',
  topic:'test',
  ParallelThreads:'4',
  PartitionKey:'merchantId'
)
FORMAT USING DSVFormatter ()
INPUT FROM TypedCSVStream;

This would create four instances of KafkaWriter with identical settings. Each instance would run in its own thread, increasing throughput. If KWSample were deployed ON ALL to multiple servers, each server would run four instances, so the total number of KafkaWriter instances would be the ParallelThreads value time the number of servers.

Events will be distributed among the writer instances by the PartitionKey field value, and each Kafka partition, Kinesis shard, or S3 folder will be written to by only one of the KafkaWriter instances.

Warning

Use ParallelThreads only when the target is not able to keep up with incoming events (that is, when its input stream is backpressured). Otherwise, the overhead imposed by additional threads could reduce the application's performance.