Powershell

Integrating Powershell with Kafka.

With my current projects switching on to a whole new stack, it was about time the projects need to have integration with this stack. An integral part of this new stack is Kafka, the distributed message queuing system..

This is specially used in the Microservices architecture as a mode of communication between 2 microservices. Won’t go in details as to what Microservices architecture is, but If you want to know more, better start here.

So the task was how to integrate Powershell with Kafka ?

The detailed tasks which I wanted to achieve were

  1. Able to connect to Kafka server
  2. Able to get list of available topics from the kafka server
  3. Able to post a message to kafka server
  4. Able to consume a message from Kafka server.

 

With the help my Architect, got reference to a already developed Powershell cmdlet library called pskafka

Pre-Requisites for installation

  1. Module ThreadJob needs to be available
  2. Powershell Version needs to be 5 or ahead of it.
  3. Admin access to run commands on powershell cmd
  4. kafka broker list , this goes with default port of 9092

 

Lets go step by Step on how I achieved the above tasks in details

  1. Installing pskafka

    1. Download the Git folder as a zip and then extract the folder, you will get a folder called pskafka-master
    2. Don’t change the name of this folder, as there are internal references made to this folder name in the code
    3. Copy the folder to your desired location and run the below commands . This will install pskafka module.
PS D:\Navin\pskafka-master> Install-Module -Name 'ThreadJob'

Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its
InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules from
'PSGallery'?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "N"): Y
PS D:\Navin\pskafka-master> Import-Module .\pskafka.psd1
PS D:\Navin\pskafka-master>


      2. Validating the instillation

    1. The below command will get all the topics from the kafka cluster.
PS D:\Navin\pskafka-master> Get-KafkaTopics -BrokerList xx.xxx.xx.xxx:9092 -Verbose
VERBOSE: D:\Navin\Git\pskafka-master\bin\win\kafkacat.exe -b xx.xxx.xx.xxx:9092 -L
__consumer_offsets
test1
test2
test3
test4
test5

     3. Post a message to Kafka Topic

    1. You can use the cmdlet Out-KafkaTopic to post a message to kafka topic. This cmdlet can be used for batch inserts as well as single inserts, more examples can be found here.

PS D:\Navin\pskafka-master> Out-KafkaTopic -Messages "1" -TopicName "test5" -BrokerList "xx.xxx.xx.xxx:9092"

PS D:\Navin\pskafka-master> Out-KafkaTopic -Messages "1" -TopicName "test5" -BrokerList "xx.xxx.xx.xxx:9092"

           2. You can validate if your message is inserted and available in kafka cluster like below.

PS D:\Navin\pskafka-master> Read-KafkaTopic -TopicName 'test5' -BrokerList 'xx.xxx.xx.xxx:9092' -Me
ssageCount 1000 -FromBeginning -Verbose
VERBOSE: D:\Navin\pskafka-master\bin\win\kafkacat.exe -b xx.xxx.xx.xxx:9092 -q -u -o beginning -c
1000 -e -C -t test5
"1"
"2"
"3"
"4"

 

So we covered all the above use cases as mentioned above.

Side note : Kafka Message Queuing system takes in messages as String, which is usually in JSON format. Its not like Kafka cannot support other type of messages format, but its more native to Kafka. So make sure you are always sending messages in terms of JSON string to kafka.

Stay tuned, I will be publishing a small post on how to build JSON string variables in powershell easily

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s