With my current projects switching on to a whole new stack, it was about time the projects need to have integration with this stack. An integral part of this new stack is Kafka, the distributed message queuing system..
This is specially used in the Microservices architecture as a mode of communication between 2 microservices. Won’t go in details as to what Microservices architecture is, but If you want to know more, better start here.
So the task was how to integrate Powershell with Kafka ?
The detailed tasks which I wanted to achieve were
- Able to connect to Kafka server
- Able to get list of available topics from the kafka server
- Able to post a message to kafka server
- Able to consume a message from Kafka server.
With the help my Architect, got reference to a already developed Powershell cmdlet library called pskafka
Pre-Requisites for installation
- Module ThreadJob needs to be available
- Powershell Version needs to be 5 or ahead of it.
- Admin access to run commands on powershell cmd
- kafka broker list , this goes with default port of 9092
Lets go step by Step on how I achieved the above tasks in details
- Download the Git folder as a zip and then extract the folder, you will get a folder called pskafka-master
- Don’t change the name of this folder, as there are internal references made to this folder name in the code
- Copy the folder to your desired location and run the below commands . This will install pskafka module.
PS D:\Navin\pskafka-master> Install-Module -Name 'ThreadJob' Untrusted repository You are installing the modules from an untrusted repository. If you trust this repository, change its InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules from 'PSGallery'? [Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "N"): Y PS D:\Navin\pskafka-master> Import-Module .\pskafka.psd1 PS D:\Navin\pskafka-master>
2. Validating the instillation
- The below command will get all the topics from the kafka cluster.
PS D:\Navin\pskafka-master> Get-KafkaTopics -BrokerList xx.xxx.xx.xxx:9092 -Verbose VERBOSE: D:\Navin\Git\pskafka-master\bin\win\kafkacat.exe -b xx.xxx.xx.xxx:9092 -L __consumer_offsets test1 test2 test3 test4 test5
3. Post a message to Kafka Topic
- You can use the cmdlet Out-KafkaTopic to post a message to kafka topic. This cmdlet can be used for batch inserts as well as single inserts, more examples can be found here.
PS D:\Navin\pskafka-master> Out-KafkaTopic -Messages "1" -TopicName "test5" -BrokerList "xx.xxx.xx.xxx:9092" PS D:\Navin\pskafka-master> Out-KafkaTopic -Messages "1" -TopicName "test5" -BrokerList "xx.xxx.xx.xxx:9092"
2. You can validate if your message is inserted and available in kafka cluster like below.
PS D:\Navin\pskafka-master> Read-KafkaTopic -TopicName 'test5' -BrokerList 'xx.xxx.xx.xxx:9092' -Me ssageCount 1000 -FromBeginning -Verbose VERBOSE: D:\Navin\pskafka-master\bin\win\kafkacat.exe -b xx.xxx.xx.xxx:9092 -q -u -o beginning -c 1000 -e -C -t test5 "1" "2" "3" "4"
So we covered all the above use cases as mentioned above.
Side note : Kafka Message Queuing system takes in messages as String, which is usually in JSON format. Its not like Kafka cannot support other type of messages format, but its more native to Kafka. So make sure you are always sending messages in terms of JSON string to kafka.
Stay tuned, I will be publishing a small post on how to build JSON string variables in powershell easily