In this project, we use Apache Kafka on Windows, using Python. Using functionalities of Kafka like: Kafka Connect, Kafka Schemas, ksqlDB, Producer and Consumer.
Normally, Apache Kafka doesn't run on Windows. In order to run Apache Kafka on Windows, I downloadeded Confluent Platform on WSL and Ubuntu LTS.
WSL is a Linux command tool that can use Windows file system.
Ubuntu LTS is an open source Linux distribution.
*To activate WSL, write this line on Power Shell in administrator mode: Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
*Install Ubuntu LTS via Microsoft Store
*After installation, open Ubuntu LTS and write rest of the lines.
*To install Confluent Platform write this command: wget https://packages.confluent.io/archive/6.1/confluent-6.1.0.tar.gz
*After the download process, extract the .tar file: tar -xvf confluent-6.1.0.tar.gz
*After the extraction, enter the paths for Ubuntu to run Confluent Platform: export CONFLUENT_HOME=~/confluent-6.1.0 and export PATH=$CONFLUENT_HOME/bin:$PATH
*Install Datagen Connector with Confluent Hub: confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:latest
*To start all of the services, enter this command: confluent local services start
*connector.py: contains the function that allows Kafka to create connector. In this project, it creates postgresql source connection. This function gets imported to producer.py.
*postgres_source_connect.json: file that contains the information that is going to be used in the creation of the connector. This file gets read by connector.py.
*producer.py: file that contains functionalities of creating topics, pushing data to topics, creating connector, creating and using schema.
*schema_workers.avro: file that contains the information of schema. Gives information about data's format.
*consumer.py: file that contains the functionalities of getting the data from topics