I help my clients acquire new users and make more money with their web businesses. I have ten years of experience with SaaS projects. If that’s something you need help with, we should get in touch!
< Back to article list

Getting started with Python+Kafka

I had a hard time getting started with communicating between a kafka app and python with Docker. The thing is that kafka and the app need to run on the same docker network.

Let’s write a simple app that sends a message in kafka using kafka-python:

from kafka import KafkaProducer
import json

producer = KafkaProducer(
	value_serializer=lambda v: json.dumps(v).encode('utf-8'),
producer.send('fizzbuzz', {'foo': 'bar'})

We need a requirements.txt:


Then we can wrap this python script and its requirements in a Dockerfile, that we’ll later build inside a container.

FROM python:3.9


COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "python", "" ]

Now we can create our app:

$ docker network create app-tier --driver bridge
$ docker run -d --name kafka-server \
    --network app-tier \
$ docker build . -t poc-python
$ docker run --network app-tier -it poc-python
$ docker run --network app-tier -it edenhill/kcat:1.7.1 -b kafka-server:9092 -L
Metadata for all topics (from broker -1: kafka-server:9092/bootstrap):
 1 brokers:
  broker 1 at 318b468cb17f:9092 (controller)
 1 topics:
  topic "fizzbuzz" with 1 partitions:
    partition 0, leader 1, replicas: 1, isrs: 1