KeiruaProd

I had a hard time getting started with communicating between a kafka app and python with Docker. The thing is that kafka and the app need to run on the same docker network.

Let’s write a simple app that sends a message in kafka using kafka-python:

from kafka import KafkaProducer
import json

producer = KafkaProducer(
	value_serializer=lambda v: json.dumps(v).encode('utf-8'),
	bootstrap_servers='kafka-server:9092',
)
producer.send('fizzbuzz', {'foo': 'bar'})

We need a requirements.txt:

kafka-python==2.0.2

Then we can wrap this python script and its requirements in a Dockerfile, that we’ll later build inside a container.

FROM python:3.9

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "python", "main.py" ]

Now we can create our app:

• a custom network (app-tier)
• run kafka on this network
• build our Python container, then run it on this network
• finally, we can listen to the events using kcat (“kafka cat”), that will display what is happening.

$ docker network create app-tier --driver bridge
$ docker run -d --name kafka-server \
    --network app-tier \
    -e ALLOW_PLAINTEXT_LISTENER=yes \
    bitnami/kafka:latest
$ docker build . -t poc-python
$ docker run --network app-tier -it poc-python
$ docker run --network app-tier -it edenhill/kcat:1.7.1 -b kafka-server:9092 -L
Metadata for all topics (from broker -1: kafka-server:9092/bootstrap):
 1 brokers:
  broker 1 at 318b468cb17f:9092 (controller)
 1 topics:
  topic "fizzbuzz" with 1 partitions:
    partition 0, leader 1, replicas: 1, isrs: 1