When developing a new feature that involves Apache Kafka I like to hook into a topic to check if everything is working as expected. Usually I do this with the Kafka command line tools but I always forget the exact command to run which I have to look from different sources.
This posts tries to put them together in order to have a single place to look them up.

If you are not familiar with Apache Kafka or want to learn about it, check out their site!
To give you a rough idea, these are the three core features of Kafka:

Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.
Store streams of records in a fault-tolerant durable way.
Process streams of records as they occur.
- https://kafka.apache.org/intro

In order to follow the example, you need to have the Kafka command line tools installed.
In brew you can do that by simply running brew install kafka.
If you don’t want to install via brew or are using linux, you can get kafka like this:

1
2
3

curl -OL http://ftp.fau.de/apache/kafka/2.1.0/kafka_2.11-2.1.0.tgz
tar -xzf kafka_2.11-2.1.0.tgz
cd kafka_2.11-2.1.0/

You find the cli commands in the bin/ subdirectory.
All commands used assume that you are using the new-consumer api and that you added the kafka commands to your PATH or running them from the bin/ directory from above.

How to consume messages from a given Kafka Topic?

Start to consume from the latest offset

1

kafka-console-consumer --bootstrap-server <one-of-your-brokers> --topic <topic-to-consume>

Consume from the beginning

1
2
3

kafka-console-consumer --bootstrap-server <one-of-your-brokers> \
                       --topic <topic-to-consume> \
                       --from-beginning

Consume a fixed amount of messages

1
2
3

kafka-console-consumer --bootstrap-server <one-of-your-brokers> \
                       --topic <topic-to-consume> \
                       --max-messages <number-of-messages>

Consume and print the message keys

1
2
3
4

kafka-console-consumer --bootstrap-server <one-of-your-brokers> \
                       --topic <topic-to-consume> \
                       --property print.key=true \
                       --property key.separator="---"

This will print your messages in a format like <message-key>---<message-payload>.

Consume messages encoded in Avro

EDIT: The kafka-avro-console-consumer is not part of the package I linked above. It is part of the confluent suite. You can get it here.
Another cool feature of Kafka is that it plays well with the Apache Avro format. To consume messages encoded in Avro simply run the following command to get the decoded messages.

1
2
3

kafka-avro-console-consumer --bootstrap-server <one-of-your-brokers> \
                            --topic <topic-to-consume> \
                            --property schema.registry.url=http://<your-schema-registry>

Move the offset of a specific consumer group forward by consuming messages

1
2
3
4

kafka-console-consumer --bootstrap-server <on-of-your-brokers> \
                       --topic <topic-to-consume> \
                       --max-messages <number-of-messages/moving the offset forward> \
                       --consumer-property group.id=<consumer-group-to-forward-the-offset> > /dev/null

I hope you can use this to speed up your work. If there are commands that you are frequently using that are not included here, let me know in the comments so I can add them.

Long time no see! I want to continue my explorational series of functional programming with side effects.
After doing some research I came across this excellent talk by Kris Jenkins about functional programming.

I my opinion he does an outstanding job in explaining the meaning of side effects and functional programming in general.
So instead of writing a long blog post, I leave you with this talk. Money quotes:

Hidden inputs and outputs are called ‘side effects’

Functional Programming is eliminating & controlling side effects

– Kris Jenkins

Let me know if things in this talk need clarification!

In my previous post I tried to explain tail call optimization.
If you read it to the end you saw that map function that took a collection and a function as its arguments.
This already was a higher order function. The definition is straight forward:

A higher order function takes a function as an argument or returns a function as its result, or both.

So lets say you work in a big company and you have to calculate the bonuses for the employees. Of course, not everyone gets the same bonus, but you’re too lazy to pass the specific multiplier at every calculation. So you create seperate functions.
In this example the C-level department members get a bonus of 1.5 of their annual salary. While the salaries may differ, the multiplier stays the same.

1
2
3
4
5
6
7
8
9

defmodule Salary do
  def bonus_calculation(percentage) do
    fn(salary) -> salary * percentage end
  end
end

# c_level_bonuses = Salary.bonus_calculation(1.5)
# c_level_bonuses.(100_000)
# => 150000.0

You may be wondering, why our c_level_bonuses function still knows about the percentage since we haven’t explicitly passed that value.
That’s because the function has knowledge about the environment it was created in - it’s a closure.

If you want to see an example for a function that takes another function as an argument, head to my previous post and take a look at the map function.

And in case you feel adventurous, try to find a use case to write a function that takes and returns a function =)