If you are not familiar with Apache Kafka or want to learn about it, check out their site!
To give you a rough idea, these are the three core features of Kafka:
Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.
Store streams of records in a fault-tolerant durable way.
Process streams of records as they occur.
- https://kafka.apache.org/intro
In order to follow the example, you need to have the Kafka command line tools installed.
In brew
you can do that by simply running brew install kafka
.
If you don’t want to install via brew
or are using linux, you can get kafka like this:1
2
3curl -OL http://ftp.fau.de/apache/kafka/2.1.0/kafka_2.11-2.1.0.tgz
tar -xzf kafka_2.11-2.1.0.tgz
cd kafka_2.11-2.1.0/
You find the cli commands in the bin/
subdirectory.
All commands used assume that you are using the new-consumer api and that you added the kafka commands to your PATH
or running them from the bin/
directory from above.
1 | kafka-console-consumer --bootstrap-server <one-of-your-brokers> --topic <topic-to-consume> |
1 | kafka-console-consumer --bootstrap-server <one-of-your-brokers> \ |
1 | kafka-console-consumer --bootstrap-server <one-of-your-brokers> \ |
1 | kafka-console-consumer --bootstrap-server <one-of-your-brokers> \ |
This will print your messages in a format like <message-key>---<message-payload>
.
EDIT: The kafka-avro-console-consumer is not part of the package I linked above. It is part of the confluent suite. You can get it here.
Another cool feature of Kafka is that it plays well with the Apache Avro format. To consume messages encoded in Avro simply run the following command to get the decoded messages.
1 | kafka-avro-console-consumer --bootstrap-server <one-of-your-brokers> \ |
1 | kafka-console-consumer --bootstrap-server <on-of-your-brokers> \ |
I hope you can use this to speed up your work. If there are commands that you are frequently using that are not included here, let me know in the comments so I can add them.
]]>I my opinion he does an outstanding job in explaining the meaning of side effects and functional programming in general.
So instead of writing a long blog post, I leave you with this talk. Money quotes:
Hidden inputs and outputs are called ‘side effects’
Functional Programming is eliminating & controlling side effects
– Kris Jenkins
Let me know if things in this talk need clarification!
]]>A higher order function takes a function as an argument or returns a function as its result, or both.
So lets say you work in a big company and you have to calculate the bonuses for the employees. Of course, not everyone gets the same bonus, but you’re too lazy to pass the specific multiplier at every calculation. So you create seperate functions.
In this example the C-level department members get a bonus of 1.5 of their annual salary. While the salaries may differ, the multiplier stays the same.
1 | defmodule Salary do |
You may be wondering, why our c_level_bonuses function still knows about the percentage since we haven’t explicitly passed that value.
That’s because the function has knowledge about the environment it was created in - it’s a closure.
If you want to see an example for a function that takes another function as an argument, head to my previous post and take a look at the map function.
And in case you feel adventurous, try to find a use case to write a function that takes and returns a function =)
]]>When talking about tail call optimization it might be helpful to have a look at what happens when you call a function recursively.
Lets say we have the following function.
1 | defmodule Recur do |
I you feel adventurous run this function and see what happens with your RAM usage(spoiler: it will go up fast).
And that’s exactly what you expect to happen - a function calls itself forever, every time it uses some memory space for the stack to save what values a and b hold on each run.
Now we are using the same example, with a small modification.
1 | defmodule Recur do |
Your RAM usage will not grow indefinitely.
Why?
Because the last thing we called in our function is just the function itself.
A function is a subject to tail call optimization if the last thing it does is calling itself - nothing but itself.
In the first case the compiler may not know if he needs to keep those a and b values, so they have to be saved before every new function call.
This might be written down like this:
1 | call multiply_forever(a) |
In contrary, in the second example, the compiler knows, that he only has to care about the new argument to multiply_forever.
So he just updates the argument and JUMPS back to when multiply_forever first got called.
That’s the difference - JUMP vs. CALL
1 | multiply_forever(a) |
That’s basically what you have to know about tail call optimization(imho).
The next section is just for the sake of a more interesting example, feel free to skip it, I hope this article was useful to you.
Lets say you want to write your own Enum module, it will need a map function. So we are going to write it.
First approach that is not suiting tail call optimization:
1 | defmodule MyEnum do |
Now the “optimized” version(disclaimer: there are better ways to implement this for sure):
1 | defmodule MyEnum do |
As you can see, the tail call version takes more code, is not as straight forward as the first approach and we even have to reverse our result.
This is just to show, that even though tail optimization is necessary and you have to use it, sometimes it takes a litte bit more work.
Merry Christmas
]]>Yeah, uhm, I think I still owe you a post for some dropbox’ish thing to sync files over multiple devices.
At first I was willing to give seafile a try, but after reading this it became a nono for me.
Fortunately there is Syncthing - a peer to peer based syncing solution that runs on almost any platform.
Another plus: there is an android app and a third party iOS app seems to be in development, awesome!
There is already an impressive number of tutorials for installation and setup out there and the syncthing documentation is great. Therefore I’ll only cover the very specific things to my setup.
Syncthing is no client-server application.
Files will only sync if at least two devices are running and online. To make syncthing a Dropbox alternative I have one device that is online all the time. It always knows the most recent state of my files.
We will use a Raspberry PI 2 running Arch Linux as our always online device.
Consider the following commands to be executed on the PI.
# Version might have changed when you are reading this.
curl -OL https://github.com/syncthing/syncthing/releases/download/v0.11.21/syncthing-linux-arm-v0.11.21.tar.gz
mkdir syncthing
tar xfvz syncthing-linux-arm-v0.11.21.tar.gz -C syncthing --strip-components=1
Nice, Syncthing is now unpacked in a folder called syncthing.
Syncthing brings a nice web GUI that lets you configure your peer. By default it is only accessible from the peer running syncthing.
Since I don’t have desktop environment installedon the PI, I want to access the web GUI from my laptop.
In order to get this working we have to tweak the syncthing config a bit.
First, run syncthing once so it creates all the default folders.
cd syncthing/
./syncthing
# hit CTRL + C after everything booted up.
Now edit the syncthing config with your favorite editor.
vim ~/.config/syncthing/config.xml
Change this line
<gui enabled="true" tls="false">
<address>127.0.0.1:8384</address>
</gui>
to this
<gui enabled="true" tls="false">
<address>0.0.0.0:8384</address>
</gui>
We can now access the web GUI of the PI via http://IP_ADDRESS_OF_YOUR_PI:8384
You may not want to start Syncthing by hand everytime you restart your Raspberry, the guys from Syncthing got your back.
# do the next stuff as root
su
cd syncthing # if your aren't already there
cp syncthing /usr/bin/
chmod a+x /usr/bin/syncthing
cp etc/linux-systemd/user/syncthing.service /usr/lib/systemd/user/
systemctl --user enable syncthing.service
reboot
Syncthing now starts at every boot!
I had some troubles figuring out how to do this proerly so I thought it would be worth mentioning.
If you are running Arch Linux on your machine it is as easy as this
pacman -S syncthing
systemctl --user enable syncthing.service
systemctl --user start syncthing.service
Please see the documentation for all the configuration and Web GUI stuff, it’s very comprehensive.
I am really satisfied with this solution as my Dropbox alternative, everything seems to work well and I hope my
few notes on setting it up on the Raspberry PI will save you some time while setting up your own Syncthing!
Yeah, uhm]]>
Once installed you are ready to go.
Assuming we have the following file structure and want all JPEG files to be resized.
1 | . |
To make them, let’s say 25% smaller, you can run the following command:
1 | mogrify -resize 25% *.jpg |
You could also run something like:
1 | mogrify -resize 1600x1200 *.jpg |
Note that mogrify will replace your original files with the resized ones, so consider backing up your originals before running this.
After resizing, now create the gif using
1 | convert -delay 35 -loop 0 *.jpg output/animated.gif |
and your done, your gif will be placed in the output folder.
]]>There are many use cases for the Pi and therefore you will find a lot of images, that will try to match your needs. Since I just want to run some server applications on the Pi, that need no GUI at all, I will go for the Arch Linux image. There are surely enough other images that will fit the needs as well or even better, feel free to try them out, the installation steps should be the same.
You can find the Arch and many other images on the download site at raspberrypi.org
Depending on your operating system, there are different ways to set up the SD card, so it will boot your Linux distribution of choice.
I will try to cover both Linux and OSX by sticking to the command line, since they have many commands in common.(differences will be marked)
First download and extract the image:
1 | curl -o arch_arm.img.zip -L http://downloads.raspberrypi.org/arch_latest |
After downloading is finished, check the integrity of the image. This can save you a lot of pain!
A corrupted download will make you to have to repeat the whole setup again.
Generate the checksum of your download and compare it to the checksum on the downloads page.
1 | # generating the checksum of the downloaded file |
If the checksums are matching, lets extract the zip file.
1 | unzip arch_arm.img.zip |
Note that the extracted file will be named something like ArchLinuxArm-some-release-date-rpi.img.
Put in the SD card format it with your tool of choice to FAT32 and remove it.
Then do
1 | df -h |
Now insert the SD card.
Again run
1 | df -h |
The additional device listed is your SD card.
In my case it’s /dev/disk2, this will differ from machine to machine. So if I use /dev/disk2 in some following commmands just see it as a placeholder for your device name.
The next step will differ a little whether you are using an OSX or a Linux OS.
The SD card needs to be unmounted now.
1 | # only for mac users |
On Linux it should look something like this:
1 | # only do this when you are on a linux machine |
Alright, now it’s time to flash the image to the disk.
This step will again slightly differ from OSX to Linux.
1 | # on a mac machine the m for bs=1m is written in lower case. |
1 | # on linux machines the M is written in upper case an the bs is set to 4 |
Ok, since the image is not that big, flashing will be quite fast.
As the final step eject the SD card.
1 | # mac |
On Linux, clear the write cache, to safely remove the card.
1 | # linux |
Now the moment of truth has come. Pull your SD card out of your computer an put it in your Pi!
Notice: put the SD card in before plugging in the power supply.
Login will be root, root. Change this immediately:
1 | # after logging in to the Pi run |
Now the Pi should be up and running, awesome!
]]>So I started with buying a Raspberry Pi(Model B) and throwing Owncloud on it.
While installation and setup worked quite fluently, the performance was a desaster. A simple request to the webinterface of Owncloud on my Pi took about 20 seconds, which brings me to the conclusion, that Owncloud will be no fit for the Pi and it’s capabilities.
Fortunately there is a alternative to Owncloud which looks promising plus it’s open source and free for personal use.
It has a free app for Android as well, therefore I will give it a shot.
My next blog posts will be about setting up useful server applications on the Pi such as
I will also cover and start with:
Disclaimer: I have no clue if the PI can handle the stuff I am going to install =)
]]>I researched some dynamic DNS providers and was surprised to find, that only a few free solutions were left.
In the past I was very happy with dnydns.org, but they stopped their free plans.
Fortunately there is no-ip.org, which is keeping up with free plans for single users.
Sadly, the stock firmware which ships with my N600 only supports tzo and dyndns.org as providers for dynDNS which are not quite the same company I guess.
So no satisfying solution here.
Which forces me install some cool stuff on my router: OpenWrt
According to their website, installation should be straight forward:
1 | 1) Download the file openwrt-ar71xx-generic-mynet-n600-squashfs-sysupgrade.bin |
Alright step 1 is easy. Step 2 can be tricky if you are using some linux-ish OS.
I did it on Mint with editing /etc/network/interfaces and adding following lines:
1 | # /etc/network/interfaces |
Did a
1 | sudo /etc/init.d/networking restart |
after saving the file.
This enables me to connect my machine and the router directly via LAN.
Step 3 to 7 worked flawlessly.
After flashing OpenWrt ist done, you can telnet on the router.
1 | telnet 192.168.1.1 |
Not lets install a web interface called LuCI.
Fortunately there are great step-by-step instructions already there!
1 | opkg update |
It just works, great.
Now we need DDNS support, since it is not avaiable out-of-the-box by luci.
1 | opkg install luci-app-ddns |
The DDNS service was disabled by default, so I turned it on via System -> Startup
Now after refreshing the page there should be a new tab called “Services” at the web interface’s navigation.
You can choose a ton of providers here. As mentioned, I will go ahead with no-ip.org.
After setting up the PI we will need these prerequisites to use our applications from outside our local network, e.g. with the smartphone.
But this should be it for now.
Please do not forget to set up a root password and all the other stuff like ssh and the wifi(if needed).
]]>1 | :sh |
in normal mode. If you type exit or hit CTR+D in your shell, you will get back to Vim. I like this behaviour when I am on a remote server via ssh.
Therefor I binded CTR+D in Vim to execute :sh, so I can toggle between shell and Vim with this shortcut.
To get it, include this snippet in your vimrc.
1 | # ~/.vimrc |
1 | InterestingModel.reflect_on_all_associations.map{ |assoc| [assoc.macro, assoc.name] } |
Addtion:
You can also see all associations for an instance of a model.
1 | InterestingModel.new.reflections |
This also works with Classes themself
1 | InterestingModel.reflections |
Note that reflect_on_all_associations returns an Array of all Associations and
reflections will return a Hash like { :association_name => ActiveRecord::Relfection::AssociationReflection }
Lately I tried to switch the database in a Rails app from SQLite to PostgreSQL.
and it greets me with
There are some encoding issues, yeah!
But adding some additional options did the trick
In summary, when you get such errors try these steps
Lately I tried to switch the database in a Ra]]> As you may remember, we are able to search for recipes, whose titles are matching our search string. Unfortunatly, the api response is not serving any link to the recipe itself. So let’s fix this and add this url directly to our method’s result! First we need to find out, how the recipe urls are built.
Now we could extend our search method by telling it to call a method named “append_recipe_link” and passing our response to that method:
And there you go. Your results should now have a url linking directly to the recipe show page. The last thing I want to show is, how you get more detailed information about your recipes.
This gives us more detailed information about a recipe in response:
Now our module has quite some functionality. Hope this was usefull for at least a few people. ]]>As you may remember, we are ab]]> some contextual issues. As I am a huge fan of Ruby and Ruby on Rails I convinced my partner to develop a web app for this seminar. Since we both like to cook, we decided to create a web app with RoR which helps you to find a suitable recipe for To get us started finding some database we can work on, I stumbled across the ChefkochApi.
… and will get a response like this one:
And this is how you can access the API with Ruby:
Now you can invoke a recipe search by simply calling
That’s it for part one. In the next part I will try to modify the result and retrieve more detailed information from it like some contextual i]]> |