Saturday, April 09, 2016

The danger in Golang infinite for-loop

This is a blog post about how my colleagues and I got bitten by simple looking infinite/forever/endless for-loop in Golang.

We were implementing a server app that waits for a client, receives data and stores it in DB. As simple as it sounds. The implementation goes like this

func main() {
    s := Server{Handler: handler}
    for {

func handler(c *Client){
    for {
        buf := make([]byte, 1024*1024)
        n, err := c.Read(buf)
        if err != nil {
        // process buf

The infinite for-loop in main function is used to prevent it getting terminated. I know it is a busy waiting and keeps the CPU active but we didn't give much thought about it.

The problem with above implementation is, the server freezes after receiving N buffers from the client. Initially we suspected the client, network, buffered IO but after adding some logs we realized that the program freezes when it tries to allocate the buffer. Yes, at the call to "make([]byte, ...)". Don't believe me, try the following snippet and see it yourself.

func main() {
    go work()

    for {

func work() {
    println("work started")
    var r int64
    for i := 0; i < 1000; i++ {
        buf := make([]byte, 1024*1024)
        r = r + int64(buf[0]) // just to make sure the allocation is not optimized away
    println("work completed")

The program will freeze after printing "work started" and it will never print "work completed". You have to terminate it forcefully. I tried with "GOMAXPROCS > 1" but no luck. I don't know the exact reason for this behaviour but I have a guess. I suppose the garbage collector(GC) kicks in after N number of allocations(i.e. make) and it tries to pause the runtime. As the runtime is busy with the infinite for-loop the GC couldn't pause runtime and the program freezes. We can avoid the freeze if we change the program in such way that the runtime can pause the infinite for-loop for the GC. There are different ways we can do this:
  • Replacing the infinite for-loop with infinite select loop. Apparently this seems to be the recommended way of blocking execution in Golang. This also keeps your CPU utilization low.
  • Sleeping within the for-loop. The sleep duration could be as small as zero.
  • Yielding the execution to runtime by calling runtime.Gosched()
I've observed the above behaviour with Go 1.5.1 on OSX. I am not sure whether it changes in different version or platform.

Saturday, April 02, 2016

Embedding DB migrations through Golang build tags

In this post I am going to explain how to embed database migration sqls within application binary and how can we utilize the Golang build tags to maintain both embedded and non-embedded versions of database migrations.

I am developing a system application in Golang that uses SQLite as database. It is destined to run on user's machine. The application runs database migrations at every startup. In a typical web application the migration sql files are usually deployed along with the web application so that it can migrate the database next time when it is launched. But in my case as it is an application distributed to the user, I don't want to keep the migrations as separate sql files. So I decided to embed the migration sqls with the application binary. I also took advantage of the Golang build tags to maintain both embedded and non-embedded versions.

Embedding Migration SQLs

I've used goose and migrate libraries in an earlier projects. But both libraries didn't support embedding the migration sqls within application binary. Quick search revealed that sql-migrate library can do both non-embedded and embedded migrations. We need to convert the migration sqls into Go source files using go-bindata and instruct sql-migrate to use embedded sqls.

The following command converts all migration sqls from "db/migrations" directory into Go source file named "bindata.go" with package name of "myapp".
$ go-bindata -pkg myapp -o bindata.go db/migrations/
The "bindata.go" exports two functions named "Asset" and "AssetDir". These functions are used to retrieved the embedded file and file list of a embedded directory.

The following code snippet wires up these functions with sql-migrate for it retrieve the embedded migration sqls.
migrations := &migrate.AssetMigrationSource{
    Asset:    Asset,
    AssetDir: AssetDir,
    Dir:      "db/migrations",

Embedding with Build Tags

I want to embed the migrations only for production build. I want to use non-embedded migrations for development builds to avoid the additional step of running go-bindata every time the migration sql changes. I achieved this workflow with help of Go build tags.

Go build tags are originally created to select platform specific source files while building multi-platform applications/libraries. The mechanism can be used for other purposes too, like my usage of embedding migration sqls. All build related go commands(e.g. go build, go test) supports specifying build tags.

So to achieve the above workflow I've created two migration source files, one for non-embedded version and another one for embedded version.

// +build !embedded_migrations
package db 
import (

func migrations() migrate.MigrationSource {
        return &migrate.FileMigrationSource{
                Dir: "db/migrations",

// +build embedded_migrations
package db 
import (

func migrations() migrate.MigrationSource {
        return &migrate.AssetMigrationSource{
                Asset:    myapp.Asset,
                AssetDir: myapp.AssetDir,
                Dir:      "db/migrations",
In the migration source files, the build tag "+build !embedded_migrations" instructs the build tool to use the source if the build tag "embedded_migrations" is not specified. Similarly the build tag "+build embedded_migrations" instructs the build tool to use the source if the build tag "embedded_migrations" is specified. Basically only one of the source file is used by the build tool based on the presence/absence of the build tag.

Regardless whether the migration sqls are embedded or not, we can retrieve the migration sqls by simply calling "db.migrations()" function and upgrade the database like below.

n, err := migrate.Exec(db, "sqlite3", db.migrations(), migrate.Up)
if err != nil {
    log.Fatal("db migrations failed: ", err)
Now to build the application with non-embedded migrations we can run the following command
$ go build
and to build with embedded migrations for production/deployment we can run the following command.
$ go-bindata -pkg myapp -o bindata.go db/migrations/
$ go build -tags embedded_migrations

Wednesday, October 21, 2015

Cross compiling libs3 for Raspberry Pi

Recently I had a need to upload files from Raspberry Pi to Amazon S3 bucket. There are at least two ways to do this: uploading through REST API and using libs3 C library. The REST API approach requires HMAC-SHA1 signature authentication. Calculating the signature is not trivial considering the various rules in constructing the signature payload. So I chose to go with libs3 approach to achieve the goal. This post explains how I cross compiled libs3 for Raspberry Pi/armhf.

Getting Source

The libs3 source is hosted in Github in the repository Clone the repository and make sure it is accessible from our build environment.

Getting Dependencies

We need header and library files of libcurl and libxml2 to build libs3. We also need library files of zlib and liblzma while building host application that uses libs3.

The easy way to get these files is to get your OS distribution packages of these libraries and extract the required files. My Raspberry Pi 2 runs Raspbian OS created from Debian Wheezy. So I downloaded Wheezy packages of these libraries for "armhf" architecture from here and extracted the files as explained here.

The hard way is to get source of all these libraries and cross compiling them for Raspberry Pi then using the resultant libraries. It is a time consuming and unnecessary exercise as the required files are readily available in the OS distribution packages. So I didn't choose to go in this way.

The libs3 can be built as static and shared library. The default build configuration builds both version. When you build libs3 static library you need the .a files from these dependencies and when you build shared library you need the .so files.

Build Environment 

Get your Raspberry Pi cross compiling environment ready as explained here. Copy the dependencies header files(i.e. "curl" and "libxml" directories) into "$HOME/raspberrypi/rootfs/usr/include". Copy the dependencies library files into "$HOME/raspberrypi/rootfs/usr/lib/arm-linux-gnueabihf".

Building The Library

Navigate to the libs3 source directory and issue the below commands to build libs3 library.

$ make CC=arm-linux-gnueabihf-gcc CURL_LIBS=$HOME/raspberrypi/rootfs/usr/lib/arm-linux-gnueabihf/libcurl.a CURL_CFLAGS=-I/opt/raspberrypi/rootfs/usr/include LIBXML2_LIBS=/$HOME/raspberrypi/rootfs/usr/lib/arm-linux-gnueabihf/libxml2.a LIBXML2_CFLAGS=-I$HOME/raspberrypi/rootfs/usr/include -f GNUmakefile

It is enough to specify the include directory "-I/opt/raspberrypi/rootfs/usr/include" once either in CURL_CFLAGS or LIBXML2_CFLAGS but if we didn't specify in both it will attempt to invoke the programs "curl-config" or "xml2-config" which we don't have.

After successfully building the library, the libs3 static and shared libraries will be available under "build/lib" directory of libs3 source directory. The directory "inc" under libs3 source directory contains the corresponding libs3 header files. Now we can either point our application build system to consume libs3 from here or copy the header and library files into the corresponding directories of "$HOME/raspberrypi/rootfs/usr" to consume them automatically via the pi.cmake toolchain file. Make sure you link zlib and lzma libraries with your application when you consume libs3, otherwise the linking process will fail with undefined symbol error.

I hope the information helps you, thanks for reading.

Thursday, December 18, 2014

Pure CMake based bin2h implementation

I had been writing CMake build script for a project that requires converting few files into C/C++ headers so that the content of those files can be embedded in the output binary.

I could've done this by executing an external bin2h program to do the conversion. But as it is a cross-platform application I need to depend on platform specific bin2h programs. A widely used approach for this is to include bin2h source in the project code base and compile it on the target platform then use the resultant executable for the bin2h conversion.

I didn't want to include the source bin2h in my project and compile it on-demand. I was thinking there should be a CMake command/function/module to do this in cross-platform way but I couldn't find anything.

So I started writing a pure CMake based bin2h implementation and end up with the following. The following module provides a CMake function "BIN2H" that can be used to convert any files into C/C++ header file. We need to specify the source file, header file and the C/C++ variable name that points to the raw bytes of the source file content. We can also specify the optional parameter 'APPEND' to append to the header file instead of overwriting and the optional parameter 'NULL_TERMINATE'  to terminate the raw bytes array with null byte.

I hope it will be useful to others.

Monday, July 07, 2014

RSA OAEP padding with SHA512 hash algorithm

Recently I wanted to encrypt a message with RSA with OAEP padding. I also wanted to use SHA512 as hashing algorithm and mask generation function(MGF) in the OAEP padding instead of SHA1. But it looks like it is not possible with the OpenSSL/libcrypto as the SHA1 hash algorithm is hard coded in OAEP padding implementation. This is confirmed by this thread in OpenSSL forum. Though the forum thread was written around 2012 but still I couldn't find a way use either SHA256 or SHA512 as my hashing algorithm and MGF in OAEP padding.

As suggested by "Dr Stephen N. Henson"(the core developer of OpenSSL) in the forum thread , I've took the implementation of RSA OAEP padding and modified to use SHA512 instead of SHA1. It is mostly just find EVP_sha1 and replace with EVP_sha512. We also need to update the usage of SHA_DIGEST_LENGTH macro to SHA512_DIGEST_LENGTH to reflect the output length of SHA512 hash. Below is the modified RSA OAEP padding implementation which uses SHA512 algorithm. Hope it helps, cheers.

Wednesday, July 02, 2014

Viewing raw RGB bitmaps with ImageMagick

I've been looking for a tool to view raw RGB/RGBA bitmaps but couldn't locate such one. It will be useful at times when we want to troubleshoot what bitmaps we are feeding to video encoder or what frames are getting produced by video decoder. The RGB bitmaps contains raw image data and doesn't carry any header info like width, height and etc. For example, if the image is 100x100 and it is a 32 bits per-pixel(RGBA) then the RGBA image will be 100 * 100 * 4 = 40,000 bytes. In this image data each pixel is represented by four bytes and the bytes present the color primary values of Red, Green, Blue & Alpha respectively.

Very recently I realized I can use ImageMagick to convert these RGB bitmaps to any viewable image formats like png or jpg and use any standard image viewer to view them. The following command uses ImageMagick's convert utility to convert the raw RGBA image to png image which can be viewed with any image viewer application.
convert.exe -depth 8 -size 2048x858 image.rgba image.png
as you can see we specify the bit depth(how many bits per pixel component) and size, without which the utility doesn't know how to interpret the image data from source image. We can also specify source and destination color space with '-colorspace' option to convert the image from one color space to another. Also the option '-alpha off' can be used if we want to ignore the alpha channel from source image.

Saturday, June 28, 2014

Enabling Direct3D fullscreen mode from non-active window

In this post I am going to tell you how can you enable Direct3D fullscreen mode from non-active window which is used as swap chain's rendering target.

Initially I didn't realize it could be challenging to enable Direct3D fullscreen mode from non-active window. I thought all I need to do is call the SetFullscreenState method of the SwapChain which is also worked fine from test application. But when I tried the same thing from the real WinForm application which was not active/foreground window at the time of enabling fullscreen, it didn't work. I was getting DXGI_ERROR_NOT_CURRENTLY_AVAILABLE error from SetFullscreenState method.

When I checked MSDN page about the SetFullscreenState method I can clearly see why it is failing.
DXGI_ERROR_NOT_CURRENTLY_AVAILABLE if the action failed. There are many reasons why a windowed-mode swap chain cannot switch to full-screen mode. For instance:
  • The application is running over Terminal Server.
  • The output window is occluded.
  • The output window does not have keyboard focus.
  • Another application is already in full-screen mode.
The third point mentioned in the MSDN page is clearly my situation. So I thought I can simply set the keyboard focus to my WinForm application window using SetForegroundWindow Win32 API. But it seems it is not as simple as it sounds, Windows impose many restrictions on SetForegroundWindow API which are explained in detail in the MSDN page. Basically a non-foreground window can't set itself or another window as foreground window. But a foreground window can set another window as foreground window.

Apparently there are ways/hacks in which we can overcome this restriction and they are explained here, here and here. I tried the first solution mentioned in the post here and it worked fine. The solution is wrapped inside the following function, the hwnd is the handle of target window which want it to be foreground/active window.

void ActivateWindow(HWND hwnd)
    DWORD foregroundThread = ::GetWindowThreadProcessId(GetForegroundWindow(), NULL);
    DWORD thisThread = ::GetCurrentThreadId();

    if (foregroundThread != thisThread)
        BOOL isAttached = ::AttachThreadInput(foregroundThread, thisThread, TRUE);

        ::ShowWindow(hwnd, SW_SHOW);

        if (isAttached)
            ::AttachThreadInput(foregroundThread, thisThread, FALSE);

The hack involves temporarily attaching our thread as the active foreground window's input processing thread and trying to bring the our window to the top most. This works as we fake our window as the active foreground window by hooking its input processing mechanism and Windows thinks we are the active foreground window and wants to change the focus. After bring our application to the top most we revert the input processing mechanism to its original thread as our goal is achieved. Once the window becomes active window then full screen mode enabled without any problem.

I hope the above description of my experience and the code snippet helps you, cheers.