Saturday, April 09, 2016

The danger in Golang infinite for-loop

This is a blog post about how my colleagues and I got bitten by simple looking infinite/forever/endless for-loop in Golang.

We were implementing a server app that waits for a client, receives data and stores it in DB. As simple as it sounds. The implementation goes like this

...
func main() {
    s := Server{Handler: handler}
    server.Start()
    for {
    }
}

func handler(c *Client){
    for {
        buf := make([]byte, 1024*1024)
        n, err := c.Read(buf)
        if err != nil {
            break;
        }
        // process buf
    }
}

The infinite for-loop in main function is used to prevent it getting terminated. I know it is a busy waiting and keeps the CPU active but we didn't give much thought about it.

The problem with above implementation is, the server freezes after receiving N buffers from the client. Initially we suspected the client, network, buffered IO but after adding some logs we realized that the program freezes when it tries to allocate the buffer. Yes, at the call to "make([]byte, ...)". Don't believe me, try the following snippet and see it yourself.

func main() {
    go work()

    for {
    }
}

func work() {
    println("work started")
    var r int64
    for i := 0; i < 1000; i++ {
        buf := make([]byte, 1024*1024)
        r = r + int64(buf[0]) // just to make sure the allocation is not optimized away
    }
    println("work completed")
}

The program will freeze after printing "work started" and it will never print "work completed". You have to terminate it forcefully. I tried with "GOMAXPROCS > 1" but no luck. I don't know the exact reason for this behaviour but I have a guess. I suppose the garbage collector(GC) kicks in after N number of allocations(i.e. make) and it tries to pause the runtime. As the runtime is busy with the infinite for-loop the GC couldn't pause runtime and the program freezes. We can avoid the freeze if we change the program in such way that the runtime can pause the infinite for-loop for the GC. There are different ways we can do this:
  • Replacing the infinite for-loop with infinite select loop. Apparently this seems to be the recommended way of blocking execution in Golang. This also keeps your CPU utilization low.
  • Sleeping within the for-loop. The sleep duration could be as small as zero.
  • Yielding the execution to runtime by calling runtime.Gosched()
I've observed the above behaviour with Go 1.5.1 on OSX. I am not sure whether it changes in different version or platform.

4 comments:

oldman69 said...

How come such a crazy design. You are really scaring me. Golang works as design.

oldman69 said...

How come such a crazy design. You are really scaring me. Golang works as design.

Anonymous said...

Relevant discussion https://github.com/golang/go/issues/11462

Yuri Volkov said...

func main() {
status := work()
<-status
}

func work() chan bool {
status := make(chan bool)

go func() {
var r int64
for i := 0; i < 1000; i++ {
buf := make([]byte, 1024*1024)
r = r + int64(buf[0]) // just to make sure the allocation is not optimized away
}
status <- true
}()
return status
}