Monthly Archives: December 2015

Are Go maps sensitive to data races ?

Panic messages from unexpected program crashes are often reported on the Go issue tracker. An overwhelming number of these panics are caused by data races, and an overwhelming number of those reports centre around Go’s built in map type.

unexpected fault address 0x0
fatal error: fault
[signal 0x7 code=0x80 addr=0x0 pc=0x40873b]

goroutine 97699 [running]:
runtime.throw(0x17f5cc0, 0x5)
    /usr/local/go/src/runtime/panic.go:527
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:21
runtime.mapassign1(0x12c6fe0, 0xc88283b998, 0xc8c9b63c68, 0xc8c9b63cd8)
    /usr/local/go/src/runtime/hashmap.go:446

Why is this so ? Why is a map commonly involved with a crash ? Is Go’s map implementation inherently fragile ?

To cut to the chase: no, there is nothing wrong with Go’s map implementation. But if there is nothing wrong with the implementation, why do maps and panic reports commonly find themselves in close proximity ?

There are three reasons that I can think of.

Maps are often used for shared state

Maps are fabulously useful data structures and this makes them perfect for tasks such as a shared cache of precomputed data or a lookup table of outstanding requests. The common theme here is the map is being used to store data shared across multiple goroutines.

Maps are more complex structures

Compared to the other built in data types like channels and slices, Go maps are more complex — they aren’t just views onto a backing array of elements. Go maps contain significant internal state, and map iterators (for k, v := range m) contain even more.

Go maps are not goroutine safe, you must use a sync.Mutex, sync.RWMutex or other memory barrier primitive to ensure reads and writes are properly synchronised. Getting your locking wrong will corrupt the internal structure of the map.

Maps move things

Of all of Go’s built in data structures, maps are the only ones that move data internally. When you insert or delete entries, the map may need to rebalance itself to retain its O(1) guarantee. This is why map values are not addressable.

Without proper synchronisation different CPUs will have different representations of the map’s internal structure in their caches. Although the language lawyers will tell you that a program with a data race exhibits undefined behaviour, it’s easy to see how having a stale copy of a map’s internal structure can lead to following a stale pointer to oblivion.

Please use the race detector

Go ships with a data race detector that works on Windows, Linux, FreeBSD and OSX. The race detector will spot this issue, and many more.

Please use it when testing your code.

How will you be programming in a decade ?

What does the computing landscape look like in a decade ?

In a word, bifurcated.

At the individual level there will be range of battery powered devices; watches, mobile phones, tablets with removable keyboards, and those without. They will be numerous, at a wide range of price points, allowing them to be dedicated to the individual. A personal computer if you will.

Of course these devices will have to always be connected to a network and the eponymous cloud, and thus the other half of the puzzle. If you think you’re going to be able to walk downstairs in a decade and touch the hardware your software runs on — you’re in for a rude shock.

What happened to the middle ?

Well, Steve Jobs blew up the desktop market, and with it the outlook for PC shipments.

Desktop PC shipments, 2010-2019

Desktop PC shipments, 2010-2019

But, but, I hear you say. You, the reader, might have a desktop computer for gaming, or enjoy software development on a workstation, rather than a laptop, or your phone.

That’s fine, nobody said you’re wrong, but you are increasingly a minority, and the economics of scale are not working in your favour.

What about us developers ?

Yongsan Electronics Market. Korea’s strategic reserve of whitebox desktop PCs stretches to the horizon.

So we know where the hardware is going, but a16z says software is eating the world. Who’s going to write all this software ? And if there are no desktop computers, how ?

Maybe, companies like Nitrous.io (now defunct) and Koding are right, and we’ll all be using online tools. In which case tablets with all day battery life and WiFi are the ticket — the market is certainly betting on that.

But I think there are serious and persistent problems with the idea of always on that cannot be fixed with money.

Broadband cellular or WiFi data has an upper limit, sure you can stack channels to make a single TCP flow go faster, but when everyone wants fast flows — and good upload bandwidth, will that scale to cities with tens of millions of individual ? Will it scale to people who don’t want to live in said Megalopolis ?

Probably not.

The other outcome is the developer PC continues to exist, in an increasingly rarified (and expensive) form as workstations migrate to the economies of scale that drive server chip sets.

Video editing suite

Video editing suite

This is what video editing looks like today, part PC, mostly custom packaged single use solution. Imagine what it would look like if this was what was required to produce software ?

How will you be programming in a decade ?

LISP Machine