Using // +build to switch between debug and release builds

Build tags are part of the conditional compilation system provided by the go tool. This is a quick post to discuss using build tags to selectively enable debug printing in a package.

This afternoon I merged a contribution to pkg/sftp which improved the packet encoding performance but introduced a bug where some packet types were incorrectly encoded.

% go test -integration
Unknown message 0
... oops

Turning on verbose got a little closer.

% go test -integration -v
=== RUN TestUnmarshalAttrs
--- PASS: TestUnmarshalAttrs (0.00s)
=== RUN TestNewClient
--- PASS: TestNewClient (0.00s)
=== RUN TestClientLstat
Unknown message 0

But each integration test sends many different packets, which one was at fault?

Rather than reaching for fmt.Println I took a few minutes to add conditional debugging to this package.

// +build debug

package sftp

import "log"

func debug(fmt string, args ...interface{}) {
	log.Printf(fmt, args...)


// +build !debug

package sftp

func debug(fmt string, args ...interface{}) {}

Adding a call to debug inside sendPacket it was easy to figure out the packet which was being incorrectly encoded.

% go test -tags debug -integration -v -run=Lstat
2014/09/28 11:18:31 send packet sftp.sshFxInitPacket, len: 38
=== RUN TestClientLstat
2014/09/28 11:18:31 send packet sftp.sshFxInitPacket, len: 5
2014/09/28 11:18:31 send packet sftp.sshFxpLstatPacket, len: 62
Unknown message 0

Debugging is optional

When I committed the fix for this bug I didn’t have to spend any time removing the debug function calls inside the package.

When -tags debug is not present, the version from release.go, effectively a no-op, is used.

Extra credit

This package includes integration tests which are not run by default. How the -integration test flag works is left as an exercise to the reader.

go list, your Swiss army knife

During my day job I’ve been working on re-factoring some the internals of Juju to reverse the trend of a growing number of dependencies in our core packages.

In this post I’ll show how I used go list to help me in this task.

The basics

Let’s start with the basics of go list. The default invocation of go list returns the name of the import path that represents the directory you are currently in, or the package path you provide.

% cd $GOPATH/src/
% go list
% go list

By itself this doesn’t appear to be particularly notable, however this simple example belies the power of go list.

The secret of the -f flag

Tucked away at the top of the documentation for go help list is this short piece

The -f flag specifies an alternate format for the list, using the syntax of package template. The default output is equivalent to -f '{{.ImportPath}}'.

Put simply, -f allows you to execute a Go template which has access to the internal data structures of the go tool, same structures that are used when building, testing or getting code.

This example, using -f '{{ .ImportPath }}', is equivalent to the previous invocation and gives us a glimpse into the power of go list

% go list -f '{{ .ImportPath }}'                

Going a step further with go list

The godoc for cmd/go lists the structures available to -f, so I won’t repeat them verbatim. Instead I’ll highlight some uses of go list.

Listing the test files that will be built

% go list -f '{{ .TestGoFiles }}' archive/tar
[reader_test.go tar_test.go writer_test.go]

Or the external test files of that package

% go list -f '{{ .XTestGoFiles }}' archive/tar

Or a summary for a set of packages

% go list -f '{{.Name}}: {{.Doc}}'{4,6}
ipv4: Package ipv4 implements IP-level socket options for the Internet Protocol version 4.
ipv6: Package ipv6 implements IP-level socket options for the Internet Protocol version 6.

Conditional builds and build tags

A important part of writing Go programs is dealing with portability issues across various operating systems or processors. I’ve written about conditional builds before, so I’ll just show an example listing the files that will be compiled on various platforms

% env GOOS=darwin go list -f '{{ .GoFiles }}'
[term.go term_bsd.go]
% env GOOS=linux go list -f '{{ .GoFiles }}'
[term.go term_linux.go]

Who depends on what?

go list can show both the packages that your package directly depends, or its complete set of transitive dependencies.

% go list -f '{{ .Imports }}'
[io/ioutil log os os/signal path/filepath runtime runtime/pprof]
% go list -f '{{ .Deps }}'
[bufio bytes errors fmt io io/ioutil log math os os/signal path/filepath reflect runtime runtime/pprof sort strconv strings sync sync/atomic syscall text/tabwriter time unicode unicode/utf8 unsafe]

The first command lists only the packages that depends on directly. This is the unique set of import statements in all Go files, adjusted for build constraints. The second command returns the set of direct and transitive dependency of

Fancy templating

The set of data structures available to the go list template is quite specialised, but don’t forget we have the whole power of the Go template package at our disposal.

In the previous examples dealing with slices of values, the default output format follows the fmt package. However it is probably more convenient for scripting applications to have one entry per line, which we can do easily as the go list template contains a function called join which delegates to strings.Join.

% go list -f '{{ join .Imports "\n" }}'

Putting it together

With the task of trying to unpick the forest of dependencies in our core packages, I can use go list to define a helper like

deps() {
        go list -f '{{ join .Deps  "\n"}}' . | grep juju

The usage is as simple as navigating to a particular package in my $GOPATH and checking to see its complete dependency list

% deps
... // many more lines elided


This short post has barely scratched the surface of the flexibility that go list provides.

If your development, build, or CI tooling is using hand rolled scripts, grep, awk, etc, to introspect Go packages and their interdependencies, consider switching to go list.

How to install multiple versions of Go

Here is a short recipe I use for installing multiple versions of Go from source. In this example I’m going to install the release (currently Go 1.3.1) and trunk versions of Go.

Step 1. Checkout

Checkout two copies of the Go source code into independent paths.

% hg clone -r release $HOME/go.release
% hg clone $HOME/go.trunk

Step 2. Build

Build and run the tests for both versions.

% cd $HOME/go.release/src && ./all.bash
% cd $HOME/go.trunk/src && ./all.bash

Step 3. Done

That’s it, we’re done. You can now invoke whichever version of Go you want by invoking the go tool like so

% $HOME/go.release/bin/go test $YOURPACKAGE # test with the release version
% $HOME/go.trunk/bin/go test $YOURPACKAGE # test with the trunk version

If you want a particular version of Go to be your default, add that version’s bin directory to your $PATH

export PATH=$PATH:$HOME/go.release/bin:$GOPATH/bin

Look Ma, no $GOROOT!

You’ll notice that I didn’t set $GOROOT. You don’t need to set $GOROOT, ever1.

  1. Unless you’re using Windows, and have decided to not follow the instructions from the site. Please refer to this helpful infographic for full details.

Go’s runtime C to Go rewrite, by the numbers

The start of September brings a close to the Go 1.4 merge window. From now until the release in December, only bug fixes and tweaks will be accepted.

A major piece of work that landed in Go 1.4 cycle was the rewrite of parts of the Go runtime from C to Go. What may not be widely known is the main Go distribution1 includes a C compiler whose primary function is to compile the C code in the runtime package.

Rewriting parts of the runtime in this cycle provided the following benefits.

  • Currently if C code in the runtime is found on a Goroutine’s call stack the runtime will fall back to the old split stacks method if it needs to grow the stack. When all the parts of the runtime called from Go code are written in Go, the copying stack method can be used more effectively.
  • The translation of the Go compilers themselves from C2 to Go is only planned to convert the Go compiler, (5g, 6g, 8g), not the C compiler. Reducing the number of lines of C code in the runtime, possibly eliminating them entirely, will make the compiler rewrite simpler.

Here are the numbers.

The data captured here is not the total line count of C and Go in the runtime package, but the count of lines compiled as part of ./all.bash. In effect

go list -f "{{ range .GoFiles }}{{ . }} {{ end }}" | wc -l
go list -f "{{ range .CFiles }}{{ . }} {{ end }}" | wc -l

for each revision that included runtime: in the commit log.


  1. This is the distribution, commonly known as gc, not gccgo.
  2. This is a different C compiler; the gc toolchain is compiled with your system’s C compiler, the gc runtime is compiled with its C compiler.
  3. The increase in number of lines converted after August 20th is probably related to this request by khr.

Go has both make and new functions, what gives ?

This is a post about Go’s built in make and new functions.

As Rob Pike noted at Gophercon this year, Go has many ways of initialising variables. Among them is the ability to take the address of a struct literal which leads to serveral ways to do the same thing.

s := &SomeStruct{}
v := SomeStruct{}
s := &v              // identical
s := new(SomeStruct) // also identical

It is fair that commenters point out this redundancy in the language and this sometimes leads them to search for other inconsistencies, most notably the redundancy between make and new.

On the surface it appears that make and new do very similar things, so what is the rationale for having both ?

Why can’t we use make for everything ?

Go does not have user defined generic types, but it does have several built in types that can operate as generic lists, maps, sets, and queues;  slices, maps and channels.

Because make is designed to create these three built in generic types, it must be provided by the runtime as there is no way to express the function signature of make directly in Go.

Although make creates generic slice, map, and channel values, they are still just regular values; make does not return pointer values.

If new was removed in favour make, how would you construct a pointer to an initialised value ?

var x1 *int
var x2 = new(int)

x1 and x2 have the same type, *intx2 points to initialised memory and may be safely dereferenced, the same is not true for x1.

Why can’t we use new for everything ?

Although the use of new is rare, its behaviour is well specified.

new(T) always returns a *T pointing to an initialised T. As Go doesn’t have constructors, the value will be initialised to T‘s zero value.

Using new to construct a pointer to a slice, map, or channel zero value works today and is consistent with the behaviour of new.

s := new([]string)
fmt.Println(len(*s))  // 0
fmt.Println(*s == nil) // true

m := new(map[string]int)
fmt.Println(m == nil) // false
fmt.Println(*m == nil) // true

c := new(chan int)
fmt.Println(c == nil) // false
fmt.Println(*c == nil) // true

Sure, but these are just rules, we can change them, right ?

For the confusion they may cause, make and new are consistent; make only makes slices, maps, and channels, new only returns pointers to initialised memory.

Yes, new could be extended to operate like make for slices, maps and channels, but that would introduce its own inconsistencies.

  1. new would have special behaviour if the type passed to new was a slice, map or channel. This is a rule that every Go programmer would have to remember.
  2. For slices and channels, new would have to become variadic, taking a possible length, buffer size, or capacity, as required. Again more special cases to have to remember, whereas before new took exactly one argument, the type.
  3. new always returns a *T for the T passed to it. That would mean code like
    func Read(buf []byte) []byte
    // assume new takes an optional length
    buf := Read(new([]byte, 4096))

    would no longer be possible, requiring more special cases in the grammar to permit *new([]byte, length).

In summary

make and new do different things.

If you are coming from another language, especially one that uses constructors, it may appear that new should be all you need, but Go is not those languages, nor does it have constructors.

My advice is to use new sparingly, there are almost always easier or cleaner ways to write your program without it.

As a code reviewer, the use of new, like the use of named return arguments, is a signal that the code is trying to do something clever and I need to pay special attention. It may be that code really is clever, but more than likely, it can be rewritten to be clearer and more idiomatic.

Tinyterm: A silly terminal emulator written in Go

This post is about Tinyterm, a silly hack that I presented as a lightning talk at last month’s Sydney Go User group 1. You can find the original slides online at

Screenshot from 2014-08-03 14:22:43

This talk is about a experiment to see if I could drive I2C devices from Go through my laptop’s VGA port. It was inspired by a recent post on Hack-a-Day.

Screenshot from 2014-08-03 14:23:26

There are several parts to this presentation. There is some Go in here, trusty me.

Screenshot from 2014-08-03 14:24:03

The first piece of the puzzle is the I2C bus.

The I2C bus is a low speed two wire serial bus mainly used for connecting sensors and microcontrollers together.

But, you don’t even need a microcontroller to use I2C. If you’re patient you can bit bang the protocol using a few resistors and tack switches.

Screenshot from 2014-08-03 14:24:42

I2C isn’t just used on microcontrollers like the Arduino. It’s has been used inside every PC and laptop for decades as a slow speed serial protocol for interfacing with simple devices like temperature sensors.

If you’ve used the lmsensors package in Linux, or have heard of SMBus, this is basically a variant of I2C.

Importantly, I2C is also used as the protocol to detect an external monitor, where it goes under the name DDC2b.


i2C pins are available on VGA, DVI and HDMI connectors. Source

Screenshot from 2014-08-03 13:37:06

Talking to I2C devices is as simple as installing a kernel module which will create devices entries in your /dev/ directory.

% ls /dev/i2c*
/dev/i2c-0  /dev/i2c-1  /dev/i2c-2  /dev/i2c-3  /dev/i2c-4 
/dev/i2c-5  /dev/i2c-6  /dev/i2c-7  /dev/i2c-8

Screenshot from 2014-08-03 13:39:18

Each device on the I2C bus has a unique address. You can use the i2cdetect command (part of the i2c-utils package on Ubuntu) to scan the bus.

In this example, the device responding at 0x50 is my laptop’s internal LCD screen. That device is an EEPROM which holds specifications of the screen.

After a bit of reverse engineering of the hack a day post, and a quick trip to Jaycar for parts I came up with this simple adapter

i2c adapter mark I

I2C adapter mark I

The adapter just breaks out pins 5, 9, 12, and 15 to the Dupont patch cables. Using a logic analyser I verified that pins 12 and 15 looked like I2C data when I ran i2cdetect.

i2c adapter talking to an i2c io  expander

I2C adapter talking to an I2C IO expander

The next step was to connect up a real I2C device to the bus and see if I could detect it with i2cdetect.

Although both the LCD and the laptop are 5 volt devices I wasn’t sure how much current the laptop could source on pin 9, so I opted to buffer the devices using a Freetronics level shifter which effectively isolates the laptop from the high current LED backlight on the LCD panel.

Screenshot from 2014-08-03 13:56:10

Now the hardware was done, it was time to write some code. Driving an I2C device from userspace in Go is pretty straight forward; open the device, then use an ioctl to tell the kernel to bind the file descriptor to a remote I2C device.

Screenshot from 2014-08-03 13:58:55

The LCD I was using is based on the Hitachi HD44780 standard which has a baroque protocol using many pins and is completely incompatible with I2C.

To interface between the HD44780 I’m using a cheap PCF8574 I2C IO expander which takes any byte received over I2C and maps it directly to its output pins.

I adapted some Python code to work with my Go I2C type which gave me a set of LCD primitives to work with.

Screenshot from 2014-08-03 14:04:51

So now I can drive the output of the LCD with Go. Here is an example



Screenshot from 2014-08-03 14:07:20

But this was kind of boring, could I do something more interesting ?

Looking back through this project it occurred to me that the recurring theme was, in the best UNIX tradition, everything is a file.

  • I2C buses are visible in userspace as files
  • Each I2C device is a file descriptor, once opened and programmed by ioctl
  • UNIX processes talk to each other over file descriptors
  • In Go, that is basically an io.Writer, right ?

So, could I connect a UNIX process’s output to the LCD screen transparently ?

Screenshot from 2014-08-03 14:11:28

Enter Tinyterm, a simple Go program that does just that.

Using an lcdWriter type (more on that in the next slide), Tinyterm spawns a child process and redirects Stdout and Stderr to the LCD.

Screenshot from 2014-08-03 14:15:43

The lcdWriter‘s Write method has a little bit of smarts to deal with making the LCD look like a 16 x 4 terminal, rather than a linear stream of characters, handles scrolling the screen, and obscures the odd addressing scheme of the video memory inside the HD44780.

Screenshot from 2014-08-03 14:17:37

Putting it all together we have the tinyterm command, which runs its arguments as a subprocess, sending the child’s stdout and stderr to the LCD. Stdin is not redirected, so it takes input from the original terminal device, eventually mapping back to my keyboard.

Tinyterm example, hopefully a video will be available soon.

Tinyterm example, hopefully a video will be available soon.

Screenshot from 2014-08-03 14:20:00

The code for the i2c and lcd types is on github,, along with the helloworld and tinyterm example programs.

1 The talk was recorded but it is not clear if the recording worked, I will update this post if/when the video is available.

Arduino SPI woes

A few months ago I upgraded the hardware my avr11 project ran on from the atmega2560 8bit micro to the SAM3x based Arduino Due. In doing so I lost access to the excellent QuadRAM memory expansion board, and had to find another solution for accessing the micro SD card.

For the moment, I’ve decided to go back to my SPI based SRAM shield that I built previously and this means I need to hook both the SPI SRAM shield and a Sparkfun micro SD card shield up to the Arduino Due.

Sparkfun micro SD card shield

Sparkfun micro SD card shield, no ICSP header connector.

This brings me to the topic for this post; why do Arduino keep moving the SPI pins!

In the beginning there was the Arduino Uno form factor, SPI was available on both pins 11, 12 and 13 as well as the dedicated ICSP header.

Arduino Uno

SPI is available on pins 11, 12 and 13, as well as the ICSP header.

Then the Arduino mega platform came out, with the Atmel 2560 chipset and the larger shield sizes.

Freetronics Ethermega 2560

SPI has moved to pins 51, 52 and 53, as well as the ICSP header.

SPI is no longer available on pins 11, 12 and 13, but has moved to pins 51, 52 and 32. It remains available on the ICSP header, which is the area that Arduino is pushing shield makers to use. Unfortunately shield makers are steadfastly ignoring the recommendations from Arduino and none of the SD card shields I ca find have a connector to route the ICSP header upwards as you add additional shields. The blanks shields from Freetronics don’t even make it an option.

This brings me to the Arduino Due, which I needed to get the grunt to run my avr11 simulator.


SPI available on the ICSP header, only.

To use SPI on the Due I need to somehow route the ISCP connector to pins 11, 12 and 13.



The best solution I had at the time was to raise the shield away from the Due using stacked headers, then route the ISCP signals to the pins that the board (and the SDFat software) expected to find them with some jumper cables. Out of shot, pins 11, 12 and 13 were bent upwards so they did not make contact with the sockets on the Due board.

This was where the project stalled for a few months.

Recently I’ve had some time to come back to this project, and the first order of business was solving the SPI problem. It was clear that pins 11, 12 and 13 were the rightful place for the SPI signals and to try to route them anywhere else would be fruitless. So, with an official Arduino expansion shield in hand, I made myself an SPI adapter board.

ICSP adapter

Pins 11, 12 and 13 are removed, but still connect to the stacked header on the opposite side of the board.

The board is very simple, all the usual Arduino Uno pins are passed through as expected, however pins 11, 12 and 13 are routed to the ICSP header to match the Arduino documentation.

icsp top view

Adapter board mounted on an Arduino Due.

Here is a picture of the shield mounted on the Due. The trace for pin 11 is run on this side of the board to avoid crossing pin 13. I felt this was important as SPI can run upwards of 16 Mhz, however I’m not sure how much improvement this will make as these traces are still long and unshielded.


The final result, more compact and much more stable.

Here is a shot of result. The SD shield is mounted without floating pins or jumper wires and additional shields can be mounted on top of the SD card shield with the original locations of the SPI pins respected.

Visualising the Go garbage collector

Update this post is also available in Japanese.

This is a post about an experimental tool that I have been working on.

gcvis is a simple way of visualising the operation of the garbage collector within a Go process. Here is a screenshot of it in operation.
The rest of this article explores how gcvis works and how to interpret its results.

How does gcvis get the data ?

There are a few ways you can interrogate a Go program.

You could use the built in profiler, via the net/http/pprof package, or my profile package. However this means modifying the source of the program, which sometimes may not be an option.

There is another source of telemetry data built into every Go program which is accessible by setting the following environment variable.


(The GODEBUG environment variable is documented in the runtime package).

When your program is started with this environment variable set, the following additional output will be printed to standard out (slightly abridged)

 % env GODEBUG=gctrace=1 godoc -http=:6060
gc76(1): 2+1+1390+1 us, 1 -> 3 MB, 16397 (1015746-999349) objects, 1436/1/0 sweeps, 0(0) handoff, 0(0) steal, 0/0/0 yields
gc77(1): 2+0+1582+1 us, 2 -> 4 MB, 14623 (1016248-1001625) objects, 1436/0/0 sweeps, 0(0) handoff, 0(0) steal, 0/0/0 yields
scvg0: inuse: 6, idle: 15, sys: 22, released: 0, consumed: 22 (MB)
scvg1: inuse: 6, idle: 15, sys: 22, released: 0, consumed: 22 (MB)
gc78(1): 5+1+4814+1 us, 2 -> 2 MB, 21076 (1023168-1002092) objects, 1436/25/0 sweeps, 0(0) handoff, 0(0) steal, 0/0/0 yields
scvg2: GC forced
scvg2: inuse: 6, idle: 15, sys: 22, released: 0, consumed: 22 (MB)

The two types of information presented are

  • A line for every garbage collection cycle, indicated by the gc prefix.
  • A set of lines for the operation of the scavenger, indicated by the scvg prefix, which is responsible for returning unused portions of the heap to the operating system.

In the next section I will discuss using, and interpreting the data from, gcvis.

Using gcvis

To use gcvis, place it in front of the Go program you want to inspect, as you would time or nice.

Here is an example of using gcvis with godoc in indexing mode (so it uses lots of memory and cpu time, generating interesting data).

% gcvis godoc -index -http=:6060
2014/07/11 16:29:12 opening browser window, if this fails, navigate to
Created new window in existing browser session.

That’s it.

gcvis takes care of setting the appropriate value of GODEBUG and filtering out the additional information generated. gcvis also tries to open a browser window to view the visualisation. This functionality is provided by pkg/browser and is somewhat operating system dependent.

Because gcvis is recording the gc debug lines in real time, it can add timestamp information to them, a feature which is currently missing from that raw GODEBUG output.

Screenshot from 2014-07-11 16:35:09
In this example you can see the frequency of gc cycles decrease as the heap grows.

The main use of the gc debug data is to record the size of the live objects on the heap, however this doesn’t reveal the total size of the heap, nor what percentage of the heap the live set represents. For that we need to add the debugging information from the scavenger.

The scavenger runs on a timer, currently every two minutes, so will only start to report its data to gcviz a few minutes after the program starts. Here is an example after running for about 15 minutes.

Screenshot from 2014-07-11 17:01:31
Some interesting points to note in this graph are

  • scvg.sys represents the total amount of memory requested from the operating system, this is roughly analogous to the VSS value reported by tools like top.
  • scvg.inuse is the amount of memory in use by the whole heap, which may include dead objects. scvg.inuse and gc.heapinuse may not track each other exactly as they are reported at different times.
  • scvg.idle represents memory that is currently unused by the garbage collector, that is, used to contain dead objects, but is now unused after garbage collection.
  • When the scavenger runs, scvg.idle grows as scvg.inuse shrinks.
  • If memory remains idle for long enough the scavenger will inform the operating system that it is no longer needed, this is reported by scvg.released and matches a drop in scvg.consumedThe operating system is free to ignore this request, and frequently does.


The code is open source on Github, so go get it and try it on your application.

go get -u -v

I’m very keen to hear from other Go users if gcvis is useful for you. Pull requests and bug reports are also most welcome.

A special thanks to Damian Gryski, Matthew Holt, and Bill Kennedy, for their suggestions and feedback.

Paranormal trivia

Like all children who grew up in the 80’s, I was, and still am a huge fan of Ghostbusters.

While recently re watching Ivan Reitman’s homage to New York, I spotted something which has gone unnoticed on the IMDB trivia page.

Shortly after being evicted from the University, Ray Stantz (Aykroyd) and Peter Venkman (Murray) are commiserating their fall from grace, when Stantz opines:


In which movie did Dan Aykroyd star a year before Ghostbusters ? Continue reading