Monthly Archives: September 2014

Using // +build to switch between debug and release builds

Build tags are part of the conditional compilation system provided by the go tool. This is a quick post to discuss using build tags to selectively enable debug printing in a package.

This afternoon I merged a contribution to pkg/sftp which improved the packet encoding performance but introduced a bug where some packet types were incorrectly encoded.

% go test -integration
Unknown message 0
... oops

Turning on verbose got a little closer.

% go test -integration -v
=== RUN TestUnmarshalAttrs
--- PASS: TestUnmarshalAttrs (0.00s)
=== RUN TestNewClient
--- PASS: TestNewClient (0.00s)
=== RUN TestClientLstat
Unknown message 0

But each integration test sends many different packets, which one was at fault?

Rather than reaching for fmt.Println I took a few minutes to add conditional debugging to this package.
debug.go

// +build debug

package sftp

import "log"

func debug(fmt string, args ...interface{}) {
	log.Printf(fmt, args...)
}

release.go

// +build !debug

package sftp

func debug(fmt string, args ...interface{}) {}

Adding a call to debug inside sendPacket it was easy to figure out the packet which was being incorrectly encoded.

% go test -tags debug -integration -v -run=Lstat
2014/09/28 11:18:31 send packet sftp.sshFxInitPacket, len: 38
=== RUN TestClientLstat
2014/09/28 11:18:31 send packet sftp.sshFxInitPacket, len: 5
2014/09/28 11:18:31 send packet sftp.sshFxpLstatPacket, len: 62
Unknown message 0

Debugging is optional

When I committed the fix for this bug I didn’t have to spend any time removing the debug function calls inside the package.

When -tags debug is not present, the version from release.go, effectively a no-op, is used.

Extra credit

This package includes integration tests which are not run by default. How the -integration test flag works is left as an exercise to the reader.

go list, your Swiss army knife

During my day job I’ve been working on re-factoring some the internals of Juju to reverse the trend of a growing number of dependencies in our core packages.

In this post I’ll show how I used go list to help me in this task.

The basics

Let’s start with the basics of go list. The default invocation of go list returns the name of the import path that represents the directory you are currently in, or the package path you provide.

% cd $GOPATH/src/code.google.com/p/go.crypto/ssh
% go list
code.google.com/p/go.crypto/ssh
% go list github.com/juju/juju
github.com/juju/juju

By itself this doesn’t appear to be particularly notable, however this simple example belies the power of go list.

The secret of the -f flag

Tucked away at the top of the documentation for go help list is this short piece

The -f flag specifies an alternate format for the list, using the syntax of package template. The default output is equivalent to -f '{{.ImportPath}}'.

Put simply, -f allows you to execute a Go template which has access to the internal data structures of the go tool, same structures that are used when building, testing or getting code.

This example, using -f '{{ .ImportPath }}', is equivalent to the previous invocation and gives us a glimpse into the power of go list

% go list -f '{{ .ImportPath }}' github.com/juju/juju                          github.com/juju/juju

Going a step further with go list

The godoc for cmd/go lists the structures available to -f, so I won’t repeat them verbatim. Instead I’ll highlight some uses of go list.

Listing the test files that will be built

% go list -f '{{ .TestGoFiles }}' archive/tar
[reader_test.go tar_test.go writer_test.go]

Or the external test files of that package

% go list -f '{{ .XTestGoFiles }}' archive/tar
[example_test.go]

Or a summary for a set of packages

% go list -f '{{.Name}}: {{.Doc}}' code.google.com/p/go.net/ipv{4,6}
ipv4: Package ipv4 implements IP-level socket options for the Internet Protocol version 4.
ipv6: Package ipv6 implements IP-level socket options for the Internet Protocol version 6.

Conditional builds and build tags

A important part of writing Go programs is dealing with portability issues across various operating systems or processors. I’ve written about conditional builds before, so I’ll just show an example listing the files that will be compiled on various platforms

% env GOOS=darwin go list -f '{{ .GoFiles }}' github.com/pkg/term
[term.go term_bsd.go]
% env GOOS=linux go list -f '{{ .GoFiles }}' github.com/pkg/term
[term.go term_linux.go]

Who depends on what?

go list can show both the packages that your package directly depends, or its complete set of transitive dependencies.

% go list -f '{{ .Imports }}' github.com/davecheney/profile
[io/ioutil log os os/signal path/filepath runtime runtime/pprof]
% go list -f '{{ .Deps }}' github.com/davecheney/profile
[bufio bytes errors fmt io io/ioutil log math os os/signal path/filepath reflect runtime runtime/pprof sort strconv strings sync sync/atomic syscall text/tabwriter time unicode unicode/utf8 unsafe]

The first command lists only the packages that github.com/davecheney/profile depends on directly. This is the unique set of import statements in all Go files, adjusted for build constraints. The second command returns the set of direct and transitive dependency of github.com/davecheney/profile.

Fancy templating

The set of data structures available to the go list template is quite specialised, but don’t forget we have the whole power of the Go template package at our disposal.

In the previous examples dealing with slices of values, the default output format follows the fmt package. However it is probably more convenient for scripting applications to have one entry per line, which we can do easily as the go list template contains a function called join which delegates to strings.Join.

% go list -f '{{ join .Imports "\n" }}' github.com/davecheney/profile
io/ioutil
log
os
os/signal
path/filepath
runtime
runtime/pprof

Putting it together

With the task of trying to unpick the forest of dependencies in our core packages, I can use go list to define a helper like

deps() {
        go list -f '{{ join .Deps  "\n"}}' . | grep juju
}

The usage is as simple as navigating to a particular package in my $GOPATH and checking to see its complete dependency list

% deps
github.com/juju/cmd
github.com/juju/errgo
github.com/juju/errors
github.com/juju/gojsonpointer
github.com/juju/gojsonreference
github.com/juju/gojsonschema
github.com/juju/juju/api/agent
... // many more lines elided

Conclusion

This short post has barely scratched the surface of the flexibility that go list provides.

If your development, build, or CI tooling is using hand rolled scripts, grep, awk, etc, to introspect Go packages and their interdependencies, consider switching to go list.

How to install multiple versions of Go

Here is a short recipe I use for installing multiple versions of Go from source. In this example I’m going to install the release (currently Go 1.3.1) and trunk versions of Go.

Step 1. Checkout

Checkout two copies of the Go source code into independent paths.

% hg clone https://code.google.com/p/go -r release $HOME/go.release
% hg clone https://code.google.com/p/go $HOME/go.trunk

Step 2. Build

Build and run the tests for both versions.

% cd $HOME/go.release/src && ./all.bash
% cd $HOME/go.trunk/src && ./all.bash

Step 3. Done

That’s it, we’re done. You can now invoke whichever version of Go you want by invoking the go tool like so

% $HOME/go.release/bin/go test $YOURPACKAGE # test with the release version
% $HOME/go.trunk/bin/go test $YOURPACKAGE # test with the trunk version

If you want a particular version of Go to be your default, add that version’s bin directory to your $PATH

export PATH=$PATH:$HOME/go.release/bin:$GOPATH/bin

Look Ma, no $GOROOT!

You’ll notice that I didn’t set $GOROOT. You don’t need to set $GOROOT, ever1.

  1. Unless you’re using Windows, and have decided to not follow the instructions from the golang.org site. Please refer to this helpful infographic for full details.

Go’s runtime C to Go rewrite, by the numbers

The start of September brings a close to the Go 1.4 merge window. From now until the release in December, only bug fixes and tweaks will be accepted.

A major piece of work that landed in Go 1.4 cycle was the rewrite of parts of the Go runtime from C to Go. What may not be widely known is the main Go distribution1 includes a C compiler whose primary function is to compile the C code in the runtime package.

Rewriting parts of the runtime in this cycle provided the following benefits.

  • Currently if C code in the runtime is found on a Goroutine’s call stack the runtime will fall back to the old split stacks method if it needs to grow the stack. When all the parts of the runtime called from Go code are written in Go, the copying stack method can be used more effectively.
  • The translation of the Go compilers themselves from C2 to Go is only planned to convert the Go compiler, (5g, 6g, 8g), not the C compiler. Reducing the number of lines of C code in the runtime, possibly eliminating them entirely, will make the compiler rewrite simpler.

Here are the numbers.

The data captured here is not the total line count of C and Go in the runtime package, but the count of lines compiled as part of ./all.bash. In effect

go list -f "{{ range .GoFiles }}{{ . }} {{ end }}" | wc -l
go list -f "{{ range .CFiles }}{{ . }} {{ end }}" | wc -l

for each revision that included runtime: in the commit log.

Notes

  1. This is the golang.org distribution, commonly known as gc, not gccgo.
  2. This is a different C compiler; the gc toolchain is compiled with your system’s C compiler, the gc runtime is compiled with its C compiler.
  3. The increase in number of lines converted after August 20th is probably related to this request by khr.