Category Archives: Programming

Evaluation order oddity

At Canonical we’re increasingly invested in gccgo. While testing various packages built with gccgo we ran across test failures which we traced to an innocent looking piece of code.

package main

import "fmt"

type T struct {
        i int
}

func (t *T) readInt32() int32 {
        t.i += 4
        return 42 // not important
}

func main() {
        var d = T{i:200}
        end := d.i - 4 + int(d.readInt32())
        fmt.Println(end)
}

So, knowing that the Go spec defines the order of evaluation of functions calls and assignments as left to right, let’s try to predict what this code will print.

200 - 4 + 42 = 238

So, is 238 the correct answer ?

% go run odd.go 
242

Wow, that is odd. Let’s see what gccgo thinks the answer is

% go run -compiler gccgo odd.go 
238

Well, at least it got the right answer, so is gc at fault or are the arguments actually evaluated in the opposite direction ?

It turns out that both compilers are correct because the evaluation order of this sort of expression is not specified. In fact, in the spec, this case is called out in an example which is strongly reminiscent of the original code.

a := 1
f := func() int { a = 2; return 3 }
x := []int{a, f()}  // x may be [1, 3] or [2, 3]: evaluation order between a and f() is not specified

The correct solution to this problem is to break the expression into two lines, removing the ambiguity about when the addition to d.i is visible.

func main() {
        var d = T{i:200}
        v := int(d.readInt32())
        end := v + d.i - 4
        fmt.Println(end) 	// prints 242
}

More simple test coverage in Go 1.2

In a previous post I blogged about the cover tool coming in Go 1.2 and a bash helper function I use to make the tool a little easier to use.

Since then I’ve extended these helpers so I wanted to blog about the improvements.

Passing arguments to the testing tool

cover () {
  t=$(tempfile)
  go test $COVERFLAGS -coverprofile=$t $@ && go tool cover -func=$t && unlink $t
}

The first improvement is to be able to pass arguments to go test, this is done by setting COVERFLAGS before invoking the cover helper.

As an example, I have defined a flag in the sftp package tests that allow me to switch the integration tests on and off.

% cover github.com/pkg/sftp
PASS
coverage: 20.4% of statements
...
 % COVERFLAGS=-integration cover github.com/pkg/sftp 
PASS
coverage: 69.2% of statements
...

Viewing coverage details

The cover in its default mode will give you a percentage coverage on a per function basis, but it doesn’t tell you which parts of the function are uncovered.

cover-web() {
  t=$(tempfile)
  go test $COVERFLAGS -coverprofile=$t $@ && go tool cover -html=$t && unlink $t
}

To explore the coverage report in detail, this helper runs the cover tool then opens a web browser with a html version of the coverage report.

Screenshot from 2013-11-14 15:50:59Enjoy!

Stupid Go declaration tricks

This is a brief post highlighting a curious aspect of the declaration syntax in Go.

Most Go programmers know that the following of import declarations are equivalent

import "fmt"
import "math/big"

// same as
import (
        "fmt"
        "math/big"
)

The same applies to const declarations

const FORTYTWO = 42
const TRUE = 1

// same as
const (
        FORTYTWO = 42
        TRUE     = 1
)

But perhaps you didn’t know that the syntax is also valid for type declarations. For example, the following is valid syntax in Go.

type (
        B struct{ a, b int }
        C interface {
                Hello() error
        }
)

However because the func declaration has been extended to place methods on types, the following is not valid syntax

func (
        main() {
                fmt.Println("wowzers")
        }
)

Subcommand handling in Go

A few days ago I was working on an example program for the sftp package and found I needed to implement subcommand handling.

https://twitter.com/davecheney/status/397686194462396416

The response was fantastic, no less than 12 different packages. I haven’t had the chance to review any of the packages in detail, but I wanted to list them here as thanks to the great #golang community,

https://twitter.com/bketelsen/status/398152055107624961

Calling for autobench contributions for Go 1.2

With the release of go1.2rc3 last week I have now merged the autobench-next branch into master in the autobench repository.

Go 1.2 is not expected to bring performance improvements of the same magnitude of Go 1.1, but moderate improvements are expected due to improvements in code generation, the runtime, the garbage collector, and the standard library.

If you would like to contribute a benchmark result from your own machine the instructions for doing so are included the README file.

How does the go build command work ?

This post explains how the Go build process works using examples from Go’s standard library.

The gc toolchain

This article focuses on the gc toolchain. The gc toolchain takes its name for the Go compiler frontend, cmd/gc, and is mainly used to distinguish it from the gccgo toolchain. When people talk about the Go compilers, they are likely referring to the gc toolchain. This article won’t be focusing on the gccgo toolchain.

The gc toolchain is a direct descendant of the Plan 9 toolchain. The toolchain consists of a Go compiler, a C compiler, an assembler and a linker. Each of these tools are found in the src/cmd/ subdirectory of the Go source and consist of a frontend, which is shared across all implementations, and a backend which is specific to the processor architecture. The backends are distinguished by their letter, which is again a Plan 9 tradition. The commands are:

  • 5g, 6g, and 8g are the compilers for .go files for arm, amd64 and 386
  • 5c, 6c, and 8c are the compilers for .c files for arm, amd64 and 386
  • 5a, 6a, and 8a are the assemblers for .s files for arm, and64 and 386
  • 5l, 6l, and 8l are the linkers for files produced by the commands above, again for arm, amd64 and 386.

It should be noted that each of these commands can be compiled on any supported platform and this forms the basis of Go’s cross compilation abilities. You can read more about cross compilation in this article.

Building packages

Building a Go package involves at least two steps, compiling the .go files then packing the results into an archive. In this example I’m going to use crypto/hmac as it is a small package, only one source and one test file. Using the -x option I’ve asked go build to print out every step as it executes them

% go build -x crypto/hmac
WORK=/tmp/go-build249279931
mkdir -p $WORK/crypto/hmac/_obj/
mkdir -p $WORK/crypto/
cd /home/dfc/go/src/pkg/crypto/hmac
/home/dfc/go/pkg/tool/linux_arm/5g -o $WORK/crypto/hmac/_obj/_go_.5 -p crypto/hmac -complete -D _/home/dfc/go/src/pkg/crypto/hmac -I $WORK ./hmac.go
/home/dfc/go/pkg/tool/linux_arm/pack grcP $WORK $WORK/crypto/hmac.a $WORK/crypto/hmac/_obj/_go_.5

Stepping through each of these steps

WORK=/tmp/go-build249279931
mkdir -p $WORK/crypto/hmac/_obj/
mkdir -p $WORK/crypto/

go build creates a temporary directory, /tmp/go-build249279931 and populates it with some skeleton subdirectories to hold the results of the compilation. The second mkdir may be redundant, issue 6538 has been created to track this.

cd /home/dfc/go/src/pkg/crypto/hmac
/home/dfc/go/pkg/tool/linux_arm/5g -o $WORK/crypto/hmac/_obj/_go_.5 -p crypto/hmac -complete -D _/home/dfc/go/src/pkg/crypto/hmac -I $WORK ./hmac.go

The go tool switches the the source directory of crypto/hmac and invokes the go compiler for this architecture, in this case 5g. In reality there is no cd, /home/dfc/go/src/pkg/crypto/hmac is the supplied as the exec.Command.Dir field when 5g is executed. This means the .go source files can be relative to their source directory, making the command line shorter.

The compiler produces a single temporary output file in $WORK/crypto/hmac/_obj/_go_.5 which will be used in the final step.

/home/dfc/go/pkg/tool/linux_arm/pack grcP $WORK $WORK/crypto/hmac.a $WORK/crypto/hmac/_obj/_go_.5

The final step is to pack the object file into an archive file, .a, which the linker and the compiler consume.

Because we invoked go build on a package, the result is discarded as $WORK is deleted after the build completes. If we invoke go install -x two additional lines appear in the output

mkdir -p /home/dfc/go/pkg/linux_arm/crypto/
cp $WORK/crypto/hmac.a /home/dfc/go/pkg/linux_arm/crypto/hmac.a

This demonstrates the difference between go build and install; build builds, install builds then installs the result to be used by other builds.

Building more complex packages

You may be wondering what the pack step in the previous example does. As the compiler and linker only accept a single file representing the contents of the package, if a package contains multiple object files, they must be packed into a single .a archive before they can be used.

A common example of a package producing more than one intermediary object file is cgo, but that is too complicated for this article, instead a simpler example is a package that contains some .s assembly files, like crypto/md5.

% go build -x crypto/md5
WORK=/tmp/go-build870993883
mkdir -p $WORK/crypto/md5/_obj/
mkdir -p $WORK/crypto/
cd /home/dfc/go/src/pkg/crypto/md5
/home/dfc/go/pkg/tool/linux_amd64/6g -o $WORK/crypto/md5/_obj/_go_.6 -p crypto/md5 -D _/home/dfc/go/src/pkg/crypto/md5 -I $WORK ./md5.go ./md5block_decl.go
/home/dfc/go/pkg/tool/linux_amd64/6a -I $WORK/crypto/md5/_obj/ -o $WORK/crypto/md5/_obj/md5block_amd64.6 -D GOOS_linux -D GOARCH_amd64 ./md5block_amd64.s
/home/dfc/go/pkg/tool/linux_amd64/pack grcP $WORK $WORK/crypto/md5.a $WORK/crypto/md5/_obj/_go_.6 $WORK/crypto/md5/_obj/md5block_amd64.6

In this example, executed on a linux/amd64 host, 6g is invoked to compile two .go files, md5.go and md5block_decl.go. The latter contains the forward declaration for the functions implemented in assembly.

6a is then invoked to assemble md5block_amd64.s. The logic for choosing which .s to compile is described in my previous article on conditional compilation.

Finally pack is invoked to pack the Go object file, _go_.6, and the assembly object file, md5block_amd64.6, into a single archive.

Building commands

A Go command is a package who’s name is main. Main packages, or commands, are compiled just like other packages, but then undergo several additional steps to be linked into final executable. Let’s investigate this process with cmd/gofmt

% go build -x cmd/gofmt
WORK=/tmp/go-build979246884
mkdir -p $WORK/cmd/gofmt/_obj/
mkdir -p $WORK/cmd/gofmt/_obj/exe/
cd /home/dfc/go/src/cmd/gofmt
/home/dfc/go/pkg/tool/linux_amd64/6g -o $WORK/cmd/gofmt/_obj/_go_.6 -p cmd/gofmt -complete -D _/home/dfc/go/src/cmd/gofmt -I $WORK ./doc.go ./gofmt.go ./rewrite.go ./simplify.go
/home/dfc/go/pkg/tool/linux_amd64/pack grcP $WORK $WORK/cmd/gofmt.a $WORK/cmd/gofmt/_obj/_go_.6
cd .
/home/dfc/go/pkg/tool/linux_amd64/6l -o $WORK/cmd/gofmt/_obj/exe/a.out -L $WORK $WORK/cmd/gofmt.a
cp $WORK/cmd/gofmt/_obj/exe/a.out gofmt

The first six lines should be familiar, main packages are compiled like any other Go package, they are even packed like any other package.

The difference is the penultimate line, which invokes the linker to produce a binary executable.

/home/dfc/go/pkg/tool/linux_amd64/6l -o $WORK/cmd/gofmt/_obj/exe/a.out -L $WORK $WORK/cmd/gofmt.a

The final line copies and renames the completed binary to its final name. If you had used go install the binary would be copied to $GOPATH/bin (or $GOBIN if set).

A little history

If you go far enough back in time, back before the go tool, back to the time of Makefiles, you can still find the core of the Go compilation process. This example is taken from the release.r60 documentation

$ cat >hello.go <<EOF
package main

import "fmt"

func main() {
        fmt.Printf("hello, world\n")
}
EOF
$ 6g hello.go
$ 6l hello.6
$ ./6.out
hello, world

It’s all here, 6g compiling a .go file into a .6 object file, 6l linking the object file against the fmt (and runtime) packages to produce a binary, 6.out.

Wrapping up

In this post we’ve talked about how go build works and touched on how go install differs in its treatment of the compilation result.

Now that you know how go build works, and how to investigate the build process with -x, try passing that flag to go test and observe the result.

Additionally, if you have gccgo installed on your system, you can pass -compiler gccgo to go build, and using -x investigate how Go code is built using this compiler.

Why I think Go package management is important

One of the stated goals of Go was to provide a language which could be built without Makefiles or other kinds of external configuration. This was realised with the release of Go 1, and the go tool which incorporated an earlier tool, goinstall, as the go get subcommand.

go get uses a cute convention of embedding the remote location of the package’s source in the import path of the package itself. In effect, the import path of the package tells go get where to find the package, allowing the dependency tree a package requires to be discovered and fetched automatically.

Compared to Go’s contemporaries, this solution has been phenomenally successful. Authors of Go code are incentivised to make there code go getable, in much the same way peer pressure drives them to go fmt their code. The installation instructions for most of the Go code you find on godoc and go-search has become, in essence, go get $REPO.

So, what is the problem?

Despite the significant improvement over some other language’s built in dependency management solutions, many Go developers are disappointed when the discover the elegance of go get is a double edged sword.

The Go import declaration references an abstract path. If this abstract path represents to a remote DVCS repository, the declaration alone does not contain sufficient information to identify which particular revision should be fetched. In almost all cases go get defaults to head/tip/trunk/master, although there is a provision for fetching from a tag called go1.

The Go Authors recognize that this is an issue, and suggest that if you need this level of control over your dependencies you should consider using alternative tools, suggesting goven as a possible solution.

Over the past two years no fewer than 19 other tools have been announced, so there is clearly a need and a desire to solve the problem. That said, none of them have achieved significant mind share, let alone challenged go get for the title of default.

Making it personal

Here are three small examples from my work life where go get isn’t sufficient for our requirements as we deliver our rewrite of Juju in Go.

Keeping the team on the same page

Juju isn’t just one project, but more than a dozen projects that we have written or rely on. Often landing a fix or feature in Juju means touching one of these upstream libraries and then making a corresponding change in Juju.

Sometimes these changes make it easy for other developers on the team to detect that they don’t have the correct version of a dependency as Juju stops compiling. Other times the effects can be more subtle, a bug fix to a library can cause no observed breakage so there is no signal to others on the team that they are compiling against the wrong version of the package.

To date, we’ve resorted to an email semaphore whenever someone fixes a bug a package, imploring everyone else to run go get -u. You can probably imagine how successful this is, and how much time is being spent chasing bugs that were already fixed.

Reproducible builds

As a maintenance programmer on Juju, I regularly switch between stable, trunk and feature branches. Each of those expects to be compiled against exactly the right version of its dependencies, but go get doesn’t provide any way of capturing this so I may reliably reproduce a build of Juju from the past.

Being a good Debian citizen

As our product eventually ends up inside Debian based distributions we have to deliver a release artifact which relies only on other Debian packages in the archive. We currently do this with a blob of shell scripts and tags on repos that we control to produce a tarball that contains the complete $GOPATH of all the source for Juju and its dependencies for exactly this release.

While it gets the job done, our pragmatism is not winning us any favors with our Debian packaging overlords as our approach makes their security tin foil hats itch. We’ve been lucky so far, but will probably at some point have a security issue in a package that Juju depends on, and because that package has been copied into every release artifact we’ve delivered fixing it will be a very involved process.

What to do?

Recently William Kennedy, Nathan Youngman and I started a Google Group in the hope of harnessing these disparate efforts of the many people who have thought, argued, hit their head against, and worked on this problem.

If you care about this issue, please consider joining the [go-pm] mailing list and contributing to the Goals document. I am particularly interested in capturing the requirements of various consumers of Go packages via user stories.

Simple test coverage with Go 1.2

Did you know that Go 1.2 will ship with a built in test coverage tool ? The tool is integrated into go test and works similarly to the profiling tool, producing an output file which is interpreted by a second command.

If you have Go 1.2rc2 or tip installed, you can use this short shell function to automate the basic usage of the cover tool to produce a per function coverage breakdown.

cover () { 
    t=$(tempfile)
    go test -coverprofile=$t $@ && go tool cover -func=$t && unlink $t
}

Usage is straight forward, just call cover with no argument for the package in your current working directory, or pass the name of a package.

% cover bytes
ok      bytes   7.770s  coverage: 91.6% of statements
bytes/buffer.go:        Bytes                   100.0%
bytes/buffer.go:        String                  100.0%
bytes/buffer.go:        Len                     100.0%
bytes/buffer.go:        Truncate                100.0%
bytes/buffer.go:        Reset                   100.0%
bytes/buffer.go:        grow                    100.0%
bytes/buffer.go:        Grow                    100.0%
bytes/buffer.go:        Write                   100.0%
bytes/buffer.go:        WriteString             100.0%
bytes/buffer.go:        ReadFrom                94.7%
bytes/buffer.go:        makeSlice               75.0%
bytes/buffer.go:        WriteTo                 78.6%
bytes/buffer.go:        WriteByte               100.0%
bytes/buffer.go:        WriteRune               100.0%
bytes/buffer.go:        Read                    100.0%
bytes/buffer.go:        Next                    100.0%
bytes/buffer.go:        ReadByte                100.0%
bytes/buffer.go:        ReadRune                83.3%
bytes/buffer.go:        UnreadRune              85.7%
bytes/buffer.go:        UnreadByte              100.0%
bytes/buffer.go:        ReadBytes               100.0%
bytes/buffer.go:        readSlice               100.0%
bytes/buffer.go:        ReadString              100.0%
bytes/buffer.go:        NewBuffer               100.0%
bytes/buffer.go:        NewBufferString         100.0%
bytes/bytes.go:         equalPortable           100.0%
bytes/bytes.go:         explode                 100.0%
bytes/bytes.go:         Count                   100.0%
bytes/bytes.go:         Contains                0.0%
bytes/bytes.go:         Index                   100.0%
bytes/bytes.go:         indexBytePortable       100.0%
bytes/bytes.go:         LastIndex               100.0%
bytes/bytes.go:         IndexRune               100.0%
bytes/bytes.go:         IndexAny                100.0%
bytes/bytes.go:         LastIndexAny            100.0%
bytes/bytes.go:         genSplit                100.0%
bytes/bytes.go:         SplitN                  100.0%
bytes/bytes.go:         SplitAfterN             100.0%
bytes/bytes.go:         Split                   100.0%
bytes/bytes.go:         SplitAfter              100.0%
bytes/bytes.go:         Fields                  100.0%
bytes/bytes.go:         FieldsFunc              100.0%
bytes/bytes.go:         Join                    100.0%
bytes/bytes.go:         HasPrefix               100.0%
bytes/bytes.go:         HasSuffix               100.0%
bytes/bytes.go:         Map                     100.0%
bytes/bytes.go:         Repeat                  100.0%
bytes/bytes.go:         ToUpper                 100.0%
bytes/bytes.go:         ToLower                 100.0%
bytes/bytes.go:         ToTitle                 100.0%
bytes/bytes.go:         ToUpperSpecial          0.0%
bytes/bytes.go:         ToLowerSpecial          0.0%
bytes/bytes.go:         ToTitleSpecial          0.0%
bytes/bytes.go:         isSeparator             80.0%
bytes/bytes.go:         Title                   100.0%
bytes/bytes.go:         TrimLeftFunc            100.0%
bytes/bytes.go:         TrimRightFunc           100.0%
bytes/bytes.go:         TrimFunc                100.0%
bytes/bytes.go:         TrimPrefix              100.0%
bytes/bytes.go:         TrimSuffix              100.0%
bytes/bytes.go:         IndexFunc               100.0%
bytes/bytes.go:         LastIndexFunc           100.0%
bytes/bytes.go:         indexFunc               100.0%
bytes/bytes.go:         lastIndexFunc           100.0%
bytes/bytes.go:         makeCutsetFunc          100.0%
bytes/bytes.go:         Trim                    100.0%
bytes/bytes.go:         TrimLeft                100.0%
bytes/bytes.go:         TrimRight               100.0%
bytes/bytes.go:         TrimSpace               100.0%
bytes/bytes.go:         Runes                   100.0%
bytes/bytes.go:         Replace                 100.0%
bytes/bytes.go:         EqualFold               100.0%
bytes/reader.go:        Len                     100.0%
bytes/reader.go:        Read                    100.0%
bytes/reader.go:        ReadAt                  100.0%
bytes/reader.go:        ReadByte                0.0%
bytes/reader.go:        UnreadByte              0.0%
bytes/reader.go:        ReadRune                0.0%
bytes/reader.go:        UnreadRune              0.0%
bytes/reader.go:        Seek                    91.7%
bytes/reader.go:        WriteTo                 83.3%
bytes/reader.go:        NewReader               100.0%
total:                  (statements)            91.6%

Release candidate 1 tarballs for ARM now available

Go 1.2 is on target for a December release and the Go team have just cut their first release candidate.

You can find the draft (no twitterverse, Go 1.2 isn’t released yet) release notes for Go 1.2 online here.

I have updated my unofficial ARM tarball distributions page with prebuilt go1.2rc1 tarballs. You can find them by following the link in the main header of this page.

If you are interested in following the performance improvements in Go 1.2, you may be interested in my autobench project.