Category Archives: Go

Cross compilation just got a whole lot better in Go 1.5

Introduction

Cross compilation is one of Go’s headline features. I’ve written about it a few times, and others have taken this work and built better tooling around it.

This morning Russ Cox committed this change which resolved the last issue in making cross compilation simpler and more accessible for all Gophers. When Go 1.5 ships in August any Go programmer will be able to cross compile their program without having to go through a fussy set up phase.

Background

In the current version of Go, (if you’re from the future, read Go 1.4 and earlier) before you could cross compile a Go program, you needed to go through a set up phase to enhance your Go installation with the bits necessary to build for other platforms.

This worked ok if you had built Go from source, but if you were using one of the binary distributions or something from your operating system (brew, apt, etc), then the process would turn into a maze of twisty passages, all alike.

This was necessary because for successful cross compilation you would need

  • compilers for the target platform, if they differed from your host platform, ie you’re on darwin/amd64 (6g) and you want to compile for linux/arm (5g).
  • a standard library for the target platform, which included some files generated at the point your Go distribution was built.

With the plan to translate the Go compiler into Go coming to fruition in the 1.5 release the first issue is now resolved. That just left the small amount of customisation to the runtime done at installation time, which has been whittled away as part of the compiler transition (most of the customisation was generation of include files for the C parts of the runtime), and as of this morning, removed.

Try it out

If you can’t wait til August, you can try this out today by building the development version of Go.

Disclaimer: the development branch is changing rapidly due to the re-factoring of the compilers. If you’re using Go in production, the release version, Go 1.4.x, is always recommended.

Prerequisites

As mentioned above to build the development version of Go, you also need Go 1.4 installed to bootstrap. Once Go 1.5 comes out this step will be unnecessary.

These steps are a simple procedure to do this which I would recommend following

  1. Uninstall any version of Go you have on your system, including any $PATH variables.
  2. Check out Go 1.4 and build it
    % git clone https://go.googlesource.com/go $HOME/go1.4
    % cd $HOME/go1.4/src
    % git checkout release-branch.go1.4
    % ./make.bash
  3. Check out the development branch and build it
    % git clone https://go.googlesource.com/go $HOME/go
    % cd $HOME/go/src
    % env GOROOT_BOOTSTRAP=$HOME/go1.4 ./all.bash
  4. Add $HOME/go/bin to your $PATH

Build something

Notice: if you are reading this from a future where Go 1.5 has been released, you can skip the previous step.

Now you have the development version of Go 1.5 installed, cross compiling this simple program is trivial

package main

import "fmt"
import "runtime"

func main() {
        fmt.Printf("Hello %s/%s\n", runtime.GOOS, runtime.GOARCH)
}

Now build for darwin/386

% env GOOS=darwin GOARCH=386 go build hello.go
# scp to darwin host
$ ./hello
Hello darwin/386

Or build for linux/arm

% env GOOS=linux GOARCH=arm GOARM=7 go build hello.go
# scp to linux host
$ ./hello
Hello linux/arm

That’s it!

Cross compilation can’t get any simpler than that.

Technical mumbo-jumbo

So what is happening under the hood when we compile a program for a different platform ? The -v flag gives us a clue.

% go build -v hello.go
command-line-arguments

% env GOOS=linux GOARCH=arm go build -v hello.go                                  runtime
errors
sync/atomic
math
unicode/utf8
sync
io
syscall
time
strconv
reflect
os
fmt
command-line-arguments

Comparing the two examples above, the first build is using the standard library that was built as part of the Go installation. That is to say, all the packages that fmt and runtime depend on are already built into $GOROOT/pkg/linux_amd64.

In the second example, the go tool detects that all the dependencies of this program need to be built for the target plaform before hello.go can be compiled and builds them, all the way back to the runtime package.

I have not included the output here because it is very verbose, but you can look at all the steps that are being performed if you pass the -x flag to go build.

Out of scope

Cross compilation while linking to libraries via cgo is the holy grail for some. Sadly the changes mentioned above do not change the situation with respect to cgo. In fact you may still need to rebuild your Go installation in the traditional way to pass environment variables like CC_FOR_TARGET. Please try it out.

Conclusion

In case you can’t tell, I’m over the moon about this improvement. Cross compilation is Go’s ace in the hole and Go 1.5 will make it even better.

Please try it out and if you find issues please let us know on the GitHub issue tracker.

Lost in translation

Over the last year I have had the privilege of travelling to meet Go communities in Japan, Korea and India. In every instance I have met experienced, passionate, pragmatic programmers ready to accept Go for what it can do for them.

At the same time the message from each of these communities was the same; where is the documentation, where are the examples, where are the tutorials ? Travelling to these communities has been an humbling experience and has made me realise my privileged position as a native English speaker.

The documentation, and the tutorials, and the examples will come, slowly; this is open source after all. But what I can offer is the fact that all the content on this blog is licensed under a Creative Commons licence.

In short, I don’t have the skills, but if you do, you are welcome to translate any content on this site, and I’ll help you in any way I can.

 

Thanks Brainman

This is a short post to recognise the incredible contribution Alex Brainman has made to the Go project.

Alex was responsible for the port of Go to Windows way back before Go 1 was even released. Since that time he has virtually single-handedly supported Go and Go users on Windows. It’s no wonder that he is the 10th most active contributor to the project.

The Windows build is consistently the most popular download from the official go site.

While I may not use Windows, and you may not use Windows, spare a thought for the large body of developers who do use Windows to develop Go programs and are able to do so because of Alex’s efforts.

Even if your entire business doesn’t use Windows, consider the moment when your product manager comes to you and asks “so, we’ve got a request from a big customer to port our product to Windows, that’s not going to be hard, right ?”. Your answer is directly attributable to Alex’s contributions.

Alex, every Go programmer owes you a huge debt of gratitude. So let me be the first to say it, thank you for everything you have done for Go. None of us would be as successful as we are today without your work.

Errors and Exceptions, redux

In my previous post, I doubled down on my claim that Go’s error handling strategy is, on balance, the best.

In this post, I wanted to take this a bit further, and prove that multiple returns and error values are the best,

When I say best, I obviously mean, of the set of choices available to programmers that write real world programs — because real world programs have to handle things going wrong.

The language we have

I am only going to use the Go language that we have today, not any version of the language which might be available in the future — it simply isn’t practical to hold my breath for that long. As I will show, additions to the language like dare I say, exceptions, would not change the outcome.

A simple problem

For this discussion, I’m going to start with a made up, but very simple function, which demonstrates the requirement for error handling.

package main

import "fmt"

// Positive returns true if the number is positive, false if it is negative.
func Positive(n int) bool {
        return n > -1
}

func Check(n int) {
        if Positive(n) {
                fmt.Println(n, "is positive")
        } else {
                fmt.Println(n, "is negative")
        }
}

func main() {
	Check(1)
	Check(0)
	Check(-1)
}

If you run this code, you get the following output

1 is positive
0 is positive
-1 is negative

which is wrong.

How can this single line function be wrong ? It is wrong because zero is neither positive or negative, and that cannot be accurately captured by the boolean return value from Positive.

This is a contrived example, but hopefully one that can be adapted to discuss the costs and benefits of the various methods of error handling.

Preconditions

No matter what solution is determined to be the best, a check will have to be added to Positive to test the non zero precondition. Here is an example with the precondition added

// Positive returns true if the number is positive, false if it is negative.
// The second return value indicates if the result is valid, which in the case
// of n == 0, is not valid.
func Positive(n int) (bool, bool) {
        if n == 0 {
                return false, false
        }
        return n > -1, true
}

func Check(n int) {
        pos, ok := Positive(n)
        if !ok {
                fmt.Println(n, "is neither")
                return
        }
        if pos {
                fmt.Println(n, "is positive")
        } else {
                fmt.Println(n, "is negative")
        }
}

Running this program we see that the bug is fixed,

1 is positive
0 is neither
-1 is negative

albeit in an ungainly way. For those interested, I also tried a version using a switch which was harder to read for the saving of one line of code.

This then is the baseline to compare other solutions.

Error

Returning a boolean is uncommon, it’s far more common to return an error value, even if the set of errors is fixed. For completeness, and because this simple example is supposed to hold up in more complex circumstances, here is an example using a value that conforms to the error interface.

// Positive returns true if the number is positive, false if it is negative.
func Positive(n int) (bool, error) {
        if n == 0 {
                return false, errors.New("undefined")
        }
        return n > -1, nil
}

func Check(n int) {
        pos, err := Positive(n)
        if err != nil {
                fmt.Println(n, err)
                return
        }
        if pos {
                fmt.Println(n, "is positive")
        } else {
                fmt.Println(n, "is negative")
        }
}

The result is a function which performs the same, and the caller must check the result in an near identical way.

If anything, this underlines the flexibility of Go’s errors are values methodology. When an error occurs, indicating only success or failure (think of the two result form of map lookup), a boolean can be substituted instead of an interface value, which removes the any confusion arising from typed nils and nilness of interface values.

More boolean

Here is an example which allows Positive to return three states, true, false, and nil (Anyone with a background in set theory or SQL will be twitching at this point).

// If the result not nil, the result is true if the number is
// positive, false if it is negative.
func Positive(n int) *bool {
        if n == 0 {
                return nil
        }
        r := n > -1
        return &r
}

func Check(n int) {
        pos := Positive(n)
        if pos == nil {
                fmt.Println(n, "is neither")
                return
        }
        if *pos {
                fmt.Println(n, "is positive")
        } else {
                fmt.Println(n, "is negative")
        }
}

Positive has grown another line, because of the requirement to capture the address of the result of the comparison.

Worse, now before the return value can be used anywhere, it must be checked to make sure that it points to a valid address. This is the situation that Java developers face constantly and leads to deep seated hatred of nil (with good reason). This clearly isn’t a viable solution.

Let’s try panicking

For completeness, let’s look at a version of this code that tries to simulate exceptions using panic.

// Positive returns true if the number is positive, false if it is negative.
// In the case that n is 0, Positive will panic.
func Positive(n int) bool {
        if n == 0 {
                panic("undefined")
        }
        return n > -1
}

func Check(n int) {
        defer func() {
                if recover() != nil {
                        fmt.Println("is neither")
                }
        }()
        if Positive(n) {
                fmt.Println(n, "is positive")
        } else {
                fmt.Println(n, "is negative")
        }
}

… this is just getting worse.

Not exceptional

For the truly exceptional cases, the ones that represent either unrecoverable programming mistakes, like index out of bounds, or unrecoverable environmental problem, like running out of stack, we have panic.

For all of the remaining cases, any error conditions that you will encounter in a Go program, are by definition not exceptional — you expect them because regardless of returning a boolean, an error, or pancing, it is the result of a test in your code.

Forgetting to check

I consider the argument that Developers forget to check error codes is cancelled out by the counter argument Developers forget to handle exceptions. Either may be true, depending on the language you are basing your argument on, but neither commands a winning position.

With that said, you only need to check the error value if you care about the result.

Knowing the difference between which errors to ignore and which to check is why we’re paid as professionals.

Conclusion

I have shown in the article that multiple returns and error values the simplest, and most reliable to use. Easier to use than any other form of error handling, including ones that do not even exist in Go as it stands today.

A challenge

So this is the best demonstration I can come up with, but I expect others can do better, particularly where the monadic style is used. I look forward to your feedback.

Inspecting errors

The common contract for functions which return a value of the interface type error, is the caller should not presume anything about the state of the other values returned from that call without first checking the error.

In the majority of cases, error values returned from functions should be opaque to the caller. That is to say, a test that error is nil indicates if the call succeeded or failed, and that’s all there is to it.

A small number of cases, generally revolving around interactions with the world outside your process, like network activity, require that the caller investigate the nature of the error to decide if it is reasonable to retry the operation.

A common request for package authors is to return errors of a known public type, so the caller can type assert and inspect them. I believe this practice leads to a number of undesirable outcomes:

  • Public error types increase the surface area of the package’s API.
  • New implementations must only return types specified in the interface’s declaration, even if they are a poor fit.
  • The error type cannot be changed or deprecated after introduction without breaking compatibility, making for a brittle API.

Callers should feel no more comfortable asserting an error is a particular type than they would be asserting the string returned from Error() matches a particular pattern.

Instead I present a suggestion that permits package authors and consumers to communicate about their intention, without having to overly couple their implementation to the caller.

Assert errors for behaviour, not type

Don’t assert an error value is a specific type, but rather assert that the value implements a particular behaviour.

This suggestion fits the has a nature of Go’s implicit interfaces, rather than the is a [subtype of] nature of inheritance based languages. Consider this example:

func isTimeout(err error) bool {
        type timeout interface {
                Timeout() bool
        }
        te, ok := err.(timeout)
        return ok && te.Timeout()
}

The caller can use isTimeout() to determine if the error is related to a timeout, via its implementation of the timeout interface, and then confirm if the error was timeout related — all without knowing anything about the type, or the original source of the error value.

Gift wrapping errors, usually by libraries that annotate the error path, is enabled by this method; providing that the wrapped error types also implement the interfaces of the error they wrap.

This may seem like an insoluble problem, but in practice there are relatively few interface methods that are in common use, so Timeout() bool and Temporary() bool would cover a large set of the use cases.

In conclusion

Don’t assert errors for type, assert for behaviour.

For package authors, if your package generates errors of a temporary nature, ensure you return error types that implement the respective interface methods. If you wrap error values on the way out, ensure that your wrappers respect the interface(s) that the underlying error value implemented.

For package users, if you need to inspect an error, use interfaces to assert the behaviour you expect, not the error’s type. Don’t ask package authors for public error types; ask that they make their types conform to common interfaces by supplying Timeout() or Temporary() methods as appropriate.

Friday pop quiz: the size of things

In this program, the size of variables of type x and y in memory varies by platform.

package main

func main() {
        const n = 4
        type x [n]uint
        type y [n]int
}

By changing only one line can you ensure that variables of type x, and y always consume 16 bytes on all platforms that Go 1.4 supports ?

Rules

The code must continue to be correctly formatted.

Bonus points will be awarded for the most creative solution.

Points will be deducted for arguing with the judge (me).


Answers

The solution obviously involved setting n to 4 on 32 bit platforms, and 2 on 64 bit. There were a wide number of variations on this, involving a menagerie of subtraction, shifting and multiplication. The solution I came up with used only one operator:

const n = 4 >> (^uint(0) >> 63)

^uint(0) gives you a number whose bits are all 1, then >> 63 shifts the number 63 binary places to the right. If we’re on a 64 bit platform, this evaluates to 1, shifting 4 one place to the right leaves 2, otherwise 32 ones shifted 63 places to the right gives zero, and 4 shifted right zero times is still 4.

So I was feeling rather chuffed with myself until Paul Hankin quietly dropped this solution:

const n = ^uint(6) % 7

Paul, my hat is off to you Sir.

Minimum one liner followup

It’s a little unfair to announce winners in some kind of order as I did post the quiz at an unfriendly hour of the day for most of the planet.

With that said, Tim and William came up with a great map based solution at roughly the same time. You’ll have to split the winnings between yourselves.

Gary came an interesting solution that works for almost all the integers.

Gustavo Niemeyer takes double points for demonstrating that the first version of this problem could be defeated easily, and then proceeded to demonstrate his very mathy solution to fix Gary’s proposal. Several others also proposed some great shift tricks.

Honourable mentions go to Charlie Somerville, for playing the man and not the ball and Francesc who proved that even with two attempts I couldn’t make the problem sufficiently water tight.

Although the prohibition on adding more than one line was lost on Brendan Tracey, I think this proposal deserves to be highlighted.

So with sensible, workable, and sometimes beautiful solutions out in the open, the race was on for the bonus points for the most creative.

The first was my entry, which was the genesis for this quiz and goes to show, this why we cannot have nice things.

func f(a int, b uint) {
        var min = 0
        min = copy(make([]struct{}, a), make([]struct{}, b))
        fmt.Printf("The min of %d and %d is %d\n", a, b, min)
}

Props for figuring this out goes to Arnaud Porterie and Gustavo Niemeyer who were both good sports and deleted their answer.

I was feeling rather pleased with myself until Paul Hankin emailed me this fabulously creative effort. After that others tweaked to the loop hole that I had inadvertently left open by importing the fmt package.

Congratulations to the winners, and thank you all for contributing.

Friday pop quiz: minimum one liner

This program is incorrect

package main

import "fmt"

func f(a, b int) {
        var min = 0
        fmt.Printf("The min of %d and %d is %d\n", a, b, min)
}

func main() {
        f(9000, 314)
}

By adding only one line can you make it print the correct answer ?

The code must continue to be correctly formatted.

Bonus points will be awarded for the most creative solution.

Points will be deducted for arguing with the judge (me).

Update: thanks to Gustavo Niemeyer who pointed out the first version of the quiz made it way to easy.

Update: a few people have provided some very sound, rational solutions. Good job, give yourself a code review gold star.

The bonus points for the most creative solution are still on the table. As a hint my solution will also work for this variant of the problem.


The answer(s) will be posted tomorrow.