Category Archives: Programming

Hear me speak about Go performance at OSCON

I’m going to be speaking at OSCON this year about Go performance. The title of the talk is High performance servers without the event loop and will focus on the features of the language and its runtime that transparently let Go programmers write high performance network servers without resorting to event loops and callback spaghetti.

As the OSCON website is a bit light on for details, here are some more to entice you.

Abstract

Conventional wisdom suggests that high performance servers require native threads, or more recently, event loops.

Neither solution is without its downside. Threads carry a high overhead in terms of scheduling cost and memory footprint. Event loops ameliorate those costs, but introduce their own requirements for a complex callback driven style.

A common refrain when talking about Go is it’s a language that works well on the server; static binaries, powerful concurrency, and high performance.

This talk focuses on the last two items, how the language and the runtime transparently let Go programmers write highly scalable network servers without having to worry about thread management or blocking I/O.

The goal of this talk is to introduce the following features of the language and the runtime:

  • Escape Analysis
  • Stack management
  • Processes and threads vs goroutines
  • Integrated network poller

These four features work in concert to build an argument for the suitability of Go as a language for writing high performance servers.

How does a 20% discount sound ?

O’Reilly have provided me with a discount code DAVEC20 that you can use to save 20% on your registration.

As a bonus, if enough people register for OSCON using the discount code DAVEC20, the organisers have said they will give me a free pass to the event which I plan to donate to Women Who Code. I know that this is a lot of ifs, but it is worth a try.

So, if you’re interested in hearing me, and hundreds of others speak at OSCON this year, you can save yourself 20% and recieve some good karma in the process. What’s too lose ?

Struct composition with Go

This is a quick Friday blog post to talk about a recent experience I had working on a piece Juju code that needed to capture the data being sent over a net.Conn.

Most Gophers know that the net package provides a net.Pipe function which returns a pair of net.Conns representing an in memory network connection. net.Pipe is ideal for testing components that expect to talk over the network without all the mucking around of actually using the network.

The Go standard library also contains the super useful io.MultiWriter function which takes any number of io.Writers and returns another io.Writer that will send a copy of any data written to it to each of its underlying io.Writers. Now I had all the pieces I needed to create a net.Conn that could record the data written through it.

func main() {
        client, server := net.Pipe()
        var buf bytes.Buffer
        client = io.MultiWriter(client, &buf)

        // ...
}

Except this code does not compile.

# command-line-arguments
/tmp/sandbox866813815/main.go:13: cannot use io.MultiWriter(client, &buf) (type io.Writer) as type net.Conn in assignment:
	io.Writer does not implement net.Conn (missing Close method)

The value returned by io.MultiWriter is an implementation of io.Writer, it doesn’t have the rest of the methods necessary to fulfil the net.Conn interface; what I really need is the ability to replace the Write method of an existing net.Conn value. We can do this with embedding by creating a structure that embeds both a net.Conn and an independant io.Writer as anonymous fields.

type recordingConn struct {
        net.Conn
        io.Writer
}

func main() {
        client, server := net.Pipe()
        var buf bytes.Buffer
        client = &recordingConn {
                Conn: client,
                Writer: io.MultiWriter(client, &buf),
        }

        // ...
}

The recodingConn embeds a net.Conn ensuring that recordingConn implements net.Conn. It also gives us a place to hang the io.MultiWriter so we can syphon off the data written by the client. There is only one small problem remaining.

# command-line-arguments
/tmp/sandbox439875759/main.go:24: recordingConn.Write is ambiguous

Because both fields in the structure are types that have a Write method, the compiler cannot decide which one should be the called. To resolve this ambiguity we can add a Write method on the recordingConn type itself:

func (c *recordingConn) Write(buf []byte) (int, error) {
        return c.Writer.Write(buf)
}

With the ambiguity resolved, the recordingConn is now usable as a net.Conn implementation. You can see the full code here.

This is just a small example of the power of struct composition using Go. Can you think of other ways to do this ?

Introducing gb, a project based build tool for the Go programming language

In April 2015 I gave a presentation to the GDG Berlin meetup group (slides, video) discussing my views on reproducible builds using the Go programming language.

Image credit, Nate Finch

Image credit, Nate Finch

As part of that presentation I demonstrated a replacement build tool that I had been developing, gb.

From the feedback I received after the meetup it was clear that many people understood and agreed with my reasoning why reproducible builds in Go are a problem today. However it was equally clear that I had not done a good enough job of explaining how gb works, and why I felt it was necessary to create an alternative build tool to solve the problem of reproducible builds.

With the help of the go-pm mailing list and the feedback from a group of early adopters of gb, I have put together a website for gb with expanded documentation and examples.

I invite you all to visit getgb.io to learn more about gb.

Simplicity and collaboration

This is the text of my closing keynote from Gophercon India. It has been slightly altered for readability from my original speaking notes.

I am indebted to the organisers of Gophercon India for inviting me to speak, and to Canonical for giving me the time off to attend the conference.

If you want to see me stumble through the real thing, the video is now available.

Simplicity and collaboration

Closing Keynote, Gophercon India
21 Feb 2015

Introduction

I want to open my presentation with a proposition.

https://twitter.com/davecheney/status/539576755254611968

Being passionate about Go means being passionate about language advocacy, and a natural hazard of dwelling too long on the nature of programming results in statements like these.

But underlying the pithy form enforced by a tweet, I believe there is a gem of truth—I cannot think of a language introduced in my life time that didn’t purport to be simple. Each new language offers as a justification, and an enticement, their inherent simplicity.

On the other hand, I cannot point to a language introduced in the same time frame with the rallying call of complexity; more complexity than its contemporaries—but many instead claim to be powerful.

The idea of proposing a language which offered higher levels of inherent complexity is clearly laughable, yet this is exactly what so many contemporary languages have become; complicated, baroque, messes. A parody of the languages they sought to replace.

Clumsy syntax and non orthogonality is justified by the difficulty of capturing nuanced corner cases of the language, many of them self inflicted by years of careless feature creep.

So, every language starts out with simplicity as a goal, yet many of them fail to achieve this goal. Eventually falling back on notions of expressiveness or power of the language as justification for a failure to remain simple.

Any why is this ? Why do so many language, launched with sincere, idealistic goals, fall afoul of their own self inflicted complexity ?

One reason, one major reason, I believe, is that to be thought successful, a language should somehow include all the popular features of its predecessors.

historyIf you would listen to language critics, they demand that any new language should push forward the boundaries of language theory.

In reality this appears to be a veiled request that your new language include all the bits they felt were important in their favourite old language, while still holding true to the promise of whatever it was that drew them to investigate your language I the first place.

I believe that this is a fundamentally incorrect view.

Why would a new language be proposed if not to address limitations of its predecessors ?

Why should a new language not aim to represent a refinement of the cornucopia of features presented in existing languages, learning from its predecessors, rather than repeating their folly.

Language design is about trade-offs; you cannot have your cake and eat it too. So I challenge the notion that every mainstream language must be a super-set of those it seeks to replace.

Simplicity

This brings me back to my tweet.

Go is a language that chooses to be simple, and it does so by choosing to not include many features that other programming languages have accustomed their users to believing are essential.

So the subtext of this thesis would be; what makes Go successful is what has been left out of the language, just as much as what has been included.

Or as Rob Pike puts it “less is exponentially more”.

Simplicity cannot be added laterIMG_0095

When raising a new building, engineers first sink long pillars, down to the bedrock to provide a stable foundation for the structure of the building.

To not do this, to just tamp the area flat, lay a concrete slab and start construction, would leave the building vulnerable to small disturbances from changes in the local area, at risk from rising damp or subsidence due to changes in environmental conditions.

As programmers, we can recognise this as the parable of leaky abstraction. Just as tall buildings can only be successfully constructed by placing them on a firm foundation, large programs can not be successful if they are placed upon a loose covering of dirt that masks decades of accumulated debris.

You cannot add simplicity after the fact. Simplicity is only gained by taking things away.

Simplicity is not easy

IMG_0245Simplicity does not mean easy, but it may mean straight forward or uncomplicated.

Something which is simple may take a little longer, it may be a little more verbose, but it will be more comprehensible.

Putting this into the context of programming languages, a simple programming language may choose to limit the number of semantic conveniences it offers to experienced programmers to avoid alienating newcomers.

Simplicity is not a synonym for easy, nor is achieving a design which is simple an easy task.

Don’t mistake simple for crude

10_ck_chef_hand_10Just because something may be simple, don’t mistake it for crude.

While lasers are fabulous technology used in manufacturing and medicine, a chef prefers a knife to prepare food.

Compared to the laser, a simple chefs knife may appear unsophisticated, but in truth it represents generations of knowledge in metallurgy, manufacturing and usability.

When considering a programming language, don’t mistake a lack of the latest features for a lack of sophistication.

Simplicity is a goal, not a by-product

“nothing went in [to the language], until all three of us [Ken, Robert and myself], agreed that it was a good idea.” — Rob Pike, Gophercon 2014

You should design your programs with simplicity as a goal, not aim to be pleasantly surprised when your solution happens to be simple.

As Rob Pike noted at Gophercon last year, Go was not designed by committee. The language represent a distillation of the experiences of Robert Griesemer, Ken Thompson, and himself, and only once all three were all convinced of a feature’s utility to the language was it included.

Choose simplicity over completeness

There is an exponential cost in completeness.

The 90% solution, a language that remains orthogonal while recognizing some things are not possible, verses a language attempting to offer 100% of its capabilities to every possible permutation will inherently be less complex, because as we engineers know,

The last 10% costs another 90% of the effort.

Complexity

four_string_braid

A lack of simplicity is, of course complexity.

Complexity is friction, a force which acts against getting things done.

Complexity is debt, it robs you of capital to invest in the future.

Good programmers write simple programs

Good programmers write simple programs.

They bring their experience, their knowledge and their failures to new designs, to learn from and avoid mistakes in the future.

Simplicity, conclusion

To steal a quote from Rich Hickey

“Simplicity is the ultimate sophistication” — Leonardo da Vinci

Go is a language designed to be simple. It is a feature, not a by-product, or an accident.

This was the message that spoke to me when I first learned about the language in 2009, and is the message that has stayed with me to this day.

The desire for simplicity is woven through every aspect of the language.

My question for you, the audience: Do you want to write simple programs, or will you settle for writing powerful programs ?

Collaboration

I hope by now I have convinced you that a need for simplicity in programming languages, is self evident, so I want to move to my second topic; collaboration.

Is programming an art or a science ? Are we artists or engineers ? This one question is a debate in itself, but I hope you will humour me that as professionals, programming is a little of both; we are both software artists, and software engineers—and as engineers we work as a team.

There is more to the success of Go than just being simple, and this is the realization that for a programming language to be successful, it must coexist inside a larger environment.

A language for collaboration

Large programs are written by large teams. I don’t believe this is a controversial statement.

The inverse is also true. Large teams of programmers, by their nature, produce large code bases.

Projects with large goals will necessitate large teams, and thus their output will be commensurate.

This is the nature of our work.

Big Problems

nasa-mainframe-980x663Go is a small language, but deliberately designed as a language for large teams of programmers.

Small annoyances such as a lack of warnings, a refusal to allow unused imports, or unused local variables, are all facets of choices designed to help Go work well for large teams.

This does not mean that Go is not suitable for the single developer working alone, or a small program written for a specific need, but speaks to the fact that a number of the choices within the language are aimed at the needs of growing software teams.

And if your project is successful, your team will grow, so you need to plan for it.

Programming languages as part of an environment

It may appear heretical to suggest this, as many of the metrics that we as professional software developers are judged by; lines of code written; the number of revisions committed to source control, and so on, are all accounted for character by character. Line by line. File by file.

But, writing a program, or more accurately solving a problem; delivering a solution, has little to do with the final act of entering the program into the computer.

Programs are designed, written, debugged and distributed in an environment significantly larger than one programmer’s editor.

Go recognizes this, it is a language designed to work in this larger environment, not in spite of it.

Because ultimately Go is a language for the problems that exist in today’s commercial programming, not just language theory.

Code is written to be decoded

cover-bigThe author Peter Seibel suggests that programs are not read, but instead decoded. In hindsight this should have been obvious, after all we call it source code, not source literature.

The source code of a program is an intermediary form, somewhere between our concept and the computer’s executable notation.

As with many transformations, this encoding of the source program is not lossless; some loss of fidelity, some ambiguity, some imprecision is present. This is why when reading source code, you must in fact decode it, to divine the original intention of the programmer.

Many of the choices relating to the way Go code is represented as source, speak to this impedance mismatch. The simplicity and regularity of the grammar, while providing few opportunities for individuality, in turn makes it easier for a new reader to decode a Go program and determine its function.

Because source code is written to be read.

How to build a Go communitygofmt

Go is a language designed from the beginning to be transposed, transformed and processed at the source level. This has opened up new fields of analysis and code generation to the wider Go community. We’ve seen several examples of this demonstrated at this conference.

While these tools are impressive, I believe the regular syntax of a Go source file belies its greatest champion; go fmt.

But what is it that is so important about go fmt, and why is it important to go fmt your source code ?

Part of the reason is, of course, to avoid needless debate. Large teams will, by their nature have a wide spectrum of views on many aspects of programming, and source code formatting is the most pernicious.

Go is a language for collaboration. So, in a large team, just as in a wider community, personal choices are moderated for a harmonious society.

The outcome is that nearly all go code is go formatted by convention. Adherence to this convention is an indicator of alignment with the values of Go.

This is important because it is a social convention leading to positive reinforcement, which is far more powerful than negative reinforcement of a chafing edict from the compiler writer.

In fact, code that is not well formatted can be a first order indicator of the suitability of the package. Now, I’m not trying to say that poorly formatted code is buggy, but poorly formatted code may be an indication that the authors have not understood the design principles that underscore Go.

So while buggy code can be fixed, design issues or impedance mismatches can be much harder to address, especially after that code is integrated into your program.

Batteries included

As Go programmers we can pick up a piece of Go code written by anyone in the world and start to read it. This goes deeper than just formatting.

Go lacks heavy libraries like Boost. There are no QT base classes, no gobject. There is no pre-processor to obfuscate. Domain specific language rarely appear in Go code.

The inclusion of maps and slices in the language side steps the most basic interoperability issues integrating packages from vendors, or other parts of your company. All Go code uses these same basic building blocks; maps, slices and channels, so all Go code is accessible to a reader who is versed in the language, not some quaint organization specific dialect.

Interfaces, the UNIX waypipe

In 1964 Doug McIlroy postulated about the power of pipes for composing programs. This was five years before the first Unix was written mind you.

McIlroy’s observations became the foundation of the UNIX philosophy; small, sharp tools which can be combined to solve larger tasks, tasks which may not have even been envisioned by the original authors.

In the last few decades, I feel that programmers have lost the ability to compose programs, lost behind waves of run time dependencies, stifling frameworks, and brittle type hierarchies that degrade the ability to move quickly and experiment cheaply.

Go programs embody the spirit of the UNIX philosophy. Go packages interact with one another via interfaces. Programs are composed, just like the UNIX shell, by combining packages together.

I can use fmt.Fprintf to write formatted output to a network connection, or a zip file, or a writer which discards its input. Conversely I can create a gzip reader that consumes data from a http connection, or a string constant, or a multireader composed of several sources.

All of these permutations are possible, in McIlroy’s vision, without any of the components having the slightest bit of knowledge about the other parts of this processing chain.

Small interfaces

Interfaces in Go are therefore a unifying force; they are the means of describing behaviour. Interfaces let programmers describe what their package provides, not how it does it.

Well designed interfaces are more likely to be small interfaces; the prevailing idiom here is that interfaces contain only a single method.

Compare this to other languages like Java or C++. In those languages interfaces are generally larger, in terms of the method count required to satisfy them, and more complex because of their entanglement with the inheritance based nature of those languages.

Interfaces in Go share none of those restrictions and so are simpler, yet at the same time, are more powerful, and more composable, and critical to the narrative of collaboration, interfaces in Go are satisfied implicitly.

Any Go type, written at any time, in any package, by any programmer, can implement an interface by simply providing the methods necessary to satisfy the interface’s contract.

It follows logically that small interfaces lead to simple implementations, because it is hard to do otherwise. Leading to packages comprised of simple implementations connected by common interfaces.

Errors and interfaces

errorI’ve written a lot about the subject of Go’s error handling, so I’ll restrict my comments here to errors as they relate to collaboration.

The error interface is the key to Go’s composable error handling story.

If you’ve worked on some large Go projects you may have come across packages like Canonical’s errgo, which provide facilities to apply a stack trace to an error value. Perhaps the project has rolled their own implementation. Maybe you have developed something similar in house.

I want to be clear that I am remaining neutral on the relative goodness or badness of the idea of gift wrapping errors.

What I do want to highlight is even though one piece of code you integrate uses fmt.Errorf, and another a third party package, and in your package you have developed your own error handling type. From the point of view of you the programmer consuming your work, the error handling strategy always looks the same. If the error is nil, the call worked.

Compare this to the variety of error handling strategies that must be managed in other languages as programs grow through accretion of dependencies.

This is the key to a Go programmer’s ability to write an application at any size without sacrificing reliability. In the context of collaboration, it must be said that Go’s error handling strategy is the only form that makes sense.

Simple build systems

Go’s lack of Makefiles is more than a convenience.

With other programming languages, when you integrate a piece of third party code, maybe it’s something complex, like v8, or something more mundane, like a database driver from your vendor, you’re integrating that code into your program, this part is obvious, but you are also integrating their build system.

This is a far less visible, and sometimes far more intractable problem. You’ve not just introduced a dependency on that piece of code, but also a dependency on its build system, be it cmake, scons, gnu autotools, what have you.

Go simply doesn’t have this problem.

Putting aside the contentious issues of package versioning, once you have the source in your $GOPATH, integrating any piece of third party Go code into your program is just an import statement.

Go programs are built from just their source, which has everything you need to know to compile a Go program. I think this is a hugely important and equally under-appreciated part of Go’s collaboration story.

This is also the key to Go’s efficient compilation. The source indicates only those things that it depends on, and nothing else. Compiling your program will touch only the lines of source necessary.

Sans runtime

Does your heart sink when you want to try the hottest new project from Hacker News or Reddit only find it requires node.js, or some assortment of Ruby dependencies that aren’t available on your operating system ? Or you have to look up what is the correct way to install a python package this week. Is it pip, is it easy_install, does that need an egg, or are they wheels ?

I can tell you mine does.

For Go programmers dependency management remains an open wound, this is a fact, and one that I am not proud of. But for users of programs written in Go their life just got a whole lot easier; compile the program, scp it to the server, job done.

Go’s ability to produce stand alone applications; and even cross compile them directly from your workstation means that programmers are rediscovering the lost art of shipping a program, a real compiled program, the exact same one they tested, to customers.

This one fact alone has allowed Go to establish a commanding position in the container orchestration market, a market which arguably would not exist in its current form if not for Go’s deployment story.

This story also illustrates how Go’s design decisions move beyond just thinking about how the programmer and the language will interact during the development phase, and extend right through the software life-cycle to the deployment phase.

Go’s choice of a single static binary is directly influenced by Google’s experiences deploying their own large complex applications, and I believe their advice should not be dismissed lightly.

Portability

C# isn’t portable, it is joined at the hip to a Windows host.

Swift and Objective-C are in the same boat, they live in the land of Apple only programming languages. Popular ? yes, but portable ? no.

Java, Scala, Groovy, and all the rest of the languages built atop the JVM may benefit from the architecture independence of the JVM bytecode format, until you realize that Oracle is only interested in supporting the JVM on its own hardware.

Java is tone deaf to the requirements of the machine it is executing on. The JVM is too sandboxed, too divorced from the reality of the environment it is working inside.

Ruby and Python are better citizens in this regard, but are hamstrung by their clumsy deployment strategies.

In the new world of metered cloud deployments in which we find ourselves, where you pay by the hour, the difference between a slow interpreted language, and a nimble compiled Go program is stark.

Go’s fresh take on portability, without the requirement to abstract yourself away from the machine your program runs on, is like no other language available today.

A command line renaissance

For the last few decades, since the rise of interpreted languages, or virtual machine run-times, programming has been less about writing small targeted tools, and more about managing the complexity of the environment those tools are deployed into.

Slow languages, or bloated deployments encourage programmers to pile additional functionality into one application to amortize the cost of installation and set up.

I believe that we are in the early stage of a command line renaissance, a renaissance driven by the rediscovery of languages which produce compiled, self contained, programs. Go is leading this charge.

A command line renaissance which enables developers to deliver simple programs that fit together, cross platform, in a way that suites the needs of the nascent cloud automation community, and reintroduces a generation of programmers to the art of writing tools which fit together, as Doug McIlroy described, “like segments in a garden hose”.

A key part of the renaissance is Go’s deployment story. I spoke earlier and many of my fellow speakers have praised Go for its pragmatic deployment story, focusing on server side deployments, but I believe there is more to this.

Over the last year we’ve seen a number of companies shift their client side tools from interpreted languages like Ruby and Python to Go. Cloud Foundry’s tools, Github’s hub, and MongoDB’s tool suite are the ones that spring to mind.

In every case their motivations were similar; while the existing tools worked well, the support load from customers who were not able to get the tool installed correctly on their machine was huge.

Go lets you write command line applications, that in turn enables developers to leverage the UNIX philosophy; small, sharp tools that work well together.

This is a command line renaissance that has been missing for a generation.

Conclusion

https://twitter.com/supermighty/status/548897982016663552

Go is a simple language, this was not an accident.

This was a deliberate decision, executed brilliantly by experienced designers who struck a chord with pragmatic developers.

Go is a language for programmers who want to get things done

Put simply, Go is a language for programmers who want to get things done.

“I just get things done instead of talking about getting them done.” — Henry Rollins

As Andrew Gerrand noted in his fifth birthday announcement

“Go arrived as the industry underwent a tectonic shift toward cloud computing, and we were thrilled to see it quickly become an important part of that movement.” — Andrew Gerrand

Go’s success is directly attributable to the factors that motivated its designers. As Rob Pike noted in his 2012 Splash paper.

“Go is a language designed by Google to help solve Google’s problems; and Google has big problems” — Rob Pike

And it turns out that Go’s design choices are applicable to the problems that an increasing number of professional programmers face today.

Go is growing

growingNovember last year, Go turned 5 years old as a public project.

In these 5 years, less if you consider that the language only reached 1.0 in April of 2012, Go, as a language, and a proposition to programmers and development teams, has been wildly successful.

In 2014 there were five international Go conferences. In 2015 there will be seven.

Plus

  • An ever growing engagement in social media; Twitter, Google plus, etc.
  • An established Reddit community.
  • Real time discussion communities like the -nuts IRC channel, or the slack gophers group.
  • Go featured in mainstream tech press, established companies are shipping Go APIs for their services.
  • Go training available in both professional and academic contexts.
  • Over 100 Go meetups around the world.
  • Sites like Damian Gryski’s Gophervids helping to disseminate the material produced by those meetups and conferences.
  • Community hackathon events like GopherGala and the Go Challenge.

Lastly, look around this room and see your peers, 350 experienced programmers, who have decided in invest in Go.

In closing

amorThis paper describes the language we have today. A language built with care and moderation. The language is what it is because of deliberate decisions that were made at every step.

Language design is about trade-offs, so learn to appreciate the care in which Go’s features were chosen and the skill in which they were combined.

While the language strives for simplicity, and is easy to learn, it does not immediately follow that the language is trivial to master.

There is a lot to love in our language, don’t be in such a hurry to dismiss it before you have explored it fully.

Learn to love the language. Really learn the language. It’ll take longer than you would think.

Learn to appreciate the choices of the designers.

Because, and I truly mean this, Go will make you a better programmer.

Cross compilation just got a whole lot better in Go 1.5

Introduction

Cross compilation is one of Go’s headline features. I’ve written about it a few times, and others have taken this work and built better tooling around it.

This morning Russ Cox committed this change which resolved the last issue in making cross compilation simpler and more accessible for all Gophers. When Go 1.5 ships in August any Go programmer will be able to cross compile their program without having to go through a fussy set up phase.

Background

In the current version of Go, (if you’re from the future, read Go 1.4 and earlier) before you could cross compile a Go program, you needed to go through a set up phase to enhance your Go installation with the bits necessary to build for other platforms.

This worked ok if you had built Go from source, but if you were using one of the binary distributions or something from your operating system (brew, apt, etc), then the process would turn into a maze of twisty passages, all alike.

This was necessary because for successful cross compilation you would need

  • compilers for the target platform, if they differed from your host platform, ie you’re on darwin/amd64 (6g) and you want to compile for linux/arm (5g).
  • a standard library for the target platform, which included some files generated at the point your Go distribution was built.

With the plan to translate the Go compiler into Go coming to fruition in the 1.5 release the first issue is now resolved. That just left the small amount of customisation to the runtime done at installation time, which has been whittled away as part of the compiler transition (most of the customisation was generation of include files for the C parts of the runtime), and as of this morning, removed.

Try it out

If you can’t wait til August, you can try this out today by building the development version of Go.

Disclaimer: the development branch is changing rapidly due to the re-factoring of the compilers. If you’re using Go in production, the release version, Go 1.4.x, is always recommended.

Prerequisites

As mentioned above to build the development version of Go, you also need Go 1.4 installed to bootstrap. Once Go 1.5 comes out this step will be unnecessary.

These steps are a simple procedure to do this which I would recommend following

  1. Uninstall any version of Go you have on your system, including any $PATH variables.
  2. Check out Go 1.4 and build it 
    % git clone https://go.googlesource.com/go $HOME/go1.4
% cd $HOME/go1.4/src
% git checkout release-branch.go1.4
% ./make.bash
  3. Check out the development branch and build it 
    % git clone https://go.googlesource.com/go $HOME/go
% cd $HOME/go/src
% env GOROOT_BOOTSTRAP=$HOME/go1.4 ./all.bash
  4. Add $HOME/go/bin to your $PATH

Build something

Notice: if you are reading this from a future where Go 1.5 has been released, you can skip the previous step.

Now you have the development version of Go 1.5 installed, cross compiling this simple program is trivial

package main

import "fmt"
import "runtime"

func main() {
        fmt.Printf("Hello %s/%s\n", runtime.GOOS, runtime.GOARCH)
}

Now build for darwin/386

% env GOOS=darwin GOARCH=386 go build hello.go
# scp to darwin host
$ ./hello
Hello darwin/386

Or build for linux/arm

% env GOOS=linux GOARCH=arm GOARM=7 go build hello.go
# scp to linux host
$ ./hello
Hello linux/arm

That’s it!

Cross compilation can’t get any simpler than that.

Technical mumbo-jumbo

So what is happening under the hood when we compile a program for a different platform ? The -v flag gives us a clue.

% go build -v hello.go
command-line-arguments

% env GOOS=linux GOARCH=arm go build -v hello.go                                  runtime
errors
sync/atomic
math
unicode/utf8
sync
io
syscall
time
strconv
reflect
os
fmt
command-line-arguments

Comparing the two examples above, the first build is using the standard library that was built as part of the Go installation. That is to say, all the packages that fmt and runtime depend on are already built into $GOROOT/pkg/linux_amd64.

In the second example, the go tool detects that all the dependencies of this program need to be built for the target plaform before hello.go can be compiled and builds them, all the way back to the runtime package.

I have not included the output here because it is very verbose, but you can look at all the steps that are being performed if you pass the -x flag to go build.

Out of scope

Cross compilation while linking to libraries via cgo is the holy grail for some. Sadly the changes mentioned above do not change the situation with respect to cgo. In fact you may still need to rebuild your Go installation in the traditional way to pass environment variables like CC_FOR_TARGET. Please try it out.

Conclusion

In case you can’t tell, I’m over the moon about this improvement. Cross compilation is Go’s ace in the hole and Go 1.5 will make it even better.

Please try it out and if you find issues please let us know on the GitHub issue tracker.

Lost in translation

Over the last year I have had the privilege of travelling to meet Go communities in Japan, Korea and India. In every instance I have met experienced, passionate, pragmatic programmers ready to accept Go for what it can do for them.

At the same time the message from each of these communities was the same; where is the documentation, where are the examples, where are the tutorials ? Travelling to these communities has been an humbling experience and has made me realise my privileged position as a native English speaker.

The documentation, and the tutorials, and the examples will come, slowly; this is open source after all. But what I can offer is the fact that all the content on this blog is licensed under a Creative Commons licence.

In short, I don’t have the skills, but if you do, you are welcome to translate any content on this site, and I’ll help you in any way I can.

 

Errors and Exceptions, redux

In my previous post, I doubled down on my claim that Go’s error handling strategy is, on balance, the best.

In this post, I wanted to take this a bit further, and prove that multiple returns and error values are the best,

When I say best, I obviously mean, of the set of choices available to programmers that write real world programs — because real world programs have to handle things going wrong.

The language we have

I am only going to use the Go language that we have today, not any version of the language which might be available in the future — it simply isn’t practical to hold my breath for that long. As I will show, additions to the language like dare I say, exceptions, would not change the outcome.

A simple problem

For this discussion, I’m going to start with a made up, but very simple function, which demonstrates the requirement for error handling.

package main

import "fmt"

// Positive returns true if the number is positive, false if it is negative.
func Positive(n int) bool {
        return n > -1
}

func Check(n int) {
        if Positive(n) {
                fmt.Println(n, "is positive")
        } else {
                fmt.Println(n, "is negative")
        }
}

func main() {
	Check(1)
	Check(0)
	Check(-1)
}

If you run this code, you get the following output

1 is positive
0 is positive
-1 is negative

which is wrong.

How can this single line function be wrong ? It is wrong because zero is neither positive or negative, and that cannot be accurately captured by the boolean return value from Positive.

This is a contrived example, but hopefully one that can be adapted to discuss the costs and benefits of the various methods of error handling.

Preconditions

No matter what solution is determined to be the best, a check will have to be added to Positive to test the non zero precondition. Here is an example with the precondition added

// Positive returns true if the number is positive, false if it is negative.
// The second return value indicates if the result is valid, which in the case
// of n == 0, is not valid.
func Positive(n int) (bool, bool) {
        if n == 0 {
                return false, false
        }
        return n > -1, true
}

func Check(n int) {
        pos, ok := Positive(n)
        if !ok {
                fmt.Println(n, "is neither")
                return
        }
        if pos {
                fmt.Println(n, "is positive")
        } else {
                fmt.Println(n, "is negative")
        }
}

Running this program we see that the bug is fixed,

1 is positive
0 is neither
-1 is negative

albeit in an ungainly way. For those interested, I also tried a version using a switch which was harder to read for the saving of one line of code.

This then is the baseline to compare other solutions.

Error

Returning a boolean is uncommon, it’s far more common to return an error value, even if the set of errors is fixed. For completeness, and because this simple example is supposed to hold up in more complex circumstances, here is an example using a value that conforms to the error interface.

// Positive returns true if the number is positive, false if it is negative.
func Positive(n int) (bool, error) {
        if n == 0 {
                return false, errors.New("undefined")
        }
        return n > -1, nil
}

func Check(n int) {
        pos, err := Positive(n)
        if err != nil {
                fmt.Println(n, err)
                return
        }
        if pos {
                fmt.Println(n, "is positive")
        } else {
                fmt.Println(n, "is negative")
        }
}

The result is a function which performs the same, and the caller must check the result in an near identical way.

If anything, this underlines the flexibility of Go’s errors are values methodology. When an error occurs, indicating only success or failure (think of the two result form of map lookup), a boolean can be substituted instead of an interface value, which removes the any confusion arising from typed nils and nilness of interface values.

More boolean

Here is an example which allows Positive to return three states, true, false, and nil (Anyone with a background in set theory or SQL will be twitching at this point).

// If the result not nil, the result is true if the number is
// positive, false if it is negative.
func Positive(n int) *bool {
        if n == 0 {
                return nil
        }
        r := n > -1
        return &r
}

func Check(n int) {
        pos := Positive(n)
        if pos == nil {
                fmt.Println(n, "is neither")
                return
        }
        if *pos {
                fmt.Println(n, "is positive")
        } else {
                fmt.Println(n, "is negative")
        }
}

Positive has grown another line, because of the requirement to capture the address of the result of the comparison.

Worse, now before the return value can be used anywhere, it must be checked to make sure that it points to a valid address. This is the situation that Java developers face constantly and leads to deep seated hatred of nil (with good reason). This clearly isn’t a viable solution.

Let’s try panicking

For completeness, let’s look at a version of this code that tries to simulate exceptions using panic.

// Positive returns true if the number is positive, false if it is negative.
// In the case that n is 0, Positive will panic.
func Positive(n int) bool {
        if n == 0 {
                panic("undefined")
        }
        return n > -1
}

func Check(n int) {
        defer func() {
                if recover() != nil {
                        fmt.Println("is neither")
                }
        }()
        if Positive(n) {
                fmt.Println(n, "is positive")
        } else {
                fmt.Println(n, "is negative")
        }
}

… this is just getting worse.

Not exceptional

For the truly exceptional cases, the ones that represent either unrecoverable programming mistakes, like index out of bounds, or unrecoverable environmental problem, like running out of stack, we have panic.

For all of the remaining cases, any error conditions that you will encounter in a Go program, are by definition not exceptional — you expect them because regardless of returning a boolean, an error, or pancing, it is the result of a test in your code.

Forgetting to check

I consider the argument that Developers forget to check error codes is cancelled out by the counter argument Developers forget to handle exceptions. Either may be true, depending on the language you are basing your argument on, but neither commands a winning position.

With that said, you only need to check the error value if you care about the result.

Knowing the difference between which errors to ignore and which to check is why we’re paid as professionals.

Conclusion

I have shown in the article that multiple returns and error values the simplest, and most reliable to use. Easier to use than any other form of error handling, including ones that do not even exist in Go as it stands today.

A challenge

So this is the best demonstration I can come up with, but I expect others can do better, particularly where the monadic style is used. I look forward to your feedback.

Inspecting errors

The common contract for functions which return a value of the interface type error, is the caller should not presume anything about the state of the other values returned from that call without first checking the error.

In the majority of cases, error values returned from functions should be opaque to the caller. That is to say, a test that error is nil indicates if the call succeeded or failed, and that’s all there is to it.

A small number of cases, generally revolving around interactions with the world outside your process, like network activity, require that the caller investigate the nature of the error to decide if it is reasonable to retry the operation.

A common request for package authors is to return errors of a known public type, so the caller can type assert and inspect them. I believe this practice leads to a number of undesirable outcomes:

  • Public error types increase the surface area of the package’s API.
  • New implementations must only return types specified in the interface’s declaration, even if they are a poor fit.
  • The error type cannot be changed or deprecated after introduction without breaking compatibility, making for a brittle API.

Callers should feel no more comfortable asserting an error is a particular type than they would be asserting the string returned from Error() matches a particular pattern.

Instead I present a suggestion that permits package authors and consumers to communicate about their intention, without having to overly couple their implementation to the caller.

Assert errors for behaviour, not type

Don’t assert an error value is a specific type, but rather assert that the value implements a particular behaviour.

This suggestion fits the has a nature of Go’s implicit interfaces, rather than the is a [subtype of] nature of inheritance based languages. Consider this example:

func isTimeout(err error) bool {
        type timeout interface {
                Timeout() bool
        }
        te, ok := err.(timeout)
        return ok && te.Timeout()
}

The caller can use isTimeout() to determine if the error is related to a timeout, via its implementation of the timeout interface, and then confirm if the error was timeout related — all without knowing anything about the type, or the original source of the error value.

Gift wrapping errors, usually by libraries that annotate the error path, is enabled by this method; providing that the wrapped error types also implement the interfaces of the error they wrap.

This may seem like an insoluble problem, but in practice there are relatively few interface methods that are in common use, so Timeout() bool and Temporary() bool would cover a large set of the use cases.

In conclusion

Don’t assert errors for type, assert for behaviour.

For package authors, if your package generates errors of a temporary nature, ensure you return error types that implement the respective interface methods. If you wrap error values on the way out, ensure that your wrappers respect the interface(s) that the underlying error value implemented.

For package users, if you need to inspect an error, use interfaces to assert the behaviour you expect, not the error’s type. Don’t ask package authors for public error types; ask that they make their types conform to common interfaces by supplying Timeout() or Temporary() methods as appropriate.