Monthly Archives: May 2015

Hear me speak about Go performance at OSCON

I’m going to be speaking at OSCON this year about Go performance. The title of the talk is High performance servers without the event loop and will focus on the features of the language and its runtime that transparently let Go programmers write high performance network servers without resorting to event loops and callback spaghetti.

As the OSCON website is a bit light on for details, here are some more to entice you.

Abstract

Conventional wisdom suggests that high performance servers require native threads, or more recently, event loops.

Neither solution is without its downside. Threads carry a high overhead in terms of scheduling cost and memory footprint. Event loops ameliorate those costs, but introduce their own requirements for a complex callback driven style.

A common refrain when talking about Go is it’s a language that works well on the server; static binaries, powerful concurrency, and high performance.

This talk focuses on the last two items, how the language and the runtime transparently let Go programmers write highly scalable network servers without having to worry about thread management or blocking I/O.

The goal of this talk is to introduce the following features of the language and the runtime:

  • Escape Analysis
  • Stack management
  • Processes and threads vs goroutines
  • Integrated network poller

These four features work in concert to build an argument for the suitability of Go as a language for writing high performance servers.

How does a 20% discount sound ?

O’Reilly have provided me with a discount code DAVEC20 that you can use to save 20% on your registration.

As a bonus, if enough people register for OSCON using the discount code DAVEC20, the organisers have said they will give me a free pass to the event which I plan to donate to Women Who Code. I know that this is a lot of ifs, but it is worth a try.

So, if you’re interested in hearing me, and hundreds of others speak at OSCON this year, you can save yourself 20% and recieve some good karma in the process. What’s too lose ?

Struct composition with Go

This is a quick Friday blog post to talk about a recent experience I had working on a piece Juju code that needed to capture the data being sent over a net.Conn.

Most Gophers know that the net package provides a net.Pipe function which returns a pair of net.Conns representing an in memory network connection. net.Pipe is ideal for testing components that expect to talk over the network without all the mucking around of actually using the network.

The Go standard library also contains the super useful io.MultiWriter function which takes any number of io.Writers and returns another io.Writer that will send a copy of any data written to it to each of its underlying io.Writers. Now I had all the pieces I needed to create a net.Conn that could record the data written through it.

func main() {
        client, server := net.Pipe()
        var buf bytes.Buffer
        client = io.MultiWriter(client, &buf)

        // ...
}

Except this code does not compile.

# command-line-arguments
/tmp/sandbox866813815/main.go:13: cannot use io.MultiWriter(client, &buf) (type io.Writer) as type net.Conn in assignment:
	io.Writer does not implement net.Conn (missing Close method)

The value returned by io.MultiWriter is an implementation of io.Writer, it doesn’t have the rest of the methods necessary to fulfil the net.Conn interface; what I really need is the ability to replace the Write method of an existing net.Conn value. We can do this with embedding by creating a structure that embeds both a net.Conn and an independant io.Writer as anonymous fields.

type recordingConn struct {
        net.Conn
        io.Writer
}

func main() {
        client, server := net.Pipe()
        var buf bytes.Buffer
        client = &recordingConn {
                Conn: client,
                Writer: io.MultiWriter(client, &buf),
        }

        // ...
}

The recodingConn embeds a net.Conn ensuring that recordingConn implements net.Conn. It also gives us a place to hang the io.MultiWriter so we can syphon off the data written by the client. There is only one small problem remaining.

# command-line-arguments
/tmp/sandbox439875759/main.go:24: recordingConn.Write is ambiguous

Because both fields in the structure are types that have a Write method, the compiler cannot decide which one should be the called. To resolve this ambiguity we can add a Write method on the recordingConn type itself:

func (c *recordingConn) Write(buf []byte) (int, error) {
        return c.Writer.Write(buf)
}

With the ambiguity resolved, the recordingConn is now usable as a net.Conn implementation. You can see the full code here.

This is just a small example of the power of struct composition using Go. Can you think of other ways to do this ?

Introducing gb, a project based build tool for the Go programming language

In April 2015 I gave a presentation to the GDG Berlin meetup group (slides, video) discussing my views on reproducible builds using the Go programming language.

Image credit, Nate Finch

Image credit, Nate Finch

As part of that presentation I demonstrated a replacement build tool that I had been developing, gb.

From the feedback I received after the meetup it was clear that many people understood and agreed with my reasoning why reproducible builds in Go are a problem today. However it was equally clear that I had not done a good enough job of explaining how gb works, and why I felt it was necessary to create an alternative build tool to solve the problem of reproducible builds.

With the help of the go-pm mailing list and the feedback from a group of early adopters of gb, I have put together a website for gb with expanded documentation and examples.

I invite you all to visit getgb.io to learn more about gb.