How the Go language improves expressiveness without sacrificing runtime performance

This week there was a discussion on the golang-nuts mailing list about an idiomatic way to update a slice of structs. For example, consider this struct representing a set of counters.

type E struct {
        A, B, C, D int
}

var e = make([]E, 1000)

Updating these counters may take the form

for i := range e {
        e[i].A += 1
        e[i].B += 2
        e[i].C += 3
        e[i].D += 4
}

Which is good idiomatic Go code. It's pretty fast too

BenchmarkManual   500000              4642 ns/op

However there is a problem with this example. Each access the ith element of e requires the compiler to insert an array bounds checks. You can avoid 3 of these checks by referencing the ith element once per iteration.

for i := range e {
        v := &e[i]
        v.A += 1
        v.B += 2
        v.C += 3
        v.D += 4
}

By reducing the number of subscript checks, the code now runs considerably faster.

BenchmarkUnroll  1000000              2824 ns/op

If you are coding a tight loop, clearly this is the more efficient method, but it comes with a cost to readability, as well as a few gotchas. Someone else reading the code might be tempted to move the creation of v into the for declaration, or wonder why the address of e[i] is being taken. Both of these changes would cause an incorrect result. Obviously tests are there to catch this sort of thing, but I propose there is a better way to write this code, one that doesn't sacrifice performance, and expresses the intent of the author more clearly.

func (e *E) update(a, b, c, d int) {
        e.A += a
        e.B += b
        e.C += c
        e.D += d
}

for i := range e {
        e[i].update(1, 2, 3, 4)
}

Because E is a named type, we can create an update() method on it. Because update is declared on with a receiver of *E, the compiler automatically inserts the (&e[i]).update() for us. Most importantly because of the simplicity of the update() method itself, the inliner can roll it up into the body of the calling loop, negating the method call cost. The result is very similar to the hand unrolled version.

BenchmarkUpdate   500000              2996 ns/op

In conclusion, as Cliff Click observed, there are Lies, Damn Lies and Microbenchmarks. This post is an example of a least one of the three. My intent in writing was not to spark a language war about who can update a struct the fastest, but instead argue that Go lets you write your code in a more expressive manner without having to trade off performance.

You can find the source code for the benchmarks presented in this post on Github,

Introducing gmx, runtime instrumentation for Go applications

What is gmx ?

gmx is an experimental package for instrumenting Go applications. gmx is similar to Java's jmx and provides a simple method of querying the internal state of your Go application by invoking anonymous functions bound to published keys. Here is an example using the included client, gmxc.

% ./gmxc -p 16378 runtime.version runtime.numcpu os.args 
os.args: [./godoc -v -http=:8080]
runtime.numcpu: 4
runtime.version: weekly.2012-01-27 11688+

How can I use gmx in my applications ?

In the example above, a stock godoc was instrumented by importing the github.com/davecheney/gmx package into main.

package main

import _ "github.com/davecheney/gmx"

The runtime and os instruments are provided by default by gmx.

How can I export my own values via gmx ?

You can publish gmx instruments at any point in your application. The most logical place is inside your package's init function.

package foo

import "github.com/davecheney/gmx"

var c1 = 1
var c2 = 2

func init() {
        // publish c1 via a closure
        gmx.Publish("c1", func() interface{} {
                return c1
        })
        // publish c2 via a function
        gmx.Publish("c2", getC2)
}

func getC2() interface{} {
        return c2
}

Using gmxc the values can be queried at runtime:

./gmxc -p 16467 c1 c2
c2: 2
c1: 1

Who is gmx aimed at ?

If you are developing an application in Go, especially a daemon or some other long running process, then gmx may appeal to you. If you live on the other side of the DevOps table, gmx will hopfully allow you to gain a greater understanding of the internal workings of applications you have been charged with caring for.

If you find gmx useful for you, please let me know. I plan to continue to develop the default instrumentation bundled with gmx and am always open to pull requests for additional functionality.

You can find the source to gmx on github, https://github.com/davecheney/gmx.

 

Why Go gets exceptions right

How does Go get exceptions right? Why, by not having them in the first place.

First, a little history.

Before my time, there was C, and errors were your problem. This was generally okay, because if you owned an 70's vintage mini computer, you probably had your share of problems. Because C was a single return language, things got a bit complicated when you wanted to know the result of a function that could sometimes go wrong. IO is a perfect example of this, or sockets, but there are also more pernicious cases like converting a string to it's integer value. A few idioms grew to handle this problem. For example, if you had a function that would mess around with the contents of a struct, you could pass a pointer to it, and the return code would indicate if the fiddling was successful. There are other idioms, but I'm not a C programmer, and that isn't the point of this article.

Next came C++, which looked at the error situation and tried to improve it. If you had a function which would do some work, it could return a value or it could throw an exception, which you were then responsible for catching and handling. Bam! Now C++ programmers can signal errors without having to conflate their single return value. Even better, exceptions can be handled anywhere in the call stack. If you don't know how to handle that exception it'll bubble up to someone who does. All the nastyness with errno and threads is solved. Achievement unlocked!

Sorta.

The downside of C++ exceptions is you can't tell (without the source and the impetus to check) if any function you call may throw an exception. In addition to worrying about resource leaks and destructors, you have to worry about RAII and transactional semantics to ensure your methods are exception safe in case they are somewhere on the call stack when an exception is throw. In solving one problem, C++ created another.

So the designers of Java sat down, stroked their beards and decided that the problem was not exceptions themselves, but the fact that they could be thrown without notice; hence Java has checked exceptions. You can't throw an exception inside a method without annotating that method's signature to indicate you may do so, and you can't call a method that may throw an exception without wrapping it in code to handle the potential exception. Via the magic of compile time bondage and discipline the error problem is solved, right?

This is about the time I enter the story, the early millennium, circa Java 1.4. I agreed then, as I do now, that the Java way of checked exceptions was more civilised, safer, than the C++ way. I don't think I was the only one. Because exceptions were now safe, developers started to explore their limits. There were coroutine systems built using exceptions, and at least one XML parsing library I know of used exceptions as a control flow technique. It's commonplace for established Java webapps to disgorge screenfuls of exceptions, dutifully logged with their call stack, on startup. Java exceptions ceased to be exceptional at all, they became commonplace. They are used from everything from the benign to the catastrophic, differentiating between the severity of exceptions falls to the caller of the function.

If that wasn't bad enough, not all Java exceptions are checked, subclasses of java.Error and java.RuntimeException are unchecked. You don't need to declare them, just throw them. This probably started out as a good idea, null references and array subscript errors are now simple to implement in the runtime, but at the same time because every exception Java extends java.Exception any piece of code can catch it, even if it makes little sense to do so, leading to patterns like

catch (e Exception) { // ignore }

So, Java mostly solved the C++ unchecked exception problem, and introduced a whole slew of its own. However I argue Java didn't solve the actual problem, the problem that C++ didn't solve either. The problem of how to signal to caller of your function that something went wrong.

Enter Go

Go solves the exception problem by not having exceptions. Instead Go allows functions to return an error type in addition to a result via it's support for multiple return values. By declaring a return value of the interface type error you indicate to the caller that this method could go wrong. If a function returns a value and an error, then you can't assume anything about the value until you've inspected the error. The only place that may be acceptable to ignore the value of error is when you don't care about the other values returned.

Go does have a facility called panic, and if you squint hard enough, you might imagine that panic is the same as throw, but you'd be wrong. When you throw and exception you're making it the caller's problem

throw new SomeoneElsesProblem();

For example in C++ you might throw an exception when you can't convert from an enum to it's string equivalent, or in Java when parsing a date from a string. In an internet connected world, where every input from a network must be considered hostile, is the failure to parse a string into a date really exceptional? Of course not.

When you panic in Go, you're freaking out, it's not someone elses problem, it's game over man.

panic("inconceivable")

panics are always fatal to your program. In panicing you never assume that your caller can solve the problem. Hence panic is only used in exceptional circumstances, ones where it is not possible for your code, or anyone integrating your code to continue.

The decision to not include exceptions in Go is an example of its simplicity and orthogonality. Using multiple return values and a simple convention, Go solves the problem of letting programmers know when things have gone wrong and reserves panic for the truly exceptional.

Three new SSH client features in Go weekly.2011-11-18

Over the last few weeks the exp/ssh package has been bubbling away quietly.

Firstly, the Client has finally grown support for publickey authentication. In place of handing over your username and password to the ClientConfig, now you can specify a number of ClientAuth implementations that will be negotiated with the remote server.

To use publickey authentication you need to supply your own type that implements the ClientKeyring interface. Here is a simple example, but more complicated intergrations with operating system keyrings is possible.

// keyring implements the ClientKeyring interface
type keyring struct {
        keys []*rsa.PrivateKey
}

func (k *keyring) Key(i int) (interface{}, error) {
        if i < 0 || i >= len(k.keys) {
                return nil, nil
        }
        return k.keys[i].PublicKey, nil
}

func (k *keyring) Sign(i int, rand io.Reader, data []byte) (sig []byte, err error) {
        hashFunc := crypto.SHA1
        h := hashFunc.New()
        h.Write(data)
        digest := h.Sum()
        return rsa.SignPKCS1v15(rand, k.keys[i], hashFunc, digest)
}

func (k *keyring) loadPEM(file string) error {
        buf, err := ioutil.ReadFile(file)
        if err != nil {
                return err
        }
        block, _ := pem.Decode(buf)
        if block == nil {
                return errors.New("ssh: no key found")
        }
        r, err := x509.ParsePKCS1PrivateKey(block.Bytes)
        if err != nil {
                return err
        }
        k.keys = append(k.keys, r)
        return nil
}

Because of the change to ClientAuth, if you want to use password authentication you will need to pass a type that implements ClientPassword.

// password implements the ClientPassword interface
type password string

func (p password) Password(user string) (string, error) {
        return string(p), nil
}

Tying it all together, Client authentication now looks like this:

func main() {
        k := new(keyring)
        err := k.LoadPEM("/path/to/your/privatekey")
        if err != nil { ... }
        config := &ssh.ClientConfig {
                User: "yourusername",
                Auth: []ssh.ClientAuth {
                        // ClientAuthPassword wraps a ClientPassword implementation
                        // in a type that implements ClientAuth.
                        ssh.ClientAuthPassword(password("yourpassword")),

                        // ClientAuthPublickey wraps a ClientKeyring implementation 
                        // in a type that implements ClientAuth.
                        ssh.ClientAuthPublickey(k),
                }
        }
        conn, err := ssh.Dial("tcp", "yourserver:22", config)
        if err != nil { ... }
}

Secondly, John Beisley has committed the first of a series of CLs that improves cipher handling in both the Client and Server. The commit includes support for AES and ARC4 ciphers and lays the groundwork for adding additional ciphers.

No changes are needed in client code to take advantage of these cipher improvements, the zero value for CryptoConfig in the ClientConfig struct defaults to a list of secure modern ciphers.

Finally, experimental support has been added to the ClientConn type for initiating direct-tcpip connections over SSH connections. The interface matches net.Dial and returns a net.Conn implementation. This allows the SSH clients, where supported, to tunnel TCP connections via the server.

// Dial an SSH server running on yourserver.com
c1, err := ssh.Dial("tcp", "yourserver.com:22", ... )
if err != nil { ... } 

// Dial news.yc via a tunnel to yourserver.com  
c2, err := c1.DialTCP("tcp", "news.ycombinator.org:80")
if err != nil { ... }

All of these are available in the current Go weekly, weekly.2011-11-18. Thanks to the intrepid souls who have already begun to explore the the exp/ssh package. I look forward to hearing about your adventures on golang-nuts.


But wait, there's more. If you're keen to live right on the bleeding +tip, this commit has made some improvements to the way interactive Sessions work. The goal is to follow the os/exec API, making Sessions more akin the to exec.Cmds. This features is very new, so I hope to post more as it solidifies after the next weekly.

Scratching my own itch, or how to publish multicast DNS records in Go

DHCP handles allocating IP addresses to the devices on my home network, but that leaves me with the problem of mapping IP addresses to names. The Macs and Linux hosts sort this out with Bonjour and Avahi, but annoyingly two devices, my router and my NAS don't support it.

Now, I could hack my /etc/hosts file for those two hosts, but that doesn't work well for embedded devices like iPad. Alternatively, I could install Avahi on my NAS, which is running Debian Sid armel, but space on it's tiny root file system is always at a premium.

Fortunately, my NAS is also one of my Go development systems, so after some prodding from +Brian Ketelsen, I wrote a package to replicate the parts of Avahi that I needed. Here it is

https://github.com/davecheney/mdns/

Here is a simple example of how to use the package.

import (
        "net"
        "github.com/davecheney/mdns"
)

func main() {
        // Publish a SVR record for ssh running on port 22 for my home NAS.

        // Publish an A record for the host
        mdns.PublishA("stora.local.", 3600, net.IPv4(192, 168, 1, 200))

        // Publish a PTR record for the _ssh._tcp DNS-SD type
        mdns.PublishPTR("_ssh._tcp.local.", 3600, "stora._ssh._tcp.local.")

        // Publish a SRV record typing the _ssh._tcp record to an A record and a port.
        mdns.PublishSRV("stora._ssh._tcp.local.", 3600, "stora.local.", 22)

        // Most mDNS browsing tools expect a TXT record for the service even if there
        // are not records defined by RFC 2782.
        mdns.PublishTXT("stora._ssh._tcp.local.", 3600, "")

        select{}
}

So that's it. This package has scratched my itch and I hope it is useful for others. I'd love to hear feedback and suggestions you've found it to be useful.

updated 16/10/2011 Thanks to @eneff for the select{} tip.

OS X has a built in WiFi scanner

Did you know that OS X has a built in WiFi survey tool? Nope, neither did I, but here it is

/System/Library/PrivateFrameworks/Apple80211.framework/Versions/A/Resources/airport

This won't be in your path, so to make things easier you could add the following to your ~/.profile

alias airport=/System/Library/PrivateFrameworks/Apple80211.framework/Versions/A/Resources/airport

Here is an example of the output.

% airport -s
         SSID BSSID             RSSI CHANNEL HT CC SECURITY (auth/unicast/group)
       fibble 00:26:de:ad:be:ef -59  9       Y  -- WEP
      flobble 00:60:ca:fe:ba:be -66  6       N  AU WPA(PSK/TKIP,AES/TKIP) WPA2(PSK/TKIP,AES/TKIP) 
       wibble 94:44:51:51:51:51 -62  6,+1    Y  -- WPA(PSK/AES,TKIP/TKIP) WPA2(PSK/AES,TKIP/TKIP) 
       wobble 00:21:af:af:af:af -60  3       N  US WPA(PSK/TKIP/TKIP) 
        ftang 00:18:3b:00:b1:15 -73  11      N  -- NONE

For the nitpickers, this data is also available in WIFi icon on the menu, with the important distinction that you can only find out the channel and signal strength if you associate with the AP. Using this tool you can easily do a quick WiFi survey to pick a good channel to avoid side lobe interference (hat tip to Matt Ryall).

Simple extended attribute support for Go

This afternoon I sat down and polished up the extended attribute library I have been working on for
a while. You can find the code and readme on github.

https://github.com/davecheney/xattr

This has been tested on linux (should work under 5g, 6g and 8g) and darwin (tested under Snow Leopard).

Installation

goinstall github.com/davecheney/xattr

Documentation

godoc github.com/davecheney/xattr

Usage

A example program is provided with the source. The simplest way to compile and install it is

make -C $GOROOT/src/pkg/github.com/davecheney/xattr/example clean install

This will install it to your $GOBIN directory. If you have trouble running this example, make sure there isn't another xattr somewhere higher in your $PATH.

Before you start

All extended attributes need a file to be associated with. In this example I'm going to create an empty file in my home directory.

touch ~/testfile

Setting an attribute

% $GOBIN/xattr -w username dave ~/testfile

Listing known attributes

% $GOBIN/xattr ~/testfile
username

Printing attribute values

% $GOBIN/xattr -p username ~/testfile
dave

Listing names and values

% $GOBIN/xattr -l ~/testfile
 username: dave

Using Clang 2.9 to build Google Go

Last night http://codereview.appspot.com/4527098 was comitted which allows Go to build under Clang 2.9.

% CC=/opt/local/bin/clang ./make.bash

Big props to Jeff Hodges for making this happen.

note: OS X users should be aware that Apple ship an older version of Clang if you have the XCode 4 tools installed. You should set CC to the absolute path to avoid confusion.

A new take on artificial scarcity

P34

Sometime in the last week, not one but two of these billboards appeared in the front yards of blocks of units on a street in Ashfield. Surprisingly spartan on details, the board implies that the wizards at Brough Real Estate have made two sets of sellers, buyers and a themselves very happy, an increasingly difficult task in the current climate. Less believable is the fact these properties are directly opposite each other.

Clearly this is a ruse.

With a reported 350,000 properties on the market across the country, the question becomes, who is the audience for this call to action?