Author Archives: Dave Cheney

About Dave Cheney

A chaotic neutral System Administrator with super cow powers. My weapons are: * fear * cynicism * an almost fanatical devotion to the command line twitter.com/davecheney

Please, vote Yes for marriage equality in Australia

I wanted to write a few words about the postal survey on marriage law currently underway in Australia.

As an Australian, our country and our government do so many things that make me ashamed as a citizen; the poverty of our indigenous population, the inhumane treatment of refugees on Manus Island, and the maniacal desire to burn every last ounce of coal in the country, come hell and high water, to name just a few.

It is, quite frankly, overwhelming how institutionally cruel our government, which is after all a representation of the majority of Australians, can be, and nothing has sharpened this meanness to a point than the way the Liberal government have approached this survey.

With everything that is wrong in the world right now; climate change, the threat of nuclear war, and an unqualified narcissist running the White House, voting yes to the survey’s simple question is, quite literally, the smallest thing you could do to bring joy to two people.

So please, when you get your postal survey, vote yes.

Thank you.

Why I joined Heptio

Everyone gets the same set of tools

Something that had long puzzled me was the question “Why do some people [in the organisation] have root, and others do not?” It seemed to me that the reason the sysadmins had the root passwords, and everyone else had to raise tickets, was a tooling problem. Giving everyone root would permit anyone in the organisation to fix their own problems, deploy their own software, or, less charitably, cowboy things or be downright naughty. And while everyone had root, it usually turned out that only the operations team had the on call pager.

After the wholesale failure of organisations to understand Devops, I’m a big fan of the “You build it, you run it” movement. So when George Barnett and I built the Atlassian OnDemand Cloud we made a deliberate decision that everyone would get the same tools, and (modulo permissions and audit logs) be empowered to use the platform to the full extent. There wouldn’t be one set of tools for regular users, and a super set of “power tools” reserved for operators.

To me you build it, you run it, means if you have a problem, we’ll help you learn to use the tools better, not fix your problem for you.

Virtualise the operating system, not the hardware

I remember playing with VMware in 1999 or early 2000. I thought it was an amazing trick, especially as the drivers for my sound card worked way better in virtualized Windows than the real thing.

Fast forward a few years and I was using VMware to maintain a fleet of foreign language Windows installations for testing. Skip forward a few more and the industry had figured out that virtualisation was a solution to the sprawl of single use Windows servers that cluttered up wiring cupboards and data centres.

Virtualisation is a neat trick taken well beyond the point of a joke, but it did shine a light on the dark corners of systems administration. Back when turning up a server involved purchase orders, waiting for hardware to be shipped, contract negotiations, and trips to the data centre, what was a few hours spent installing the operating system? But when virtual hardware could be conjured out of thin air in seconds, it cast a long shadow over the need to automate operating system installation and management.

This was the age of Puppet and Chef, who re-plowed the ground sowed a decade earlier by CFEngine. Now sysadmins could configure and manage servers at the speed they could be virtually provisioned. I remember, thinking back to when I started to use Puppet, and imagining about what it would have been like to have those tools in previous jobs, where automation involved SVN repositories full of perl scripts, and crontab entries lovingly copy pasta’d between machines. And so everything was good for a time in the age of configuration as code.

But, simulating the entirety of an x86 host on another, just so people can share a computer, is a ridiculous waste. This shouldn’t be a surprise, FreeBSD Jails and Solaris Zones (rest in peace) had been coughing loudly about this for decades. Bryan Cantrill said it best when he exclaimed that we should “virtualise the operating system, not the hardware“, or as we’ve come to know them: containers.

The death of the operating system

I remember where I saw Docker for the first time. The product wasn’t even a year old and they were carpet bombing any meetup that would have them to promote it. Canonical were sprinting at a hotel near SFO and I convinced several of my teammates to squeeze into a taxi for the first meetup in San Mateo. What I saw that night shook me to my core. It wasn’t just the speed–oh the speed, after spending two years waiting for EC2 and slow apt mirrors–it was the clarity of that Californian mindset. What would happen if I checked my entire application deployment into git?

It was clear to me that night that virtual machines were virtualising stuff that people didn’t care about; virtual video cards, virtual floppy drives, virtual ram that swapped to virtual disks. What people wanted was a virtual kernel–their own pid 1. Orchestration tools like Chef, Puppet, and Juju were trying to orchestrate an entire operating system when what developers really wanted was a way to take a single program, the one that they had written, and deploy it to a server. Filesystems, crontabs, init/upstart/systemd, apt-get and dpkg-reconfigure, weren’t just someone else’s problem, they were irrelevant.

Anyone who’s endured to my rants about product knows my unwavering belief in the Innovator’s Dilemma. Through the window of Christensen’s logic, it was clear that the server orchestration market had been upended in that moment. Squeezed between Docker images at the low end and Netflix’s “everything is an AMI” model at the top end, was a large middle ground filled with orchestration tools that expected to be given a running operating system to configure. The Chefs and Puppets and whatnots would be desperately trying to convince the biggest orchestration users–the Netflixs of the world, with their CI/CD pipelines that pooped AMIs–to adopt agent based tooling, while all the while each developer faced with the question “How should I deploy my application?” would default to docker push.

Orchestration as table stakes

If you’re building your own orchestration layer, then you are betting on the wrong horse–I say this as someone who’s built a bespoke container based PaaS.

Within the next year or two you’ll be able to buy access to a Kubernetes API server at every price point; on your laptop, shared as a VPS, in your own VPC, or even as an appliance. Building on top of the Kubernetes primitives is where the value lies. Building on top of the shared tooling the Kubernetes API provides the level playing field that every development team who is responsible for supporting their own software in production is entitled to.

Why did I join Heptio? Because I believe that the administration of operating systems has reached its endgame. Kubernetes is going to revolutionise the way software is developed, and deployed, and I’m honoured to be given the opportunity to join the company that is going to make that happen.

I’m talking about Go at DevFest Siberia 2017

In September i’ll be speaking about Go at events in Russia and Taiwan.

DevFest Siberia 2017, September 23rd and 24th

I’ve been accepted to give two presentations at the GDG Novosibirsk DevFest Siberia 2017 event in Russia.

High performance servers without the event loop

Conventional wisdom suggests that the key to high performance servers are native threads, or more recently event loops. Neither solution is without downside. Threads carry a high overhead in terms of scheduling cost and memory footprint. Event loops lessen those costs, but introduce their own requirements for a complex callback driven style.

Go is a general purpose programming language in use in a wide range of domains and is well suited to writing network software. Go was introduced in 2009 with the explicit goal of helping programmers write programs that could solve problems of Google’s scale, and that means writing high performance servers.

This talk will focus on the features of the Go language and runtime environment, that allow programmers to write simple, high performance network services without resorting to native threads or event loop-driven callbacks.

Workshop: Exploring the Go execution tracer

As a complement to my conference talk I’ll be teaching a workshop on the Go execution tracer. This workshop follows on from my GolangUK presentation from last year and my High Performance Go workshop, and specifically focuses on the Go execution tracer,

The execution tracer is a new profiling and tracing facility integrated into Go since version 1.5. Unlike “external” profiling tools like pprof, valgrind, or perf, the execution tracer is integrated directly into the Go runtime, giving it detailed knowledge of the scheduler, the network poller, and the garbage collector.

In this workshop I will explain the operation of the execution tracer, how to collect, then analyse, the results of a trace. The audience will step through a set of problems, framed as the trace output of unknown programs to learn how to interpret the results from the execution tracer, improve our code to address performance or scalability bottlenecks, and verify the results.

You can find more information and purchase tickets for the event at the DevFest 2017 website.

Go Taiwan Meetup, Taipei, September 26th

I’ll be visiting the Go meetup in Taipei, Taiwan on the 26th of September. You can find details of the meetup soon on the GolangTW website.


Russian translation by Elena Grahovac

В сентябре я расскажу о Go на мероприятиях в России и Тайване.

DevFest Siberia 2017, Новосибирск, 23-24 сентября

Оргкомитет конференции DevFest Siberia 2017, которая пройдет в Новосибирске (Россия), принял мои заявки на два выступления.

Высокопроизводительные серверы без цикла событий

Бытует мнение, что ключом к написанию высокопроизводительных серверов является использование собственных потоков (native threads), место которых в последнее время занимают циклы событий (event loops). Однако, у обоих этих решений есть свои недостатки. Потоки, с точки зрения затрат на планирование и объем памяти, несут высокие накладные расходы. Циклы событий уменьшают эти затраты, но ставят определенные требования к витиеватым принципам разработки, основанной на callback’ах.

Go – это универсальный язык программирования, который используется в широком диапазоне областей и отлично подходит для написания сетевого программного обеспечения. Go был представлен в 2009 году, его цель – помочь разработчикам писать программы, которые могли бы решать задачи масштаба Google, то есть задачи написания высокопроизводительных серверов.

В этом докладе будут рассмотрены особенности языка и среды выполнения (runtime) Go, которые позволяют программистам писать простые высокопроизводительные сетевые сервисы, не прибегая к собственным потокам или callback’ам, связанным с циклом событий.

Мастер-класс: Изучаем трассировщик выполнения Go

В качестве дополнения к докладу я проведу мастер-класс по трассировщику выполнения (execution tracer) Go. Этот мастер-класс вытекает из моего доклада «Семь способов профилирования программы, написанной на Go» с прошлогодней конференции GolangUK и из моего мастер-класса «Высокая производительность Go». Новый мастер-класс фокусируется на трассировщике выполнения Go.

Трассировщик выполнения – это новое средство профилирования и трассировки, интегрированное в Go, начиная с версии 1.5. В отличие от «внешних» инструментов профилирования, таких как pprof, valgrind или perf, трассировщик выполнения интегрируется непосредственно в среду выполнения Go, предоставляя подробные сведения о планировщике (scheduler), сетевом поллере (network poller) и сборщике мусора (garbage collector).

В рамках мастер-класса я объясню, как работает трассировщик выполнения, и расскажу о том, как собрать, а затем проанализировать результаты трассировки. Шаг за шагом участники пройдут через набор задач, оформленных как вывод трассировки неизвестных программ, и узнают, как интерпретировать результаты трассировщика, улучшить код, устранить узкие места производительности или масштабируемости и проверить результаты.

Найти больше информации и приобрести билеты можно на сайте DevFest Siberia 2017.

Go Taiwan Meetup, Тайбэй, 26-е сентября

Я приеду на Go-митап в Тайбее (Тайвань) 26-го сентября. Детали мероприятия скоро появятся на сайте GolangTW.

The HERE IS key

The Lear Siegler ADM-3A terminal is a very important artefact in computing history.

ADM-3A keyboard (image credit vintagecomputer.ca)

If you want to know why your shell abbreviates $HOME to ~, it’s because of the label on the ~ key on the ADM-3A. If you want to know why hjkl are the de facto cursor keys in vi, look at the symbols above the letters. The ADM-3A was the “dumb terminal” which Bill Joy used to develop vi.

Recently the ADM-3A came up in a twitter discussion about the wretched Apple touch bar when Bret Victor dropped this tweet:

Which settled the argument until Paul Brousseau asked:

Indeed, what does the HERE IS1 key do? Its prominent position adjacent to the RETURN key implies whatever it does, it is important.

Fortunately the answer to Paul’s question was easy to find. The wonderful BitSavers archive has the user manual for the ADM-3A available (cached to avoid unnecessary bandwidth costs to BitSavers). On page 29 we find this diagram

Page 29, ADM-3A Users Manual (courtesy bitsavers.org)

So HERE IS, when pressed, transmits a predefined identification message. But what do to the words “message is displayed in half-duplex” mean? The answer to that riddle lies in the ADM-3A’s Answerback facility.

Scanning forward to page 36, section 3.3.6 describes the configuration of the Answerback facility–programming the identification message transmitted when HERE IS is pressed.

Section 3.3.6, page 36, ADM-3A Users Manual (courtesy bitsavers.org)

Pressing the HERE IS key or receiving an ENQ from the host … causes the answerback message to be transmitted to the host and to be displayed if the terminal is in half duplex mode.

This is interesting, the remote side can ask the terminal “who are you?”.

The HERE IS key is a vestige of a an older facility called Enquiry. Enquiry allowed one end of the connection to query if the remote side was still connected, and if it was, exactly who was connected.

ANSWERBACK Message
Answerback is a question and answer sequence where the host computer asks the terminal to identify itself. The VT100 answerback feature provides the terminal with the capability to identify itself by sending a message to the host. The entire answerback sequence takes place automatically without affecting the screen or requiring operator action. The answerback message may also be transmitted by typing CTRL-BREAK.

This description is from the 1978 Digital VT100 user guide. It was certainly a simpler time when the server could ask a terminal to identify itself, and trust the answer.


Notes

  1. I’ve chosen to write the name of the key in all caps as the base model of the ADM-3A was only capable of displaying upper case letters. If you wanted lower case (above 0x5F hex), that was an optional extra.

Context isn’t for cancellation

This is an experience report about the use of, and difficulties with, the context.Context facility in Go.

Many authors, including myself, have written about the use of, misuse of, and how they would changecontext.Context in a future iteration of Go. While opinions differs on many aspects of context.Context, one thing is clear–there is almost unanimous agreement that the Context.WithValue method on the context.Context interface is orthogonal to the type’s role as a mechanism to control the lifetime of request scoped resources.

Many proposals have emerged to address this apparent overloading of context.Context with a copy on write bag of values. Most approximate thread local storage so are unlikely to be accepted on ideological grounds.

This post explores the relationship between context.Context and lifecycle management and asks the question, are attempts to fix Context.WithValue solving the wrong problem?

Context is a request scoped paradigm

The documentation for the context package strongly recommends that context.Context is only for request scoped values:

Do not store Contexts inside a struct type; instead, pass a Context explicitly to each function that needs it. The Context should be the first parameter, typically named ctx:

func DoSomething(ctx context.Context, arg Arg) error {
        // ... use ctx ...
}

Specifically context.Context values should only live in function arguments, never stored in a field or global. This makes context.Context applicable only to the lifetime of resources in a request’s scope. Given Go’s lineage on the server, this is a compelling use case. However, there exist other use cases for cancellation where the lifetime of the resource extends beyond a single request. For example, a background goroutine as part of an agent or pipeline.

Context as a hook for cancellation

The stated goal of the context package is:

Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.

Which sounds great, but belies its catch-all nature. context.Context is used in three independent, yet sometimes conflated, scenarios:

  • Cancellation via context.WithCancel.
  • Timeout via context.WithDeadline.
  • A bag of values via context.WithValue.

At any point, a context.Context value can represent any one, or all three of these independent concerns. However, context.Context‘s most important facility, broadcasting a cancellation signal, is incomplete as there is no way to wait for the signal to be acknowledged.

Looking to the past

As this is an experience report, it would be germane to highlight some actual experience. In 2012 Gustavo Niemeyer wrote a package for goroutine lifecycle management called tomb which is used by Juju for the management of the worker goroutines within the various agents in the Juju system.

tomb.Tombs are concerned only with lifecycle management. Importantly, this is a generic notion of a lifecycle, not tied exclusively to a request, or a goroutine. The scope of the resource’s lifetime is defined simply by holding a reference to the tomb value.

A tomb.Tomb value has three properties:

  1. The ability to signal the owner of the tomb to shut down.
  2. The ability to wait until that signal has been acknowledged.
  3. A way to capture a final error value.

However, tomb.Tombs have one drawback, they cannot be shared across multiple goroutines. Consider this prototypical network server where a tomb.Tomb cannot replace the use of sync.WaitGroup.

func serve(l net.Listener) error {
        var wg sync.WaitGroup
        var conn net.Conn
        var err error
        for {
                conn, err = l.Accept()
                if err != nil {
                        break
                }
                wg.Add(1)
                go func(c net.Conn) {
                        defer wg.Done()
                        handle(c)
                }(conn)
        }
        wg.Wait()
        return err
}

To be fair, context.Context cannot do this either as it provides no built in mechanism to acknowledge cancellation. What is needed is a form of sync.WaitGroup that allows cancellation, as well as waiting for its participants to call wg.Done.

Context should become, well, just context

The purpose of the context.Context type is in it’s name:

context /kɒntɛkst/ noun
The circumstances that form the setting for an event, statement, or idea, and in terms of which it can be fully understood.

I propose context.Context becomes just that; a request scoped association list of copy on write values.

Decoupling lifetime management from context.Context as a store of request scoped values will hopefully highlight that request context and lifecycle management are orthogonal concerns.

Best of all, we don’t need to wait til Go 2.0 to explore these ideas like Gustavo’s tomb package.

Typed nils in Go 2

This is an experience report about a gotcha in Go that catches every Go programmer at least once. The following program is extracted from a larger version that caused my co-workers to lose several hours today.

package main

import "fmt"

type T struct{}

func (t T) F() {}

type P interface {
        F()
}

func newT() *T { return new(T) }

type Thing struct {
        P
}

func factory(p P) *Thing { 
        return &Thing{P: p}
}

const ENABLE_FEATURE = false

func main() {
        t := newT()
        t2 := t
        if !ENABLE_FEATURE {
                t2 = nil
        }
        thing := factory(t2)
        fmt.Println(thing.P == nil)
}

This distilled version of the program in question, while non-sensical, contains all the attributes of the original. Take some time to study the program and ask yourself, does the program print true or false?

nil != nil

Not to spoil the surprise, but the program prints false. The reason is, while nil is assigned to t2, when t2 is passed to factory it is “boxed” into an variable of type P; an interface. Thus, thing.P does not equal nil because while the value of P was nil, its concrete type was *T.

Typed nil

You’ve probably realised the cause of this problem is the dreaded typed nil, a gotcha that has its own entry in the Go FAQ. The typed nil emerges as a result of the definition of a interface type; a structure which contains the concrete type of the value stored in the interface, and the value itself. This structure can’t be expressed in pure Go, but can be visualised with this example:

var n int = 200 
var i interface{} = n

The interface value i is assigned a copy of the value of n, so i‘s type slot holds n‘s type; int, and it’s data slot holds the value 200. We can write this more concisely as (int, 200).

In the original program we effectively have the following:

var t2 *T = nil
var p P = t2

Which results in p, using our nomenclature, holding the value (*T, nil). So then, why does the expression p == nil evaluate to false? The explanation I prefer is:

  • nil is a compile time constant which is converted to whatever type is required, just as constant literals like 200 are converted to the required integer type automatically.
  • Given the expression p == nil, both arguments must be of the same type, therefore nil is converted to the same type as p, which is an interface type. So we can rewrite the expression as (*T, nil) == (nil, nil).
  • As equality in Go almost always operates as a bitwise comparison it is clear that the memory bits which hold the interface value (*T, nil) are different to the bits that hold (nil, nil) thus the expression evaluates to false.

Put simply, an interface value is only equal to nil if both the type and the value stored inside the interface are both nil.

For a detailed explanation of the mechanics behind Go’s interface implementation, Russ Cox has a great post on his blog.

The future of typed nils in Go 2

Typed nils are an entirely logical result of the way dynamic types, aka interfaces, are implemented, but are almost never what the programmer wanted. To tie this back to Russ’s GopherCon keynote, I believe typed nils are an example where Go fails to scale for programming teams.

This explanation has consumed 700 words–and several hours over chat today–to explain, and in the end my co-workers were left with a bad taste in their mouths. The clarity of interfaces was soured by a suspicion that gotchas like this were lurking in their codebase. As an experienced Go programmer I’ve learnt to be wary of the possibility of a typed nil during code review, but it is unfortunate that they remain something that each Go programmer has to learn the hard way.

For Go 2.0 I’d like to start the discussion of what it would mean if comparing an interface value to nil considered the value portion of the interface such that the following evaluated to true:

var b *bytes.Buffer
var r io.Reader = b
fmt.Println(r == nil)

There are obviously some subtleties that this pithy demand fails to capture, but a desire to make this seemingly straight forward comparison less error prone would, at least in my mind, make Go 2 easier to scale to larger development teams.

Should Go 2.0 support generics?

A long time ago, someone–I normally attribute this to David Symonds, but I can’t be sure he was the first to say it–said that the reason for adding generics to Go would be the reason for calling it Go 2.0. That is to say, adding generics to the language would be half baked if they were not used throughout the standard library. I wrote about this in a series of blog posts where I explored what I felt would be the repercussions of integrating templated types into Go.

Do I think Go should have generics? Well, there are really two answers to that question.

As I argued in my Simplicity Debt posts, mainstream programmers in 2017 expect a set of features in their languages. Many of us work in polyglot environments. Even if we want to be writing in Go as much as possible, there’s usually some Javascript, some CSS, some Python, maybe some Java, Swift, C#, PHP or even C++ in the project. Maybe this will change in the future, but right now, if you’re a commercial programmer working for a crust, every day you’ll touch a bunch of languages, so their differences tend to rub against one another.

  • Mainstream programmers expect static typing, not for performance, but for readability and maintainability–just look at what Typescript and Dart are bringing to Javascript, and Python’s formative efforts with optional typing.
  • Mainstream programmers expect concurrency. They expect to be able to do more than one thing at a time–just look at node.js and the compromises programmers were prepared to make to move away from heavy-weight thread per connection models. Go is obviously well positioned here.
  • Mainstream programmers expect some form of templated types because they’re used it in the other languages they interact with alongside Go.

So my first answer is: Go should have some form of generics because it is a mainstream, imperative, block scoped language and it is expected these days.

My second answer is if the designers of the language choose not to add templated types or parameterised functions–and keep in mind that I am not one of the language designers, only an exuberant fan–because, as I wrote in my series of posts, the repercussions for the simplicity and readability of the language may prove too jarring. If that were to happen, my recommendation would be that Go should own that decision.

What do I mean by that? Well, the best explanation I can give is a counterexample. Let’s look at Haskell. Haskell is what most functional programmers consider to be the baseline for a real FP language, and thus it looks pretty much like nothing programmers schooled in imperative, side effect ridden, block structured, languages are used to. But Haskell programmers own that. They own their difference, they don’t see it as a reason to make their language work more like PHP, or C++, or Rust, or even Go, and they are happy to explain the Haskell way of doing things to anyone who asks. My point is that if Go is not going to have a story for templated types, then we need to own it, just like Haskell programmers own their decisions.

This isn’t simply a case of saying “nope, sorry, no generics for Go 2.0, maybe in another 5 years”, but a more fundamental statement that they are not something that will be implemented in Go because we believe there is a better way to solve the underlying problem. Note that I did not say a better way to implement a templated type or parameterised function, but a better way to solve the underlying business problem. There is a difference.

This isn’t without precedent, Go was one of the first C style languages to eschew type inheritance, a decision which lead to a radical simplification of the language and a focus on the mantras of communicating intent via interfaces, and encapsulation over inheritance. Before Go, it was assumed that a mainstream language would have classes and a type hierarchy, nowadays that is less true.

So, should Go 2.0 have generics? If the decision is to add them then I’m sure it can be done, after all the syntax is the least important part of the decision, and there is a wealth of prior art in other languages to guide us. However, if the decision is not to add templated types, then it should be made so explicitly. Then it is incumbent upon all Go programmers to explain the Go Way of solving problems.

How to find out which Go version built your binary

This is a short post describing the procedure for discovering which version of Go was used to compile a Go binary.

This procedure relies on the fact that each Go program includes a copy of the version string reported by runtime.Version() . Linker magic ensures that this value will be present in the final binary irrespective of whether runtime.Version() is called by the resulting program. The value in question is stored in the runtime.buildVersion variable and can be recovered by a debugger.

The rest of this post describes the mechanisms for recovering the contents of runtime.buildVersion on various platforms.

Linux/FreeBSD/OpenBSD/NetBSD

If you’re on a Linux or *BSD platform, you can recover the binary build version with gdb.

% gdb $HOME/bin/godoc
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
(gdb) p 'runtime.buildVersion'
$1 = 0xa9ceb8 "go1.8.3"

Darwin

The debugging situation on OS X isn’t great, but here are several options.

gdb

gdb was removed from the XCode toolchain following the switch from gcc to llvm. If you are running a version of XCode that has gdb, you should used the instructions from the previous section.

Delve

Delve can be used to print the value of runtime.buildVersion.

% dlv exec $HOME/bin/godoc
Type 'help' for list of commands.
(dlv) b main.main
Breakpoint 1 set at 0x15596eb for main.main() ./golang.org/x/tools/cmd/godoc/main.go:156
(dlv) c
> main.main() ./golang.org/x/tools/cmd/godoc/main.go:156 (hits goroutine(1):1 total:1) (PC: 0x15596eb)
   151:                 }
   152:         }
   153:         log.Fatalf("too many redirects")
   154: }
   155:
=> 156: func main() {
   157:         flag.Usage = usage
   158:         flag.Parse()
   159:
   160:         playEnabled = *showPlayground
   161:
(dlv) p runtime.buildVersion
"go1.8.1"

lldb

Christian Witts reports on Twitter that XCode 8.3.3 ships with a version of lldb, version 370.0.42, that can interpret the Go string syntax.

$ lldb $HOME/bin/godoc
(lldb) b main.main
(lldb) run
(lldb) p runtime.buildVersion

I’ve tested earlier versions of lldb and found they do not work. Instread, use delve

Windows

Good news, everyone. Brian Ketelsen of GopherCon and GoTime.fm fame, reports that delve works perfectly on Windows for recovering this binaries’ build version.

PS C:\Users\bkete\go\src\http://github.com \derekparker\delve\cmd\dlv> dlv exec C:\Users\bkete\go\bin\dlv.exe
Type 'help' for list of commands.
(dlv) b main.main
Breakpoint 1 set at 0x8ec666 for main.main() c:/Users/bkete/go/src/github.com/derekparker/delve/cmd/dlv/main.go:11
(dlv) c
> main.main() c:/Users/bkete/go/src/github.com/derekparker/delve/cmd/dlv/main.go:11 (hits goroutine(1):1 total:1) (PC: 0x8ec666)
     6: )
     7:
     8: // Build is the git sha of this binaries build.
     9: var Build string
    10:
=>  11: func main() {
    12:         http://version.DelveVersion.Build  = Build
    13:         http://cmds.New ().Execute()
    14: }
(dlv) p runtime.buildVersion
"go1.8.1"

If someone wants to figure out the correct WinDbg or Visual Studio Debugger incantation, please let me know and I’ll link to you from this post.

Simplicity Debt Redux

In my previous post I discussed my concerns the additional complexity adding generics or immutability would bring to a future Go 2.0. As it was an opinion piece, I tried to keep it around 500 words. This post is an exploration of the most important (and possibly overlooked) point of that post.

Indeed, the addition of [generics and/or immutability] would have a knock-on effect that would profoundly alter the way error handling, collections, and concurrency are implemented. 

Specifically, what I believe would be the possible knock-on effect of adding generics or immutability to the language.

Error handling

A powerful motivation for adding generic types to Go is to enable programmers to adopt a monadic error handling pattern. My concerns with this approach have little to do with the notion of the maybe monad itself. Instead I want to explore the question of how this additional form of error handling might be integrated into the stdlib, and thus the general population of Go programmers.

Right now, to understand how io.Reader works you need to know how slices work, how interfaces work, and know how nil works. If the if err != nil { return err } idiom was replaced by an option type or maybe monad, then everyone who wanted to do basic things like read input or write output would have to understand how option types or maybe monads work in addition to discussion of what templated types are, and how they are implemented in Go.

Obviously it’s not impossible to learn, but it is more complex than what we have today. Newcomers to the language would have to integrate more concepts before they could understand basic things, like reading from a file.

The next question is, would this monadic form become the single way errors are handled? It seems confusing, and gives unclear guidiance to newcomers to Go 2.0, to continue to support both the error interface model and a new monadic maybe type. Also, if some form of templated maybe type was added, would it be a built in, like error, or would it have to be imported in almost every package. Note: we’ve been here before with os.Error.

What began as the simple request to create the ability to write a templated maybe or option type has ballooned into a set of question that would affect every single Go package ever written.

Collections

Another reason to add templated types to Go is to facilitate custom collection types without the need for interface{} boxing and type assertions.

On the surface this sounds like a grand idea, especially as these types are leaking into the standard library anyway. But that leaves the question of what to do with the built in slice and map types. Should slices and maps co-exist with user defined collections, or should they be removed in favour of defining everything as a generic type?

To keep both sounds redundant and confusing, as all Go developers would have to be fluent in both and develop a sophisticated design sensibility about when and where to choose one over the other. But to remove slices and maps in favour of collection types provided by a library raises other questions.

Slicing

For example, if there is no slice type, only types like a vector or linked list, what happens to slicing? Does it go away, if so, how would that impact common operations like handling the result a call to io.Reader.Read? If slicing doesn’t go away, would that require the addition of operator overloading so that user defined collection types can implement a slice operator?

Then there are questions on how to marry the built in map type with a user defined map or set. Should user defined maps support the index and assignment operators? If so, how could a user defined map offer both the one and two return value forms of lookup without requiring polymophic dispatch based on the number of return arguments? How would those operators work in the presence of set operations which have no value, only a key?

Which types could use the delete function? Would delete need to be modified to work with types that implement some kind of Deleteable interface? The same questions apply to append, lencap, and copy.

What about addressability? Values in the built in map type are not addressable, but should that be permitted or disallowed for user defined map types? How would that interact with operator overloading designed to make user defined maps look more like the built in map?

What sounded like a good idea on paper—make it possible for programmers to define their own efficient collection data types—has highlighted how deeply integrated the built in map and slice are and spawned not only a requirement for templated types, but operator overloading, polymorphic dispatch, and some kind of return value addressability semantics.

How could you implement a vector?

So, maybe you make the argument that now we have templated types we can do away with the built in slice and map, and replace them with a Java-esque list of collection types.

Go’s Pascal-like array type has a fixed size known at compile time. How could you implement a growable vector without resorting to unsafe hacks? I’ll leave that as an exercise to the reader. But I put it to you that if you cannot implement simple templated vector type with the memory safety we enjoy today with slices, then that is a very strong design smell.

Iteration

I’ll admit that the inability to use the for ... range statement over my own types was something that frustrated me for a long time when I came to Go, as I was accustomed to the flexibility of the iterator types in the Java collections library.

But iterating over in-memory data structures is boring—what you really want to be able to do is compose iterators over database results and network requests. In short, data from outside your process—and when data is outside your process, retrieving it might fail. In that case you have a choice, does your Iterable interface return a value, a value and an error, or perhaps you go down the option type route. Each would require a new form of range loop semantic sugar in an area which already contains its share of footguns.

You can see that adding the ability to write template collection types sounds great on paper, but in practice it would perpetuate a situation where the built in collection types live on in addition to their user defined counterparts. Each would have their strengths and weaknesses, and a Go developer would have to become proficient in both. This is something that Go developers just don’t have to think about today as slices and maps are practically ubiquitous.

Immutability

Russ wrote at the start of the year that a story for reference immutability was an important area of exploration for the future of Go. Having surveyed hundreds of Go packages and found few which are written with an understanding of the problem of data races—let alone actually tried running their tests under the race detector—it is tempting to agree with Russ that the ‘after the fact’ model of checking for races at run time has some problems.

On balance, after thinking about the problems of integrating templated types into Go, I think if I had to choose between generics and immutability, I’d choose the latter.

But the ability to mark a function parameter as const is insufficient, because while it restricts the receiver from mutating the value, it does not prohibit the caller from doing so, which is the majority of the data races I see in Go programs today. Perhaps what Go needs is not immutability, but ownership semantics.

While the Rust ownership model is undoubtedly correctiff your program complies, it has no data races—nobody can argue that the ownership model is simple or easy for newcomers. Nor would adding an extra dimension of immutability to every variable declaration in Go be simple as it would force every user of the language to write their programs from the most pessimistic standpoint of assuming every variable will be shared and will be mutated concurrently.

In conclusion

These are some of the knock on effects that I see of adding generics or immutability to Go. To be clear, I’m not saying that it should not be done, in fact in my previous post I argued the opposite.

What I want to make clear is adding generics or immutability has nothing to do with the syntax of those features, little to do with their underlying implementation, and everything to do with the impact on the overall complexity budget of the language and its libraries, that these features would unlock.

David Symonds argued years ago that there would be no benefit in adding generics to Go if they were not used heavily in the stdlib. The question, and concern, I have is; would the result be more complex than what we have today with our quaint built in slice, map, and error types?

I think it is worth keeping in mind the guiding principals of the language—simplicity and readability. The design of Go does not follow the accretive model of C++ or Java The goal is not to reinvent those languages, minus the semicolons.

Simplicity Debt

Fifteen years ago Python’s GIL wasn’t a big issue. Concurrency was something dismissed as probably unnecessary. What people really was needed was a faster interpreter, after all, who had more than one CPU? But, slowly, as the requirement for concurrency increased, the problems with the GIL increased.

By the time this decade rolled around, Node.js and Go had arrived on the scene, highlighting the need for concurrency as a first class concept. Various async contortions papered over the single threaded cracks of Python programs, but it was too late. Other languages had shown that concurrency must be a built-in facility, and Python had missed the boat.

When Go launched in 2009, it didn’t have a story for templated types. First we said they were important, but we didn’t know how to implement them. Then we argued that you probably didn’t need them, instead Go programmers should focus on interfaces, not types. Meanwhile Rust, Nim, Pony, Crystal, and Swift showed that basic templated types are a useful, and increasingly, expected feature of any language—just like concurrency.

There is no question that templated types and immutability are on their way to becoming mandatory in any modern programming language. But there is equally no question that adding these features to Go would make it more complex.

Just as efforts to improve Go’s dependency management situation have made it easier to build programs that consume larger dependency graphs, producing larger and more complex pieces of software, efforts to add templated types and immutability to the language would unlock the ability to write more complex, less readable software. Indeed, the addition of these features would have a knock on effect that would profoundly alter the way error handling, collections, and concurrency are implemented.

I have no doubt that adding templated types to Go will make it a more complicated language, just as I have no doubt that not adding them would be a mistake–lest Go find itself, like Python, on the wrong side of history. But, no matter how important and useful templated types and immutability would be, integrating them into a hypothetical Go 2 would decrease its readability and increase compilation times—two things which Go was designed to address. They would, in effect, impose a simplicity debt.

If you want generics, immutability, ownership semantics, option types, etc, those are already available in other languages. There is a reason Go programmers choose to program in Go, and I believe that reason stems from our core tenets of simplicity and readability. The question is, how can we pay down the cost in complexity of adding templated types or immutability to Go?

Go 2 isn’t here yet, but its arrival is a lot more certain than previously believed. As it stands now, generics or immutability can’t just be added to Go and still call it simple. As important as the discussions on how to add these features to Go 2 would be, equal weight must be given to the discussion of how to first offset their inherent complexity.

We have to build up a bankroll to spend on the complexity generics and immutability would add, otherwise Go 2 will start its life in simplicity debt.

Next: Simplicity Debt Redux