Practical Go: Real world advice for writing maintainable Go programs

Why should we strive for simplicity? Why is important that Go programs be simple?

We’ve all been in a situation where you say "I can’t understand this code", yes? We’ve all worked on programs where you’re scared to make a change because you’re worried it’ll break another part of the program; a part you don’t understand and don’t know how to fix. This is complexity.

Complexity turns reliable software in unreliable software. Complexity is what kills software projects. Therefore simplicity is the highest goal of Go. Whatever programs we write, we should be able to agree that they are simple.

Why is it important that Go code be readable? Why should we strive for readability?

Readability is important because all software, not just Go programs, is written by humans to be read by other humans. The fact that software is also consumed by machines is secondary.

Code is read many more times than it is written. A single piece of code will, over its lifetime, be read hundreds, maybe thousands of times.

Readability is key to being able to understand what the program is doing. If you can’t understand what a program is doing, how can you hope to maintain it? If software cannot be maintained, then it will be rewritten; and that could be the last time your company will invest in Go.

If you’re writing a program for yourself, maybe it only has to run once, or you’re the only person who’ll ever see it, then do what ever works for you. But if this is a piece of software that more than one person will contribute to, or that will be used by people over a long enough time that requirements, features, or the environment it runs in changes, then your goal must be for your program to be maintainable.

The first step towards writing maintainable code is making sure the code is readable.

The last underlying principle I want to highlight is productivity. Developer productivity is a sprawling topic but it boils down to this; how much time do you spend doing useful work verses waiting for your tools or hopelessly lost in a foreign code-base. Go programmers should feel that they can get a lot done with Go.

The joke goes that Go was designed while waiting for a C++ program to compile. Fast compilation is a key feature of Go and a key recruiting tool to attract new developers. While compilation speed remains a constant battleground, it is fair to say that compilations which take minutes in other languages, take seconds in Go. This helps Go developers feel as productive as their counterparts working in dynamic languages without the reliability issues inherent in those languages.

More fundamental to the question of developer productivity, Go programmers realise that code is written to be read and so place the act of reading code above the act of writing it. Go goes so far as to enforce, via tooling and custom, that all code be formatted in a specific style. This removes the friction of learning a project specific dialect and helps spot mistakes because they just look incorrect.

Go programmers don’t spend days debugging inscrutable compile errors. They don’t waste days with complicated build scripts or deploying code to production. And most importantly they don’t spend their time trying to understand what their coworker wrote.

Productivity is what the Go team mean when they say the language must scale.

Go is not a language that optimises for clever one liners. Go is not a language which optimises for the least number of lines in a program. We’re not optimising for the size of the source code on disk, nor how long it takes to type the program into an editor.

Key to this clarity is the names we choose for identifies in Go programs. Let’s talk about the qualities of a good name:

Let’s talk about each of these properties in depth.

Sometimes people criticise the Go style for recommending short variable names. As Rob Pike said, "Go programmers want the right length identifiers". ^[1]

Andrew Gerrand suggests that by using longer identifies to indicate to the reader things of higher importance.

From this we can draw some guidelines:

Let’s look at an example to

In this example, the range variable p is declared on line 10 and only referenced once, on the following line. p lives for a very short time both on the page, and during the execution of the function. A reader who is interested in the effect values of p have on the program need only read two lines.

By comparison people is declared in the function parameters and lives for seven lines. The same is true for sum, and count, thus they justify their longer names. The reader has to scan a wider number of lines to locate them so they are given more distinctive names.

I could have chosen s for sum and c (or possibly n) for count but this would have reduced all the variables in the program to the same level of importance. I could have chosen p instead of people but that would have left the problem of what to call the for …​ range iteration variable. The singular person would look odd as the loop iteration variable which lives for little time has a longer name than the slice of values it was derived from.

You shouldn’t name your variables after their types for the same reason you don’t name your pets "dog" and "cat". You also probably shouldn’t include the name of your type in the name of your variable’s name for the same reason.

The name of the variable should describe its contents, not the type of the contents. Consider this example:

What’s good about this declaration? We can see that its a map, and it has something to do with the *User type, that’s probably good. But usersMap is a map, and Go being a statically typed language won’t let us accidentally use it where a scalar variable is required, so the Map suffix is redundant.

Now, consider what happens if we were to declare other variables like:

Now we have three map type variables in scope, usersMap, companiesMap, and productsMap, all mapping strings to different types. We know they are maps, and we also know that their map declarations prevent us from using one in place of another—​the compiler will throw an error if we try to use companiesMap where the code is expecting a map[string]*User. In this situation it’s clear that the Map suffix does not improve the clarity of the code, its just extra boilerplate to type.

My suggestion is to avoid any suffix that resembles the type of the variable.

This advice also applies to function parameters. For example:

Naming the *Config parameter config is redundant. We know its a *Config, it says so right there.

In this case consider conf or maybe c will do if the lifetime of the variable is short enough.

If there is more that one *Config in scope at any one time then calling them conf1 and conf2 is less descriptive than calling them original and updated as the latter are less likely to be mistaken for one another.

Another property of a good name is it should be predictable. The reader should be able to understand the use of a name when they encounter it for the first time. When they encounter a common name, they should be able to assume it has not changed meanings since the last time they saw it.

For example, if your code passes around a database handle, make sure each time the parameter appears, it has the same name. Rather than a combination of d *sql.DB, dbase *sql.DB, DB *sql.DB, and database *sql.DB, instead consolidate on something like;

Doing so promotes familiarity; if you see a db, you know it’s a *sql.DB and that it has either been declared locally or provided for you by the caller.

Similar advice applies to method receivers; use the same receiver name every method on that type. This makes it easier for the reader to internalise the use of the receiver across the methods in this type.

Finally, certain single letter variables have traditionally been associated with loops and counting. For example, i, j, and k are commonly the loop induction variable for simple for loops. n is commonly associated with a counter or accumulator. v is a common shorthand for a value in a generic encoding function, k is commonly used for the key of a map, and s is often used as shorthand for parameters of type string.

As with the db example above programmers expect i to be a loop induction variable. If you ensure that i is always a loop variable, not used in other contexts outside a for loop. When readers encounter a variable called i, or j, they know that a loop is close by.

Go has at least six different ways to declare a variable

I’m sure there are more that I haven’t thought of. This is something that Go’s designers recognise was probably a mistake, but its too late to change it now. With all these different ways of declaring a variable, how do we avoid each Go programmer choosing their own style?

I want to present a suggestions for how I declare variables in my programs. This is the style I try to use where possible.

In the first and third examples, because in Go there are no automatic conversions from one type to another; the type on the left hand side of the assignment operator must be identical to the type on the right hand side. The compiler can infer the type of the variable being declared from the type on the right hand side, to the example can be written more concisely like this:

This leaves us with explicitly initialising players to 0 which is redundant because 0 is `players’ zero value. So its better to make it clear that we’re going to use the zero value by instead writing

What about the second statement? We cannot elide the type and write

Because nil does not have a type. ^[2] Instead we have a choice, do we want the zero value for a slice?

or do we want to create a slice with zero elements?

If we wanted the latter then this is not the zero value for a slice so we should make it clear to the reader that we’re making this choice by using the short declaration form:

Which tells the reader that we have chosen to initialise things explicitly.

This brings us to the third declaration,

Which is both explicitly initialising a variable and introduces the uncommon use of the new keyword which some Go programmer dislike. If we apply our short declaration syntax recommendation then the statement becomes

Which makes it clear that thing is explicitly initialised to the result of new(Thing)--a pointer to a Thing--but still leaves us with the unusual use of new. We could address this by using the compact literal struct initialiser form,

Which does the same as new(Thing), hence why some Go programmers are upset by the duplication. However this means we’re explicitly initialising thing with a pointer to a Thing{}, which is the zero value for a Thing.

Instead we should recognise that thing is being declared as its zero value and use the address of operator to pass the address of thing to json.Unmarshall

In summary:

I talked about a goal of software engineering to produce readable, maintainable, code. Therefore you will likely spend most of your career working on projects of which you are not the sole author. My advice in this situation is to follow the local style.

Changing styles in the middle of a file is jarring. Uniformity, even if its not your preferred approach, is more valuable for maintenance than your personal preference. My rule of thumb is; if it fits through gofmt then its usually not worth holding up a code review for.

I stated earlier that the name of a variable, or a constant, should describe its purpose. When you add a comment to a variable or constant, that comment should describe the variables contents, not the variables purpose.

In this example the comment describes why randomNumber is assigned the value six, and where the six was derived from. The comment does not describe where randomNumber will be used. Here are some more examples:

In the context of HTTP the number 100 is known as StatusContinue, as defined in RFC 7231, section 6.2.1.

Because godoc is the documentation for your package, you should always add a comment for every public symbol—​variable, constant, function, and method—​declared in your package.

Here are two rules from the Google Style guide

There is one exception to this rule; you don’t need to document methods that implement an interface. Specifically don’t do this:

This comment says nothing. It doesn’t tell you what the method does, in fact it’s worse, it tells you to go look somewhere else for the documentation. In this situation I suggest removing the comment entirely.

Here is an example from the io package

Note that the LimitedReader declaration is directly preceded by the function that uses it, and the declaration of LimitedReader.Read follows the declaration of LimitedReader itself. Even though LimitedReader.Read has no documentation itself, its clear from that it is an implementation of io.Reader.

Writing a good Go package starts with the package’s name. Think of your package’s name as an elevator pitch to describe what it does using just one word.

Just as I talked about names for variables in the previous section, the name of a package is very important. The rule of thumb I follow is not, "what types should I put in this package?". Instead the question I ask "what does service does package provide?" Normally the answer to that question is not "this package provides the X type", but "this package let’s you speak HTTP".

A common cause of poor package names is what call utility packages. These are packages where common helpers and utility code congeals over time. As these packages contain an assortment of unrelated functions, their utility is hard to describe in terms of what the package provides. This often leads to the package’s name being derived from what the package contains--utilities.

Package names like utils or helpers are commonly found in larger projects which have developed deep package hierarchies and want to share helper functions without encountering import loops. By extracting utility functions to new package the import loop is broken, but because the package stems from a design problem in the project, its name doesn’t reflect its purpose, only its function of breaking the import cycle.

My recommendation to improve the name of utils or helpers packages is to analyse where they are called and if possible move the relevant functions into their caller’s package. Even if this involves duplicating some helper code this is better than introducing an import dependency between two packages.

In the case where utility functions are used in many places prefer multiple packages, each focused on a single aspect, to a single monolithic package.

Packages with names like base or common are often found when functionality common to two or more implementations, or common types for a client and server, has been refactored into a separate package. I believe the solution to this is to reduce the number of packages, to combine the client, server, and common code into a single package named after the function of the package.

For example, the net/http package does not have client and server sub packages, instead it has a client.go and server.go file, each holding their respective types, and a transport.go file for the common message transport code.

As Go does not use exceptions for control flow there is no requirement to deeply indent your code just to provide a top level structure for the try and catch blocks. Rather than the successful path nesting deeper and deeper to the right, Go code is written in a style where the success path continues down the screen as the function progresses. My friend Mat Ryer calls this practice 'line of sight' coding. ^[4]

This is achieved by using guard clauses; conditional blocks with assert preconditions upon entering a function. Here is an example from the bytes package,

Upon entering UnreadRune the state of b.lastRead is checked and if the previous operation was not ReadRune an error is returned immediately. From there the rest of the function proceeds with the assertion that b.lastRead is greater that opInvalid.

Compare this to the same function written without a guard clause,

The body of the successful case, the most common, is nested inside the first if condition and the successful exit condition, return nil, has to be discovered by careful matching of closing braces. The final line of the function now returns an error, and the called must trace the execution of the function back to the matching opening brace to know when control will reach this point.

This is more error prone for the reader, and the maintenance programmer, hence why Go prefer to use guard clauses and returning early on errors.

Every variable declaration, assuming no explicit initialiser is provided, will be automatically initialised to a value that matches the contents of zeroed memory. This is the values zero value. The type of the value determines the value’s zero value; for numeric types it is zero, for pointer types nil, the same for slices, maps, and channels.

This property of always setting a value to a known default is important for safety and correctness of your program and can make your Go programs simpler and more compact. This is what Go programmers talk about when they say "give your structs a useful zero value".

Consider the sync.Mutex type. sync.Mutex contains two unexported integer fields, representing the mutex’s internal state. Thanks to the zero value those fields will be set to will be set to 0 whenever a sync.Mutex is declared. sync.Mutex has been deliberately coded to take advantage of this property, making the type usable without explicit initialisation.

Another example of a type with a useful zero value is bytes.Buffer. You can declare a bytes.Buffer and start writing to it without explicit initialisation.

A useful property of slices is their zero value is nil. This makes sense if we look at the runtime’s definition of a slice header.

The zero value of this struct would imply len and cap have the value 0, and array, the pointer to memory holding the contents of the slice’s backing array, would be nil. This means you don’t need to explicitly make a slice, you can just declare it.

A useful, albeit surprising, property of uninitialised pointer variables—​nil pointers—​is you can call methods on types that have a nil value. This can be used to provide default values simply.

The key to writing maintainable programs is that they should be loosely coupled—​a change to one package should have a low probability of affecting another package that does not directly depend on the first.

There are two excellent ways to achieve loose coupling in Go

In Go we can declare variables at the function or method scope, and also at the package scope. When the variable is public, given a identifier starting with a capital letter, then its scope is effectively global to the entire program—​any package may observe the type and contents of that variable at any time.

Mutable global state introduces tight coupling between independent parts of your program as global variables become an invisible parameter to every function in your program! Any function that relies on a global variable can be broken if that variable’s type changes. Any function that relies on the state of a global variable can be broken if another part of the program changes that variable.

If you want to reduce the coupling a global variable creates,

One of the things I tend to pick up in code review for programmers who are transitioning from other languages to Go is they tend to overuse packages.

Go does not provide elaborate ways of establishing visibility. Go lacks Java’s public, protected, private, and implicit default access modifiers. There is no equivalent of C++'s notion of a friend classes.

In Go we have only two access modifiers, public and private, indicated by the capitalisation of the first letter of the identifier. If an identifier is public, it’s name starts with a capital letter, that identifier can be referenced by any other Go package.

Given the limited controls available to control access to a package’s symbols, what practices should Go programmers follow to avoid creating over-complicated package hierarchies?

The advice I find myself repeating is to prefer fewer, larger packages. Your default position should be to not create a new package. That will lead to too many types being made public creating a wide, shallow, API surface for your package..

The sections below explores this suggestion in more detail.

Your main function, and main package should do as little as possible. This is because main.main acts as a singleton; there can only be one main function in a program, including tests.

Because main.main is a singleton there are a lot of assumptions built into the things that main.main will call that they will only be called during main.main or main.init, and only called once. This makes it hard to write tests for code written in main.main, thus you should aim to move as much of your business logic out of your main function and ideally out of your main package.

If you take anything away from this presentation, it should be this advice from Josh Bloch. If an API is hard to use for simple things, then every invocation of the API will look complicated. When the actual invocation of the API is complicated it will be less obvious and more likely to be overlooked.

A few years ago I gave a talk ^[6] about using functional options ^[7] to make APIs easier to use for their default case.

The gist of this talk was you should design your APIs for the common use case. Said another way, your API should not require the caller to provide parameters which they don’t care about.

Let’s say I’ve been given a task to write a function that persists a Document structure to disk.

I could specify this function, Save, which takes an *os.File as the destination to write the Document. But this has a few problems

The signature of Save precludes the option to write the data to a network location. Assuming that network storage is likely to become requirement later, the signature of this function would have to change, impacting all its callers.

Save is also unpleasant to test, because it operates directly with files on disk. So, to verify its operation, the test would have to read the contents of the file after being written.

And I would have to ensure that f was written to a temporary location and always removed afterwards.

*os.File also defines a lot of methods which are not relevant to Save, like reading directories and checking to see if a path is a symlink. It would be useful if the signature of the Save function could describe only the parts of *os.File that were relevant.

What can we do ?

Using io.ReadWriteCloser we can apply the interface segregation principle to redefine Save to take an interface that describes more general file shaped things.

With this change, any type that implements the io.ReadWriteCloser interface can be substituted for the previous *os.File.

This makes Save both broader in its application, and clarifies to the caller of Save which methods of the *os.File type are relevant to its operation.

And as the author of Save I no longer have the option to call those unrelated methods on *os.File as it is hidden behind the io.ReadWriteCloser interface.

But we can take the interface segregation principle a bit further.

Firstly, it is unlikely that if Save follows the single responsibility principle, it will read the file it just wrote to verify its contents—​that should be responsibility of another piece of code.

So we can narrow the specification for the interface we pass to Save to just writing and closing.

Secondly, by providing Save with a mechanism to close its stream, which we inherited in this desire to make it still look like a file, this raises the question of under what circumstances will wc be closed.

Possibly Save will call Close unconditionally, or perhaps Close will be called in the case of success.

This presents a problem for the caller of Save as it may want to write additional data to the stream after the document is written.

A better solution would be to redefine Save to take only an io.Writer, stripping it completely of the responsibility to do anything but write data to a stream.

By applying the interface segregation principle to our Save function, the results has simultaneously been a function which is the most specific in terms of its requirements—​it only needs a thing that is writable—​and the most general in its function, we can now use Save to save our data to anything which implements io.Writer.

If you were in my presentation yesterday I talked about the draft proposals for improving error handling. But do you know what is better than an improved syntax for handling errors? Not needing to handle errors at all.

This section draws inspiration from John Ousterhout’s recently book, A philosophy of Software Design ^[9]. One of the chapters in that book is called "Define Errors Out of Existence". We’re going to try to apply this advice to Go.

Lastly, I want to mention that you should only handle errors once. Handling an error means inspecting the error value, and making a single decision.

If you make less than one decision, you’re ignoring the error. As we see here, the error from w.WriteAll is being discarded.

But making more than one decision in response to a single error is also problematic. The following is code that I come across frequently.

In this example if an error occurs during w.Write, a line will be written to a log file, noting the file and line that the error occurred, and the error is also returned to the caller, who possibly will log it, and return it, all the way back up to the top of the program.

The caller is probably doing the same

So you get a stack of duplicate lines in your log file,

but at the top of the program you get the original error without any context.

I want to dig into this a little further because I don’t see the problems with logging and returning as just a matter of personal preference.

The problem I see a lot is programmers forgetting to return from an error. As we talked about earlier, Go style is to use guard clauses, checking preconditions as the function progresses and returning early.

In this example the author checked the error, logged it, but forgot to return. This has caused a subtle bug.

The contract for error handling in Go says that you cannot make any assumptions about the contents of other return values in the presence of an error. As the JSON marshalling failed, the contents of buf are unknown, maybe it contains nothing, but worse it could contain a half written JSON fragment.

Because the programmer forgot to return after checking and logging the error, the corrupt buffer will be passed to WriteAll, which will probably succeed and so the config file will be written incorrectly. However the function will return just fine, and the only indication that a problem happened will be a single log line complaining about marshalling JSON, not a failure to write the config.

What is the problem with this program?

The program does what we intended, it serves a simple web server. However it also does something else at the same time, it wastes CPU in an infinite loop. This is because the for{} on the last line of main is going to block the main goroutine because it doesn’t do any IO, wait on a lock, send or receive on a channel, or otherwise communicate with the scheduler.

As the Go runtime is mostly cooperatively scheduled, this program is going to spin fruitlessly on a single CPU, and may eventually end up live-locked.

How could we fix this? Here’s one suggestion.

This might look silly, but it’s a common common solution I see in the wild. It’s symptomatic of not understanding the underlying problem.

Now, if you’re a little more experienced with go, you might instead write something like this.

An empty select statement will block forever. This is a useful property because now we’re not spinning a whole CPU just to call runtime.GoSched(). However, we’re only treating the symptom, not the cause.

I want to present to you another solution, one which has hopefully already occurred to you. Rather than run http.ListenAndServe in a goroutine, leaving us with the problem of what to do with the main goroutine, simply run http.ListenAndServe on the main goroutine itself.

So this is my first piece of advice: if your goroutine cannot make progress until it gets the result from another, oftentimes it is simpler to just do the work yourself rather than to delegate it.

This often eliminates a lot of state tracking and channel manipulation required to plumb a result back from a goroutine to its initiator.

What is the difference between these two APIs?

Firstly, the obvious differences; the first example reads a directory into a slice then returns the whole slice, or an error if something went wrong. This happens synchronously, the caller of ListDirectory blocks until all directory entries have been read. Depending on how large the directory, this could take a long time, and could potentially allocate a lot of memory building up the slide of directory entry names.

Lets look at the second example. This is a little more Go like, ListDirectory returns a channel over which directory entries will be passed. When the channel is closed, that is your indication that there are no more directory entries. As the population of the channel happens after ListDirectory returns, ListDirectory is probably starting a goroutine to populate the channel.

The channel version of ListDirectory has two further problems:

The solution to the problems of both implementations is to use a callback, a function that is called in the context of each directory entry as it is executed.

Not surprisingly this is how the filepath.WalkDir function works.

The previous example showed using a goroutine when one wasn’t really necessary. But one of the driving reasons for using Go is the first class concurrency features the language offers. Indeed there are many instances where you want to exploit the parallelism available in your hardware. To do so, you must use goroutines.

This simple application serves http traffic on two different ports, port 8080 for application traffic and port 8001 for access to the /debug/pprof endpoint.

Although this program isn’t very complicated, it represents the basis of a real application.

There are a few problems with the application as it stands which will reveal themselves as the application grows, so lets address a few of them now.

By breaking the serveApp and serveDebug handlers out into their own functions we’ve decoupled them from main.main. We’ve also followed the advice from above and make sure that serveApp and serveDebug leave their concurrency to the caller.

But there are some operability problems with this program. If serveApp returns then main.main will return causing the program to shutdown and be restarted by whatever process manager you’re using.

However, serveDebug is run in a separate goroutine and if it returns just that goroutine will exit while the rest of the program continues on. Your operations staff will not be happy to find that they cannot get the statistics out of your application when they want too because the /debug handler stopped working a long time ago.

What we want to ensure is that if any of the goroutines responsible for serving this application stop, we shut down the application.

Now serverApp and serveDebug check the error returned from ListenAndServe and call log.Fatal if required. Because both handlers are running in goroutines, we park the main goroutine in a select{}.

This approach has a number of problems:

What we’d really like is to pass any error that occurs back to the originator of the goroutine so that it can know why the goroutine stopped, can shut down the process cleanly.

We can use a channel to collect the return status of the goroutine. The size of the channel is equal to the number of goroutines we want to manage so that sending to the done channel will not block, as this will block the shutdown the of goroutine, causing it to leak.

As there is no way to safely close the done channel we cannot use the for range idiom to loop of the channel until all goroutines have reported in, instead we loop for as many goroutines we started, which is equal to the capacity of the channel.

Now we have a way to wait for each goroutine to exit cleanly and log any error they encounter. All that is needed is a way to forward the shutdown signal from the first goroutine that exits to the others.

It turns out that asking a http.Server to shut down is a little involved, so I’ve spun that logic out into a helper function. The serve helper takes an address and http.Handler, similar to http.ListenAndServe, and also a stop channel which we use to trigger the Shutdown method.

Now, each time we receive a value on the done channel, we close the stop channel which causes all the goroutines waiting on that channel to shut down their http.Server. This in turn will cause all the remaining ListenAndServe goroutines to return. Once all the goroutines we started have stopped, main.main returns and the process stops cleanly.