Practical Go: Real world advice for writing maintainable Go programs

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Before I begin I want to mention that the more I research, read, talk to folks, and think about programming, the more I’m convinced that there are no rules. There are only guidelines.

What’s the difference? I’d say probably experience.

The mastery of a topic, Malcolm Gladwell’s 10,000 hour yardstick, is equal parts observation and practice. You watch someone else do the thing, while concurrently you practice doing the thing.

Knowing when to stick to the code and when to bend the rules probably comes down to experience, gained from observation, experimentation and research.

With that said, these are my experiences, and the goal today is not to be prescriptive. This is a discussion, not a lecture.

Code is read many more times than it is written. A single piece of code will, over its lifetime, be read hundreds, maybe thousands of times.

Clarity is important because all software, not just Go programs, is written by humans to be read by other humans. The fact that software is also consumed by machines is secondary.

If you’re writing a program for yourself, maybe it only has to run once, or you’re the only person who’ll ever see it, then do what ever works for you. But if this is a piece of software that more than one person will contribute to, or that will be used by people over a long enough time that requirements, features, or the environment it runs in may change, then your goal must be for your program to be maintainable.

Why should we strive for simplicity? Why is important that Go programs be simple?

We’ve all been in a situation where you say "I can’t understand this code", yes? We’ve all worked on programs where you’re scared to make a change because you’re worried it’ll break another part of the program; a part you don’t understand and don’t know how to fix. This is complexity.

Complexity turns reliable software in unreliable software. Complexity is what kills software projects. Therefore simplicity is the highest goal of Go. Whatever programs we write, we should be able to agree that they are simple.

The last underlying principle I want to highlight is productivity. Developer productivity is a sprawling topic but it boils down to this; how much time do you spend doing useful work verses waiting for your tools or hopelessly lost in a foreign code-base. Go programmers should feel that they can get a lot done with Go.

The joke goes that Go was designed while waiting for a C++ program to compile. Fast compilation is a key feature of Go and a key recruiting tool to attract new developers. While compilation speed remains a constant battleground, it is fair to say that operations which take minutes in other languages, take seconds in Go. This helps Go developers feel as productive as their counterparts working in dynamic languages without the reliability issues inherent in those languages.

More fundamental to the question of developer productivity, Go programmers realise that code is written to be read and so place the act of reading code above the act of writing it. Go goes so far as to enforce, via tooling and custom, that all code be formatted in a specific style. This removes the friction of learning a project specific dialect and helps spot mistakes because they just look incorrect.

Go programmers don’t spend days debugging inscrutable compile errors. They don’t waste days with complicated build scripts or deploying code to production. And most importantly they don’t spend their time trying to understand what their coworker wrote.

To say that Go is a language designed to be productive is an understanding it is built for software design in the large, at industrial scale.

Productivity is what the Go team mean when they say the language must scale.

Go is not a language that optimises for clever one liners. Go is not a language which optimises for the least number of lines in a program. We’re not optimising for the size of the source code on disk, nor how long it takes to type the program into an editor. Rather, we want to optimise our code to be clear to the reader.

Key to this clarity is the names we choose for identifies in Go programs. Let’s talk about the qualities of a good name:

Let’s talk about each of these properties in depth.

Sometimes people criticise the Go style for recommending short variable names. As Rob Pike said, "Go programmers want the right length identifiers". ^[1]

Andrew Gerrand suggests that by using longer identifies to indicate to the reader things of higher importance.

From this we can draw some guidelines:

Let’s look at an example:

In this example, the range variable p is declared on line 10 and only referenced once, on the following line. p lives for a very short time on the page and in limited scope during the execution of the function. A reader who is interested in the effect values of p have on the program need only read the three loop’s three lines.

By comparison people is declared in the function parameters, is live for the body of the function, and is referenced three times over seven lines. The same is true for sum, and count, thus they justify their longer names. The reader has to scan a wider number of lines to locate them so they are given more distinctive names.

I could have chosen s for sum and c (or possibly n) for count but this would have reduced all the variables in the program to the same level of importance. I could have chosen p instead of people but that would have left the problem of what to call the for …​ range iteration variable and the singular person would look odd as the loop iteration variable which lives for little time has a longer name than the slice of values it was derived from.

You shouldn’t name your variables after their types for the same reason you don’t name your pets "dog" and "cat". You shouldn’t include the name of your type in the name of your variable’s name for the same reason.

The name of the variable should describe its contents, not the type of its contents. Consider this example:

What’s good about this declaration? We can see that its a map, and it has something to do with the *User type, that’s probably good. But usersMap is a map, and Go being a statically typed language won’t let us accidentally use it where a different type is required. The Map suffix is redundant from the point of view of the compiler. Hence utility of the suffix is entirely down to whether we can prove it is of use to the reader.

Now, consider what happens if we were to declare other variables:

Now we have three map type variables in scope, usersMap, companiesMap, and productsMap, all mapping strings to different types. We know they are maps; it’s right there in their declaration. We also know that their map declarations prevent us from using one in place of another—​the compiler will throw an error if we try to use companiesMap where code was expecting a map[string]*User. In this situation it’s clear that the Map suffix does not improve the clarity of the code, its just extra boilerplate to type.

Removing the suffix leaves us with the more concise and equally descriptive:

My suggestion is to avoid any suffix that resembles the type of the variable.

This advice also applies to function parameters. For example:

Naming the *Config parameter config is redundant. We know its a *Config, it says so right there.

In this case consider conf or maybe c will do if the lifetime of the variable is short enough.

If there is more that one *Config in scope at any one time then calling them conf1 and conf2 is less descriptive than calling them original and updated as the latter are less likely to be mistaken for one another.

Another property of a good name is it should be predictable. The reader should be able to understand the use of a name when they encounter it for the first time. When they encounter a common name, they should be able to assume it has not changed meanings since the last time they saw it. You could also say that a good name should feel familiar.

For example, if your code passes around a database handle, make sure each time the parameter appears it has the same name. Rather than a combination of d *sql.DB, dbase *sql.DB, DB *sql.DB, and database *sql.DB, instead consolidate on something like;

and use it consistently across parameters, return values, local declarations, and potentially receivers. Doing so promotes familiarity; if you see a db, you know it’s a *sql.DB and that it has either been declared locally or provided for you by the caller.

Similar advice applies to method receivers; use the same receiver name every method on that type. This makes it easier for the reader to internalise the use of the receiver across the methods in this type.

Finally, certain single letter variables have traditionally been associated with loops and counting. For example, i, j, and k are commonly the loop induction variable for simple for loops. ^[2] n is commonly associated with a counter or accumulator. v is a common shorthand for a value in a generic encoding function, k is commonly used for the key of a map. a and b are generic names for parameters comparing two variables of the same type. x and y are generic names for local variables created for comparision, and s is often used as shorthand for parameters of type string.

As with the db example above programmers expect i to be a loop induction variable. If you ensure that i is always a loop variable, not used in other contexts outside a for loop. When readers encounter a variable called i, or j, they know that a loop is close by.

Go has at least six different ways to declare a variable

This list does not include receivers, formal parameters and named return values. I’m sure there are more that I haven’t thought of.

This is something that Go’s designers recognise was probably a mistake, but its too late to change it now, and, they argue, the bigger problem is shadowing. With all these different ways of declaring a variable, how do we avoid each Go programmer choosing their own style?

In Go each variable has a purpose because each variable we declare has to be used within the same scope. Here is a suggestion for how to make the purpose of each declaration clear to the reader. This is the style I try to use where possible.

To explain why, Let’s look at the previous example, but this time deliberately initialising each variable:

In the first and third examples, because in Go there are no automatic conversions from one type to another; the type on the left hand side of the assignment operator must be identical to the type on the right hand side. The compiler can infer the type of the variable being declared from the type on the right hand side, to the example can be written more concisely like this:

This leaves us with explicitly initialising players to 0 which is redundant because 0 is `players’ zero value. So it’s better to make it clear that we’re going to use the zero value by instead writing

What about the second statement? We cannot elide the type and write

Because nil does not have a type. ^[3] Instead we have a choice, do we want the zero value for a slice?

or do we want to create a slice with zero elements?

If we wanted the latter then this is not the zero value for a slice so we should make it clear to the reader that we’re making this choice by using the short declaration form:

Which tells the reader that we have chosen to initialise things explicitly.

This brings us to the third declaration,

Which is both explicitly initialising a variable and introduces the uncommon use of the new keyword which some Go programmer dislike. If we apply our short declaration syntax recommendation then the statement becomes

Which makes it clear that thing is explicitly initialised to the result of the expression new(Thing)--a pointer to a Thing--but still leaves us with the unusual use of new. We could address this by using the compact literal struct initialiser form,

Which does the same as new(Thing), hence why some Go programmers are upset by the duplication. However this means we’re explicitly initialising thing with a pointer to the literal Thing{}, which itself is the zero value for a Thing.

Instead we should recognise that thing is being declared as its zero value and use the address of operator to pass the address of thing to json.Unmarshall

In summary:

I talked about a goal of software engineering to produce maintainable code. Therefore you will likely spend most of your career working on projects of which you are not the sole author. My advice in this situation is: follow the local style.

Changing styles in the middle of a file is jarring. Uniformity, even if its not your preferred approach, is more valuable for maintenance over the long run than your personal preference. The rule I try to follow is; if it fits through go fmt then it’s usually not worth holding up a code review for.

I stated earlier that the name of a variable, or a constant, should describe its purpose. When you add a comment to a variable or constant, that comment should describe the variables contents, not the variables purpose.

In this example the comment describes why RandomNumber is assigned the value six, and where the six was derived from. The comment does not describe where RandomNumber will be used. This is deliberate, RandomNumber may be used many times by any package that references it. It is not possible to keep a record of all those uses at the site that RandomNumber is declared. Instead the name of the constant should be a guide the appropriate use for potential users.

Here are some more examples:

In general use the untyped constant 100 is just the number one hundred. In the context of HTTP the number 100 is known as StatusContinue, as defined in RFC 7231, section 6.2.1. The comment included with that declaration helps the reader understand why 100 has special significance as a HTTP response code.

Because godoc is the documentation for your package, you should always add a comment for every public symbol—​variable, constant, function, and method—​declared in your package.

Here are two rules from the Google Style guide:

There is one exception to this rule; you don’t need to document methods that implement an interface. Specifically don’t do this:

This comment says nothing. It doesn’t tell you what the method does, in fact it’s worse, it tells you to go look somewhere else for the documentation. In this situation I suggest removing the comment entirely.

Here is an example from the io package

Note that the LimitedReader declaration is directly preceded by the function that uses it, and the declaration of LimitedReader.Read follows the declaration of LimitedReader itself. Even though LimitedReader.Read has no documentation itself, its clear from that it is an implementation of io.Reader.

If the goal of a well designed Go package is to provide a set of related behaviours, writing a good package starts with choosing a good name. Think of your package’s name as a one world elevator pitch to describe what your package can do for you elevator companion.

Just as I talked earlier about nameing variables, the name of a package is very important. I start by asking myself questions like, "what is the purpose of this package" or "what does service does package provide?". Hopefully the answer to that question is "this package let’s you speak HTTP", not "this package provides the X type", otherwise its time to go back to the drawing board.

Within your project, each package name should be unique. This should pretty easy to if you’ve followed the previous advice that a package’s name should derive from its purpose. If you find you have two packages which need the same name, it is likely either;

A common cause of poor package names is what I call utility packages. These are packages where helpers and utility code congeals over time. As these packages contain an assortment of unrelated functions, their utility is hard to describe in terms of what the package provides. This often leads to the package’s name being derived from what the package contains--utilities.

Package names like utils or helpers are commonly found in larger projects which have developed deep package hierarchies and want to share helper functions without encountering import loops. By extracting utility functions to new package the import loop is broken, but because the package stems from a design problem in the project its name doesn’t reflect its purpose, only its function of breaking the import cycle.

My recommendation to improve the name of utils or helpers packages is to analyse where they are called and if possible move the relevant functions into their caller’s package. Even if this involves duplicating some helper code this is better than introducing an import dependency between two packages.

In the case where utility functions are used in many places prefer multiple packages, each focused on a single aspect, to a single monolithic package.

Packages with names like base or common are often found when functionality common to two or more implementations, or common types for a client and server, has been refactored into a separate package. Their names also represent design holdovers from languages like Java and C++ where the relationship between packages followed similar rules to those of inheretence. I believe the solution to packeges like base or common is to reduce the number of packages, combine the client, server, and common code into a single package named after the behaviour delivered from the previously fractured packages.

For example, the net/http package does not have client and server sub packages, instead it has a client.go and server.go file, each holding their respective types, and a transport.go file for the common message transport code.

As Go does not use exceptions for control flow there is no requirement to deeply indent your code just to provide a top level structure for the try and catch blocks. Rather than the successful path nesting deeper and deeper to the right, Go code is written in a style where the success path continues down the screen as the function progresses. My friend Mat Ryer calls this practice 'line of sight' coding. ^[5]

This is achieved by using guard clauses; conditional blocks with assert preconditions upon entering a function. Here is an example from the bytes package,

Upon entering UnreadRune the state of b.lastRead is checked and if the previous operation was not ReadRune an error is returned immediately. From there the rest of the function proceeds with the assertion that b.lastRead is greater that opInvalid.

Compare this to the same function written without a guard clause,

The body of the successful case, the most common, is nested inside the first if condition and the successful exit condition, return nil, has to be discovered by careful matching of closing braces. The final line of the function now returns an error, and the reader must trace the execution of the function back to the matching opening brace to know when control will reach this point.

This is more error prone for the reader, and the maintenance programmer, hence why Go prefer to use guard clauses and returning early on errors.

Every variable declaration, assuming no explicit initialiser is provided, will be automatically initialised to a value that matches the contents of zeroed memory. This is the value’s zero value. The type of the value determines it’s zero value; for numeric types it is zero, for string types it is "", for pointer types nil, the same for slices, maps, and channels.

This property of always setting a value to a known default is important for safety and correctness of your program and can make your Go programs simpler and more compact. This is what Go programmers talk about when they say "give your structs a useful zero value".

Consider the sync.Mutex type. sync.Mutex contains two unexported integer fields, representing the mutex’s internal state. Thanks to the zero value those fields will be set to will be set to 0 whenever a sync.Mutex is declared. sync.Mutex has been deliberately coded to take advantage of this property, making the type usable without explicit initialisation.

Another example of a type with a useful zero value is bytes.Buffer. You can declare a bytes.Buffer and start writing to it without explicit initialisation.

A useful property of slices is their zero value is nil. This makes sense if we look at the runtime’s (pseudo) definition of a slice header.

The zero value of this struct would imply len and cap have the value 0, and array, the pointer to memory holding the contents of the slice’s backing array, would be nil. This means unless you need to specify a size you don’t need to explicitly make a slice, you can just declare it.

A useful, albeit surprising, property of uninitialised pointer variables—​nil pointers—​is you can call methods on types that have a nil value. This can be used to provide default values simply.

The key to writing maintainable programs is that they should be loosely coupled. A change to one package should have a low probability of affecting another.

There are two excellent ways to achieve loose coupling in Go:

In Go we can declare variables at the block, function, or method scope, and also at the package scope. When the variable is public, given a identifier starting with a capital letter, then its scope is effectively global to the entire program—​any package may observe the type and contents of that variable at any time.

Mutable global state introduces tight coupling between independent parts of your program as global variables become an invisible parameter to every function in your program! Any function that relies on a global variable can be broken if that variable’s type changes. Any function that relies on the state of a global variable can be broken if another part of the program changes that variable.

If you want to reduce the coupling a global variable creates,

One of the things I tend to pick up in code review for programmers who are transitioning from other languages to Go is they tend to overuse packages. To be fair, mastery of Go is effectively the art of moderation in the use of all Go’s features, but package overuse seems to be one of the most common misstep. I suspect driven by programmers' innate drive to categorise and neatly organise the things they see.

Go does not provide elaborate ways of establishing visibility. Go lacks Java’s public, protected, private, and implicit default access modifiers. There is no equivalent of C++'s notion of a friend classes.

In Go we have only two access modifiers, public and private, the former indicated by the capitalisation of the first letter of the identifier. If an identifier is public, it’s name starts with a capital letter, that identifier can be referenced by any other Go package.

Given the limited controls available to control access to a package’s symbols, what practices should Go programmers follow to avoid creating over-complicated package hierarchies?

The advice I find myself repeating is to prefer fewer, larger packages. Your default position should be to not create a new package. That will lead to too many types being made public creating a wide, shallow, API surface for your package..

The sections below explores this suggestion in more detail.

Your main function, and main package should do as little as possible. This is because main.main acts as a singleton; there can only be one main function in a program.

Because main.main is a singleton there are a lot of assumptions built into the things that main.main will call, that they will only be called during main.main or main.init, and only called once. This makes it hard to write tests for code written in main.main. Main packages often invoke singletons, parse command line flags, expect files to be on disk in a certain place, and never expect to be executed concurrently. You can’t even reference main.main from a test.

Thus you should aim to move as much of your business logic out of your main function and ideally out of your main package. func main() should parse flags, open connections to databases, loggers, and such, then hand off execution to a high level object.

If you take anything away from this section, it should be this advice from Josh Bloch. If an API is hard to use for simple things, then every invocation will look complicated. When the actual invocation of the API is complicated it will be less obvious and more likely to be overlooked.

A few years ago I gave a talk ^[7] about using functional options ^[8] to make APIs easier to use for their default case.

The gist of this talk was you should design your APIs for the common, or default, use case. Said another way, your API should not require the caller to provide parameters which they don’t care about. More than being hard to use, you place the user in the position of guessing reasonable values. If they are lucky, the values they YOLO’d have no impact. That’s if they are lucky.

Let’s say I’ve been given a task to write a method that persists a Document structure to disk.

I could specify this method, Save, which takes an *os.File as the destination to write the Document. But this has a few problems.

The signature of Save precludes the option to write the data to a network location. Assuming that in the new world of lambda functions and microservices, network storage is likely to become requirement, the signature of this function would have to change, impacting all its callers.

Save is also unpleasant to test, because it operates directly with files on disk. To verify its operation the test would have to read the contents of the file after being written. You would also have to ensure that f was written to a temporary location and always removed afterwards.

Moreover *os.File defines a lot of methods which are not relevant to Save, like reading directories and checking to see if a path is a symlink. It would be useful if the signature of Save could describe only the parts of *os.File that were relevant.

Using io.ReadWriteCloser we can apply the interface segregation principle to redefine Save to take an interface that describes more general file shaped things. With this change, any type that implements the io.ReadWriteCloser interface can be substituted for the previous *os.File. This makes Save both broader in its application, and clarifies to the caller of Save which methods of the *os.File type are relevant to its operation. As the author of Save I no longer have the option to call those unrelated methods on *os.File as it is hidden behind the io.ReadWriteCloser interface. But we can take the interface segregation principle a bit further.

Firstly, it is unlikely that if Save follows the single responsibility principle, it will read the file it just wrote to verify its contents—​that should be responsibility of another piece of code.

We can narrow the specification for the interface we pass to Save to just writing and closing.

Secondly, by providing Save with a mechanism to close its stream, which we inherited in this desire to make it still look like a file, this raises the question of under what circumstances will wc be closed. Possibly Save will call Close unconditionally, or perhaps Close will be called in the case of success. Neither of these is a good option. Unconditionally closing wc after the call to Save precludes the caller from writing additional data after the document is written. Conditionally closing the WriteCloser — it doesn’t matter if its on success, or failure—​means the caller must grow intricate knowledge of the operation of Save.

A better solution would be to redefine Save to take only an io.Writer, stripping it completely of the responsibility to do anything but write data to a stream.

By applying the interface segregation principle to our Save function, the results has simultaneously been a function which is the most specific in terms of its requirements—​it only needs a thing that is writable—​and the most general in its function, we can now use Save to save our data to anything which implements io.Writer.

As a side effect it is clear that the name of the method is no longer accurate. A better name may be

nil is a curious beast.

nil is a constant, as I’ll show in a minute, but nil cannot be assigned to a named constant. nil can be assigned to a value, and values can be compared to nil, but nil cannot be compared to itself.

So, given all these restrictions, nil sounds out of place in the orthogonal Go world, why would such a concept exist? The answer is, while nil may appear inconsistent, it makes a lot of other interactions in Go simpler.

nil 's meaning, or it’s type, is fully determined by the static type of the variable it’s assigned to. When you write a statement like

The rule of expressions dictates that all the variables in the expression must have the same type. We know the type of f, it is a *os.File, therefore we know that nil has been coerced from an ideal constant to an expression which evaluates to the value also of type *os.File.

Here is a more complicated example

Again, the type of s is known, and as there are no conversions in the expression, the type of nil on the other side of the comparison must be the same, []string.

This statement is almost universal in the Go programmer’s phrase book, but what do Go programmers mean when they say "errors are just values", and what does this technique imply? By way of explanation, consider the counter example of panic and recover, often mistaken for exceptions.

panic and recover, two keywords added to the language for a single purpose. recover can only be used for one purpose; to access a value previously passed to panic. If that wasn’t enough recover’s use case is so specific, it can only be used inside a `defer block. You cannot use recover for any other purpose, it can only be used in concert with panic.

This pair of features sit by themselves in a corner of the language. How’s that for non orthogonal?

By contrast, error values are not limited to the rarefied semantics of panic and recover.

Programmers will rely on whatever behaviour, guaranteed or not, they observe from your API. Simply put, the more observable state your API returns, the larger the yoke of backwards compatibility you are implicitly committing to.

To the caller, the type and contents of an error value, if not nil, should be considered opaque. To do otherwise introduces brittle coupling between the function and its caller.

The exception to this rule are are sentinel values like io.EOF. These are however, the exception to the rule, not a pattern to be emulated.

The common contract for functions which return a value of the interface type error, is the caller should not presume anything about the state of the other values returned from that call without first checking the error. In the majority of cases, error values returned from functions should be opaque to the caller. That is to say, a test that error is nil indicates if the call succeeded or failed, and that’s all there is to it.

The methodology I follow is; if a function can return an error, you cannot make any assumptions about the state of any other values returned until you check the error. If it was found that the error was set (ie, not nil), then the state of those other values is unknown.

A small number of cases, require that the caller investigate the nature of the error to decide if it is reasonable to retry the operation. A common request for package authors is to return errors of a known public type, so the caller can type assert and inspect them. I believe this practice leads to a number of undesirable outcomes:

You should feel no more comfortable asserting an error is a particular type than they would be asserting the string returned from Error() matches a particular pattern.

Instead I present a suggestion that permits package authors and consumers to communicate about their intention, without having to overly couple their implementation to the caller. This suggestion fits the has a [behaviour] nature of Go’s implicit interfaces, rather than the is a [subtype of] nature of inheritance based languages. Consider this example:

The caller can use isTimeout to determine if the error is related to a timeout, and if so confirm if the error was timeout related, all without knowing anything about the type, or the original source of the error value.

Gift wrapping errors, usually by libraries that annotate the error path, is enabled by this method; providing that the wrapped error types also implement the interfaces of the error they wrap. This may seem like a generally intractable problem, but in practice there are relatively few interface methods that are in common use, so Timeout() bool and Temporary() bool cover a large set of use cases.

For package authors, if your package generates errors of a temporary nature, ensure you return error types that implement the respective interface methods. If you wrap error values on the way out, ensure that your wrappers respect the interface(s) that the underlying error value implemented.

For package users, if you need to inspect an error—​and hopefully this should be infrequent—​declare and assert an interface to assert the behaviour you expect, not the error’s type. Don’t ask package authors for public error types; instead ask that they make their types conform to common interfaces as appropriate.

When Go programmers discover that you can call a method on nil receiver it generally blows their mind.

This is because a method in Go is just syntactic sugar for a function who’s first parameter is the receiver.

Restated like this it is clear to see why passing a nil value for b is uneventful. However, a problem arises when the code attempts to access b or one of it’s fields.

Faced with this realisation Go programmers are grippe with fear that someone could call their code’s methods on an accidental nil method. Their usual reaction is a creeping panic that they will have to pepper their code with nil checks like this protect against this scenario.

The solution is to realise that the check for a nil receiver before attempting the call is in the right place.

Rather than checking inside the method when it is too late, the check should be executed by the caller. But this is seen as unsatisfactory because it force the check to every call site, rather than in one place, the receiver.

For arguments sake let’s explore the options to effectively handle a nil receiver inside the method. What are the options for an author to handle this situation?

Go’s error handling strategy is via the error interface and returning error values. Go does have panic, which is a by-product of the counterpart in the runtime’s internal throw function. There are few cases of using recover that I know of, and all of those are used to simulate non local transfer of control not exception handling. Using recover has all the problems of sensing errors by type, with the added complication that the set of types returned is unbounded.

While it is true that any Go function can call panic, any Go procedure can fail due to out of memory, the program can be killed by a process manager, or the serve can simply fail. Always write your programs to assume failure, not success. Avoid panic and eschew recover, they’re not the tool you are looking for.

Some of what I’ve discussed in this chapter will be obsoluted by the Go 2 changes to error handling I talked about the draft proposals for improving error handling. But do you know what is better than an improved syntax for handling errors? Not needing to handle errors at all.

This section draws inspiration from John Ousterhout’s recently book, A philosophy of Software Design ^[10]. One of the chapters in that book is called "Define Errors Out of Existence". We’re going to try to apply this advice to Go.

Lastly, I want to mention that you should only handle errors once. Handling an error means inspecting the error value, and making a single decision.

If you make less than one decision, you’re ignoring the error. As we see here, the error from w.WriteAll is being discarded.

But making more than one decision in response to a single error is also problematic. The following is code that I come across frequently.

In this example if an error occurs during w.Write, a line will be written to a log file, noting the file and line that the error occurred, and the error is also returned to the caller, who possibly will log it, and return it, all the way back up to the top of the program.

The caller is probably doing the same

So you get a stack of duplicate lines in your log file,

but at the top of the program you get the original error without any context.

I want to dig into this a little further because I don’t see the problems with logging and returning as just a matter of personal preference.

The problem I see a lot is programmers forgetting to return from an error. As we talked about earlier, Go style is to use guard clauses, checking preconditions as the function progresses and returning early.

In this example the author checked the error, logged it, but forgot to return. This has caused a subtle bug.

The contract for error handling in Go says that you cannot make any assumptions about the contents of other return values in the presence of an error. As the JSON marshalling failed, the contents of buf are unknown, maybe it contains nothing, but worse it could contain a half written JSON fragment.

Because the programmer forgot to return after checking and logging the error, the corrupt buffer will be passed to WriteAll, which will probably succeed and so the config file will be written incorrectly. However the function will return just fine, and the only indication that a problem happened will be a single log line complaining about marshalling JSON, not a failure to write the config.

In an earlier section I asserted that your packages should be designed around the behaviour they present. Unit tests at the package level should lock in the behaviour of the package’s API. They describe, in code, what the package promises to do.

If there is a unit test for each input permutation, you have defined the contract for what the code will do in code, not documentation. This is a contract you can assert as simply as typing go test. At any stage, you can know with a high degree of confidence, that the behaviour people relied on before your change continues to function after your change.

Let’s say we have a function to split strings.

How can we write a unit test for this function?

We start with a file in the same directory, with the same package name, strings. Tests are just regular Go functions with a few rules.

In our test we call Split with some inputs, then compare it to the result we expected.

The go tool supports writing your testing package tests in two places. Assuming your package is called http2, you can write a http2_test.go file and use the package http2 declaration. Doing so will compile the code in http2_test.go as if it were part of the http2 package. This is known colloquially as an internal test.

The go tool also supports a special package declaration, ending in test, ie., package http_test. This allows your test files to live alongside your code in the same package, however when those tests are compiled they are not part of your package’s code, they live in their own package. This allows you to write your tests as if you were another package calling into your code. This is known as an _external test.

I recommend using internal tests when writing unit tests for your package. This allows you to test each function or method directly, avoiding the bureaucracy of external testing.

However, you should place your Example test functions in an external test file. This ensures that when viewed in godoc, the examples have the appropriate package prefix and can be easily copy pasted.

Testing is more than just the hand to hand combat of software development. Tests are a critical component of making sure you can always ship your master branch.

For all the La Croix and ping pong tables, at the end of the day, as a development team, you are judged on your ability to deliver working software to the business. Your super power is at any time anyone on the team should be confident that the master branch of your code is shippable—​it may not have all the desired features, but what features are there should work as expected. This means at any time you can deliver a release of your software to the business and the business can recoup its investment in your development R&D.

I cannot emphasise this enough. If you want the non technical parts of the business to believe you are heros, you must never put yourself in the situation where you tell them "well, we can’t release right now because we’re in the middle of an important refactoring. It’ll be a few weeks. We hope.”

I’m not saying you cannot refactor, but at every stage your product has to be shippable. To give yourself the confidence that your product works as described, your tests have to pass.

Maybe the most important aspect of testing is good coverage gives you the confidence to change code that you didn’t write. This is critical when working in a codebase that is shared with others, or more likely on a codebase that has evolved alongside a team.

Fundamentally solid test coverage gives you the confidence to change the codebase you’re working in.

Go programmers quickly grasp the idea of a channel as a queue of values and are comfortable with the notion that channel operations may block when full or empty. In this section I want to explore several of the less common properties of channels.

When dealing with an unknown producer or consumer choose a buffer size of zero or one.

A buffer size of zero is ideal for coordination. A buffer size of one is idea to permit the sender to deposit the value without blocking and move on.

A buffer size greater than one is useful in the case where you know that exact number of values that will be deposited in the channel before it is drained. The common case is multiple workers operating in parallel, and a coordinator waiting on that result.

The most reasonable channels sizes are usually zero and one. Most other sizes are guesses. When you guess incorrectly, the program is unreliable.

What is the problem with this program?

The program does what we intended, it serves a simple web server. However it also does something else at the same time, it wastes CPU in an infinite loop. This is because the for{} on the last line of main is going to block the main goroutine because it doesn’t do any IO, wait on a lock, send or receive on a channel, or otherwise communicate with the scheduler.

As the Go runtime is mostly cooperatively scheduled, this program is going to spin fruitlessly on a single CPU, and may eventually end up live-locked.

How could we fix this? Here’s one suggestion.

This might look silly, but it’s a common common solution I see in the wild. It’s symptomatic of not understanding the underlying problem.

Now, if you’re a little more experienced with go, you might instead write something like this.

An empty select statement will block forever. This is a useful property because now we’re not spinning a whole CPU just to call runtime.GoSched(). However, we’re only treating the symptom, not the cause.

I want to present to you another solution, one which has hopefully already occurred to you. Rather than run http.ListenAndServe in a goroutine, leaving us with the problem of what to do with the main goroutine, simply run http.ListenAndServe on the main goroutine itself.

So this is my first piece of advice: if your goroutine cannot make progress until it gets the result from another, oftentimes it is simpler to just do the work yourself rather than to delegate it.

This often eliminates a lot of state tracking and channel manipulation required to plumb a result back from a goroutine to its initiator.

What is the difference between these two APIs?

The obvious differences are the first example reads a directory into a slice then returns the whole slice, or an error if something went wrong. This happens synchronously, the caller of ListDirectory blocks until all directory entries have been read. Depending on how large the directory, this could take a long time, and could potentially allocate a lot of memory building up the slide of directory entry names.

Lets look at the second example. This is a little more Go like, ListDirectory returns a channel over which directory entries will be passed. When the channel is closed, that is your indication that there are no more directory entries. As the population of the channel happens after ListDirectory returns, ListDirectory is probably starting a goroutine to populate the channel.

The channel version of ListDirectory has two further problems:

The solution to the problems of both implementations is to use a callback, a function that is called in the context of each directory entry as it is executed.

Not surprisingly this is how the filepath.WalkDir function works.

Perhaps fitting for the final topic in this presentation, we’re going to talk about stopping.

A previous example showed using a goroutine when one wasn’t really necessary. But one of the driving reasons for using Go is the first class concurrency features the language offers. Indeed there are many instances where you want to exploit the parallelism available in your hardware. To do so, you must use goroutines.

This simple application serves http traffic on two different ports, port 8080 for application traffic and port 8001 for access to the /debug/pprof endpoint.

Although this program isn’t very complicated, it represents the basis of a real application.

There are a few problems with the application as it stands which will reveal themselves as the application grows, so lets address a few of them now.

By breaking the serveApp and serveDebug handlers out into their own functions we’ve decoupled them from main.main. We’ve also followed the advice from above and make sure that serveApp and serveDebug leave their concurrency to the caller.

But there are some operability problems with this program. If serveApp returns then main.main will return causing the program to shutdown and be restarted by whatever process manager you’re using.

However, serveDebug is run in a separate goroutine and if it returns just that goroutine will exit while the rest of the program continues on. Your operations staff will not be happy to find that they cannot get the statistics out of your application when they want too because the /debug handler stopped working a long time ago.

What we want to ensure is that if any of the goroutines responsible for serving this application stop, we shut down the application.

Now serverApp and serveDebug check the error returned from ListenAndServe and call log.Fatal if required. Because both handlers are running in goroutines, we park the main goroutine in a select{}.

This approach has a number of problems:

What we’d really like is to pass any error that occurs back to the originator of the goroutine so that it can know why the goroutine stopped, can shut down the process cleanly.

We can use a channel to collect the return status of the goroutine. The size of the channel is equal to the number of goroutines we want to manage so that sending to the done channel will not block, as this will block the shutdown the of goroutine, causing it to leak.

As there is no way to safely close the done channel we cannot use the for range idiom to loop of the channel until all goroutines have reported in, instead we loop for as many goroutines we started, which is equal to the capacity of the channel.

Now we have a way to wait for each goroutine to exit cleanly and log any error they encounter. All that is needed is a way to forward the shutdown signal from the first goroutine that exits to the others.

It turns out that asking a http.Server to shut down is a little involved, so I’ve spun that logic out into a helper function. The serve helper takes an address and http.Handler, similar to http.ListenAndServe, and also a stop channel which we use to trigger the Shutdown method.

Now, each time we receive a value on the done channel, we close the stop channel which causes all the goroutines waiting on that channel to shut down their http.Server. This in turn will cause all the remaining ListenAndServe goroutines to return. Once all the goroutines we started have stopped, main.main returns and the process stops cleanly.