Practical Go: Real world advice for writing maintainable Go programs

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

100 years ago David Hilbert was deeply unhappy with the state of mathematics. Mathematics was beset with inconsistencies and paradoxes. Seeking closure the German mathematican embarked on what became know as Hilbert’s Program, an attempt to reduce mathematics to a set of axiomatic proofs. From those foundations, mathematics could be reconstructed. Incontravertbily, provably, correct.

Sadly for Hilbert and his compatriots, their work was undone in 1931 when Kurt Gödel published his incompleteness theorum. Gödel demonstrated that, parodoxically, any system powerful enough to formalise mathematics would be unable to resolve its own inconsistences. Yet as fundamentally flawed as mathematics is, it has permitted us to understand the universe and the space between atoms.

A few years ago I met a man who told me that the Wright Brothers built the first successful aeroplane years before the laws of hydrodynamics were fully understood. They weren’t the first to fly, people had parachuted from balloons or glided from cliff tops, but they were the first to transition from ground to airborne.

This was 1903 and the theories of aerodynamics wouldn’t be developed for another 60 years, yet they still flew. How did they do this? How did the brothers' succeed in flight without an understanding of the physics that underpins the principle of lift?

Wilbur and Orville didn’t understand the science of why they flew, but they did learn from previous attempts. Their success was based on the experience gained from all those who had failed before them. Said another way, the Kitty Hawk flew because the Wrights were the first to fail to build an airplane which wouldn’t fly.

My friend asserted that just because we don’t understand the laws that underlie the development of software that doesn’t mean they don’t exist and, perhaps more importantly, the search for those theoretical underpinnings should not preclude an experimental approach.

Some argue that computer science is not a science. Alan Kay famously called it a pop culture. And this was my friends point. It’s not that computer science is a pop culture, or sociology, it’s that the practice of programming is running ahead of the science. We may be reasonably certain that a practice works, but we won’t know why until later.

So, I put it to you that the period of development of the practice of computer programming is running ahead of the scientific foundations on which is will ultimately be based. In a few hundred years we may have a solid scientific basis for computer programming, but right now all we have is empirical evidence.

Knowing when to stick to the code and when to bend the rules probably comes down to experience, gained from observation, experimentation, and research. What’s the difference between a rule and a guideline? I’d say, experience. The mastery of a topic, Malcolm Gladwell’s 10,000 hour yardstick, is equal parts observation and practice. You watch someone else do the thing, while concurrently you practice doing the thing.

I love writing software. I love discovering the patterns and structure that underlies design problems, but I do not pretend to represent the truth about the right way to program. Such a thing may not be knowable, but that should not preclude us from having a discussion based on experience.

With that said, these are my experiences, and the goal today is not to be prescriptive. This is a discussion, not a lecture.

If I’m going to talk about best practices for a programming language I need some framework by which to define what I mean by best.

Over the years that I’ve been thinking about this material I’ve tried to identify a common theme, and abstract that can summarise the tens of thousands of words I’ve written.

In forming this abstract many words have come and gone; simplicity, readability, clarity, productivity, but ultimately they are all synonyms for one word — m_aintainabilty_.

If there was a quote that summarises the ethos of writing Pratical Go code it would be this quote from Sandi Metz.

Go is not a language that optimises for clever one liners. Go is not a language which optimises for the least number of lines in a program. We’re not optimising for the size of the source code on disk, nor how long it takes to type the program into an editor.

Rather, we want to optimise our code to be clear to the reader. Key to this clarity is the names we choose for identifiers in Go programs. To get technical, when I’m talking about naming, I’m talking about naming identifiers in Go programs. But that’s a bit lengthy, so lets just call it naming from now on — you understand what I mean.

First, lets set the ground rules. Anything in Go that is an identifier has a name. What are the things in Go we can name?

Go programmers care about the name of things—​a lot. Naming is key to readability, hence the names of identifiers used in your program is critical. Poorly chosen names contribute to a program that is harder to comprehend thus harder to maintain.

Although not defined by go fmt, the canonical Go style for naming things descends from its original authors and can be identified by the following observations:

These two observations interlock to form general advice. For example, the names of local variables should be shorter than the names of the formal parameters.

Let’s talk about the qualities of a good name:

Let’s talk about each of these properties in depth.

You should avoid naming your variables after their types for the same reason you don’t name your pets "dog" and "cat". You shouldn’t include the name of your type in your variable’s name for the same reason.

Consider this example:

What’s good about this declaration? We can see that its a map, and it has something to do with the *User type, that’s probably good. But usersMap is a map, and Go being a statically typed language won’t let us accidentally use it where a different type is required. The Map suffix is redundant from the point of view of the compiler. Hence utility of the suffix is entirely down to whether we can prove it is of use to the reader.

Now, consider what happens if we were to declare other variables:

Now we have three map type variables in scope, usersMap, companiesMap, and productsMap, all mapping strings to different types. We know they are maps; it’s right there in their declaration. We also know that their map declarations prevent us from using one in place of another—​the compiler will throw an error if we try to use companiesMap where code was expecting a map[string]*User. In this situation it’s clear that the Map suffix does not improve the clarity of the code, its just extra boilerplate to type.

Removing the suffix leaves us with the more concise and equally descriptive:

My suggestion is to avoid any suffix that resembles the type of the variable. This advice also applies to function parameters. For example:

Naming the *Config parameter config is redundant. We know its a *Config, it says so right there.

In this case consider conf, or maybe c, if the lifetime of the variable is short enough. If there is more that one *Config in scope at any one time then calling them conf1 and conf2 is less descriptive than calling them original and updated as the latter are less likely to be mistaken for one another.

Where I live we have three levels of government; local, state and federal. It is universally accepted that this is one too many, however consensus on which level to eliminate is lacking. In much the same way, Go has at least six different ways to declare a variable:

This list does not include receivers, formal parameters and named return values. I’m sure there are more that I haven’t thought of.

This is something that Go’s designers recognise was probably a mistake, but its too late to change it now, and, they argue, the bigger problem is shadowing. With all these different ways of declaring a variable, how do we avoid each Go programmer choosing their own style?

In Go each variable has a purpose because each variable we declare has to be used within the same scope. Due to Go’s automatic type deduction it is uncommon to declare a type and initialise its value; you either do one or the other, declare without initialisation, or assign without initalisation. Here is a suggestion for how to make the purpose of each declaration clear to the reader. This is the style I try to use where possible.

To explain why, Let’s look at the previous example, but this time deliberately initialising each variable:

In the first and third examples, because in Go there are no automatic conversions from one type to another; the type on the left hand side of the assignment operator must be identical to the type on the right hand side. The compiler can infer the type of the variable being declared from the type on the right hand side, to the example can be written more concisely like this:

This leaves us with explicitly initialising players to 0 which is redundant because 0 is players' zero value. So it’s better to make it clear that we’re going to use the zero value by instead writing

What about the second statement? We cannot elide the type and write

Because nil does not have a type. ^[3] Instead we have a choice, do we want the zero value for a slice?

or do we want to create a slice with zero elements?

If we wanted the latter then this is not the zero value for a slice so we should make it clear to the reader that we’re making this choice by using the short declaration form:

Which tells the reader that we have chosen to initialise things explicitly.

This brings us to the third declaration,

Which is both explicitly initialising a variable and introduces the uncommon use of the new keyword which some Go programmer dislike. If we apply our short declaration syntax recommendation then the statement becomes

Which makes it clear that thing is explicitly initialised to the result of the expression new(Thing)--a pointer to a Thing--but still leaves us with the unusual use of new. We could address this by using the compact literal struct initialiser form,

Which does the same as new(Thing), hence why some Go programmers are upset by the duplication. However this means we’re explicitly initialising thing with a pointer to the literal Thing{}, which itself is the zero value for a Thing.

Instead we should recognise that thing is being declared as its zero value and use the address of operator to pass the address of thing to json.Unmarshall

The import statement declares a new identifier at the package level (technically the file level, but files which do not import the identifiers they need will not compile, so the distinction is mostly academic).

Consider the problems naming a package that deals with file descriptors, fd. fd would be the natural identifier for a file descriptor value returned by some hypothetical fd.Open function.

However don’t think up a convoluted package name just to retain the use of a convenient identifier.

FD is a bad name for a package, you want it for the variable, it’s also a bad name for a type, for the same reason.

An identifier’s name includes its package name. This means you should think about the name of your types, symbols, etc with their qualified name, package.Symbol.

A symbol’s name always includes the name of it’s package. A symbol’s name never includes the "full path" of its package, so applicationserver/v2/cache, is just cache. /apis/meta/v1 isn’t the v1 package of the meta package for the api, it’s just v1, and potentially conflicts with all the other v1 packages you imported.

If you find you have a lot of packages that have the same name, you place the user in the position that they’re going to have to rename your imports on import. This is undesirable as the name that symbols inside the file refer to your package as, is not the same name as the package’s declaration. Moving code between files is more laborious as goimports won’t work for you, and now looking at the name of the package in the symbols name you’re going to have to memorise the name you renamed the package too because it conflicted on import.

Scope and shadowing in Go are tightly linked. The former is considered to be a powerful tool in avoiding bugs, the latter, to the uninitiated, is a source of bugs. However the solution to both is adopting a consistent style which will highlight possible errors due to the irregularity.

The goal of scoping variables tightly is to turn the accidental use of a variable into a compile error. For example, compare:

to

The former restricts the scope of the binding err to the block from the start of the if statement to the closing brace. If this check was moved higher or lower in the function it will continue to compile without issue.

Compare this to the other example which requires that err had not been previously declared. If you move these four lines later in the function, it is possible that some other method expects err to be declared and will just be using assignment.

which will cause two compile errors.

Because godoc is the documentation for your package, you should always add a comment for every public symbol—​variable, constant, function, and method—​declared in your package.

Here are two rules from the Google Style guide:

There is one exception to this rule; you don’t need to document methods that implement an interface. Specifically don’t do this:

This comment says nothing. It doesn’t tell you what the method does, in fact it’s worse, it tells you to go look somewhere else for the documentation. In this situation I suggest removing the comment entirely (although some linters disagree).

Here is an example from the io package

Note that the LimitedReader declaration is directly preceded by the function that uses it, and the declaration of LimitedReader.Read follows the declaration of LimitedReader itself. Even though LimitedReader.Read has no documentation itself, its clear from that it is an implementation of io.Reader.

I stated earlier that the name of a variable, or a constant, should describe its purpose. When you add a comment to a variable or constant, that comment should describe the variable’s contents, not the variable’s purpose.

In this example the comment describes why RandomNumber is assigned the value six, and where the six was derived from. The comment does not describe where RandomNumber will be used. This is deliberate, RandomNumber may be used many times by any package that references it. It is not possible to keep a record of all those uses at the site that RandomNumber is declared. Instead the name of the constant should be a guide the appropriate use for potential users.

Here are some more examples:

In general use the untyped constant 100 is just the number one hundred. In the context of HTTP the number 100 is known as StatusContinue, as defined in RFC 7231, section 6.2.1. The comment included with that declaration helps the reader understand why 100 has special significance as a HTTP response code.

For variables without an initial value, the comment should describe who is responsible for initialising this variable.

This example comes deep from the bowels of the Go compiler. Here, the comment lets the reader know that the dowidth function is responsible for maintaining the state of sizeCalculationDisabled.

The fact that this advice runs contrary to previous advice that comments should not describe who uses them is a hint that dowidth and sizeCalculationDisabled are intimately entwined. The comments presence suggests a possible design weakness.

The comment on a function signature should describe what the function intends to do, not how it does it. If the name of the function is all the description it needs — even better. Similarly they should describe the inputs and outputs of a function, not be overly perscriptive of how those should be used. Rather than describe the type of the return value, the function’s comment should describe the value’s meaning.

The description should be sufficient to write a unit test for the documented behaviour.

Comments highlighting the grossness of a particular piece of code are not sufficient. If you encounter one of these comments, you should raise an issue as a reminder to refactor it later. It is okay to live with technical debt, as long as the amount of debt is known.

The tradition in the standard library is to annotate a TODO style comment with the username of the person who noticed it.

The username is not a promise that that person has comitted to fixing the issue, but they may be the best person to ask when the time comes to address it. Other project annotate todos with a date and or an issue number, which is a benficial tradition.

Functions should do one thing only. If you find yourself commenting a piece of code because it is unrelated to the rest of the function, consider extracting it into a function of its own.

In addition to being easier to comprehend, smaller functions are easier to test in isolation. Once you’ve isolated the orthogonal code into its own function, its name may be all the documentation required.

gofmt solved so many unproductive style wars, intentation, alignment, and so on, are a thing of the past, but vertical whitespace is still an open question.

This quote from the Google C++ style guide is most apt:

Just as you begin each paragraph—​each new thought—​with a line break, do so with a new set of statements in a function. This allows you to understand each part of a function, each set of statements. Ideally each function only has one set of statements, so no padding is necessary.

Each function should be written in terms of a single level of abstraction. Ideally a function should do one, and only one, thing.

This should place an upper limit on the length of a function which is beneificial because, besides longer functions being harder to read, longer functions are more likely to mix more than one idea. The required disentanglement must then be performed by the reader.

Every time you indent you add another precondition to the programmers stack consuming one of the 7 ±2 slots in their short term memory.

Go does not use exceptions for control flow thus there is no requirement to deeply indent your code to provide a top level structure for try and catch blocks. Rather than the successful path nesting deeper and deeper to the right, Go code is written in a style where the success path continues down the screen as the function progresses. Mat Ryer calls this practice 'line of sight' coding. ^[6]

This is achieved by using guard clauses; conditional blocks which assert preconditions upon entering a function. Here is an example from the bytes package,

Upon entering UnreadRune the state of b.lastRead is checked and if the previous operation was not ReadRune an error is returned immediately. From there the rest of the function proceeds with the assertion that b.lastRead is greater that opInvalid.

Compare this to the same function written without a guard clause,

The body of the successful case, the most commonly executed, is nested inside the first if condition and the successful exit condition, return nil, has to be discovered by careful matching of closing braces. The final line of the function now returns an error, and the reader must trace the execution of the function back to the matching opening brace to know when control will reach this point.

This is more error prone for the reader, and the maintenance programmer, hence why Go prefers to use guard clauses and returning early on errors.

Every variable declaration, assuming no explicit initaliser is provided, will be automatically intialised to a value that matches the contents of zero’d memory. This is the value’s zero value. The type of the value determines it’s zero value; for numeric types it is zero, for string types it is "", for pointer types, nil, the same for slices, maps, and channels.

This property of always setting a value to a known default is important for safety and correctness of your program and can make your Go programs simpler and more compact. This is what Go programmers talk about when they say "give your structs a useful zero value".

Consider the sync.Mutex type. sync.Mutex contains two unexported integer fields, representing the variable’s internal state. Thanks to the zero value those fields will be set to will be set to 0 whenever a sync.Mutex is declared. sync.Mutex has been deliberately written to take advantage of this property, making the type usable without explicit initialisation.

Another example of a type with a useful zero value is bytes.Buffer You can decare a bytes.Buffer and start writing to it without explicit initialisation.

A useful property of slices is their zero value is nil. This makes sense if we look at the runtime’s (pseudo) definition of a slice header.

The zero value of this struct would imply len and cap have the value 0, and array, the pointer to memory holding the contents of the slice’s backing array, would be nil. This means unless you need to specify a size you don’t need to explicitly make a slice, you can just declare it.

A useful, albeit surprising, property of uninitialised pointer variables—​nil pointers—​is you can call methods on types that have a nil value. This can be used to provide default values simply.

I despise configuration structs, a type who’s only purpose is to provide facts—​and they must be facts—​not variables, to another type. Instead, figure out how to make the original type configurable. This often means making its zero value usable.

Without exception, everything in Go is a copy. Fundamental to the understanding of Go are the three following axioms of Go values;

We also know that method calls are just syntactic sugar for calling a function an passing the receiver as the first parameter. So, what are the rules for how the receiver should be passed, as a value or a pointer to that value?

K&D[gopl] points out that if some of your methods have pointer receivers, all your methods should have pointer receivers. The same logic applies in reverse if you really want one method to have a value receiver; all the others must follow suite.

One argument is to always declare all methods on *T as it avoids copying and is thus faster. However Go developers are acutely attuned to this sort of absolutist thinking and tend to reject it without further proof. After all, we pass around slice and string values, both of which are several words in length without a care, so a blanket rule that every method must be declared on a pointer receiver for performance reads like dogma.

In the end I’m left with the unhappy compromise of;

In practical terms, pointer receiver should be your go to unless you are working on a specific type that you want to exploit the properties of the copying behaviour of value receivers. Methods on values should be used sparingly, and with great consideration.

When should something be a method on a type vs a free function? I recommend using methods where state is retained, functions where it is not.

Public functions are the way to communicate across packages, and interfaces are the mechanism to define behaviour across packages.

Pure functions are easier to test than methods because methods live on types and are inherently impure — they contain state via the receiver. The opposite is also true, if a method never mutates its receiver, should it be a method?

If you prefer a function to a method, which lets you add methods to interfaces, continue to place the receiver in the first formal parameter. As you wrap and abstract, parameters should move to the left, from the format parameters, to the receiver, to a type embedded in the receiver

Named return values permit the function’s author to;

Each of which are a net negative on the readability of the function.

In short, named return values are a symptom of a clever piece of code which should be reviewed with suspicion. If the method is infact simple, then named returns values are playing the short game of brevity over readabilty.

Its’s my opinion that names return arguments should not be used unless required to provide something that could not reasonably be done another way. For example, to modify the return arguments in a defer block, where it is required to name return arguments to capture them.

What is clear is that this function is complex, and named return values are part of that complexity.

All things being equal, you should aim to write simple code, not clever code. And so should avoid designs that require named return values.

Naked returns combine the declaration of a return value in the function declaration with an unspecified assignment somewhere in the body of the function. Everything about the use of naked returns admits a set of actions that hides bugs, in even small functions.

Naked returns are inconsistent; they make it look like the function or method returns no values, when infact it does, as they were declared in the function signature.

Naked returns are often used inconsistently, especially in an error path where nil is returned explicitly, or the zero value of a named return value is used. Combined with early returns [link] this results in multiple, sometimes conflicting, return stamements

“Use struct literal initialization to avoid invalid intermediate state. Inline struct declarations where possible.” — Peter Bourgon[bourgon2016]

Where possible values should be completely intitalised by construction, rather than by convention. We see examples of this failure at the most basic levels, for instance when declaring a value then overriding the default zero initalisation.

One the first line, userCount is in scope, but misleadingly holds the value 0. Only after countUsers has been called is userCount valid.

The incomplete initialsation pattern tends to show up in more complicated declarations

In this example vhost is incomplete, it has not yet had a set of routes appended too it. Compare this with

In the revised version, the routes slices is populated fully, then assigned to the &Virtualhost literal, noting that this literal is never given a name so cannot appear partially initialised.

Specifically avoid public Init functions.

Go contains a finalisation facility that lets the programmer register a function to be run when no live references to the object remain. Finalisation’s siren song of garbage collection for non memory resources can be beguiling. Inherently the idea is sound; in some programs it can be difficult to identify the owner of a resource like a file, a lock, or a socket. Do not use it.

At one point in the Go runtime’s development there were serious discussions about making finalisation a noop; functions registered for finalisation would simply be ignored. Fortuntately cooler heads prevailed, but had they not this would not have been a violation of the specification or the Go 1 guarentee. Runtime finalisation does not guarentee timely execution of finalisations; that can be delayed until after the program has exited, and still be compliant.

There is only one place in the entire standard library where finalisation is used; for variables of type *os.File. This use is at best a historical artifact. Given the serious problems with finalisation, and near prohibition in the standard library, do not design your software to rely on timely finalisation.

As we saw above nil can be simple, or complicated, depending on how you reason about it. One area where nil is complicated, until you memorise the rule is the dreaded typed nil.

If your method returns an interface type, be sure to always return nil explicitly. Assigning nil to a value of a concrete type and returning that will convert it to a typed nil

Go’s inclusion of nil tends to upset people who come from a Java, C#, or C++ background because they are traumatised by how nil operated in those languages.

In other languages, especially those that don’t support multiple return values, it is extremely common to return a nil like value a failure happens inside the method. On one hand this is eminently sensible, exceptions are overused, and for most failures they are hardly unexpected, so some mechanism of representing a failure that doesn’t warrant the four alarm fire of an exception is called for. Obviously this has some major downsides. As the flow of execution is not redirected a catch block this nil (or null, not naming any names) sentinel value now represent a silent failure condition.

Fortuntaly Go does not suffer from these drawbacks. This is for two reasons main reasons:

When Go programmers discover that you can call a method on nil receiver it generally blows their mind.

This is because a method in Go is just syntactic sugar for a function who’s first parameter is the receiver.

Restated like this it is clear to see why passing a nil value for b is uneventful. However, a problem arises when the code attempts to access b or one of it’s fields.

Faced with this realisation Go programmers are gripped with fear that someone could call their code’s methods on an accidental nil method. Their usual reaction is a creeping panic that they will have to pepper their code with nil checks like this protect against this scenario.

The solution is to realise that the check for a nil receiver before attempting the call is in the right place.

Rather than checking inside the method when it is too late, the check should be executed by the caller. But this is seen as unsatisfactory because it force the check to every call site, rather than in one place, the receiver.

For arguments sake let’s explore the options to effectively handle a nil receiver inside the method. What are the options for an author to handle this situation?

Given there is no reasonable way for the method executed on a nil receiver to protected against this, the remaining option is to simply not worry about it. After all a nil receiver is a symptom of a bug that happened elsewhere in your code. The most likely cause was a failure to check the error from a previous call. That is the place where you should spend your efforts, not defensively trying to code around a failure to follow proper error handling.

The method sets of a value of type T and type *T are disjoint.

You may convert a value of T to *T with the address of operator, &T. Conversely values of *T can be converted to values of T with the dereference operator.

Sadly Go retains the syntactic difficulty between the *T declaration and the *T dereference, although the former is a type, and the latter is a value of the result of the expression *T.

Although the method sets of T and *T are different, the compiler will help you by automatically inserting the relevant conversion operation, assuming the value is addressable.

This allows the caller to operate as if the methods available on T and *T are a union, almost all of the time (save where addressability is not present, see maps)

However, when a value is assigned to an interface type this illusion is broken.

Why is this so?

Deference is easy, however address of, converting a T into a *T would take the address of the copy of the value stored in the interfaces' value slot, not the original value before assignment.

Said another way, because everything in Go is a copy, there is no way to "wrap" an interface around an existing value, a copy must be taken, if the value is a not pointer type, then the copy of the original value placed inside an interface

TODO:unfinshed

If you take away one thing from this section, it should be this advice from Josh Bloch. If an API is hard to use for simple things, then every invocation will look complicated. When the actual invocation of the API is complicated it will be less obvious and more likely to be overlooked.

TODO: unfinished

Take away, libraries define concrete types, helpers, free function, clients define the behaviour they want with their own interfaces.

Good api design should make the default behaviour trivial to implement and none trivial behaviour possible to implement. Each public function that takes an argument must have a obvious and defensible default behaviour.

A few years ago I gave a talk ^[7] about using functional options ^[8] to make APIs easier to use for their default case.

The gist of this talk was you should design your APIs for the common, or default, use case. Said another way, your API should not require the caller to provide parameters which they don’t care about. More than being hard to use, you place the user in the position of guessing reasonable values. If they are lucky, the values they YOLO’d have no impact. That’s if they are lucky.

In this talk, I have presented many of the existing configuration patterns, those considered idiomatic and commonly in use today, and at every stage asked questions like: - Can this be made simpler? - Is that parameter necessary? - Does the signature of this function make it easy for it to be used safely? - Does the API contain traps or confusing misdirection that will frustrate?

Declarations provide the groundwork for a straightforward design, but it is the active elements of a Go program; the functions, the methods and it’s interfaces which bear the weight of the design of a Go program.

If a function is public and does not have anything to do with the package; uses none of the packages symbols * move it away * this might be a utility function

A good example of a simple looking, but hard to use correctly, API is one which takes two or more parameters of the same type. Let’s compare two function signatures:

What’s the difference between these two functions? Obviously one returns the maximum of two numbers, the other copies a file, but that’s not the important thing.

Max is commutative; the order of its parameters does not matter. The maximum of eight and ten is ten regardless of if I compare eight and ten or ten and eight.

However, this property does not hold true for CopyFile.

Which one of these statements made a backup of your presentation and which one overwrite your presentation with last week’s version? You can’t tell without consulting the documentation. A code reviewer cannot know if you’ve got the order correct without consulting the documentation.

The general advice is to try to avoid this situation. Just like long parameter lists, indistinct parameter lists are a design smell.

However, a possible solution to this class of problem is to introduce a helper type which will be responsible for calling CopyFile correctly.

In this way CopyFile is always called correctly and, given its poor API can possibly be made private, further reducing the likelihood of misuse.

It’s very common to write a function or method that takes a slice of values.

This is just an example I made up, but its common to a lot of code I’ve worked on. The problem with signatures like these is they presume that they will be called with more than one entry. However, what I have found is many times these type of functions are called with only one argument, which has to be "boxed" inside a slice just to meet the requirements of the functions signature.

Additionally, because the ids parameter is a slice, you can pass an empty slice or nil to the function and the compiler will be happy. This adds extra testing load because you should cover these cases in your testing.

To give an example of this class of API, recently I was refactoring a piece of logic that required me to set some extra fields if at least one of a set of parameters was non zero. The logic looked like this:

As the if statement was getting very long I wanted to pull the logic of the check out into its own function. This is what I came up with:

This enabled me to make the condition where the inner block will be executed clear to the reader:

However there is a problem with anyPositive, someone could accidentally invoke it like this

In this case anyPositive would return false because it would execute zero iterations and immediately return false. This isn’t the worst thing in the world — that would be if anyPositive returned true when passed no arguments.

Nevertheless it would be be better if we could change the signature of anyPositive to enforce that the caller should pass at least one argument. We can do that by combining normal and vararg parameters like this:

Now anyPositive cannot be called with less than one argument.

Go’s variable arguments syntax, the elipsis, is a very powerful tool in designing functions and methods that are easier to use. If you have a functions like

You can combine them into one function with

Which operates the same as DeployMany because the type of t inside Deploy is []*Thing.

Well designed interfaces are more likely to be small interfaces; the prevailing idiom is that interfaces contain only a single method. It follows that small interfaces lead to simpler implementations, because it is hard to do otherwise. This leads to programs comprised of simple implementations connected by small interfaces.

The opposite is also true, the larger the interface, the less abstraction it provides.

Interfaces must represent a generalisation of the behaviour of a set of implementations. The larger the interface, the larger the visible state of the implementation.

TODO: unfinished

Where-ever possible avoid reading data into a []byte and passing it around.

Depending on the request you may end up reading megabytes of data into memory. This places huge pressure on the GC, which will increase the average latency of your application. Instead use io.Reader and io.Writer to construct processing pipelines to cap the amount of memory in use per request. For efficiency, consider implementing io.ReaderFrom and io.WriterTo if you use a lot of io.Copy. These interface are more efficient and avoid copying memory into a temporary buffer.

Much programming involves the movement and manipulation of data. If you are lucky this data is already in memory, in convenient types like float64 or a struct. If you are unlucky, which is more often the case, you will be dealing with data in the form of raw []byte slices. This last statement is especially true for Go’s raison d’etre, client/server programming.

[]byte is always as big as it says, it’s a concrete value. io.Reader and io.Writer let you work at a higher level while working with a window of data.

Also solves ownership issues, if you have an io.Reader value, you have little control over the ownership of the underlying data, all you can ask is for some of that data to be read into a []byte slice that you provide.

Q\. What is the difference between a function and a method?

A\. There is no difference between a function and a method; a method is simply syntactic sugar for the receiver as the implicit first parameter to a function.

So, if there is no difference between a function and a method then when should one be used in favour of the other.

A function implicitly has no state, unless it’s accessing global state. A method therefore must be used when there is state local to its instance; and you could further draw the conclusion that a function that operates on global state is a singleton jnstange of a method on a singleton instance of an unnamed type

Just as it is a mistake to ask for formal parameters which go unused, asking for an interface type with methods that go uncalled is a smell.

Let’s say I’ve been given a task to write a method that persists a Document structure to disk.

I could specify this method, Save, which takes an *os.File as the destination to write the Document. But this has a few problems.

The signature of Save precludes the option to write the data to a network location. Assuming that in the new world of lambda functions and microservices, network storage is likely to become requirement, the signature of this function would have to change, impacting all its callers.

Save is also unpleasant to test, because it operates directly with files on disk. To verify its operation the test would have to read the contents of the file after being written. You would also have to ensure that f was written to a temporary location and always removed afterwards.

Moreover *os.File defines a lot of methods which are not relevant to Save, like reading directories and checking to see if a path is a symlink. It would be useful if the signature of Save could describe only the parts of *os.File that were relevant.

Using io.ReadWriteCloser we can apply the interface segregation principle to redefine Save to take an interface that describes more general file shaped things. With this change, any type that implements the io.ReadWriteCloser interface can be substituted for the previous *os.File. This makes Save both broader in its application, and clarifies to the caller of Save which methods of the *os.File type are relevant to its operation. As the author of Save I no longer have the option to call those unrelated methods on *os.File as it is hidden behind the io.ReadWriteCloser interface. But we can take the interface segregation principle a bit further.

Firstly, it is unlikely that if Save follows the single responsibility principle, it will read the file it just wrote to verify its contents—​that should be responsibility of another piece of code.

We can narrow the specification for the interface we pass to Save to just writing and closing.

Secondly, by providing Save with a mechanism to close its stream, which we inherited in this desire to make it still look like a file, this raises the question of under what circumstances will wc be closed. Possibly Save will call Close unconditionally, or perhaps Close will be called in the case of success. Neither of these is a good option. Unconditionally closing wc after the call to Save precludes the caller from writing additional data after the document is written. Conditionally closing the WriteCloser — it doesn’t matter if its on success, or failure—​means the caller must grow intricate knowledge of the operation of Save.

A better solution would be to redefine Save to take only an io.Writer, stripping it completely of the responsibility to do anything but write data to a stream.

By applying the interface segregation principle to our Save function, the results has simultaneously been a function which is the most specific in terms of its requirements—​it only needs a thing that is writable—​and the most general in its function, we can now use Save to save our data to anything which implements io.Writer.

As a side effect it is clear that the name of the method is no longer accurate. A better name may be

When declaring an interface the variable name is not needed — that’s a property of the type that implements the interface. Thus

Is valid, but not very useful descriptive.

However there is a clue for how we can make this interface declaration more expressive. Part of it is aleady there, can you see it. Run takes three parameters, a context.Context, a string, and an int. What does the string value denote, maybe its a name of the thing being run, maybe its a operation too run? What does the int valye denote, maybe its an identifier for this job, maybe its a trace ID, maybe its a timeout?

But the first parameter, context.Context is unequivocal. It its a context, more importnatly it is the context, not just a generic context like thing.

Perhaps the take away is for interface declarations, the avoid primative types (those declared in the universe block).

If the goal of a well designed Go package is to provide a set of related behaviours, writing a good package starts with choosing a good name. Think of your package’s name as an elevator pitch to describe what it does, using just one word.

Each Go package is in effect it’s own small Go program. You have access to all the symbols in this package, public and private (let’s ignore external tests for the moment), and can use the all the features of the language on all parts of this package.

But, when you combine those pieces, from the perspective of one module / function / package / piece, you want to take about the other as abstractly as possible.

A pkg’s name should be a one word elevator pitch for_what_it_provides (not what it contains) https://twitter.com/davecheney/status/793306691370557440

A package should confirm to SRP, it should have a single purpose and a single responsiblity

https://blog.golang.org/package-names .

Models and other package names . They re fine, but not very reusable. Imagine the task of merging two projects, both with their own models directories. It should be clear that models is just a taxonomy, a place to put data, not reuse behavior

Initially it appeared attractive for every major piece of functinality to have a testing sub package

This package could contain mocks or helper functions.

The problem is most functional tests will need to import the mocks for more than one of these areas, causing a name clash on testing

As you have to rename the imports anyway,

Why not start with that initially

cf, no utils packages, cf, io/ioutils

A package’s name should describe the function of the package, not the contents. http is a good package name, it describes that this package provides something to do with HTTP.

is not as good name because it does not describe what this package provides. Providing "utility" functions is of no value to the reader, especially if every package author were to follow this trend.

is not a good name for two reasons

Within your project, each package name should be unique. This should pretty easy to if you’ve followed the previous advice that a package’s name should derive from its purpose. If you find you have two packages which need the same name, it is likely either;

Provide something, cannot provide multiple things, ie http not https or http servers.

A common cause of poor package names is what I call utility packages. These are packages where helpers and utility code congeals over time. As these packages contain an assortment of unrelated functions, their utility is hard to describe in terms of what the package provides. This often leads to the package’s name being derived from what the package contains--utilities.

Package names like utils or helpers are commonly found in larger projects which have developed deep package hierarchies and want to share helper functions without encountering import loops. By extracting utility functions to new package the import loop is broken, but because the package stems from a design problem in the project its name doesn’t reflect its purpose, only its function of breaking the import cycle.

My recommendation to improve the name of utils or helpers packages is to analyse where they are called and if possible move the relevant functions into their caller’s package. Even if this involves duplicating some helper code this is better than introducing an import dependency between two packages.

In the case where utility functions are used in many places prefer multiple packages, each focused on a single aspect, to a single monolithic package.

Packages with names like base or common are often found when functionality common to two or more implementations, or common types for a client and server, has been refactored into a separate package. Their names also represent design holdovers from languages like Java and C++ where the relationship between packages followed similar rules to those of inheretence. I believe the solution to packeges like base or common is to reduce the number of packages, combine the client, server, and common code into a single package named after the behaviour delivered from the previously fractured packages.

For example, the net/http package does not have client and server sub packages, instead it has a client.go and server.go file, each holding their respective types, and a transport.go file for the common message transport code.

The key to writing maintainable programs is that they should be loosely coupled. A change to one package should have a low probability of affecting another.

In Go we can declare variables at the block, function, or method scope, and also at the package scope. When the variable is public—​the identifier starts with a capital letter—​then its scope is effectively global to the entire program. Any package may observe the type and contents of that variable at any time.

Mutable global state introduces tight coupling between independent parts of your program as global variables become an invisible parameter to every function in your program! Any function that relies on a global variable can be broken if that variable’s type changes. Any function that relies on the state of a global variable can be broken if another part of the program changes that variable.

Specifically, while six possible declarations exist at the package level, consider restricting your programs to package, import, const, type, and func. By avoiding package level var declarations, you remove the opportunity for package global state to leak into your program.

If you follow this advice then the use of the explicit func init becomes less useful, or at least less stateful.

If you want to reduce the coupling a global variable creates,

TODO: unfinished

Avoid getter type methods because they break encapsulation. In the case where they are unavoidable—​perhaps because of a poorly designed interface—​be wary that you do not leak a reference to an internal data structure which can be retained and mutated long after the original getter was accessed.

For example, many would recognise the dangers with the following function

By retaining a reference to u.first, the caller can both observe the state of u.first without going through FirstName and can, if it wishes, mutate u.first. The difficulties with this design are obvious.

Somehwhat less obvious are accessors like this

Slices, as we know, are value types, however the first field in a slice is a pointer to its backing array.

While the value returned from Siblings is a copy of u.siblings, its backing array pointes to the u.siblings array—​the share the same backing array.

This means, assuming u.siblings is not being appended too, the copy returned from Siblings can be used to access and mutate u’s `siblings slice. In the case that other code appends to u.siblings at some point the various copies returned from Siblings may point to different backing arrays.This is perhaps, although this might be seem to be splitting hairs, a worse outcome. The situation has evovled from a simple data race, to a non deterministic data race.

The solution, I belive is twofold

Which may present a performance issue depending on the size, frequency, and use of Siblings. . Refactor the code to reflect the use of the result of Siblings.

Go eschewed type hierarchy, and that is generally considered to be a good thing, don’t make the mistake or replacing that with an elaborate package hierarchy. That will lead to too many internal types being made public. If you have two packages that are always imported together, maybe combine them. Alternatively, if you have one package that is never imported directly, only via a third, maybe roll it up.

cmd has a long legacy, going all the way back to the arrangement of the 1972 unix source code.

One of the things I tend to pick up in code review for programmers who are transitioning from other languages to Go is they tend to overuse packages. To be fair, mastery of Go is effectively the art of moderation in the use of all Go’s features, but package overuse seems to be one of the most common misstep. I suspect driven by programmers' innate drive to categorise and neatly organise the things they see.

Go does not provide elaborate ways of establishing visibility. Go lacks Java’s public, protected, private, and implicit default access modifiers. There is no equivalent of C++'s notion of a friend classes.

In Go we have only two access modifiers, public and private, the former indicated by the capitalisation of the first letter of the identifier. If an identifier is public, it’s name starts with a capital letter, that identifier can be referenced by any other Go package.

Given the limited controls available to control access to a package’s symbols, what practices should Go programmers follow to avoid creating over-complicated package hierarchies?

The advice I find myself repeating is to prefer fewer, larger packages. Your default position should be to not create a new package. That will lead to too many types being made public creating a wide, shallow, API surface for your package..

The sections below explores this suggestion in more detail.

If you’re arranging your packages by what they provide to callers, should you do the same for files within a Go package? How do you know when you should break up a .go file into multiple ones? How do you know when you’ve gone to far and should instead consolidate several .go files together?

Here are the guidelines I use:

My rule of thumb for splitting up a large file is to use imports as a guideline. That is, all the things that import network stuff go in one file, all the thing importing strings/regex and parsing go in another and so on.

This tends to also make testing more straight forward. If you have a file, say, conn.go, that deals with network type stuff, then its counterpart conn_test.go should deal only with testing the network functionality of this package. If you didn’t you’ve got parsing or business logic tests in that file as well, that’s a sign that your design isn’t right, or that you carved your file along the wrong boundary.

Obviously some judgement needs to be applied as packages like fmt and errors tend to be used everywhere. Also, don’t take this to the extreme and split too much, otherwise you’ll replace the problem of a large file with the problem of constantly searching a directory of files. If your package starts to look like Java, with one function or type per file then you’ve gone to far and it’s time to consolidate. Again, imports help guide you here, if you find two files with identical imports and related functionality, then try meeting them.

If your project contains multiple packages you may find you have some exported functions which are intended to be used by other packages in your project, but are not intended to be part of your project’s public API. If you find yourself in this situation the go tool recognises a special folder name—​not package name--internal/ which can be used to indicate code which is public to your project, but private to others.

To create an internal package, place it within a directory named internal/ or in a sub-directory of a directory named internal/. When the go command sees an import of a package with internal in its path, it verifies that the importing package is within the tree rooted at the parent of the internal directory.

For example, a package …​/a/b/c/internal/d/e/f can be imported only by code in the directory tree rooted at …​/a/b/c. It cannot be imported by code in …​/a/b/g or in any other repository. ^[9]

If you’re designing an application—​one that will will span multiple packages—​your main function, and main package should do as little as possible. This is because main.main acts as a singleton; there can only be one main function in a program.

Main packages are harder to test. Instead extract all the functionality to other packages, possibly internal/, and leave main for - flag parsing - constructing the top level object - managing the top level lifetime of components

Because main.main is a singleton there are a lot of assumptions built into the things that main.main will call, that they will only be called during main.main or main.init, and only called once. This makes it hard to write tests for code written in main.main. Main packages often invoke singletons, parse command line flags, expect files to be on disk in a certain place, and never expect to be executed concurrently. You can’t even reference main.main from a test.

Thus you should aim to extract as much of your business logic out of your main function and ideally out of your main package. func main() should parse flags, open connections to databases, loggers, and such, then hand off execution to a high level object.

Every package should have a unique name within your project, within the code that you are writing. This means that when multiple packages are imported into your package, they will not conflict.

As your project grows, you will naturally have to add more packages (but please consider the other topics in this chapter)

don’t do

This package organisation highlights a fundamental issue with the design. This also speaks to writing larger packages and writing less packages.

Go packages are not for creating taxonomies, net/http is not a type of net thing. Go packages are for avoid namespace collisions.

As an aside, I find that many programmers have a strong desire to aptly taxonomies to their work, they want to find differences between two things so they can be separated and classified. I think to write successful Go, you should look for similarities and combine things with a similar purpose into a package. For example, net/http contains both http client and server facilities, because they both relate to the use of http. The prevailing style in other languages would have these placed in separate packages, one for client, one for server, and a third for things which are common to both. I think this is unnecessary.

This statement is almost universal in the Go programmer’s phrase book, but what do Go programmers mean when they say "errors are just values", and what does this technique imply? By way of explanation, consider the counter example of panic and recover, often mistaken for exceptions.

panic and recover, two keywords added to the language for a single purpose. recover can only be used for one purpose; to access a value previously passed to panic. If that wasn’t enough recover’s use case is so specific, it can only be used inside a `defer block. You cannot use recover for any other purpose, it can only be used in concert with panic.

This pair of features sit by themselves in a corner of the language. How’s that for non orthogonal?

By contrast, error values are not limited to the rarefied semantics of panic and recover.

The common contract for functions which return a value of the interface type error, is the caller should not presume anything about the state of the other values returned from that call without first checking the error. In the majority of cases, error values returned from functions should be opaque to the caller. That is to say, a test that error is nil indicates if the call succeeded or failed, and that’s all there is to it.

The methodology I follow is; if a function can return an error, you cannot make any assumptions about the state of any other values returned until you check the error. If it was found that the error was set (ie, not nil), then the state of those other values is unknown.

A small number of cases, require that the caller investigate the nature of the error to decide if it is reasonable to retry the operation. A common request for package authors is to return errors of a known public type, so the caller can type assert and inspect them. I believe this practice leads to a number of undesirable outcomes:

You should feel no more comfortable asserting an error is a particular type than they would be asserting the string returned from Error() matches a particular pattern.

Assert that error value implements a particular behaviour. Don’t assert an error value is a specific type, but rather assert that the value implements a particular behaviour.

Instead I present a suggestion that permits package authors and consumers to communicate about their intention, without having to overly couple their implementation to the caller. This suggestion fits the has a [behaviour] nature of Go’s implicit interfaces, rather than the is a [subtype of] nature of inheritance based languages. Consider this example:

The caller can use isTimeout to determine if the error is related to a timeout, and if so confirm if the error was timeout related, all without knowing anything about the type, or the original source of the error value.

Gift wrapping errors, usually by libraries that annotate the error path, is enabled by this method; providing that the wrapped error types also implement the interfaces of the error they wrap. This may seem like a generally intractable problem, but in practice there are relatively few interface methods that are in common use, so Timeout() bool and Temporary() bool cover a large set of use cases.

For package authors, if your package generates errors of a temporary nature, ensure you return error types that implement the respective interface methods. If you wrap error values on the way out, ensure that your wrappers respect the interface(s) that the underlying error value implemented.

For package users, if you need to inspect an error—​and hopefully this should be infrequent—​declare and assert an interface to assert the behaviour you expect, not the error’s type. Don’t ask package authors for public error types; instead ask that they make their types conform to common interfaces as appropriate.

The simple rule of thumb is panic should only be used in truely exceptional cases, which, as their name suggests are rare. To paraphrase the words of Dash Parr, if everything is exceptional, the nothing is.

Go’s error handling strategy is via the error interface and returning error values. Go does have panic, which is a by-product of the counterpart in the runtime’s internal throw function. There are few cases of using recover that I know of, and all of those are used to simulate non local transfer of control not exception handling. Using recover has all the problems of sensing errors by type, with the added complication that the set of types returned is unbounded.

While it is true that any Go function can call panic, any Go procedure can fail due to out of memory, the program can be killed by a process manager, or the serve can simply fail. Always write your programs to assume failure, not success. Avoid panic and eschew recover, they’re not the tool you are looking for.

Panic will, during unwinding the stack, execute any deferred statements. However just as a panic in one goroutine cannot be recovered in another, a panic in one goroutine will not allow defer statements in other goroutines to exit. For a goroutine spawned by a library to panic the entire program is selfish.

As part of the Go 2 design goals, improvements (although few can agree on what improve means) to error handling are highlighted as a candidate. This surfaced in 2018 with the check/handle proposal, which was later refined into the highly contentious try proposal. But do you know what is better than an improved syntax for handling errors? Not needing to handle errors at all.

This section draws inspiration from John Ousterhout’s book, A philosophy of Software Design ^[11]. One of the chapters in that book is called "Define Errors Out of Existence".

One of Osterhout’s examples was the operation of UNIX’s write(2) API, specifically that if an error occurs during writing, you can perhaps ignore it as the error will surface during close(2) on that file. This example plays a little fast and loose with POSIX, but illustrates the idea; done well, error handling can be repetitive, thus, where possible, use the emergent properties of the problem to avoid error handling where possible.

We’ll see an example of Osterhout’s write/close solution a little later as I try to apply his advice to Go.

To close this chapter, I want to recommend that your code should only handle errors once.

If you make less than one decision, you’re ignoring the error. As we see here, the error from w.WriteAll is being discarded.

But making more than one decision in response to a single error is also problematic. The following is code that I come across frequently.

In this example if an error occurs during w.Write, a line will be written to a log file, noting the file and line that the error occurred, and the error is also returned to the caller, who possibly will log it, and return it, all the way back up to the top of the program.

The caller is probably doing the same

So you get a stack of duplicate lines in your log file,

but at the top of the program you get the original error without any context.

I want to dig into this a little further because I don’t see the problems with logging and returning as just a matter of personal preference.

The problem I see a lot is programmers forgetting to return from an error. As we talked about earlier, Go style is to use guard clauses, checking preconditions as the function progresses and returning early.

In this example the author checked the error, logged it, but forgot to return. This has caused a subtle bug.

The contract for error handling in Go says that you cannot make any assumptions about the contents of other return values in the presence of an error. As the JSON marshalling failed, the contents of buf are unknown, maybe it contains nothing, but worse it could contain a half written JSON fragment.

Because the programmer forgot to return after checking and logging the error, the corrupt buffer will be passed to WriteAll, which will probably succeed and so the config file will be written incorrectly. However the function will return just fine, and the only indication that a problem happened will be a single log line complaining about marshalling JSON, not a failure to write the config.

Channels are a signature feature of the Go programming language. Channels provide a powerful way to reason about the flow of data from one goroutine to another without the use of locks or critical sections.

I want to talk about two important properties of channels that make them useful for controlling not just data flow within your program, but the flow of control as well.

What is the difference between these two APIs?

The obvious differences are the first example reads a directory into a slice then returns the whole slice, or an error if something went wrong. This happens synchronously, the caller of ListDirectory blocks until all directory entries have been read. Depending on how large the directory, this could take a long time, and could potentially allocate a lot of memory building up the slide of directory entry names.

Lets look at the second example. This is a little more Go like, ListDirectory returns a channel over which directory entries will be passed. When the channel is closed, that is your indication that there are no more directory entries. As the population of the channel happens after ListDirectory returns, ListDirectory is probably starting a goroutine to populate the channel.

The channel version of ListDirectory has two further problems:

The solution to the problems of both implementations is to use a callback, a function that is called in the context of each directory entry as it is executed.

Not surprisingly this is how the filepath.WalkDir function works.