Automatically fetch your project’s dependencies with gb

gb has been in development for just over a year now. Since the announcement in May 2015 the project has received over 1,600 stars, produced 16 releases, and attracted 41 contributors.

Thanks to a committed band of early adopters, gb has grown to be a usable day to day replacement for the go tool. But, there is one area where gb has not lived up to my hopes, and that is dependency management.

gb’s $PROJECT/vendor/ directory was the inspiration for the go tool’s vendor/ directory (although their implementations differ greatly) and has delivered on its goal of reproducible builds for Go projects. However, the success of gb’s project based model, and vendoring code in general, has a few problems. Specifically, wholesale copying (or forking if you prefer) of one code base into another continues to sidestep the issue of adoption of a proper release and versioning culture amongst Go developers.

To be fair, for Go developers using the tools they have access to today–including gb–there is no incentive to release their code. As a Go package author, you get no points for doing proper versioned releases if your build tool just pulls from HEAD anyway. There is similarly limited value in adopting a version numbering policy like SemVer if your tools only memorise the git revision you last copied your code at.

A second problem, equally poorly served by gb or the vendor/ support in the go tool, are developers and projects who cannot, usually for legal reasons, or do not wish to, copy code wholesale into their project. Suggestions of using git submodules have been soundly dismissed as unworkable.

With the release of gb 0.4.3, there is a new way to manage dependencies with gb. This new method does not replace gb vendor or $PROJECT/vendor as the recommended method for achieving reproducible builds, but it does acknowledge that vendoring is not appropriate for all use cases.

To be clear, this new mode of managing dependencies does not supersede or deprecate the existing mechanisms of cloning source code into $PROJECT/vendor. The automatic download feature is optional and is activated by the project author creating a file in their project’s root called, $PROJECT/depfile.

If you have a gb project that is currently vendoring code, or you’re using gb vendor restore to actively avoid cloning code into your project, you can try this feature today, with the following caveats:

  1. Currently only GitHub is supported. This is because the new facility uses the GitHub API to download release tarballs via https. Vanity urls that redirect to GitHub are also not supported yet, but will be supported soon.
  2. The repository must have made a release of its code, and that release must be tagged with a tag containing a valid SemVer 2.0.0 version number. The format of the tag is described in this proposal. If a dependency you want to consume in your gb project has not released their code, then please ask them to do so.

Polishing this feature will be the remainder of the 0.4.x development series. After this work is complete gb vendor will be getting some attention. Ultimately both gb vendor and $PROJECT/depfile do the same thing–one copies the source of your dependencies into your project, the other into your home directory.

Using gb as my every day build tool

For the longest time I had this alias in my .bashrc

alias gb='go install -v'

as an homage to John Asmuth’s gb tool which I was very fond of way back before we had the go tool.

Once gb was written, I had to remove that alias and live in a world where I used the go tool and gb concurrently. Now, with the improvements that have landed since GopherCon I’ve decided it’s time to get serious about eating my own dog food and use gb as a full time replacement for the go tool.

I use gb not just to build gb projects, but to work on Juju, my $DAYJOB, as well as using gb to develop gb itself. Because the definition of a gb project is backwards compatible with $GOPATH I don’t even need to rewrite every Go package I want to work on as a gb project, I can work with them with their source happily in my $GOPATH.

gb isn’t perfect yet, there are still some major missing pieces. I hope by aggressively dogfooding I’ll be able to close the gaps before the end of the year. Specifically:

  • gb build works well, but the gb test story is less solid; I hope to reach parity with go test in the next few months by adding support for test flags, -race and coverage options.
  • Support for cross compilation was always high on my list, and is being actively worked on, and will be available to experiment within the next two weeks. At this point I envisage it will be Go 1.5 only, Go 1.4 support may come in time, but is not a high priority.

If others want to join me and make gb their default Go build tool, I’d welcome the company, and of course, your bug reports.

gb, a project based build tool for the Go programming language

A few months ago I introduced gb as a proof of concept to the audience at GDG Berlin. Since then, together with a small band of contributors and an enthusiastic cabal of early adopters, gb has moved from proof of concept, written mostly on trains during a trip through Europe, to something approaching a usable build tool.

This post gives an introduction to gb, and explains the benefits of adopting a project based workflow for building solutions in Go. If you want to read more about the motivations for gb, please read the previous post in this series.

gb

https://getgb.io/

gb is a project based build tool for the Go programming language.

gb projects define an on disk layout that permits repeatable builds via source vendoring. When vendoring (copying) code into a gb project, the original source is not rewritten or modified.

As gb is written in Go, its packages can be used to create plugins to gb that extend its functionality.

Why is a project based approach useful ?

Why is a project based approach, as opposed to a workspace based approach like $GOPATH, useful ?

First and foremost, by structuring your Go application as a project, rather than a piece of a distributed global workspace, you gain control over all the source that goes into your application, including that of its dependencies.

Second, while your project’s layout contains both the code that you have written, and the code that your code depends on, there is a clear separation between the two; they live in different subdirectory trees.

Thirdly, now your project, and by extension the repository that houses it, contains all the source to build your application, having multiple working copies on disk is trivial. If you are part of a team responsible for maintaining multiple releases, this is a very useful property.

Lastly. As your project can be built at any time without going out to the internet to fetch code, you are insulated from political, financial, or technical events that can unexpectedly break your build. Conversely, as the dependencies for your project are included with the project, upgrading those dependencies is atomic and affects everyone on the team automatically without them having to run external steps like go get -u or godeps -u dependencies.txt.

How is gb different ?

In the previous section I outlined the advantages I see in using a project based tool to build Go applications. I want to digress for a moment to explain why gb is different to other existing go build tools.

gb is not a wrapper around the go tool. Not wrapping the go tool means gb is not constrained to solutions that can be implemented with $GOPATH tricks. Not relying on the go tool means gb can ship faster, and fix bugs faster than the fixed pace of releases of the Go toolchain.

You can read more about the rationale for gb here, and reasons for not wrapping the go tool here.

Being a project owner

In the discussions I’ve had about gb, I’ve tried to emphasise the role of the project owner. The owner of the project has a special responsibility; they are responsible for admitting new dependencies into a project, and for curating those dependencies once they are part of the shipping product.

Whether the role of project owner falls to a single engineer, or is distributed across your whole team, it is the project owner who is ultimately responsible for shipping your product, so gb gives you the tools to achieve this without having to rely on a third party.

Project ownership is important. You, the developer, the project owner, the build engineer, needs to own all the source that goes into your product whether you wrote it or not. Don’t be the person who cannot deliver a release because GitHub is down.

github

No import rewriting

gb is built around a philosophy of leaving the source of a project’s dependency untouched.

The are various technical reasons why I believe import rewriting is a bad idea for Go projects, I won’t repeat them here.

It is my hope that maybe one day, build tools like gb can get a bit smarter about managing dependencies, and avoid the need for whole cloth vendoring, but this cannot happen if imports are rewritten.

Demo time

Enough background, let’s show off gb.

Creating a gb project

Creating a gb project is as simple as creating a directory. This directory can be anywhere of your choosing; it does not need to be inside $GOPATH, in fact gb does not use $GOPATH.

% mkdir -p demo/src
% cd demo

demo is now a gb project. A gb project is defined as any directory that contains a directory called src/. We’ll refer to the root of the project, demo in this case, as the project root, or $PROJECT for short. Let’s go ahead and create a single main package.

% mkdir -p src/cmd/helloworld
% cat > src/cmd/helloworld/main.go <<EOF
package main

import "fmt"

func main() {
       fmt.Println("Hello world from gb")
}
EOF

Commands (main packages) don’t have to be placed in src/cmd/, but that is a nice tradition that has emerged from the Go standard library, so we’ll follow it here. Also note that although gb does not use the go tool to compile Go code, that code must still be structured into packages.

In fact gb is much stricter in this respect, Go code can only be built if it is inside a package, there are no facilities to build or run a single .go source file.

gb supports all the usual ways of compiling one package by passing the name of the package to gb build, but it is simpler to just build the entire project by staying at the root of your project and issuing gb build to build all the source in your project.

gb has support for incremental compilation, so even though gb build is told to build all the source in the project, it will only recompile the parts that have changed; there is no need to point them out to the compiler. Also note that there is no gb install command. gb build both builds and installs (caches packages forincremental compilation later).

With all this said, let’s go ahead an build this project, then run the resulting program

% gb build
cmd/helloworld
% bin/helloworld 
Hello world from gb

By default gb prints out the names of packages it is compiling, you can use the -q flag if you want to suppress this output. When compiling, packages will be built and placed in $PROJECT/pkg/ for possible reuse by latter compilation cycles, main packages (commands), will be placed in $PROJECT/bin/.

If this project contained multiple commands, they would all be built and placed in $PROJECT/bin/. To demonstrated this I created a few more main packages in this project, let’s compile them and look at the result

% gb build
cmd/client
cmd/helloworld
cmd/server
% ls bin/
client  helloworld  server

gb project layout

The previous section walked through the creation of a gb project from scratch and showed using gb build to compile the project.

Let’s have a look at the directory tree of this project an add some annotations to reinforce the gb project concepts.

% tree $(pwd)
/home/dfc/demo
├── bin
│   ├── client
│   ├── helloworld
│   └── server
└── src
    └── cmd
        ├── client
        │   └── main.go
        ├── helloworld
        │   └── main.go
        └── server
            └── main.go

6 directories, 6 files

Starting from the top, we have a bin/ directory, this is created by gb build when building main packages to hold the final output of linking executable programs. Inside bin/ we have the the binaries that were built.

Next is the src/ which contains the subdirectory cmd/ and inside that, three packages, client, helloworld, and server.

The final directories you will find inside a gb project is $PROJECT/pkg/ for compiled go packages, and $PROJECT/vendor/ for the source of your project’s dependencies. We’ll discuss vendoring dependencies later in this piece.

Source control

gb doesn’t care about source control, all it cares about is the source of your project is arranged in the format it expects. How those files got there, or who is responsible for tracking changes to them is outside gb’s concern.

Of course, source control is a great idea, and you should be tracking the source of your project using source control. Let’s create a git repo in the $PROJECT root now

% git init .
Initialized empty Git repository in /home/dfc/demo/.git/
% git add src/
% git commit -am 'initial import'
[master (root-commit) aa1acfd] initial import
 3 files changed, 21 insertions(+)
 create mode 100644 src/cmd/client/main.go
 create mode 100644 src/cmd/helloworld/main.go
 create mode 100644 src/cmd/server/main.go

Then of course add a git remote and push to it.

You should not place $PROJECT/bin/ or $PROJECT/pkg/ under source control, as they are temporary directories. You may wish to add a .gitignore or similar to prevent doing so accidentally.

Dependency management

A project which doesn’t have any dependencies, apart from the standard library, is not going to be very a compelling use case for gb. For this next section I’ll walk through creating a new gb project which has several dependencies.

The source for this project is online at github.com/constabulary/example-gsftp. Let’s start by creating the project structure and adding some source

% mkdir example-gsftp
% cd example-gsftp
% mkdir -p src/cmd/gsftp # this is our main package
% vim src/cmd/gsftp/main.go

The source for cmd/gsftp/main.go is too long to include, but is available here.

This project depends on golang.org/x/crypto/ssh package and github.com/pkg/sftp package, which itself has a dependency on github.com/kr/fs. In its current state, if you were to gb build this project it would fail with an error like this

% gb build
FATAL command "build" failed: failed to resolve import path "cmd/gsftp": cannot find package "github.com/pkg/sftp" in any of:
        /home/dfc/go/src/github.com/pkg/sftp (from $GOROOT)
        /home/dfc/example-gsftp/src/github.com/pkg/sftp (from $GOPATH)
        /home/dfc/example-gsftp/vendor/src/github.com/pkg/sftp

gb is unable to find the a package with an import path of github.com/pkg/sftp, in either the project’s source directory, $PROJECT/src/, or the project’s vendored source directory, $PROJECT/vendor/src/.

note the references to $GOPATH are a side effect of reusing the go/build package. gb does not use $GOPATH, this message will be addressed in the future.

Now, I have copies of all of the source of these packages in my $GOPATH, so I can copy them into the $PROJECT/vendor/src directory by hand to satisfy the build.

% mkdir -p vendor/src/github.com/pkg/sftp
% cp -r $GOPATH/src/github.com/pkg/sftp/* vendor/src/github.com/pkg/sftp
% mkdir -p vendor/src/github.com/kr/fs
% cp -r $GOPATH/src/github.com/kr/fs/* vendor/src/github.com/kr/fs
% mkdir -p vendor/src/golang.org/x/crypto/ssh
% cp -r $GOPATH/src/golang.org/x/crypto/ssh/* vendor/src/golang.org/x/crypto/ssh
% gb build
github.com/kr/fs
golang.org/x/crypto/ssh
golang.org/x/crypto/ssh/agent
github.com/pkg/sftp
cmd/gsftp
% ls -l bin/gsftp
-rwxrwxr-x 1 dfc dfc 5949744 Jun  8 14:05 bin/gsftp

For completeness’s sake, let’s take a look at the directory structure of the project with these vendored dependencies.

% tree -d $(pwd)                                                                                                                         
/home/dfc/example-gsftp
├── bin
├── pkg
│   └── linux
│       └── amd64
│           ├── github.com
│           │   ├── kr
│           │   └── pkg
│           └── golang.org
│               └── x
│                   └── crypto
│                       └── ssh
├── src
│   └── cmd
│       └── gsftp
└── vendor
    └── src
        ├── github.com
        │   ├── kr
        │   │   └── fs
        │   └── pkg
        │       └── sftp
        │           └── examples
        │               ├── buffered-read-benchmark
        │               ├── buffered-write-benchmark
        │               ├── streaming-read-benchmark
        │               └── streaming-write-benchmark
        └── golang.org
            └── x
                └── crypto
                    └── ssh
                        ├── agent
                        ├── terminal
                        ├── test
                        └── testdata

34 directories

Using the gb-vendor plugin

gb’s answer to dependency management is vendoring, copying the source of your project’s dependencies into $PROJECT/vendor/src/. As you saw above, this process can be quite tedious, especially if you do not have the source of the dependency easily to hand.

To assist with this process, gb ships with a plugin called gb-vendor, which aims to automate a lot of this work.

gb-vendor can fetch the dependencies of your project. Let’s use it to automate the steps we just did above.

% rm -rf vendor/src
% gb vendor fetch github.com/pkg/sftp
% gb vendor fetch github.com/kr/fs
% gb vendor fetch golang.org/x/crypto/ssh
% gb build
github.com/kr/fs
golang.org/x/crypto/ssh
golang.org/x/crypto/ssh/agent
github.com/pkg/sftp
cmd/gsftp

At this point it is a good idea to add your project’s vendor/ directory to source control.

% git add vendor/
% git commit -am 'added vendored dependencies'

gb-vendor also provides commands to update, delete, and report on the vendor dependencies of a project. For example

% gb vendor update github.com/pkg/sftp

will replace the source of github.com/pkg/sftp with the latest available upstream.

% gb vendor delete github.com/pkg/kr/fs

Will remove $PROJECT/vendor/src/github.com/pkg/kr/fs from disk, and remove its entry from the manifest file.

Lastly, the list subcommand behaves similarly to go list and lets you report on the dependencies recorded in the manifest file.

% gb vendor list
github.com/pkg/sftp     https://github.com/pkg/sftp     master  f234c3c6540c0358b1802f7fd90c0879af9232eb
github.com/kr/fs        https://github.com/kr/fs        master  2788f0dbd16903de03cb8186e5c7d97b69ad387b
golang.org/x/crypto/ssh https://go.googlesource.com/crypto/ssh  master  c10c31b5e94b6f7a0283272dc2bb27163dcea24b

gb-vendor is completely optional

At this point you’re probably saying, “Hang on, aren’t you the person who made a big song and dance about no metadata files ?”.

Yes, it is true that gb-vendor records the dependencies it fetches in a manifest file, ($PROJECT/vendor/manifest), but this manifest file is only used by gb-vendor, and is not part of gb.

gb-vendor is a plugin, it adds a little bit of smarts on top of git clone, or hg checkout, but it isn’t mandatory to use gb-vendor to build gb projects.

All gb cares about is the source on disk, you don’t have to use it. If your workflow works well with svn externals or git subtrees, or maybe just copying the package and recording the revision you copied in the commit message, you can use that approach as well.

gb-vendor is not required to use gb, and gb is completely oblivious to its operation. All that gb cares about is finding the source on disk with the correct layout. You are free to use any method of managing the contents of your $PROJET/vendor/src directory.

How does gb handle the diamond dependency problem ?

In every Go program, regardless of which tool built it (gb, the go tool, Makefile, or by hand), there may only be one copy of a package linked into the final binary.

For project owners this means that if they encounter a situation where two dependencies of their project expect different exported API’s of a third package, they must resolve this problem at the point they introduce these dependencies into their project.

Resolving a diamond dependency conflict requires the project owner choose which copy (Go packages do not have a notion of versions) of the source of that third dependency they will place in $PROJECT/vendor/src/ and adjusting, updating, or replacing other dependencies as required.

Can a gb project be a library ?

In the presentations I’ve made about gb, I have focussed on development teams shipping products written in Go. I see these teams as the ones who have the most to gain, and the least to lose, from adopting gb, so it is reasonable to focus on those teams first.

You can also use gb to build libraries (effectively gb projects that don’t have main packages), and then vendor the source of that project’s src/ directory into another gb project as demonstrated above.

At the moment no automated tools exist to assist with this process, but it is likely that gb-vendor will acquire this ability if there is significant demand in developing libraries in the gb project format.


Wrapping up

This post has described the theory and the practice of using gb. I hope that you have found it useful, and in turn that you may find gb useful if you are part of a team charged with delivering solutions using Go.

https://getgb.io/

getgb.io

  • Project based workflow
  • Repeatable builds via source vendoring without import rewriting
  • Reusable components with a plugin interface

Introducing gb, a project based build tool for the Go programming language

In April 2015 I gave a presentation to the GDG Berlin meetup group (slides, video) discussing my views on reproducible builds using the Go programming language.

Image credit, Nate Finch
Image credit, Nate Finch

As part of that presentation I demonstrated a replacement build tool that I had been developing, gb.

From the feedback I received after the meetup it was clear that many people understood and agreed with my reasoning why reproducible builds in Go are a problem today. However it was equally clear that I had not done a good enough job of explaining how gb works, and why I felt it was necessary to create an alternative build tool to solve the problem of reproducible builds.

With the help of the go-pm mailing list and the feedback from a group of early adopters of gb, I have put together a website for gb with expanded documentation and examples.

I invite you all to visit getgb.io to learn more about gb.