Configuring Your GOPATH with Go and Google App Engine

When I started working with Google App Engine and Go, I wasn’t sure how to best configure my GOPATH when developing Google App Engine applications.  You can find documentation on this aspect on the Go and App Engine page, but being new to both Go and App Engine I was not aware of what options were available, and what their pros and cons would be.

If you are not sure what a GOPATH is, I recommend this video tutorial on setting up a Go workspace and running and testing code as it helped explain the concept to me.  It is also worth noting, that some Go programmers have a single GOPATH they use for all their projects on a given computer, however, for the sake of this article, we will be considering the use of a separate GOPATH per-project instead.

Up until a point, you can actually get away with not having a GOPATH at all when developing with App Engine. If you have a simple project with a single directory and no dependencies, you have no need to set a GOPATH and wouldn’t notice any difference if it was missing.  On top of this, if you do have dependencies, goapp will actually store anything you goapp get in the Google App Engine SDK’s gopath subdirectory.

However, I strongly advocate having a GOPATH set as a mainstay for developing with Go and App Engine.  Managing your code through an idiomatically Go way, i.e. with a GOPATH, will ensure that your code remains manageable as it gets more sophisticated and complex, and that all your dependencies for a specific project are retained within its specific GOPATH, not shared between any projects using the SDK.  Using GOPATH has an added benefit if you ever switch between regular Go development and App Engine development — in that case, there should be minimal context switching on your development approach and toolchains.

The aim of this post, and the attached sample code, is to show several options for GOPATH and dependency management when working with Go and App Engine, while exploring some of the pros and cons of each approach. Understanding these options will enable you to start with an initial code layout and GOPATH strategy that will work with your project at its start and well into its lifecycle.

 

Initial Configuration

To get started, let’s git clone this project, and have a look at its structure.

image16

This looks very much like a regular Go project. We have a src folder that contains our Go code. Within that, we have a modules folder that contains three different App Engine
Modules
 (basic, vendored, and gb), each implemented with a different GOPATH strategy.  Within each module subfolder, there exists an app.yaml file that has the App Engine settings for that module, and a routes.go file that specifies the http endpoints for that module.  We also have a lib folder, which contains code that is shared by all three of these modules, to show one possibility of how we can share code between App Engine Modules with all three GOPATH structures.

For the sake of this article, I’m making the assumption that the Google Cloud SDK and the Go App Engine SDK is already installed on your system.  That being said, it is worth noting at this step that this whole project is totally Make driven.  The Makefiles specify what the GOPATH is set to, and perform all our operations on this code base. So, if you want to follow along at home, you don’t need to worry about corrupting an already set GOPATH or other environment variables as this example code will not alter them in any way.

I used a few general tools, such as golint and goimports, in developing this project, some of which we will look at while we go through this example, so you will need to install them if you decide to run through the code yourself:

image01

Now that the tools are in your ./bin folder, your Makefiles can reference them.

 

A Basic GOPATH

This is the simplest implementation. We have a GOPATH with a single entry (more on that later), and we are using the basic goapp tooling that is provided with the App Engine SDK (also more on that later).

Let’s take the opportunity to look at the code in ./src/modules/basic/routes.go

This is a simple HTTP handler, which uses the template display in template.go to return some HTML that shows us the module name, and uses our lib dependency reverse and a third-party dependency github.com/nu7hatch/gouuid to output several values on screen.  We also call a GiveMeANumber() function that is implemented in number.go in the same directory as the routes.go file.

First things first, let’s have a look at what GOPATH the Makefile has set. There is a handy Make target named debug-env that shows us all GO environment variables that are set.

image00

We can see here that the GOPATH is set to the directory we cloned this repository to, which keeps things very simple.

To install our third-party dependencies, we have a deps target in our Makefile that uses the goapp get tool to download the third-party dependency of github.com/nu7hatch/gouuid

Let’s run this target, and see where our code ends up.

image14

We can see here that the gouuid package is stored under /src/github.com/nu7hatch/gouuid, which anyone who is used to working with regular GOPATHs would have expected.

This single level GOPATH approach works well in that it is very clear and easy to understand. Every piece of Go code you use is placed in the same directory and you know exactly where it all sits.  The downside to this approach is that all your third-party dependencies can get mixed into your custom code base, which can feel kind of messy and could potentially be confusing.

Let’s run this, and see it in action. Our Makefile has a serve target that will spin up a local App Engine instance for development:

image07

Browsing to http://localhost:8080 we can see the result we wanted: a UUID, an integer, and our UUID reversed:

image03

 

A Vendored GOPATH

This GOPATH implementation is slightly more complex, but nicely separates our third-party dependencies from our own custom code.  We are still using the standard goapp tool, but we implement a two-level GOPATH to allow our dependencies to be placed in a different location.

Let’s have a look at the code in ./src/modules/vendored/routes.go

We can see that the code is essentially the same as before. We still have the dependency on the third-party library github.com/nu7hatch/gouuid, we have a local function in this module named GiveMeACapitalLetter(), and we are outputting several values to a HTML page through our template display.

Again, let’s look at the GOPATH for this module using the debug-env Makefile target

image15

We can see that the GOPATH that has been set here has a : in the middle of it. This makes Go look in both /home/mark/workspace/appengine-golang-gopath/vendor and also /home/mark/workspace/appengine-golang-gopath when looking for Go source code. It’s also worth noting that goapp get (and go get) will place any dependencies it retrieves in the first path it finds in that GOPATH list, which, as you’ll see shortly, is a very useful behaviour.

Let’s clean out our old dependencies, re-run make deps in this module, and see where our uuid third-party dependency ends up:

image09

This is getting interesting! Rather than our third-party dependency being stored in the same directory as our custom code, it gets placed in a vendor directory. This means that there is a very clear separation between our dependencies and what we are authoring, and there is very little chance for confusion between the two, at the expense of having a slightly more complex GOPATH configuration.

Let’s run make serve to see our code run.

image17

Browsing to http://localhost:8080 we again can see the result we wanted: a UUID, a letter, and our UUID reversed:

image06

 

GB

This approach is a bit more interesting, in that it uses no GOPATH at all. Instead it uses a tool called gb recently written by Dave Cheney. This tool is one of many Go dependency management tools in existence, but it has risen quickly in popularity, and has become one of my favourite tools when developing Go applications across the board. It rewrites the Go tool chain to make project-based development easier, and it has an ecosystem of plugins to help write Go and, in our case, Google App Engine applications.

Having a look at our routes source code in ./src/modules/gb/routes.go, we can see that the code is almost identical to our last two examples:

The only difference from our previous routes.go is that we have a different package local function GiveMeASymbol(), which returns a random ASCII symbol.

While the overall code structure looks the same, let’s have a look at our GOPATH:

image10

Wow, there is no GOPATH at all! Gb instead goes looking for a directory that has a src subdirectory, which a GOPATH oriented project usually does — so no changes needed there either.  This is one of the nice things about gb, you don’t have to worry about environment variables; all your code organisation happens through convention.

When we run our make deps target, you can see that the usual goapp get commands have been switched out for gb vendor fetch commands. This is powered by the optional gb-vendor plugin, which downloads third-party dependencies and works slightly differently from the standard goapp get.

image05

The gb-vendor plugin downloads third-party dependencies into a vendor folder, almost exactly the same as we had before, but without having to directly specify it in the GOPATH. This approach gives you the same separation of third-party dependencies from your custom code, but without the extra work of managing your own GOPATH configuration.

Note that gb-vendor also creates a manifest file in the vendor directory:

This is a central repository of which exact version of the dependency you have downloaded. This is useful, as you can then share this manifest your vendor directory in your source control, and other team members and build systems can ensure that they all have exactly the same dependencies. without having to store your third-party code in your repository if you don’t want to. (Update: 16th, July 2015 – gb-vendor plugin is of the opinion that you should store your vendored dependencies in your source control. See this blog post for reasons.)

gb allows for plugins, and there is a community contributed gb-gae plugin that integrates gb with Google App Engine. In the gb Makefile, we use this to start up the local App Engine development server when we run the make serve target:

image11

Browsing to http://localhost:8080 we can see the result we wanted: a UUID, a symbol, and our UUID reversed:

image13

It is worth noting that the gb-gae plugin can also be used to build, test, and deploy App Engine applications as well, so it can be used for all your Go and App Engine needs.

 

Conclusion

To recap, we’ve gone through several GOPATH solutions here that can work for building Go applications on App Engine. The key things to remember are:

If you like simplicity above all else, the single layer, basic GOPATH may be the right option for you.

If you like clear separation between your third-party dependencies and your own code, the dual layer GOPATH that vendors your dependencies may be right for you.

If you like a tool that not only vendors your dependencies, but also manages which version is being used across teams and platforms and has an ecosystem of plugins as well, the gb approach may be right for you.

Hopefully that has given you some ideas on how you would like to structure the code in your next Go App Engine product.  Good luck, and happy Go coding!

All code show here is licensed under Apache 2. For more details find the original source on GitHub.

Leave a Comment