GO EP11: Go Commands - go get, go mod tidy,...

The go.mod file, or go module, is basically a way to organize and manage a bunch of Go packages together.

Photo by David Iskander on Unsplash

This discussion centers around modules in Go, especially the go.mod file. We also cover how Go resolves dependencies when two modules have the same dependency but different versions.

To connect all the dots from the previous section, we also discuss other Go commands, such as go build, go mod, and the difference between go get and go install.

If you haven’t read the previous section about the fundamental Go environment, covering environment variables like GOROOT, GOPATH, GOCACHE, etc, or how to speed up the build image pipeline, here is the link: GO EP10: GOROOT, GOPATH, GOCACHE.

GO EP10: GOROOT, GOPATH, GOCACHE

We also talk about module-aware mode, which is introduced in Go 1.11 and enabled by default starting with Go 1.13, and how to use it to manage dependencies in a project.

“Wait, what the heck is module-aware mode?”

Module-aware mode is a feature that allows the Go commands to manage dependencies using modules.

In other words, commands like go build, go test, go get, and others operate within the context of the module, making sure that all dependencies are resolved according to the versions specified in the go.mod file.

Ah, this mode also introduces the go.sum file, which contains checksums of the module's dependencies to verify their integrity and prevent tampering. But that's for the next story.

What is go.mod?

The go.mod file, or go module, is basically a way to organize and manage a bunch of Go packages together.

These packages are versioned, released, and distributed as a single unit, a single module. You might be familiar with fetching these modules from repositories like GitHub or even specialized module proxy servers.

People often call it a “package” — like, “fetch this package” or “use that package” — but technically, it’s more about “module.” So, when we talk about fetching a package, we’re really fetching the entire module.

What does the go.mod file do?

It’s the file that defines the module itself. It tells you the module’s name, the Go version it uses, the dependencies it needs, whether directly or indirectly, etc.

A module name is just the import path of the module. If you want to use any package from that module, you need to import it with the module name as a prefix.

# go.mod
module thisismodulename

// import package from this module
import "thisismodulename/any/package"

To create or initialize a module in the current directory, you can use go mod init <module-name> command, of course the directory should not be a module before

This is all basic stuff, now let’s dive into the details about its functionality.

go get: manages (and not build, not install) dependencies

The go get command in Go is used to update module dependencies in the go.mod file of the main module. It's also used to build and install the packages you specify on the command line.

When you run go get, it figures out which modules to update based on what you tell it. There are a few ways to do this: you can specify a package (github.com/user/project/package), a module path (github.com/user/project), or use a pattern like .

“Isn’t there a way to use go get without any arguments?”

Yep, if you just run go get, it behaves as if you’re specifying the current directory ("."). It will update any missing modules for all the imported packages ONLY in the current directory. Often, people use go get ./... or go mod tidy for this.

When you use go get ./..., the pattern is ./..., which represents all subdirectories. The query recognizes this as a wildcard pattern and resolves to all packages in the current directory and its subdirectories, of course within the main module.

How go get Works

When you run go get, the first thing it does is figure out which modules need to be updated. You give it a list of modules, packages, or path patterns as arguments.

Each argument can include a version query suffix and this fancy term just means you can add an @ symbol followed by the version you want:

• Specific version @v1.2.3: Tells go get to use that exact version.

• Version prefix @v1.2: Grabs the latest version that starts with that prefix.

• Branch @main or tag @v1.2.3: Points to the latest version on that branch or tag.

• Commit hash or revision @abcdef: Uses that specific commit.

• Latest version @latest or latest patch version @patch: Grabs the newest version or patch.

Once go get figures out which modules and versions to use based on your arguments, it updates your project’s go.mod file which keeps track of the minimum required versions for your dependencies. If new dependencies need higher versions, go get will also automatically update the go.mod file to reflect that.

Now, what if you have two dependencies, A and B, and they both rely on the same module C but at different versions? How does Go sort this out?

How Go Resolves Dependencies

Go uses something called Minimal Version Selection (MVS) to handle dependencies and resolve version conflicts.

MVS sounds complicated, but here’s the gist: it picks the lowest version of each module that satisfies all the requirements from other modules. This way, it keeps dependencies as minimal as possible while still meeting the needs of everything involved.

For example, let’s say you have three modules: A, B, and C.

• Module A@1.3.2 and module B@1.0.0 both depend on module C.

• But A needs C@2.3.2, and B needs C@2.3.0.

• C also has a newer version, 2.4.1.

How does Go decide which version of C to use?

MVS checks all the requirements for module C from modules A and B. Since A requires C version 2.3.2 and B requires C version 2.3.0, MVS will pick the lowest version of C that satisfies both requirements.

In this case, the smallest version that satisfies both 2.3.0 and 2.3.2 is 2.3.2. Choosing C at 2.4.1 wouldn’t be smallest, so it sticks with 2.3.2.

“What if the chosen version 2.3.2 isn’t compatible with module B?”

Compatibility is handled by semantic versioning (semver), which Go modules use.

If a module follows semver, any breaking changes should bump the major version (major.minor.patch). Since both versions 2.3.0 and 2.3.2 have the same major versions (2.3.x), they should be backward-compatible with each other.

It’s up to the module author to make sure different versions stay compatible.

“What if A needs C at version 3.0.0, which would break compatibility with B?”

When a module hits major version 2 or higher, its path includes the major version number (like /v2 or /v3):

github.com/user/project/v2
github.com/user/project/v3

Modules with different major versions are treated as separate modules. So, you can have both C@2.3.2 and C@3.0.0 in the same project.

For instance, your go.mod file might look like this:

module yourproject

require (
    github.com/user/A v1.0.0
    github.com/user/B v1.0.0
)
require (
    github.com/user/C/v2 v2.3.2 // indirect
    github.com/user/C/v3 v3.0.0 // indirect
)

The // indirect comment means your project doesn't directly import the C module, it's there because A or B needs it.

One small thing: go get doesn’t update or add missing test dependencies. To include those, use the -t flag, like go get -t ./...

Let’s look at some examples to see how go get works in different situations.

go get .

Using go get . or go get ./... finds all the missing dependencies in the current directory or its subdirectories and adds them to the go.mod file.

“Missing” is the key point here.

This means it checks for any dependencies that aren’t already listed and adds them. It doesn’t update existing dependencies to their latest versions unless you specifically ask it to, like with the -u flag in the next example.

go get -u .

Using the -u flag with go get . updates the existing dependencies in the current directory to their latest minor or patch versions. Remember, it won’t update to a new major version because those are treated as different modules.

To update all dependencies of your main module to their latest versions, you can use go get -u ./...

In many cases below, even without specifying the -u flag, go get might still update dependencies if they are out of date or missing.

go get github.com/user/project

This command downloads the module github.com/user/project and adds it to your go.mod file. If it’s already listed there, it updates the module to the latest minor or patch version.

Basically, if you don’t specify a version (or a version query suffix), it assumes you want to upgrade to the latest version, just like using go get github.com/user/project@upgrade.

go get github.com/user/project/package

Even though a package isn’t a module itself, go get still updates the module that provides that package.

It works similarly to go get github.com/user/project, with the hidden @upgrade behavior to get the latest version of the module that contains the package.

go get github.com/user/project@v1.2.3

This command updates the module to the specified version, v1.2.3. Depending on your current version, it could either upgrade or downgrade the module to match that version.

go install: builds and installs packages

What does it mean to build and install packages?

Unlike just downloading dependencies to use their source code in your project, go install builds the source code of the dependencies into a binary file and installs it by moving it to the $GOPATH/bin directory. This makes it available for use from the terminal.

$ go install golang.org/x/tools/gopls@latest

When you run this command and check your $GOBIN folder (which we talked about earlier), you’ll see an executable file named gopls there. You can then run gopls in the terminal, assuming $GOBIN is in your $PATH.

$ gopls version
golang.org/x/tools/gopls v0.15.3

“What if I run go install without any arguments?”

Go install will download the missing dependencies and build the current module in the current directory.

This leads to some folks mistakenly using go install to manage dependencies because it does download them. But that’s not its main job, it’s actually meant to build your project and install the resulting binary to the $GOBIN directory.

So, what’s the difference between go install and go get? 

go install is for building and installing packages, while go get is for managing dependencies. People often get confused because, in older Go versions, go get indeed used to build packages after updating the go.mod file and then install them to $GOPATH/bin.

But since Go 1.16, go install is the go-to command for building and installing, leaving go get to focus on managing requirements in go.mod file.

go mod

The go mod command family is used to manage the main module and its dependencies. These commands help create, edit, and maintain the go.mod file for your project.

go mod init

As the name suggests, this command initializes a module in the current directory.

go mod init github.com/user/project

After running this, you’ll see a go.mod file in the current directory and there’s not much else to it.

One small thing: the module name or path is optional since Go can often figure it out based on existing code, config files, or even the current directory structure:

“If the module path argument is omitted, init will attempt to infer the module path using import comments in .go files, vendoring tool configuration files (like Gopkg.lock), and the current directory (if in GOPATH).”

go mod tidy

go mod tidy is actually one of the most handy commands in Go, it checks all the packages in your project and their dependencies to clean up and optimize your go.mod file.

How does it work?

1. It looks at all the packages in your main module and their dependencies recursively, as if all build tags are enabled (except // +build ignore).

2. It ensures all the modules used by packages in your main module are listed in the go.mod file. If any module isn’t listed, it finds and downloads the latest version.

3. It removes any dependencies that aren’t actually used by your main module or its dependencies.

4. It adds any transitive dependencies used indirectly by your main module, marking them with // indirect.

A transitive dependency means it’s not directly used by your code but is needed by another module you depend on.

Unlike go get, go mod tidy doesn’t update dependencies to their latest versions (you might need the -u flag for that), but it also includes test dependencies in the go.mod file.

go mod download

If you’ve read about the module cache directory ($GOPATH/pkg/mod), go mod download is the command that populates this cache by downloading all the dependencies—both direct and indirect—listed in the go.mod file.

It doesn’t need to look at your source code like go mod tidy does, it just reads the go.mod file and downloads everything.

This is why go mod download is often used in a Dockerfile to cache dependencies before building the project. With just the go.mod file, go mod download takes care of downloading everything you need.

go mod why

If you’re wondering why a package is in your go.mod file, you can use the go mod why command to find out.

$ go mod why github.com/user/project

# github.com/user/project
myproject/module/logic
github.com/user/anotherproject
github.com/user/project

The go mod why command shows you the shortest path from your main module to the package in question.

In this example, myproject/module/logic is a package in your main module that imports github.com/user/anotherproject, which then imports github.com/user/project.

There are other commands in the go mod family, but they aren’t used as often, so we’ll just take a quick look at them.

go mod edit

Normally, we edit the go.mod file directly, but go mod edit is handy for tools and scripts that need to make changes to the go.mod file without manual editing.

go mod graph

go mod graph helps you understand your project’s dependencies by printing out a module requirement graph. It shows how each module version is linked to the versions of the modules it depends on.

Though it’s called a “graph,” it’s just a list of modules and their dependencies for now, not a visual graph.

go mod vendor

We’ll dive deeper into the topic of vendoring next, but here’s a quick overview:

Instead of downloading dependencies from the internet and caching them in $GOPATH/pkg/mod, go mod vendor lets you download and store them right in your project’s root folder. This makes your project portable.