Go Sum You Should Know in Golang

The Power of Go. Sum: Ensuring Consistency

Golang Go Sum (from github.com/MariaLetta/free-gophers-pack)|300

Hello, here is Wesley, Today’s article is about go.sum in Go. Without further ado, let’s get started.💪

Introduction

To ensure consistency in building, Go introduced the go.mod file to mark each dependency package’s version. During the build process, the go command downloads dependencies from go.mod, caches them locally for future builds.

However, considering that downloaded dependencies may be maliciously tampered with by hackers, and cached dependencies may also be tampered with, a single go.mod file cannot guarantee consistency in building.

Therefore, introducing the go.sum file is necessary, used to record each dependency package’s hash value, during the build process, if the local dependency package’s hash value does not match the recorded hash value in the go.sum file, it will reject the build.

The go.sum file is used together with the go.mod file and automatically generates and records all modules required by a Go language project, along with their corresponding hash values. When you execute commands like go build, go test, or go mod tidy in your project, the Go toolchain will generate or update this file.

go.sum File Format

1
<module> <version>/go.mod <hash>

or

1
2
<module> <version> <hash>
<module> <version>/go.mod <hash>

Where:

  • module is the path of the dependency.
  • version is the version number of the dependency.
  • hash is a string starting with h1: representing the checksum algorithm (sha256) used to generate the hash value.

Normally, each dependency package version will have two records:

  • The first record represents the overall hash value of the entire package (all files).
  • The second record only represents the hash value of the go.mod file within that package version.
    If a package version does not contain a go.mod file, there will be only one record.

Example:

1
github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=  github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=

In this example, v1.1.1 represents the overall package version, while v1.1.1/go.mod represents the hash value of the go.mod file within that package version.

If any file (including go.mod) in a package version is modified, it will change the overall hash value, and an additional record will be kept to calculate the dependency tree without downloading the entire package version, using only the go.mod file.

go.sum Versions

  1. If a project does not have a tag, it will generate a version number in the format v0.0.0-commitDate-commitID.
    Example: github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=.

  2. If a project references a specific branch, such as the develop branch, it will also generate a similar version number in the format v+1-commitDate-commitID.
    Example: `github.com/DATA-DOG/go-sqlmock v1.3.4-0.20191205000432-012d92843b00 h1:Cnt/xQ9MO4BiAjZrVpl0BiqqtTJjXUkWhIqwuOCVtWo=.

  3. If a project uses Go modules, the version number will be normal and use tags.
    Example: `github.com/DATA-DOG/go-sqlmock v1.3.3 h1:CWUqKXe0s8A2z6qCgkP4Kru7wC11YoAnoupUKFDnH08=.

  4. If a project has a tag but does not use Go modules, it will add the +incompatible flag to distinguish it from projects that use Go modules.
    Example: `github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=.

  5. For projects that use the v2+ Go module, the project path will have a version number suffix.
    Example: `github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM=.

go.sum Generation and Verification

The environment variable GOSUMDB identifies a checksum database, which is essentially a web server that provides a service for querying the hash values of dependency package versions.
This database records many dependency package version hash values, such as Google’s official sum.golang.org, which records all publicly available dependency packages.

In addition to using the official database, you can also specify your own custom database or even disable it (export GOSUMDB=off).

If the system is configured with GOSUMDB, before writing a dependency package version to go.sum, the system will query the hash value of that dependency package version from the database and perform a secondary verification. If the verification passes, the system will then write the hash value to go.sum.

If the system disables GOSUMDB, before writing a dependency package version to go.sum, the system will not perform a secondary verification, and the Go command will trust all downloaded dependencies and record their hash values in go.sum.

go get

When we introduce a new dependency in GOMODULE mode, we usually use the go get command to retrieve that dependency.

Before updating go.sum, to ensure that the downloaded dependency package is genuine and reliable, the Go command will query the GOSUMDB server after downloading the dependency package to obtain an authoritative hash value for the dependency package version.

If the Go command’s calculated hash value does not match the hash value provided by the GOSUMDB server, the Go command will refuse to proceed and will not update go.sum.

go build

The Go command will search for all dependencies recorded in go.mod from the local cache and calculate the hash values of the dependencies, then compare them with the records in go.sum, which detects whether the dependency package versions used in the local cache directory meet the expectations of the project’s go.sum file.

If the verification fails, it means that the hash value of the dependency package version in the local cache directory does not match the record in go.sum, and the Go command will refuse to build.
When the verification fails, it is necessary to confirm whether the local cache or the go.sum record is incorrect. Both are possible errors; the local cache directory may have been intentionally or unintentionally modified, and the go.sum record may have been tampered with.

In the event of a failure, the Go command tends to trust go.sum, because a new dependency package version must be verified by GOSUMDB before being added to go.sum.

Advantages and Disadvantages of go.sum

  1. Distributed Package Management with Content Verification

In mainstream package management mechanisms, a central repository is typically used to ensure that each published version’s content cannot be tampered with.
Go, on the other hand, does not have a central repository. After releasing a project with a tag like 0.1, the author can still delete the original tag and submit different content before re-tagging it. Both authors and platforms may tamper with the content. Therefore, only storing checksums of all dependencies in each project can ensure that no dependency is tampered with.

  1. Strengthening Security with Transparent Log

go.sum not only records the current dependency’s checksum but also preserves the checksums of every previous dependency. This approach is similar to the concept of transparent log 1.
Transparent log aims to maintain an Append-Only log record, increasing the cost of tampering and making it easier to review which records have been tampered with.

  1. Potential Merge Conflicts

Each commit is equivalent to a new release, leading to occasional insertion of a new record into the go.sum file when pulling code.
In scenarios where multiple people collaborate on internal public libraries that frequently update version numbers, go.sum may require manual intervention to select the correct version.

  1. Lack of Constraint Ability for Third-Party Libraries

In package managers with central repositories, users can restrict those troublemakers at their source. However, go.sum’s constraint is purely moral. If a library randomly modifies already published versions, it will cause build failures in projects that depend on this library.
The mistake is the author’s, but the hassle falls to the user. Apart from cursing and reporting issues or elsewhere, there seems to be no other solution.

Common Practices

  1. Dependency Retrieval: Each time you run go get, go build, or go mod tidy, Go checks the go.mod file to determine which dependencies and versions your project requires. Then, Go downloads these dependencies and calculates their hash values.
  2. Verification: The calculated hash values are compared with those in the go.sum file. If they match, it indicates that the dependencies have not been tampered with. If they don’t match, Go will report an error, alerting developers to potential dependency tampering or version mismatch.
  3. Update: When you add new dependencies or modify existing ones, go.mod updates to reflect these changes. Simultaneously, go.sum also updates, adding new hash values to ensure security. Running the go mod tidy command helps clean up unused dependencies and synchronize go.mod and go.sum files.
  4. Version Control: Typically, both go.mod and go.sum files are committed together to a version control system (such as Git), ensuring that team members and CI/CD systems obtain the same dependency versions, achieving consistent build results.

Conclusion

The go.sum file is an essential component of the Go module system. It records each module’s dependencies and corresponding checksums, ensuring that the same versions are used across different development environments.

References

Go版本管理–go.sum - failymao - 博客园
谈谈go.sum - spacewander - SegmentFault 思否
Go Modules: v2 and Beyond - The Go Programming Language

More

Recent Articles:

Random Article:


More Series Articles about You Should Know In Golang:

https://wesley-wei.medium.com/list/you-should-know-in-golang-e9491363cd9a

And I’m Wesley, delighted to share knowledge from the world of programming. 

Don’t forget to follow me for more informative content, or feel free to share this with others who may also find it beneficial. It would be a great help to me.

Give me some free applauds, highlights, or replies, and I’ll pay attention to those reactions, which will determine whether I continue to post this type of article.

See you in the next article. 👋

中文文章: https://programmerscareer.com/zh-cn/golang-go-sum/
Author: Medium,LinkedIn,Twitter
Note: Originally written at https://programmerscareer.com/golang-go-sum/ at 2025-01-12 11:39.
Copyright: BY-NC-ND 3.0

Summary for 2024 - Remarkable progress Common Optimizations You Should Know in Golang

Comments

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×