Go Errgroup: How to Use Goroutines Effectively
Have you ever written a function that's long because it does many tasks, even if they don't rely on each other?
Have you ever written a function that’s long because it does many tasks, even if they don’t rely on each other? I’ve been in your shoes.
Think about this, you have a function to do 3 things:
Fetching user details from a database by userID.
Calling an external service to get the user’s recent activity by userID
Accessing a logging service to get the user’s last login details by userID
func FetchUserData(userID string) error {
g := errgroup.Group{}
userDetail, _ := fetchUserDetails(userID)
userAct, _ := fetchUserActivity(userID)
userLoginInfo, _ := fetchUserLoginDetails(userID)
// ...
}All these tasks just need the userID and don’t use data from the others.
So, doing them at the same time would save time, right? you may try using goroutines for this, but you need to handle the details yourself. Let’s answer these question:
Synchronize: How do you make sure everything finishes?
Error handling: What do you do if one task goes wrong? Or if two out of three don’t work?
Cancellation: If one task has a problem, how do you stop all the other running goroutines?
Limiting: Have you thought about setting a limit on how many goroutines run at the same time?
Reusuable: Once you find a solution, how can you use it again in similar situations?
…
There’s a lot to think about, right? What if I told you there’s already a solution for this? A tool you can use in many parts of your code. Let’s talk about errgroup, learn how to use it, and find out why it can help.
1. What is errgroup?
The errgroup package lets you handle many tasks at once.
It makes it easy to run things together in a safe way, keep them in sync, deal with errors, and control when to stop goroutines. Here’s a quick example of how to use it:
func FetchUserData() error {
g := errgroup.Group{}
// Fetch user details
g.Go(func() error {
time.Sleep(1 * time.Second)
fmt.Println("Fetched user details...")
return nil
})
// Fetch user activity
g.Go(func() error {
time.Sleep(1 * time.Second)
fmt.Println("Fetched user activity...")
return nil
})
// Fetch user login details
g.Go(func() error {
time.Sleep(2 * time.Second)
fmt.Println("Fetched user login details...")
return nil
})
// Wait for all goroutines to finish and return the first error (if any)
return g.Wait()
}So, what errgroup does is run these tasks and waits for them to end with g.Wait(), all we need to do is add the tasks.
This approach is really useful when you have a lot of tasks, like 10, 20, or even more.
But there’s something to think about, running too many goroutines at the same time can use up resources if you don’t keep an eye on it. How do we deal with that? Let’s explore in the next section.”
2. SetLimit: Limit the number of goroutines
This package is understand us and provide a way to limit the number of goroutines running at the same time, let’s see how to use it:
func FetchUserData() error {
g := errgroup.Group{}
// Set the limit to 2
g.SetLimit(2)
// Fetch user details
g.Go(func() error {
time.Sleep(1 * time.Second)
fmt.Println("Fetched user details...")
return nil
})
// Fetch user activity
g.Go(func() error {
time.Sleep(1 * time.Second)
fmt.Println("Fetched user activity...")
return nil
})
// Fetch user login details
g.Go(func() error {
time.Sleep(2 * time.Second)
fmt.Println("Fetched user login details...")
return nil
})
// Wait for all goroutines to finish and return the first error (if any)
return g.Wait()
}By doing this, we make sure only 2 goroutines run at the same time. If you try it out, the first two tasks will show their messages together and the third one will show its message 3 seconds after starting.
“Can I change the limit after setting it?”
The answer is yes, but be careful.
If any goroutine is running, trying to change the limit will cause errgroup.SetLimit() to panic.
Now, let’s think about something else, what if you don’t want to add more goroutines when there are already maybe 50 in the errgroup?
3. TryGo: A Controlled Approach to Adding Goroutines
TryGo shares similarities with the Go function, yet it offers a nuanced approach to handling goroutines in an errgroup. Specifically, it prevents the addition of a new goroutine if the current count reaches the set limit
The signature of TryGo is a litle bit different:
// TryGo: Checks and adds a goroutine
func (g *Group) TryGo(fn func() error) bool
// Go: Just adds a goroutine
func (g *Group) Go(fn func() error)If you use TryGo and it successfully adds a goroutine to the errgroup, it communicates that success by returning true. If it doesn't, you'll get a false in response.
But here’s where it gets interesting.
When an errgroup is full, the errgroup.Go() blocks until a goroutine finishes before adding a new one, in contrast, errgroup.TryGo() doesn't wait. If the errgroup is at its limit, TryGo returns false right away.
4. WithContext: Handle cancellation
How can you stop all the running goroutines if just one of them faces an error, so you don’t waste resources?
The WithContext function is used to make a new errgroup along with a context:
erg, ctx := errgroup.WithContext(context.Background())This function gives you the context but not the cancel function, so you can’t stop the context yourself.
“Will the errgroup stop all the goroutines for me if an error comes up?”
No, the errgroup cancels the context but doesn’t stop the goroutines. The rest of the goroutines will keep running until they are done, unless you do this:
func main() {
g, ctx := errgroup.WithContext(context.Background())
g.Go(func() error { return fetchUserDetails(ctx) })
g.Go(func() error { return fetchUserActivity(ctx) })
g.Go(func() error { return fetchUserPaymentHistory(ctx) })
// Wait for tasks to finish and print the error
if err := g.Wait(); err != nil {
fmt.Printf("Encountered an error: %v\n", err)
}
}It’s important to make these tasks depend on the context. So, when the context is canceled, all the goroutines will be stopped too.
Next time, I’ll talk about how errgroup functions. Follow along to know when the next article is out.
This method not only cuts down on extra code but also gives a strong way to deal with errors and control the lifetimes of goroutines.
Connect with me on Twitter or LinkedIn at www.linkedin.com/in/quocphuong
How do we return an object along with the error from the go routine?