Golang for the Pros: Networking, Multithreading, Data Structures, and Machine Learning with Go

Hello Habitants!



Are you already familiar with the basics of the Go language? Then this book is for you. Michalis Tsukalos will demonstrate the capabilities of the language, give clear and simple explanations, give examples and suggest effective programming patterns. As you explore the nuances of Go, you will master the language's data types and structures, as well as packaging, concurrency, network programming, compiler design, optimization, and more. The materials and exercises at the end of each chapter will help reinforce your new knowledge. A unique material will be the chapter on machine learning in Go, which will walk you from basic statistical techniques to regression and clustering. You will learn classification, neural networks, and anomaly detection techniques. In the Applied Sections, you will learn how to use Go with Docker and Kubernetes, Git, WebAssembly, JSON, and more.

And again about Go-channels

Once the select keyword is used, there are several unique ways to use Go channels that do much more than what you saw in Chapter 9. In this section, you will learn about the different uses of Go channels.



Let me remind you that the zero value for channels is nil, and if you send a message to a closed channel, the program will go into panic mode. But if you try to read data from a closed channel, you will get a zero value for this type of channel. Thus, after closing the channel, you can no longer write data to it, but you can still read it.



For a channel to be closed, it does not have to be designed to only receive data. In addition, channel zero is always blocked, that is, an attempt to read or write from channel zero will block the channel. This property of channels is very useful when you want to disable a branch of a select statement by setting the channel variable to nil.



Finally, when trying to close the zero channel, the program will raise a panic. Let's look at the closeNilChannel.go example:

package main

func main() {
      var c chan string
      close(c)
}

Executing closeNilChannel.go will produce the following result:

$ go run closeNilChannel.go
panic: close of nil channel
goroutine 1 [running]:
main.main()
       /Users/mtsouk/closeNilChannel.go:5 +0x2a
exit status 2

Signal channels

A signaling channel is a channel that is used only for signaling. Simply put, the signaling channel can be used when you want to inform another program about something. Signaling channels do not need to be used to transfer data.

Signaling channels should not be confused with the UNIX signal handling discussed in Chapter 8, they are completely different things.

An example of code that uses signaling channels is discussed later in this chapter.

Buffered channels

The topic of this subsection is buffered pipes. These are channels that allow the Go scheduler to quickly queue jobs to handle more requests. In addition, they can be used as semaphores to limit the bandwidth of an application.



The method presented here works like this: all incoming requests are redirected to a channel, which processes them in turn. When the channel finishes processing the request, it sends a message to the original caller that the channel is ready to process a new request. Thus, the buffer capacity of a channel limits the number of concurrent requests that channel can store.



We'll look at this method using the bufChannel.go program code as an example. Let's divide it into four parts.



The first part of the bufChannel.go code looks like this:

package main

import (
       "fmt"
)

The second portion of the bufChannel.go file contains the following Go code:

func main() {
      numbers := make(chan int, 5)
      counter := 10

The numbers definition presented here allows you to store up to five integers in this pipe.



The third part of bufChannel.go contains the following Go code:

for i := 0; i < counter; i++ {
     select {
     case numbers <- i:
     default:
            fmt.Println("Not enough space for", i)
     }
}

In this code, we tried to put ten numbers in the numbers channel. However, since numbers only have room for five integers, we cannot store all ten integers in it.



The rest of the Go code from bufChannel.go looks like this:

   for i := 0; i < counter+5; i++ {
        select {
              case num := <-numbers:
                    fmt.Println(num)
              default:
                    fmt.Println("Nothing more to be done!")
              break
        }
   }
}

In this Go code, we tried to read the contents of the numbers channel using a for loop and a select statement. As long as there is something to read in the numbers channel, the first branch of the select statement will execute. When the numbers channel is empty, the default branch is executed.



Executing bufChannel.go will produce the following result:

$ go run bufChannel.go
Not enough space for 5
Not enough space for 6
Not enough space for 7
Not enough space for 8
Not enough space for 9
0
1
2
3
4
Nothing more to be done!
Nothing more to be done!
Nothing more to be done!
Nothing more to be done!
Nothing more to be done!
Nothing more to be done!
Nothing more to be done!
Nothing more to be done!
Nothing more to be done!
Nothing more to be done!

Zero channels

In this section, you will learn about channel zero. This is a special kind of channel that is always blocked. We'll look at these channels using the nilChannel.go program as an example. Let's split it into four pieces of code.



The first part of nilChannel.go looks like this:

package main

import (
       "fmt"
       "math/rand"
       "time"
)

The second part of nilChannel.go contains the following Go code:

func add(c chan int) {
      sum := 0
      t := time.NewTimer(time.Second)

      for {
           select {
           case input := <-c:
                 sum = sum + input
           case <-t.C:
                 c = nil
                 fmt.Println(sum)
           }
      }
}

Here, using the add () function as an example, shows how the zero channel is used. The <-tC operator blocks the C channel of timer t for the time specified in the call to time.NewTimer (). Do not confuse channel c, which is a function argument, with channel tC, which belongs to timer t. When the time expires, the timer sends a value to the tC channel, which initiates the execution of the corresponding branch of the select statement - it sets channel c to nil and displays the value of the sum variable.



The third nilChannel.go code snippet looks like this:

func send(c chan int) {
      for {
           c <- rand.Intn(10)
      }
}

The purpose of the send () function is to generate random numbers and send them to the channel as long as the channel is open.



The rest of the Go code in nilChannel.go looks like this:

func main() {
      c := make(chan int)
      go add(c)
      go send(c)
      time.Sleep(3 * time.Second)
}

The time.Sleep () function is needed so that the two goroutines have enough time to execute.



Running nilChannel.go will produce the following results:

$ go run nilChannel.go
13167523
$ go run nilChannel.go
12988362

Since the number of times the first branch of the select statement in add () is executed is not fixed, running nilChannel.go multiple times will produce different results.

Channel channels

A channel channel is a special kind of channel variable that works with other channels instead of the usual variable types. However, you still need to declare a datatype for a channel of channels. To define the channel of channels, use the chan keyword twice in a row, as shown in the following statement:

c1 := make(chan chan int)

Other types of channels presented in this chapter are more popular and useful than channel channels.

We'll walk through the use of channel channels using the example code found in the chSquare.go file. Let's divide it into four parts.



The first part of chSquare.go looks like this:

package main

import (
       "fmt"
       "os"
       "strconv"
       "time"
)

var times int

The second part of chSquare.go contains the following Go code:

func f1(cc chan chan int, f chan bool) {
      c := make(chan int)
      cc <- c
      defer close(c)

      sum := 0
      select {
      case x := <-c:
            for i := 0; i <= x; i++ {
                 sum = sum + i
            }
            c <- sum
      case <-f:
            return
      }
}

Having declared a regular channel of int type, we pass it to the channel channel variable. Then, using the select statement, we get the opportunity to read data from a regular int channel or exit the function using the signal channel f.



After reading one value from channel c, we run a for loop that calculates the sum of all integers from 0 to the integer value we just read. Then we send the calculated value to the int channel c, and that's it.



The third part of chSquare.go contains the following Go code:

func main() {
      arguments := os.Args
      if len(arguments) != 2 {
          fmt.Println("Need just one integer argument!")
          return
      }
      times, err := strconv.Atoi(arguments[1])
      if err != nil {
           fmt.Println(err)
           return
      }

      cc := make(chan chan int)

In the last line of this code snippet, we declare a channel variable named cc. This variable is the star of this program, because everything depends on it. The cc variable is passed to f1 () and used in the next for loop.



The rest of the chSquare.go Go code looks like this:

   for i := 1; i < times+1; i++ {
        f := make(chan bool)
        go f1(cc, f)
        ch := <-cc
        ch <- i
        for sum := range ch {
             fmt.Print("Sum(", i, ")=", sum)
        }
        fmt.Println()
        time.Sleep(time.Second)
        close(f)
    }
}

Channel f is the signal channel for the end of the goroutine when all the work is done. The ch: = <-cc instruction allows you to get a regular channel from a channel variable to pass an int value there using the ch <- i operator. After that, we read data from the pipe using a for loop. The f1 () function is programmed to return one value, but we can read multiple values ​​as well. Note that each i value is served by its own goroutine.



The signal channel type can be anything, including the bool used in the previous code and the struct {}, which will be used for the signal channel in the next section. The main advantage of a signaling channel of type struct {} is that data cannot be sent to such a channel, this prevents errors from occurring.



Executing chSquare.go will produce results like this:

$ go run chSquare.go 4
Sum(1)=1
Sum(2)=3
Sum(3)=6
Sum(4)=10
$ go run chSquare.go 7
Sum(1)=1
Sum(2)=3
Sum(3)=6
Sum(4)=10
Sum(5)=15
Sum(6)=21
Sum(7)=28

Choosing the sequence of execution of goroutines

You are not required to make any assumptions about the sequence of execution of the goroutines. However, there are times when it is necessary to control this order. In this subsection, you will learn how to do this using signaling channels.

You may ask, "Why create goroutines and then execute them in a given order when it's much easier to do the same with regular functions?" The answer is simple: goroutines can run concurrently and wait for other goroutines to complete, whereas functions cannot because they execute sequentially.

In this subsection, we'll look at a Go program called defineOrder.go. Let's divide it into five parts. The first part of defineOrder.go looks like this:

package main

import (
       "fmt"
       "time"
)

func A(a, b chan struct{}) {
      <-a
      fmt.Println("A()!")
      time.Sleep(time.Second)
      close(b)
}

Function A () is blocked by the channel stored in parameter a. As soon as this channel is unlocked in main (), the A () function will start working. Finally, it will close channel b, thereby unblocking another function - in this case, B ().



The second part of defineOrder.go contains the following Go code:

func B(a, b chan struct{}) {
      <-a
      fmt.Println("B()!")
      close(b)
}

The logic of the function B () is the same as that of A (). This function is blocked until channel a is closed. Then it does its job and closes channel b. Note that channels a and b refer to the function parameter names.



The third piece of code for defineOrder.go looks like this:

func C(a chan struct{}) {
      <-a
      fmt.Println("C()!")
}

The C () function is blocked and is waiting for channel a to close before starting.



The fourth part of defineOrder.go contains the following code:

func main() {
      x := make(chan struct{})
      y := make(chan struct{})
      z := make(chan struct{})

These three channels will become parameters for three functions.



The last snippet of defineOrder.go contains the following Go code:

     go C(z)
     go A(x, y)
     go C(z)
     go B(y, z)
     go C(z)

     close(x)
     time.Sleep(3 * time.Second)
}

Here the program performs all the necessary functions, and then closes channel x and sleeps for three seconds.



Executing defineOrder.go will produce the desired result, even though the C () function will be called multiple times:

$ go run defineOrder.go
A()!
B()!
C()!
C()!
C()!

Calling C () multiple times as a goroutine will not cause problems, because C () does not close any channels. But if you call A () or B () more than once, then, most likely, an error message will be displayed, for example:

$ go run defineOrder.go
A()!
A()!
B()!
C()!
C()!
C()!
panic: close of closed channel
goroutine 7 [running]:
main.A(0xc420072060, 0xc4200720c0)
       /Users/mtsouk/Desktop/defineOrder.go:12 +0x9d
created by main.main
       /Users/mtsouk/Desktop/defineOrder.go:33 +0xfa
exit status 2

As you can see, here the A () function was called twice. However, when A () closes a channel, one of its goroutines detects that the channel is already closed and creates a panic situation when it tries to close that channel again. If we try to call the B () function more than once, we get a similar panic situation.

How not to use goroutines

In this section, you will learn a naive way of sorting natural numbers using goroutines. The program we'll be looking at is called sillySort.go. Let's divide it into two parts. The first part of sillySort.go looks like this:

package main

import (
       "fmt"
       "os"
       "strconv"
       "sync"
       "time"
)

func main() {
      arguments := os.Args

      if len(arguments) == 1 {
          fmt.Println(os.Args[0], "n1, n2, [n]")
          return
      }

      var wg sync.WaitGroup
      for _, arg := range arguments[1:] {
           n, err := strconv.Atoi(arg)
           if err != nil || n < 0 {
                fmt.Print(". ")
                continue
           }

The second part of sillySort.go contains the following Go code:

           wg.Add(1)
           go func(n int) {
                defer wg.Done()
                time.Sleep(time.Duration(n) * time.Second)
                fmt.Print(n, " ")
           }(n)
      }

      wg.Wait()
      fmt.Println()
}

Sorting is performed by calling the time.Sleep () function - the larger the natural number, the longer it takes before the fmt.Print () operator is executed!



Executing sillySort.go will produce results like this:

$ go run sillySort.go a -1 1 2 3 5 0 100 20 60
. . 0 1 2 3 5 20 60 100
$ go run sillySort.go a -1 1 2 3 5 0 100 -1 a 20 hello 60
. . . . . 0 1 2 3 5 20 60 100
$ go run sillySort.go 0 0 10 2 30 3 4 30
0 0 2 3 4 10 30 30

about the author

Mihalis Tsoukalos is a UNIX administrator, programmer, database administrator and mathematician. Likes to write technical books and articles, learn something new. In addition to this book, Michalis has written Go Systems Programming as well as over 250 technical articles for many magazines, including Sys Admin, MacTech, Linux User and Developer, Usenix; login :, Linux Format, and Linux Journal. Michalis's research interests are databases, visualization, statistics and machine learning.

About the scientific editor

Mat Ryer has been writing computer programs since the age of six: first on BASIC for the ZX Spectrum, and then, with his father, on AmigaBASIC and AMOS for the Commodore Amiga. He spent a lot of time manually copying code from the Amiga Format magazine, changing the values ​​of variables or GOTO statement references to see what came of it. The same spirit of exploration and obsession with programming led 18-year-old Matt to work for a local organization in Mansfield, UK, where he began building websites and other online services.



After several years of working with various technologies in various fields, not only in London but around the world, Mat turned his attention to a new systems programming language called Go, first used at Google. Because Go was solving very hot and hot technical problems, Mat started using the language for problem solving when Go was still in beta and has continued to program in it ever since. Mat has worked on various open source projects, created several Go packages, including Testify, Moq, Silk and Is, and the MacOS developer toolkit BitBar.



Since 2018 Mat has been a co-founder of Machine Box, but he still participates in conferences, writes about Go on his blog, and is an active member of the Go community.



»More details about the book can be found on the publisher's website

» Table of Contents

» Excerpt



For Habitants a 25% discount on coupon - Golang



Upon payment for the paper version of the book, an e-book is sent to the e-mail.