Go with multiple workers - Goroutines

What I wanted is give a plan with golang as a job workers . So build very small project initially as a prototype.

Requirements :

1. Language should be golang
2. Wanted to plan without multiple servers(service)
3. Queue management already with rabbitmq
4. Wanted to run concurrently multiple jobs(any no of jobs define)
5. To run a job takes some times (minutes)

Let’s plan how we achieve this.

1. We need to add multiple subscribers for rabbitmq queue.
2. We need to implement rabbitmq acknowledgement (that will help to system pick same job if job not running. Until its acknowledged info will not delete from the rabbitmq queue). once Job is complete we need to ack

To run concurrently in go we need to use Goroutines (link : https://tour.golang.org/concurrency/1)

Assume following things

• you have basic idea about golang.
• rabbitmq connection already available.
• rabbitmq queue already available.

I will provide another post basic rabbitmq with golang later (how to connect and how to create queue.)

will write a function for generate multiple goroutines . will call that function as “multipleWorkers”

func multipleWorkers() {
  
  for i := 1; i <= 10; i++ {
    go worker(i)
  }
}

inside that function will create 10 goroutines for run 10 jobs.

Now inside worker function we need to read rabbitmq queue and get one job and ack once job is done.

“failOnError” function is very straight forward that will handle errors for now just show the error.

keep in mind auto-ack is false so we need to send ack status.

var job *Job is job struct

json.Unmarshal(d.Body, &job) will help to json to struct

status := job.Process() run the job and give the bool status

d.Ack(!status) according to the above status, will ack for rabbitmq.

following lines to help to run these forever

forever := make(chan bool)
//and 
<-forever

THATS IT…..!

Next post will talk sync.WaitGroup, limited concurrent for many http requests.