how to use callback in beanstalkd ？

There is 3 machines and one publisher, two consumers. I am using golang to publish order to some machine. And the machine is used python to be consumer. I want to know how can I get the result that the order is finished or failed in publisher.

And if the order is not belong to machine one, what should i do? release or bury it?

python: consumer:

import beanstalkc

def get_beanstalk_data(conf):
    beanstalk = beanstalkc.Connection(host='127.0.0.1',port=11300)
    beanstalk.use('cloud')
    beanstalk.watch('cloud')
    beanstalk.ignore('default')
    job = beanstalk.reserve()

    if job.body == "one":  #job.body == "two"
        #TODO
        job.delete()
        return job.body
    else:
        #TODO    what should I do in here, because there is two consumer and get different orders

while True:
    data = get_beanstalk_data(conf)
    print data

golang: publish:

package main

import (
    "fmt"
    "github.com/kr/beanstalk"
    "time"
)

func main() {
    c, err := beanstalk.Dial("tcp", "127.0.0.1:11300")
    id, err := c.Put([]byte("hello"), 1, 0, 120*time.Second)

    if err != nil {
        fmt.Println(err)
    }
    fmt.Println(id)
}


Asked by: Daniel324 | Posted: 27-01-2022






Answer 1

The right way to let publisher know the status of a job is using callbacks

With job let the publisher put a callback url (a queue or http) and on success or failure of the job th consumer may send a status message to the status call back

so the job structure may loom like

//JobRequest has the struct storing request name and body
type JobRequest struct {
        ID              string
        RequestBody     []byte
        CallbackURL  *string
}

The json strng of the above struct will be the job body.The consumer would get the CallbackURL and will senbd status to that url

Trying to explain as reqiuested details

Let's call producer and consumer(s) master and workers(s) .

  1. When ever a job is available master would create a job object which has

    • job id (A unique value to identify teh job)
    • RequestBody (The details of the job)
    • StatusCallbackURL (URL the worker would hit with the job status)

  2. One of the workers listening to the queue reserve the job,there buy telling i will try to do this job

  3. Decode the json and get the job details .Now does the job
  4. On success delete the job from queue and send status to the CallbackURL
  5. On failure if non temporary failure send status as fail to the CallbackURL and delete the job
  6. If temporary failure do nothing as after reserve timeout over it wil be re-enqueued

Now this object is converted to json and put in queue

PS: Do not delete the job before succesfully completing .On Completion or Permanant Failure only delte the job

Answered by: Daniel805 | Posted: 28-02-2022



Similar questions

BeanStalkd on Solaris doesnt return anything when called from the python library

i am using Solaris 10 OS(x86). i installed beanstalkd and it starts fine by using command "beanstalkd -d -l hostip -p 11300". i have Python 2.4.4 on my system i installed YAML and beanstalkc python libraries to connect beanstalkd with python my problem is when i try to write some code: import beanstalkc beanstalk = beanstalkc.Connection(host='hostip', port=11300) no error so far but when i try to ...

python - Celery or beanstalkd or both?

I know (but I do not understand) that Celery can use Beanstalk as delivery mechanism. From the Beanstalkd Homepage "Beanstalk is a simple, fast work queue." From the Celery Homepage "It’s a task queue with focus on real-time processing, while also supporting task scheduling." That seems pretty simi...

python - Getting jobs from beanstalkd - timed out exception

I am using Python 2.7, beanstalkd server with beanstalkc as the client library. It takes about 500 to 1500 ms to process each job, depending on the size of the job. I have a cron job that will keep adding jobs to the beanstalkd queue and a "worker" that will run in an infinite loop getting jobs and processing them. eg: def ge...

python - Django server stuck on Beanstalkd

I'm trying to setup a django server with existing code. But I seemed to be stuck on this while trying to start server using python manage.py runserver this the traceback: (camdyvirtualenv) Mac-mini-3:big-picture-api Sqooge_Ahmed$ python manage.py runserver Traceback (most recent call last): File "manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/U...





Still can't find your answer? Check out these communities...



PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python



© Patterns & Practices 2022 Rights Reserved | Contact us
top