How do I handle exceptions when using threading and Queue?

If I have a program that uses threading and Queue, how do I get exceptions to stop execution? Here is an example program, which is not possible to stop with ctrl-c (basically ripped from the python docs).

from threading import Thread
from Queue import Queue
from time import sleep

def do_work(item):
    print "working" , item

def worker():
        while True:
            item = q.get()

q = Queue()

num_worker_threads = 10

for i in range(num_worker_threads):
     t = Thread(target=worker)
    # t.setDaemon(True)

for item in range(1, 10000):

q.join()       # block until all tasks are done

Asked by: Caroline617 | Posted: 06-12-2021

Answer 1

The simplest way is to start all the worker threads as daemon threads, then just have your main loop be

while True:

Hitting Ctrl+C will throw an exception in your main thread, and all of the daemon threads will exit when the interpreter exits. This assumes you don't want to perform cleanup in all of those threads before they exit.

A more complex way is to have a global stopped Event:

stopped = Event()
def worker():
    while not stopped.is_set():
            item = q.get_nowait()
        except Empty:      # import the Empty exception from the Queue module

Then your main loop can set the stopped Event to False when it gets a KeyboardInterrupt

    while not stopped.is_set():
except KeyboardInterrupt:

This lets your worker threads finish what they're doing you want instead of just having every worker thread be a daemon and exit in the middle of execution. You can also do whatever cleanup you want.

Note that this example doesn't make use of q.join() - this makes things more complex, though you can still use it. If you do then your best bet is to use signal handlers instead of exceptions to detect KeyboardInterrupts. For example:

from signal import signal, SIGINT
def stop(signum, frame):
signal(SIGINT, stop)

This lets you define what happens when you hit Ctrl+C without affecting whatever your main loop is in the middle of. So you can keep doing q.join() without worrying about being interrupted by a Ctrl+C. Of course, with my above examples, you don't need to be joining, but you might have some other reason for doing so.

Answered by: Melanie453 | Posted: 07-01-2022

Similar questions

python - Avoid exceptions?

This particular example relates to Django in Python, but should apply to any language supporting exceptions: try: object = ModelClass.objects.get(search=value) except DoesNotExist: pass if object: # do stuff The Django model class provides a simple method get which allows me to search for one and only one object from the database, if it finds more or less it ...

Catching Python exceptions using 'expect' method?

import sys try: file = open("words.txt") expect(IOError): if file: print "%s" % file else: print "Cant the %s file" % "words.txt" this gives me an a error - File "", line 4 expect(IOError): SyntaxError: invaild syntax What im going wrong/ how do you fix this

python - How to properly ignore exceptions

When you just want to do a try-except without handling the exception, how do you do it in Python? Is the following the right way to do it? try: shutil.rmtree(path) except: pass

Python - Possible to force raise exceptions regardless try/except blocks?

In Python is there any language (or interpreter) feature to force the python interpreter to always raise exceptions even if the exception offending code is inside a try/except block ? I've just inherited a larger and old codebase written in python, whose purpose is to communicate with some custom designed hardware we also developed. Many communication errors and timeouts are being masked/missed due to the followin...

Best Practices for Python Exceptions?

What are the best practices for creating exceptions? I just saw this, and I don't know if I should be horrified, or like it. I read several times in books that exceptions should never ever hold a string, because strings themselves can throw exceptions. Any real truth to this? Basically from my understanding from the scripts is that this was done so all the inhouse Python libraries will have a common error message f...

unit testing - Python unittest: how do I test the argument in an Exceptions?

I am testing for Exceptions using unittest, for example: self.assertRaises(UnrecognizedAirportError, func, arg1, arg2) and my code raises: raise UnrecognizedAirportError('From') Which works well. How do I test that the argument in the exception is what I expect it to be? I wish to somehow assert that capturedException.argument == 'Fro...

python - FastCgi crashes -- Want to catch all exceptions but how?

I have a django app running on apache with fastcgi (uses Flup's WSGIServer). This gets setup via dispatch.fcgi, concatenated below: #!/usr/bin/python import sys, os sys.path.insert(0, os.path.realpath('/usr/local/django_src/django')) PROJECT_PATH=os.environ['PROJECT_PATH'] sys.path.insert(0, PROJECT_PATH) os.chdir(PROJECT_PATH) os.environ['DJANGO_SETTINGS_MODULE'] = "settings" from django.cor...

cleaning up when using exceptions and files in python

I'm learning python for a couple of days now and am struggling with its 'spirit'. I'm comming from the C/C++/Java/Perl school and I understand that python is not C (at all) that's why I'm trying to understand the spirit to get the most out of it (and so far it's hard)... My question is especially focused on exception handling and cleaning: The code at the end of this post is meant to simulate a fairly common case o...

Correct way of handling exceptions in Python?

I have searched for other posts, as I felt this is a rather common problem, but all other Python exception questions I have found didn't reflect my problem. I will try to be as specific here as I can, so I will give a direct example. And pleeeeease do not post any workarounds for this specific problem. I am not specifically interested how you can send an email much nicer with xyz. I want to know how you generally d...

python - Logging All Exceptions in a pyqt4 app

What's the best way to log all of the exceptions in a pyqt4 application using the standard python logging api? I've tried wrapping exec_() in a try, except block, and logging the exceptions from that, but it only logs exceptions from the initialization of the app. As a temporary solution, I wrapped the most important methods in try, except blocks, but that can't be the only way to do it.

Still can't find your answer? Check out these communities...

PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python