Answer 1

Twisted contains a large number of examples. One in particular, the "evolution of Finger" tutorial, contains a thorough explanation of how an asynchronous program grows from a very small kernel up to a complex system with lots of moving parts. Another one that might be of interest to you is the tutorial about simply writing servers.

The key thing to keep in mind about Twisted, or even other asynchronous networking libraries (such as asyncore, MINA, or ACE), is that your code only gets invoked when something happens. The part that I've heard most often sound like "voodoo" is the management of callbacks: for example, Deferred. If you're used to writing code that runs in a straight line, and only calls functions which return immediately with results, the idea of waiting for something to call you back might be confusing. But there's nothing magical, no "voodoo" about callbacks. At the lowest level, the reactor is just sitting around and waiting for one of a small number of things to happen:

1. Data arrives on a connection (it will call dataReceived on a Protocol)
2. Time has passed (it will call a function registered with callLater).
3. A connection has been accepted (it will call buildProtocol on a factory registered with a listenXXX or connectXXX function).
4. A connection has been dropped (it will call connectionLost on the appropriate Protocol)

Every asynchronous program starts by hooking up a few of these events and then kicking off the reactor to wait for them to happen. Of course, events that happen lead to more events that get hooked up or disconnected, and so your program goes on its merry way. Beyond that, there's nothing special about asynchronous program structure that are interesting or special; event handlers and callbacks are just objects, and your code is run in the usual way.

Here's a simple "event-driven engine" that shows you just how simple this process is.

# Engine
import time
class SimplestReactor(object):
    def __init__(self):
        self.events = []
        self.stopped = False

    def do(self, something):
        self.events.append(something)

    def run(self):
        while not self.stopped:
            time.sleep(0.1)
            if self.events:
                thisTurn = self.events.pop(0)
                thisTurn()

    def stop(self):
        self.stopped = True

reactor = SimplestReactor()

# Application    
def thing1():
    print 'Doing thing 1'
    reactor.do(thing2)
    reactor.do(thing3)

def thing2():
    print 'Doing thing 2'

def thing3():
    print 'Doing thing 3: and stopping'
    reactor.stop()

reactor.do(thing1)
print 'Running'
reactor.run()
print 'Done!'

At the core of libraries like Twisted, the function in the main loop is not sleep, but an operating system call like select() or poll(), as exposed by a module like the Python select module. I say "like" select, because this is an API that varies a lot between platforms, and almost every GUI toolkit has its own version. Twisted currently provides an abstract interface to 14 different variations on this theme. The common thing that such an API provides is provide a way to say "Here are a list of events that I'm waiting for. Go to sleep until one of them happens, then wake up and tell me which one of them it was."

Answered by: Kellan677 | Posted: 01-03-2022

Similar questions

asynchronous programming in python

Is there a generic notion of asynchronous programming in python? Could I assign a callback to a function, execute it and return to the main program flow immediately, no matter how long the execution of that function would take?

---

asynchronous - Async web programming in python on windows

I would like to do some async web programming in python. I would like to use http://www.tornadoweb.org, but my code has to run on windows. My next try was TwistedWeb, but the Resource-based programming model does not fit my needs very well. I would prefer a Flask-like toolkit. I tried to dig thr...

---

Approach for Python wrapper around C++ Asynchronous Programming Model

I need to create a python wrapper around C++ code that is heavily modeled around APM model. I was researching, but cannot find good approach how to model this in Python. Options that I could think of, are: Use concurrent.futures.Future it is from 3.4, but I would like this wrapper to be used...

---

python - Modifying list in asynchronous programming

Let's say I have a class containing only one member, and it's a list. class List: def __init__(self): self.l = [] def add_to_list(self, element): self.l.append(element) And now, I have a lot of coroutines running in parallel in asyncio.Task's that executes add_to_list on List instance. Is this justififed to protect list by asyncio.Lo...

---

python - Best solution for asynchronous socket programming

Closed. This question is opinion-based. It is not c...

---

asynchronous - Async programming in Python with Dask

is there a way to implement the below example in dask? import time from celery import Celery app = Celery('celery_blog', broker='redis://localhost:6379/0', backend='redis://localhost:6379/0') @app.task def sl(): time.sleep(1) if __name__ == "__main__": sleeper = sl.delay() print('submitted') print(f"sleeper done: {sleeper.ready()}") time.sleep(2...

---

python - How to make selenium faster using asynchronous programming?

My objective with is to scrape as many profile links as possible on Khan Academy. And then scrape some specific data on each of these profiles to write them into a CSV file. Here is the script: from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium....

---

python - Asynchronous programming for calculating hashes of files

I'm trying to calculate hash for files to check if any changes are made. i have Gui and some other observers running in the event loop. So, i decided to calculate hash of files [md5/Sha1 which ever is faster] asynchronously. Synchronous code : import hashlib import time chunk_size = 4 * 1024 def getHash(filename): md5_hash = hashlib.md5() with open(...

---

python - Hooking up GUI interface with asynchronous (s)ftp operation

Trying to implement a progress dialog window for file uploads that looks like a cross between IE download dialog and Firefox download dialog with a python GUI library on Windows. What asynchronous (S)FTP lib...

---

readline - How to implement a python REPL that nicely handles asynchronous output?

I have a Python-based app that can accept a few commands in a simple read-eval-print-loop. I'm using raw_input('> ') to get the input. On Unix-based systems, I also import readline to make things behave a little better. All this is working fine. The problem is that there are asynchronous events coming in, and I'd like to print output as soon as they happen. Unfortunately, this makes thi...

---

python - Are asynchronous Django model queries possible?

I'm new to Django, but the application that I have in mind might end up having URLs that look like this: http://mysite/compare/id_1/id_2 Where "id_1" and "id_2" are identifiers of two distinct Model objects. In the handler for "compare" I'd like to asynchronously, and in parallel, query and retrieve objects id_1 and id_2. Is there any way to do this using a standard Django syntax?...

---

Asynchronous method call in Python?

I was wondering if there's any library for asynchronous method calls in Python. It would be great if you could do something like @async def longComputation(): <code> token = longComputation() token.registerCallback(callback_function) # alternative, polling while not token.finished(): doSomethingElse...

---

Is there any way to make an asynchronous function call from Python [Django]?

I am creating a Django application that does various long computations with uploaded files. I don't want to make the user wait for the file to be handled - I just want to show the user a page reading something like 'file is being parsed'. How can I make an asynchronous function call from a view? Something that may look like that: def view(request): ... if form.is_valid(): for...

---

asynchronous - Tornado and Python 3.x

I really like Tornado and I would like to use it with Python 3, though it is written for Python versions 2.5 and 2.6. Unfortunately it seems like the project's source doesn't come with a test suite. If I understand correctly the WSGI part of it wouldn't be that easy to port as it's spec is not ready for

---

Python: asynchronous tcp socketserver

I'm looking http://docs.python.org/library/socketserver.html to try and handle asynchronous requests with the socketserver in python. At the very bottom there is an example, but it doesn't make sense. It says you use port 0 which assigns an arbitrary unused port. But how do you know what port to use for the client if they are not in th...

---

Python asynchronous callbacks and generators

I'm trying to convert a synchronous library to use an internal asynchronous IO framework. I have several methods that look like this: def foo: .... sync_call_1() # synchronous blocking call .... sync_call_2() # synchronous blocking call .... return bar For each of the synchronous functions (sync_call_*), I have written a corresponding async function that takes ...

---

Does anyone know of a asynchronous mysql lib for python?

I've been looking into non-blocking servers for python (tornado, twisted etc) but a lot of the benefits seems to be lost if there's no non-blocking connection to the database. Does anyone know if there are any projects that take care of this? (by non-blocking a la node.js) Edit: Clarified my question

---

Asynchronous data through Bloomberg's new data API (COM v3) with Python?

Does anyone know how to get asynchronous data through Bloomberg's new data API (COM v3) with Python? I found this code below on wilmott.com and it works just fine, but it's for the old API version. Does anyone know the corresponding code for the new version? from win32com.client import DispatchWithEvents from pythoncom import PumpWaitingMessages, Empty, Missing from time import time class BBCommEve...

---

Still can't find your answer? Check out these communities...

PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python