In chronological order of contribution: +
In chronological order of contribution: Stephen Diehl Jérémy Bethmont sww @@ -101,20 +101,40 @@
Also thanks to Denis Bilenko for writing gevent and guidance in constructing this tutorial.
This is a collaborative document published under MIT license. Have something to add? See a typo? Fork and issue a pull request Github. Any and all contributions are welcome.
-This page is also available in Japanese.
+This page is also available in Japanese and Italian.
The primary pattern used in gevent is the Greenlet, a lightweight coroutine provided to Python as a C extension module. Greenlets all run inside of the OS process for the main -program but are scheduled cooperatively.
+program but are scheduled cooperatively.@@ -127,7 +147,7 @@Only one greenlet is ever running at any given time.
A context switch in gevent is done through yielding. In this case +
A context switch in gevent is done through yielding. In this
example we have two contexts which yield to each other through invoking
gevent.sleep(0).
@@ -341,7 +362,7 @@@@ -177,18 +197,18 @@Synchronous & Asynchronous Execu def gr1(): # Busy waits for a second, but we don't want to stick around... - print('Started Polling: ', tic()) + print('Started Polling: %s' % tic()) select.select([], [], [], 2) - print('Ended Polling: ', tic()) + print('Ended Polling: %s' % tic()) def gr2(): # Busy waits for a second, but we don't want to stick around... - print('Started Polling: ', tic()) + print('Started Polling: %s' % tic()) select.select([], [], [], 2) - print('Ended Polling: ', tic()) + print('Ended Polling: %s' % tic()) def gr3(): - print("Hey lets do some stuff while the greenlets poll, at", tic()) + print("Hey lets do some stuff while the greenlets poll, %s" % tic()) gevent.sleep(1) gevent.joinall([ @@ -200,11 +220,11 @@
Synchronous & Asynchronous Execu
-Started Polling: at 0.0 seconds -Started Polling: at 0.0 seconds -Hey lets do some stuff while the greenlets poll, at at 0.0 seconds -Ended Polling: at 2.0 seconds -Ended Polling: at 2.0 seconds +Started Polling: at 0.0 seconds +Started Polling: at 0.0 seconds +Hey lets do some stuff while the greenlets poll, at 0.0 seconds +Ended Polling: at 2.0 seconds +Ended Polling: at 2.0 secondsAnother somewhat synthetic example defines a
taskfunction which is non-deterministic @@ -221,10 +241,10 @@Synchronous & Asynchronous Execu Some non-deterministic task """ gevent.sleep(random.randint(0,2)*0.001) - print('Task', pid, 'done') + print('Task %s done' % pid) def synchronous(): - for i in range(1,10): + for i in xrange(10): task(i) def asynchronous(): @@ -241,6 +261,7 @@
Synchronous & Asynchronous Execu
Synchronous: +Task 0 done Task 1 done Task 2 done Task 3 done @@ -252,15 +273,15 @@Synchronous & Asynchronous Execu Task 9 done Asynchronous: Task 1 done -Task 2 done -Task 7 done Task 6 done -Task 5 done Task 0 done Task 3 done -Task 8 done -Task 9 done Task 4 done +Task 5 done +Task 7 done +Task 9 done +Task 2 done +Task 8 done
In the synchronous case all the tasks are run sequentially, which results in the main programming blocking ( @@ -268,7 +289,7 @@
Synchronous & Asynchronous Execu while each task executes.
The important parts of the program are the
gevent.spawnwhich wraps up the given function -inside of a Greenlet thread. The list of initialized greenlets +inside of a Greenlet thread. The list of initialized greenlets are stored in the arraythreadswhich is passed to thegevent.joinallfunction which blocks the current program to run all the given greenlets. The execution will step @@ -277,8 +298,8 @@Synchronous & Asynchronous Execu the async case is essentially random and that the total execution time in the async case is much less than the sync case. In fact the maximum time for the synchronous case to complete is when -each tasks pauses for 2 seconds resulting in a 20 seconds for the -whole queue. In the async case the maximum runtime is roughly 2 +each tasks pauses for 0.002 seconds resulting in a 0.02 seconds for the +whole queue. In the async case the maximum runtime is roughly 0.002 seconds since none of the tasks block the execution of the others.
In a more common use case, asynchronously fetching data from a server, @@ -292,12 +313,12 @@
Synchronous & Asynchronous Execu import simplejson as json def fetch(pid): - response = urllib2.urlopen('http://json-time.appspot.com/time.json') + response = urllib2.urlopen('http://jsontime.herokuapp.com/') result = response.read() json_result = json.loads(result) datetime = json_result['datetime'] - print 'Process ', pid, datetime + print('Process %s: %s' % (pid, datetime)) return json_result['datetime'] def synchronous(): @@ -310,10 +331,10 @@
Synchronous & Asynchronous Execu threads.append(gevent.spawn(fetch, i)) gevent.joinall(threads) -print 'Synchronous:' +print('Synchronous:') synchronous() -print 'Asynchronous:' +print('Asynchronous:') asynchronous()
A common pattern is to listen SIGQUIT events on the main program
-and to invoke gevent.shutdown before exit.
gevent.kill or gevent.killall before exit.
import gevent
import signal
@@ -524,8 +545,8 @@ Program Shutdown
gevent.sleep(1000)
if __name__ == '__main__':
- gevent.signal(signal.SIGQUIT, gevent.shutdown)
thread = gevent.spawn(run_forever)
+ gevent.signal(signal.SIGQUIT, gevent.kill, thread)
thread.join()
@@ -549,7 +570,7 @@ In addition, gevent also provides timeout arguments for a -variety of Greenlet and data stucture related calls. For example:
+variety of Greenlet and data structure related calls. For example:## Long Polling @@ -1347,7 +1379,7 @@ WSGIServer(('', 8000), ajax_endpoint).serve_forever() ## Websockets -Websocket example which requires gevent-websocket. +Websocket example which requires gevent-websocket.import gevent from gevent import Timeout @@ -614,24 +635,24 @@-Timeouts
Monkeypatching
Alas we come to dark corners of Gevent. I've avoided mentioning monkey patching up until now to try and motivate the powerful -coroutine patterns but the time has come to discuss the dark arts -of monkey-patching. If you noticed above we invoked the commnad +coroutine patterns, but the time has come to discuss the dark arts +of monkey-patching. If you noticed above we invoked the command
+modify the standard library's socket library.monkey.patch_socket(). This is a purely side-effectful command to -modify the standard library's socket library@@ -648,7 +669,7 @@import socket -print( socket.socket ) +print(socket.socket) -print "After monkey patch" +print("After monkey patch") from gevent import monkey monkey.patch_socket() -print( socket.socket ) +print(socket.socket) import select -print select.select +print(select.select) monkey.patch_select() -print "After monkey patch" -print( select.select ) +print("After monkey patch") +print(select.select)Monkeypatching
Python's runtime allows for most objects to be modified at runtime including modules, classes, and even functions. This is generally an -astoudingly bad idea since it creates an "implicit side-effect" that is +astoundingly bad idea since it creates an "implicit side-effect" that is most often extremely difficult to debug if problems occur, nevertheless in extreme situations where a library needs to alter the fundamental behavior of Python itself monkey patches can be used. In this case gevent @@ -669,36 +690,45 @@
Events
Greenlets.-import gevent -from gevent.event import AsyncResult +from gevent.event import Event -a = AsyncResult() +''' +Illustrates the use of events +''' + +evt = Event() def setter(): - """ - After 3 seconds set wake all threads waiting on the value of - a. - """ + '''After 3 seconds, wake all threads waiting on the value of evt''' + print('A: Hey wait for me, I have to do something') gevent.sleep(3) - a.set() + print("Ok, I'm done") + evt.set() def waiter(): - """ - After 3 seconds the get call will unblock. - """ - a.get() # blocking - print 'I live!' - -gevent.joinall([ - gevent.spawn(setter), - gevent.spawn(waiter), -]) + '''After 3 seconds the get call will unblock''' + print("I'll wait for you") + evt.wait() # blocking + print("It's about time") + +def main(): + gevent.joinall([ + gevent.spawn(setter), + gevent.spawn(waiter), + gevent.spawn(waiter), + gevent.spawn(waiter), + gevent.spawn(waiter), + gevent.spawn(waiter) + ]) + +if __name__ == '__main__': main()A extension of the Event object is the AsyncResult which +
An extension of the Event object is the AsyncResult which allows you to send a value along with the wakeup call. This is -sometimes called a future or a deferred, since it holds a +sometimes called a future or a deferred, since it holds a reference to a future value that can be set on an arbitrary time schedule.
@@ -718,7 +748,7 @@Events
After 3 seconds the get call will unblock after the setter puts a value into the AsyncResult. """ - print a.get() + print(a.get()) gevent.joinall([ gevent.spawn(setter), @@ -733,7 +763,7 @@Queues
operations but are written in a way such that they can be safely manipulated across Greenlets.For example if one Greenlet grabs an item off of the queue, the -same item will not grabbed by another Greenlet executing +same item will not be grabbed by another Greenlet executing simultaneously.
import gevent @@ -768,36 +798,36 @@Queues
Worker john got task 2 Worker nancy got task 3 Worker steve got task 4 -Worker nancy got task 5 -Worker john got task 6 +Worker john got task 5 +Worker nancy got task 6 Worker steve got task 7 Worker john got task 8 Worker nancy got task 9 Worker steve got task 10 -Worker nancy got task 11 -Worker john got task 12 +Worker john got task 11 +Worker nancy got task 12 Worker steve got task 13 Worker john got task 14 Worker nancy got task 15 Worker steve got task 16 -Worker nancy got task 17 -Worker john got task 18 +Worker john got task 17 +Worker nancy got task 18 Worker steve got task 19 Worker john got task 20 Worker nancy got task 21 Worker steve got task 22 -Worker nancy got task 23 -Worker john got task 24 +Worker john got task 23 +Worker nancy got task 24 Quitting time! Quitting time! Quitting time!Queues can also block on either
+putorgetas the need arises.Queues can also block on either
putorgetas the need arises.Each of the
+putandgetoperations has a non-blocking -counterpart,put_nowaitand +counterpart,put_nowaitandget_nowaitwhich will not block, but instead raise eithergevent.queue.Emptyor -gevent.queue.Fullin the operation is not possible.gevent.queue.Fullif the operation is not possible.In this example we have the boss running simultaneously to the workers and have a restriction on the Queue preventing it from containing more than three elements. This restriction means that the
put@@ -805,7 +835,7 @@Queues
Conversely thegetoperation will block if there are no elements on the queue to fetch, it also takes a timeout argument to allow for the queue to exit with the exception -gevent.queue.Emptyif no work can found within the +gevent.queue.Emptyif no work can be found within the time frame of the Timeout.But regardless, performance on Gevent servers is phenomenal compared to other Python servers. libev is a very vetted technology and its derivative servers are known to perform well at scale. -To benchmark, try Apache Benchmark ``ab`` or see this -[Benchmark of Python WSGI Servers](http://nichol.as/benchmark-of-python-web-servers) +To benchmark, try Apache Benchmark ``ab`` or see this +[Benchmark of Python WSGI Servers](http://nichol.as/benchmark-of-python-web-servers) for comparison with other servers.import gevent @@ -813,26 +843,28 @@Queues
tasks = Queue(maxsize=3) -def worker(n): +def worker(name): try: while True: task = tasks.get(timeout=1) # decrements queue size by 1 - print('Worker %s got task %s' % (n, task)) + print('Worker %s got task %s' % (name, task)) gevent.sleep(0) except Empty: print('Quitting time!') def boss(): """ - Boss will wait to hand out work until a individual worker is + Boss will wait to hand out work until an individual worker is free since the maxsize of the task queue is 3. """ for i in xrange(1,10): + print('Assigned work %s in iteration 1' % (i)) tasks.put(i) print('Assigned all work in iteration 1') for i in xrange(10,20): + print('Assigned work %s in iteration 2' % (i)) tasks.put(i) print('Assigned all work in iteration 2') @@ -846,25 +878,44 @@Queues
++Assigned work 1 in iteration 1 +Assigned work 2 in iteration 1 +Assigned work 3 in iteration 1 +Assigned work 4 in iteration 1 Worker steve got task 1 Worker john got task 2 Worker bob got task 3 +Assigned work 5 in iteration 1 +Assigned work 6 in iteration 1 +Assigned work 7 in iteration 1 Worker steve got task 4 -Worker bob got task 5 -Worker john got task 6 +Worker john got task 5 +Worker bob got task 6 +Assigned work 8 in iteration 1 +Assigned work 9 in iteration 1 Assigned all work in iteration 1 +Assigned work 10 in iteration 2 Worker steve got task 7 Worker john got task 8 Worker bob got task 9 +Assigned work 11 in iteration 2 +Assigned work 12 in iteration 2 +Assigned work 13 in iteration 2 Worker steve got task 10 -Worker bob got task 11 -Worker john got task 12 +Worker john got task 11 +Worker bob got task 12 +Assigned work 14 in iteration 2 +Assigned work 15 in iteration 2 +Assigned work 16 in iteration 2 Worker steve got task 13 Worker john got task 14 Worker bob got task 15 +Assigned work 17 in iteration 2 +Assigned work 18 in iteration 2 +Assigned work 19 in iteration 2 Worker steve got task 16 -Worker bob got task 17 -Worker john got task 18 +Worker john got task 17 +Worker bob got task 18 Assigned all work in iteration 2 Worker steve got task 19 Quitting time! @@ -920,7 +971,7 @@Groups and Pools
group = Group() def hello_from(n): - print('Size of group', len(group)) + print('Size of group %s' % len(group)) print('Hello from Greenlet %s' % id(getcurrent())) group.map(hello_from, xrange(3)) @@ -946,11 +997,11 @@Groups and Pools
@@ -1014,14 +1065,14 @@Size of group 3 -Hello from Greenlet 10769424 +Hello from Greenlet 4405439216 Size of group 3 -Hello from Greenlet 10770544 +Hello from Greenlet 4405439056 Size of group 3 -Hello from Greenlet 10772304 +Hello from Greenlet 4405440336 Ordered ('task', 0) ('task', 1) @@ -971,7 +1022,7 @@Groups and Pools
pool = Pool(2) def hello_from(n): - print('Size of pool', len(pool)) + print('Size of pool %s' % len(pool)) pool.map(hello_from, xrange(3))Locks and Semaphores
A semaphore is a low level synchronization primitive that allows greenlets to coordinate and limit concurrent access or execution. A semaphore exposes two methods,
acquireandreleaseThe -difference between the number of times and a semaphore has been +difference between the number of times a semaphore has been acquired and released is called the bound of the semaphore. If a semaphore bound reaches 0 it will block until another greenlet releases its acquisition.-from gevent import sleep from gevent.pool import Pool -from gevent.coros import BoundedSemaphore +from gevent.lock import BoundedSemaphore sem = BoundedSemaphore(2) @@ -1096,10 +1147,10 @@Thread Locals
2 x is not local to f2Many web framework thats integrate with gevent store HTTP session -objects inside of gevent thread locals. For example using the +
Many web frameworks that use gevent store HTTP session +objects inside gevent thread locals. For example, using the Werkzeug utility library and its proxy object we can create -Flask style request objects.
+Flask-style request objects.from gevent.local import local from werkzeug.local import LocalProxy @@ -1152,14 +1203,14 @@Subprocess
def cron(): while True: - print "cron" + print("cron") gevent.sleep(0.2) g = gevent.spawn(cron) sub = Popen(['sleep 1; uname'], stdout=PIPE, shell=True) out, err = sub.communicate() g.kill() -print out.rstrip() +print(out.rstrip())@@ -1243,7 +1294,7 @@In addition, gevent also provides timeout arguments for a -variety of Greenlet and data stucture related calls. For example: +variety of Greenlet and data structure related calls. For example: [[[cog import gevent @@ -521,18 +541,18 @@ modify the standard library's socket library.Actors
collection of independent Actors which have an inbox from which they receive messages from other Actors. The main loop inside the Actor iterates through its messages and takes action according to -its desired behavior. +its desired behavior.Gevent does not have a primitive Actor type, but we can define one very simply using a Queue inside of a subclassed Greenlet.
@@ -1280,13 +1331,13 @@@@ -473,7 +493,7 @@ with Timeout(time_to_wait, TooLong):Actors
class Pinger(Actor): def receive(self, message): - print message + print(message) pong.inbox.put('ping') gevent.sleep(0) class Ponger(Actor): def receive(self, message): - print message + print(message) ping.inbox.put('pong') gevent.sleep(0) @@ -1306,19 +1357,18 @@Gevent ZeroMQ
ZeroMQ is described by its authors as "a socket library that acts as a concurrency framework". It is a very powerful messaging layer for building concurrent and -distributed applications.
+distributed applications.ZeroMQ provides a variety of socket primitives, the simplest of which being a Request-Response socket pair. A socket has two methods of interest
+normally blocking operations. But this is remedied by a brilliant +library (is now part of PyZMQ) +by Travis Cline which uses gevent.socket +to poll ZeroMQ sockets in a non-blocking manner.sendandrecv, both of which are -normally blocking operations. But this is remedied by a briliant -library by Travis Cline which -uses gevent.socket to poll ZeroMQ sockets in a non-blocking -manner. You can install gevent-zeromq from PyPi via:pip install -gevent-zeromq@@ -308,7 +328,7 @@ def foo(message, n): # foo thread1 = Greenlet.spawn(foo, "Hello", 1) -# Wrapper for creating and running a new Greenlet from the named +# Wrapper for creating and running a new Greenlet from the named # function foo, with the passed arguments thread2 = gevent.spawn(foo, "I live!", 2) @@ -393,7 +413,7 @@ print(loser.ready()) # True print(winner.successful()) # True print(loser.successful()) # False -# The exception raised in fail, will not propogate outside the +# The exception raised in fail, will not propagate outside the # greenlet. A stack trace will be printed to stdout but it # will not unwind the stack of the parent. @@ -414,7 +434,7 @@ This results in so called "zombie processes" which need to be killed from outside of the Python interpreter. A common pattern is to listen SIGQUIT events on the main program -and to invoke ``gevent.shutdown`` before exit. +and to invoke ``gevent.kill`` or ``gevent.killall`` before exit.-# Note: Remember to ``pip install pyzmq gevent_zeromq`` +# Note: Remember to ``pip install pyzmq`` import gevent -from gevent_zeromq import zmq +import zmq.green as zmq # Global Context context = zmq.Context() @@ -1329,7 +1379,7 @@Gevent ZeroMQ
for request in range(1,10): server_socket.send("Hello") - print('Switched to Server for ', request) + print('Switched to Server for %s' % request) # Implicit context switch occurs here server_socket.recv() @@ -1340,7 +1390,7 @@Gevent ZeroMQ
for request in range(1,10): client_socket.recv() - print('Switched to Client for ', request) + print('Switched to Client for %s' % request) # Implicit context switch occurs here client_socket.send("World") @@ -1353,29 +1403,29 @@Gevent ZeroMQ
-Switched to Server for 1 -Switched to Client for 1 -Switched to Server for 2 -Switched to Client for 2 -Switched to Server for 3 -Switched to Client for 3 -Switched to Server for 4 -Switched to Client for 4 -Switched to Server for 5 -Switched to Client for 5 -Switched to Server for 6 -Switched to Client for 6 -Switched to Server for 7 -Switched to Client for 7 -Switched to Server for 8 -Switched to Client for 8 -Switched to Server for 9 -Switched to Client for 9 +Switched to Server for 1 +Switched to Client for 1 +Switched to Server for 2 +Switched to Client for 2 +Switched to Server for 3 +Switched to Client for 3 +Switched to Server for 4 +Switched to Client for 4 +Switched to Server for 5 +Switched to Client for 5 +Switched to Server for 6 +Switched to Client for 6 +Switched to Server for 7 +Switched to Client for 7 +Switched to Server for 8 +Switched to Client for 8 +Switched to Server for 9 +Switched to Client for 9Simple Servers
@@ -247,7 +267,7 @@ run2 = [a for a in p.imap_unordered(echo, xrange(10))] run3 = [a for a in p.imap_unordered(echo, xrange(10))] run4 = [a for a in p.imap_unordered(echo, xrange(10))] -print( run1 == run2 == run3 == run4 ) +print(run1 == run2 == run3 == run4) # Deterministic Gevent Pool @@ -259,7 +279,7 @@ run2 = [a for a in p.imap_unordered(echo, xrange(10))] run3 = [a for a in p.imap_unordered(echo, xrange(10))] run4 = [a for a in p.imap_unordered(echo, xrange(10))] -print( run1 == run2 == run3 == run4 ) +print(run1 == run2 == run3 == run4)-# On Unix: Access with ``$ nc 127.0.0.1 5000`` +# On Unix: Access with ``$ nc 127.0.0.1 5000`` # On Window: Access with ``$ telnet 127.0.0.1 5000`` from gevent.server import StreamServer @@ -1400,7 +1450,7 @@WSGI Servers
In earlier versions of gevent before 1.0.x, gevent used libevent instead of libev. Libevent included a fast HTTP server which was -used by gevent's
+used by gevent'swsgiserver.wsgiserver.In gevent 1.0.x there is no http server included. Instead
@@ -1469,8 +1519,8 @@gevent.wsgiis now an alias for the pure Python server ingevent.pywsgi.Streaming Servers
But regardless, performance on Gevent servers is phenomenal compared to other Python servers. libev is a very vetted technology and its derivative servers are known to perform well at scale.
-To benchmark, try Apache Benchmark
abor see this -Benchmark of Python WSGI Servers +To benchmark, try Apache Benchmark
abor see this +Benchmark of Python WSGI Servers for comparison with other servers.$ ab -n 10000 -c 100 http://127.0.0.1:8000/ @@ -1514,7 +1564,7 @@Long Polling
Websockets
-Websocket example which requires gevent-websocket.
+Websocket example which requires gevent-websocket.
# Simple gevent-websocket server import json @@ -1577,7 +1627,7 @@Websockets
Chat Server
The final motivating example, a realtime chat room. This example -requires Flask ( but not neccesarily so, you could use Django, +requires Flask ( but not necessarily so, you could use Django, Pyramid, etc ). The corresponding Javascript and HTML files can be found here.
@@ -1615,7 +1665,7 @@Chat Server
def add(self, message): for user in self.users: - print user + print(user) user.queue.put_nowait(message) self.messages.append(message) @@ -1644,7 +1694,7 @@Chat Server
active_room = rooms[room] active_room.subscribe(user) - print 'subscribe', active_room, user + print('subscribe %s %s' % (active_room, user)) messages = active_room.backlog() diff --git a/requirements.txt b/requirements.txt index 9a10e26..b17eeac 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,4 +2,4 @@ Markdown==2.1.1 cogapp==2.3 Jinja2==2.6 gevent -gevent-zeromq +pyzmq diff --git a/tutorial.md b/tutorial.md index 5940fe5..bcf78f9 100644 --- a/tutorial.md +++ b/tutorial.md @@ -11,7 +11,7 @@ applications today. ### Contributors -In chronological order of contribution: +In chronological order of contribution: [Stephen Diehl](http://www.stephendiehl.com) [Jérémy Bethmont](https://github.com/jerem) [sww](https://github.com/sww) @@ -23,6 +23,26 @@ In chronological order of contribution: [Eddie Hebert](https://github.com/ehebert) [Alexis Metaireau](http://notmyidea.org) [Daniel Velkov](https://github.com/djv) +[Sean Wang](https://github.com/sww) +[Inada Naoki](https://github.com/methane) +[Balthazar Rouberol](https://github.com/brouberol) +[Glen Baker](https://github.com/iepathos) +[Jan-Philip Gehrcke](https://gehrcke.de) +[Matthijs van der Vleuten](https://github.com/zr40) +[Simon Hayward](http://simonsblog.co.uk) +[Alexander James Phillips](https://github.com/AJamesPhillips) +[Ramiro Morales](https://github.com/ramiro) +[Philip Damra](https://github.com/djheru) +[Francisco José Marques Vieira](https://github.com/fvieira) +[David Xia](https://www.davidxia.com) +[satoru](https://github.com/satoru) +[James Summerfield](https://github.com/jsummerfield) +[Adam Szkoda](https://github.com/adaszko) +[Roy Smith](https://github.com/roysmith) +[Jianbin Wei](https://github.com/jianbin-netskope) +[Anton Larkin](https://github.com/ToxicWar) +[Matias Herranz](https://github.com/matiasherranz-santex) +[Pietro Bertera](http://www.bertera.it) Also thanks to Denis Bilenko for writing gevent and guidance in constructing this tutorial. @@ -32,7 +52,7 @@ Have something to add? See a typo? Fork and issue a pull request [Github](https://github.com/sdiehl/gevent-tutorial). Any and all contributions are welcome. -This page is also [available in Japanese](http://methane.github.com/gevent-tutorial-ja). +This page is also [available in Japanese](http://methane.github.com/gevent-tutorial-ja) and [Italian](http://pbertera.github.io/gevent-tutorial-it/). # Core @@ -41,7 +61,7 @@ This page is also [available in Japanese](http://methane.github.com/gevent-tutor The primary pattern used in gevent is the Greenlet, a lightweight coroutine provided to Python as a C extension module. Greenlets all run inside of the OS process for the main -program but are scheduled cooperatively. +program but are scheduled cooperatively. > Only one greenlet is ever running at any given time. @@ -57,7 +77,7 @@ into a collection of subtasks which are scheduled to run simultaneously or *asynchronously*, instead of one at a time or *synchronously*. A switch between the two subtasks is known as a *context switch*. -A context switch in gevent is done through *yielding*. In this case +A context switch in gevent is done through *yielding*. In this example we have two contexts which yield to each other through invoking ``gevent.sleep(0)``. @@ -106,18 +126,18 @@ tic = lambda: 'at %1.1f seconds' % (time.time() - start) def gr1(): # Busy waits for a second, but we don't want to stick around... - print('Started Polling: ', tic()) + print('Started Polling: %s' % tic()) select.select([], [], [], 2) - print('Ended Polling: ', tic()) + print('Ended Polling: %s' % tic()) def gr2(): # Busy waits for a second, but we don't want to stick around... - print('Started Polling: ', tic()) + print('Started Polling: %s' % tic()) select.select([], [], [], 2) - print('Ended Polling: ', tic()) + print('Ended Polling: %s' % tic()) def gr3(): - print("Hey lets do some stuff while the greenlets poll, at", tic()) + print("Hey lets do some stuff while the greenlets poll, %s" % tic()) gevent.sleep(1) gevent.joinall([ @@ -144,10 +164,10 @@ def task(pid): Some non-deterministic task """ gevent.sleep(random.randint(0,2)*0.001) - print('Task', pid, 'done') + print('Task %s done' % pid) def synchronous(): - for i in range(1,10): + for i in xrange(10): task(i) def asynchronous(): @@ -169,7 +189,7 @@ while each task executes. The important parts of the program are the ``gevent.spawn`` which wraps up the given function -inside of a Greenlet thread. The list of initialized greenlets +inside of a Greenlet thread. The list of initialized greenlets are stored in the array ``threads`` which is passed to the ``gevent.joinall`` function which blocks the current program to run all the given greenlets. The execution will step @@ -179,8 +199,8 @@ The important fact to notice is that the order of execution in the async case is essentially random and that the total execution time in the async case is much less than the sync case. In fact the maximum time for the synchronous case to complete is when -each tasks pauses for 2 seconds resulting in a 20 seconds for the -whole queue. In the async case the maximum runtime is roughly 2 +each tasks pauses for 0.002 seconds resulting in a 0.02 seconds for the +whole queue. In the async case the maximum runtime is roughly 0.002 seconds since none of the tasks block the execution of the others. @@ -196,12 +216,12 @@ import urllib2 import simplejson as json def fetch(pid): - response = urllib2.urlopen('http://json-time.appspot.com/time.json') + response = urllib2.urlopen('http://jsontime.herokuapp.com/') result = response.read() json_result = json.loads(result) datetime = json_result['datetime'] - print 'Process ', pid, datetime + print('Process %s: %s' % (pid, datetime)) return json_result['datetime'] def synchronous(): @@ -214,10 +234,10 @@ def asynchronous(): threads.append(gevent.spawn(fetch, i)) gevent.joinall(threads) -print 'Synchronous:' +print('Synchronous:') synchronous() -print 'Asynchronous:' +print('Asynchronous:') asynchronous()@@ -451,7 +471,7 @@ def wait(): try: gevent.spawn(wait).join() except Timeout: - print 'Could not complete' + print('Could not complete')import gevent @@ -424,8 +444,8 @@ def run_forever(): gevent.sleep(1000) if __name__ == '__main__': - gevent.signal(signal.SIGQUIT, gevent.shutdown) thread = gevent.spawn(run_forever) + gevent.signal(signal.SIGQUIT, gevent.kill, thread) thread.join()@@ -549,7 +569,7 @@ function select at 0x1924de8 Python's runtime allows for most objects to be modified at runtime including modules, classes, and even functions. This is generally an -astoudingly bad idea since it creates an "implicit side-effect" that is +astoundingly bad idea since it creates an "implicit side-effect" that is most often extremely difficult to debug if problems occur, nevertheless in extreme situations where a library needs to alter the fundamental behavior of Python itself monkey patches can be used. In this case gevent @@ -576,36 +596,47 @@ Greenlets.import socket -print( socket.socket ) +print(socket.socket) -print "After monkey patch" +print("After monkey patch") from gevent import monkey monkey.patch_socket() -print( socket.socket ) +print(socket.socket) import select -print select.select +print(select.select) monkey.patch_select() -print "After monkey patch" -print( select.select ) +print("After monkey patch") +print(select.select)-A extension of the Event object is the AsyncResult which +An extension of the Event object is the AsyncResult which allows you to send a value along with the wakeup call. This is -sometimes called a future or a deferred, since it holds a +sometimes called a future or a deferred, since it holds a reference to a future value that can be set on an arbitrary time schedule. @@ -626,7 +657,7 @@ def waiter(): After 3 seconds the get call will unblock after the setter puts a value into the AsyncResult. """ - print a.get() + print(a.get()) gevent.joinall([ gevent.spawn(setter), @@ -643,7 +674,7 @@ operations but are written in a way such that they can be safely manipulated across Greenlets. For example if one Greenlet grabs an item off of the queue, the -same item will not grabbed by another Greenlet executing +same item will not be grabbed by another Greenlet executing simultaneously. [[[cog @@ -674,13 +705,13 @@ gevent.joinall([ ]]] [[[end]]] -Queues can also block on either ``put`` or ``get`` as the need arises. +Queues can also block on either ``put`` or ``get`` as the need arises. Each of the ``put`` and ``get`` operations has a non-blocking -counterpart, ``put_nowait`` and +counterpart, ``put_nowait`` and ``get_nowait`` which will not block, but instead raise either ``gevent.queue.Empty`` or -``gevent.queue.Full`` in the operation is not possible. +``gevent.queue.Full`` if the operation is not possible. In this example we have the boss running simultaneously to the workers and have a restriction on the Queue preventing it from containing @@ -689,7 +720,7 @@ operation will block until there is space on the queue. Conversely the ``get`` operation will block if there are no elements on the queue to fetch, it also takes a timeout argument to allow for the queue to exit with the exception -``gevent.queue.Empty`` if no work can found within the +``gevent.queue.Empty`` if no work can be found within the time frame of the Timeout. [[[cog @@ -698,26 +729,28 @@ from gevent.queue import Queue, Empty tasks = Queue(maxsize=3) -def worker(n): +def worker(name): try: while True: task = tasks.get(timeout=1) # decrements queue size by 1 - print('Worker %s got task %s' % (n, task)) + print('Worker %s got task %s' % (name, task)) gevent.sleep(0) except Empty: print('Quitting time!') def boss(): """ - Boss will wait to hand out work until a individual worker is + Boss will wait to hand out work until an individual worker is free since the maxsize of the task queue is 3. """ for i in xrange(1,10): + print('Assigned work %s in iteration 1' % (i)) tasks.put(i) print('Assigned all work in iteration 1') for i in xrange(10,20): + print('Assigned work %s in iteration 2' % (i)) tasks.put(i) print('Assigned all work in iteration 2') @@ -772,7 +805,7 @@ from gevent.pool import Group group = Group() def hello_from(n): - print('Size of group', len(group)) + print('Size of group %s' % len(group)) print('Hello from Greenlet %s' % id(getcurrent())) group.map(hello_from, xrange(3)) @@ -809,7 +842,7 @@ from gevent.pool import Pool pool = Pool(2) def hello_from(n): - print('Size of pool', len(pool)) + print('Size of pool %s' % len(pool)) pool.map(hello_from, xrange(3)) ]]] @@ -849,7 +882,7 @@ class SocketPool(object): A semaphore is a low level synchronization primitive that allows greenlets to coordinate and limit concurrent access or execution. A semaphore exposes two methods, ``acquire`` and ``release`` The -difference between the number of times and a semaphore has been +difference between the number of times a semaphore has been acquired and released is called the bound of the semaphore. If a semaphore bound reaches 0 it will block until another greenlet releases its acquisition. @@ -857,7 +890,7 @@ releases its acquisition. [[[cog from gevent import sleep from gevent.pool import Pool -from gevent.coros import BoundedSemaphore +from gevent.lock import BoundedSemaphore sem = BoundedSemaphore(2) @@ -918,10 +951,10 @@ gevent.joinall([g1, g2]) ]]] [[[end]]] -Many web framework thats integrate with gevent store HTTP session -objects inside of gevent thread locals. For example using the +Many web frameworks that use gevent store HTTP session +objects inside gevent thread locals. For example, using the Werkzeug utility library and its proxy object we can create -Flask style request objects. +Flask-style request objects.import gevent -from gevent.event import AsyncResult +from gevent.event import Event -a = AsyncResult() +''' +Illustrates the use of events +''' + + +evt = Event() def setter(): - """ - After 3 seconds set wake all threads waiting on the value of - a. - """ - gevent.sleep(3) - a.set() + '''After 3 seconds, wake all threads waiting on the value of evt''' + print('A: Hey wait for me, I have to do something') + gevent.sleep(3) + print("Ok, I'm done") + evt.set() -def waiter(): - """ - After 3 seconds the get call will unblock. - """ - a.get() # blocking - print 'I live!' -gevent.joinall([ - gevent.spawn(setter), - gevent.spawn(waiter), -]) +def waiter(): + '''After 3 seconds the get call will unblock''' + print("I'll wait for you") + evt.wait() # blocking + print("It's about time") + +def main(): + gevent.joinall([ + gevent.spawn(setter), + gevent.spawn(waiter), + gevent.spawn(waiter), + gevent.spawn(waiter), + gevent.spawn(waiter), + gevent.spawn(waiter) + ]) + +if __name__ == '__main__': main()from gevent.local import local @@ -979,14 +1012,14 @@ from gevent.subprocess import Popen, PIPE def cron(): while True: - print "cron" + print("cron") gevent.sleep(0.2) g = gevent.spawn(cron) sub = Popen(['sleep 1; uname'], stdout=PIPE, shell=True) out, err = sub.communicate() g.kill() -print out.rstrip() +print(out.rstrip())@@ -1073,7 +1106,7 @@ by the language Erlang. In short the main idea is that you have a collection of independent Actors which have an inbox from which they receive messages from other Actors. The main loop inside the Actor iterates through its messages and takes action according to -its desired behavior. +its desired behavior. Gevent does not have a primitive Actor type, but we can define one very simply using a Queue inside of a subclassed Greenlet. @@ -1114,13 +1147,13 @@ from gevent import Greenlet class Pinger(Actor): def receive(self, message): - print message + print(message) pong.inbox.put('ping') gevent.sleep(0) class Ponger(Actor): def receive(self, message): - print message + print(message) ping.inbox.put('pong') gevent.sleep(0) @@ -1142,21 +1175,20 @@ gevent.joinall([ping, pong]) [ZeroMQ](http://www.zeromq.org/) is described by its authors as "a socket library that acts as a concurrency framework". It is a very powerful messaging layer for building concurrent and -distributed applications. +distributed applications. ZeroMQ provides a variety of socket primitives, the simplest of which being a Request-Response socket pair. A socket has two methods of interest ``send`` and ``recv``, both of which are -normally blocking operations. But this is remedied by a briliant -library by [Travis Cline](https://github.com/traviscline) which -uses gevent.socket to poll ZeroMQ sockets in a non-blocking -manner. You can install gevent-zeromq from PyPi via: ``pip install -gevent-zeromq`` +normally blocking operations. But this is remedied by a brilliant +[library](https://github.com/tmc/gevent-zeromq) (is now part of PyZMQ) +by [Travis Cline](https://github.com/tmc) which uses gevent.socket +to poll ZeroMQ sockets in a non-blocking manner. [[[cog -# Note: Remember to ``pip install pyzmq gevent_zeromq`` +# Note: Remember to ``pip install pyzmq`` import gevent -from gevent_zeromq import zmq +import zmq.green as zmq # Global Context context = zmq.Context() @@ -1167,7 +1199,7 @@ def server(): for request in range(1,10): server_socket.send("Hello") - print('Switched to Server for ', request) + print('Switched to Server for %s' % request) # Implicit context switch occurs here server_socket.recv() @@ -1178,7 +1210,7 @@ def client(): for request in range(1,10): client_socket.recv() - print('Switched to Client for ', request) + print('Switched to Client for %s' % request) # Implicit context switch occurs here client_socket.send("World") @@ -1194,8 +1226,8 @@ gevent.joinall([publisher, client])Using pywsgi we can however write our handler as a generator and yield the result chunk by chunk. @@ -1292,20 +1324,20 @@ def application(environ, start_response): WSGIServer(('', 8000), application).serve_forever() -+-# On Unix: Access with ``$ nc 127.0.0.1 5000`` -# On Window: Access with ``$ telnet 127.0.0.1 5000`` +# On Unix: Access with ``$ nc 127.0.0.1 5000`` +# On Window: Access with ``$ telnet 127.0.0.1 5000`` from gevent.server import StreamServer @@ -1220,7 +1252,7 @@ Henceforth called ``wsgi`` and ``pywsgi``: In earlier versions of gevent before 1.0.x, gevent used libevent instead of libev. Libevent included a fast HTTP server which was -used by gevent's ``wsgi`` server. +used by gevent's ``wsgi`` server. In gevent 1.0.x there is no http server included. Instead ``gevent.wsgi`` is now an alias for the pure Python server in @@ -1270,7 +1302,7 @@ def application(environ, start_response): WSGIServer(('', 8000), application).serve_forever()-+$ ab -n 10000 -c 100 http://127.0.0.1:8000/-
@@ -1415,7 +1447,7 @@ HTML Page: ## Chat Server The final motivating example, a realtime chat room. This example -requires Flask ( but not neccesarily so, you could use Django, +requires Flask ( but not necessarily so, you could use Django, Pyramid, etc ). The corresponding Javascript and HTML files can be found here. @@ -1455,7 +1487,7 @@ class Room(object): def add(self, message): for user in self.users: - print user + print(user) user.queue.put_nowait(message) self.messages.append(message) @@ -1484,7 +1516,7 @@ def join(room, uid): active_room = rooms[room] active_room.subscribe(user) - print 'subscribe', active_room, user + print('subscribe %s %s' % (active_room, user)) messages = active_room.backlog()