1+ {
2+ "metadata" : {
3+ "name" : " "
4+ },
5+ "nbformat" : 3 ,
6+ "nbformat_minor" : 0 ,
7+ "worksheets" : [
8+ {
9+ "cells" : [
10+ {
11+ "cell_type" : " heading"
12+ "level" : 1 ,
13+ "metadata" : {},
14+ "source" : [
15+ " 7.11. Inlining Callback Functions"
16+ ]
17+ },
18+ {
19+ "cell_type" : " markdown"
20+ "metadata" : {},
21+ "source" : [
22+ " # Problem\n "
23+ " You\u2019 re writing code that uses callback functions, but you\u2019 re concerned about the pro\u2010 \n "
24+ " liferation of small functions and mind boggling control flow. You would like some way\n "
25+ " to make the code look more like a normal sequence of procedural steps.\n "
26+ " \n "
27+ " # Solution\n "
28+ " Callback functions can be inlined into a function using generators and coroutines. To\n "
29+ " illustrate, suppose you have a function that performs work and invokes a callback as\n "
30+ " follows (see Recipe 7.10):"
31+ ]
32+ },
33+ {
34+ "cell_type" : " code"
35+ "collapsed" : false ,
36+ "input" : [
37+ " def apply_async(func, args, *, callback):\n "
38+ " # Compute the result\n "
39+ " result = func(*args)\n "
40+ " # Invoke the callback with the result\n "
41+ " callback(result)"
42+ ],
43+ "language" : " python"
44+ "metadata" : {},
45+ "outputs" : [],
46+ "prompt_number" : 1
47+ },
48+ {
49+ "cell_type" : " markdown"
50+ "metadata" : {},
51+ "source" : [
52+ " Now take a look at the following supporting code, which involves an `Async` class and\n "
53+ " an `inlined_async` decorator:"
54+ ]
55+ },
56+ {
57+ "cell_type" : " code"
58+ "collapsed" : false ,
59+ "input" : [
60+ " from queue import Queue\n "
61+ " from functools import wraps\n "
62+ " \n "
63+ " class Async:\n "
64+ " def __init__(self, func, args):\n "
65+ " self.func = func\n "
66+ " self.args = args\n "
67+ " \n "
68+ " def inlined_async(func):\n "
69+ " @wraps(func)\n "
70+ " def wrapper(*args):\n "
71+ " f = func(*args)\n "
72+ " result_queue = Queue()\n "
73+ " result_queue.put(None)\n "
74+ " while True:\n "
75+ " result = result_queue.get()\n "
76+ " try:\n "
77+ " a = f.send(result)\n "
78+ " apply_async(a.func, a.args, callback=result_queue.put)\n "
79+ " except StopIteration:\n "
80+ " break\n "
81+ " return wrapper"
82+ ],
83+ "language" : " python"
84+ "metadata" : {},
85+ "outputs" : [],
86+ "prompt_number" : 2
87+ },
88+ {
89+ "cell_type" : " markdown"
90+ "metadata" : {},
91+ "source" : [
92+ " These two fragments of code will allow you to inline the callback steps using `yield`\n "
93+ " statements. For example:"
94+ ]
95+ },
96+ {
97+ "cell_type" : " code"
98+ "collapsed" : false ,
99+ "input" : [
100+ " def add(x, y):\n "
101+ " return x + y\n "
102+ " \n "
103+ " @Async.inlined_async\n "
104+ " def test():\n "
105+ " r = yield Async(add, (2, 3))\n "
106+ " print(r)\n "
107+ " r = yield Async(add, ('hello', 'world'))\n "
108+ " print(r)\n "
109+ " for n in range(10):\n "
110+ " r = yield Async(add, (n, n))\n "
111+ " print(r)\n "
112+ " print('Goodbye')"
113+ ],
114+ "language" : " python"
115+ "metadata" : {},
116+ "outputs" : [],
117+ "prompt_number" : 3
118+ },
119+ {
120+ "cell_type" : " markdown"
121+ "metadata" : {},
122+ "source" : [
123+ " If you call `test()`, you\u2019 ll get output like this:\n "
124+ " ```\n "
125+ " 5\n "
126+ " helloworld\n "
127+ " 0\n "
128+ " 2\n "
129+ " 4\n "
130+ " 6\n "
131+ " 8\n "
132+ " 10\n "
133+ " 12\n "
134+ " 14\n "
135+ " 16\n "
136+ " 18\n "
137+ " Goodbye\n "
138+ " ```"
139+ ]
140+ },
141+ {
142+ "cell_type" : " code"
143+ "collapsed" : false ,
144+ "input" : [
145+ " test()"
146+ ],
147+ "language" : " python"
148+ "metadata" : {},
149+ "outputs" : [
150+ {
151+ "output_type" : " stream"
152+ "stream" : " stdout"
153+ "text" : [
154+ " 5\n "
155+ " helloworld\n "
156+ " 0\n "
157+ " 2\n "
158+ " 4\n "
159+ " 6\n "
160+ " 8\n "
161+ " 10\n "
162+ " 12\n "
163+ " 14\n "
164+ " 16\n "
165+ " 18\n "
166+ " Goodbye\n "
167+ ]
168+ }
169+ ],
170+ "prompt_number" : 5
171+ },
172+ {
173+ "cell_type" : " markdown"
174+ "metadata" : {},
175+ "source" : [
176+ " Aside from the special decorator and use of `yield`, you will notice that no callback\n "
177+ " functions appear anywhere (except behind the scenes)."
178+ ]
179+ },
180+ {
181+ "cell_type" : " markdown"
182+ "metadata" : {},
183+ "source" : [
184+ " # Discussion\n "
185+ " This recipe will really test your knowledge of callback functions, generators, and control\n "
186+ " flow.\n "
187+ " \n "
188+ " First, in code involving callbacks, the whole point is that the current calculation will\n "
189+ " suspend and resume at some later point in time (e.g., asynchronously). When the \n "
190+ " calculation resumes, the callback will get executed to continue the processing. The \n "
191+ " `apply_async()` function illustrates the essential parts of executing the callback, although\n "
192+ " in reality it might be much more complicated (involving threads, processes, event \n "
193+ " handlers, etc.).\n "
194+ " \n "
195+ " The idea that a calculation will suspend and resume naturally maps to the execution\n "
196+ " model of a generator function. Specifically, the `yield` operation makes a generator\n "
197+ " function emit a value and suspend. Subsequent calls to the `__next__()` or `send()`\n "
198+ " method of a generator will make it start again.\n "
199+ " \n "
200+ " With this in mind, the core of this recipe is found in the `inline_async()` decorator\n "
201+ " function. The key idea is that the decorator will step the generator function through all\n "
202+ " of its `yield` statements, one at a time. To do this, a result queue is created and initially\n "
203+ " populated with a value of `None`. A loop is then initiated in which a result is popped off\n "
204+ " the queue and sent into the generator. This advances to the next yield, at which point\n "
205+ " an instance of `Async` is received. The loop then looks at the function and arguments,\n "
206+ " and initiates the asynchronous calculation `apply_async()`. However, the sneakiest part\n "
207+ " of this calculation is that instead of using a normal callback function, the callback is set\n "
208+ " to the queue `put()` method.\n "
209+ " \n "
210+ " At this point, it is left somewhat open as to precisely what happens. The main loop\n "
211+ " immediately goes back to the top and simply executes a `get()` operation on the queue.\n "
212+ " If data is present, it must be the result placed there by the `put()` callback. If nothing is\n "
213+ " there, the operation blocks, waiting for a result to arrive at some future time. How that\n "
214+ " might happen depends on the precise implementation of the `apply_async()` function.\n "
215+ " \n "
216+ " If you\u2019 re doubtful that anything this crazy would work, you can try it with the \n "
217+ " multiprocessing library and have async operations executed in separate processes:\n "
218+ " ```python\n "
219+ " if __name__ == '__main__':\n "
220+ " import multiprocessing\n "
221+ " pool = multiprocessing.Pool()\n "
222+ " apply_async = pool.apply_async\n "
223+ " \n "
224+ " # Run the test function\n "
225+ " test()\n "
226+ " ```\n "
227+ " Indeed, you\u2019 ll find that it works, but unraveling the control flow might require more\n "
228+ " coffee.\n "
229+ " \n "
230+ " Hiding tricky control flow behind generator functions is found elsewhere in the \n "
231+ " standard library and third-party packages. For example, the `@contextmanager` decorator in\n "
232+ " the `contextlib` performs a similar mind-bending trick that glues the entry and exit\n "
233+ " from a context manager together across a yield statement. The popular \n "
234+ " [Twisted package](http://twistedmatrix.com/) has inlined callbacks that are also similar."
235+ ]
236+ }
237+ ],
238+ "metadata" : {}
239+ }
240+ ]
241+ }
0 commit comments