《 smooth Python》 Luciano · Ramallo The first 7 Chapter Function decorators and closures Reading notes
Catalog
7.1 Decorator basics
7.2 Python When to execute the decorator
7.3 Use a decorator to improve “ Strategy ” Pattern
7.4 Variable scope rule
7.5 Closure
7.6 nonlocal Statement
7.7 Implement a simple decorator
7.8 Decorators in standard library
7.8.1 Use functools.lru_cache Do cheat
7.9 Stacked ornaments
7.10 Parametric decorators
7.10.1 A parameterized registration decorator
7.10.2 A parameterized clock Decorator
This chapter begins with the following topics ：
Python Decorator syntax
Python How to judge whether a variable is local Ch7.4
Why closures exist and how they work Ch7.5
nonlocal What problems can be solved Ch7.6
After mastering these basic knowledge , We can further explore decorators （ A little difficult , First, make a simple record of what you can understand ）：
Achieve a well behaved decorator
Useful decorators in the standard library
Implement a parameterized decorator
Decorators are callable objects , Its argument is another function （ Decorated function ）. Decorators may
- Handle decorated functions , And then I'm going to return it
- Replace the decorated function with another function or callable object
If there's one called decorate The decorator ：
@decorate
def target():
print('running target()')The effect of the above code is the same as the following ：
def target():
print('running target()')
target = decorate(target) Two characteristics of decorators ：
- Can replace the decorated function with other functions
- The decorator executes immediately when the module is loaded
One of the key features of decorators is that , They run immediately after the decorated function definition . This is usually done when importing （ namely Python When loading modules ）.
Example 7-2 registration.py modular
The following examples mainly want to emphasize , The function decorator executes immediately when the module is imported , The decorated function only runs when explicitly called . This highlights the The difference between import time and runtime .
registry = [] #1
def register(func): #2
print('running register(%s)' % func) #3
registry.append(func) #4
return func #5
@register #6
def f1():
print('running f1()')
@register
def f2():
print('running f2()')
def f3(): #7
print('running f3()')
def main(): #8
print('running main()')
print('registry ->', registry)
f1()
f2()
f3()
if __name__=='__main__':
main() #9
（1） hold registration.py The output from running as a script is as follows ：
# python3 registration.py
running register(<function f1 at 0x7ff58ca96310>)
running register(<function f2 at 0x7ff58ca963a0>)
running main()
registry -> [<function f1 at 0x7ff58ca96310>, <function f2 at 0x7ff58ca963a0>]
running f1()
running f2()
running f3()Be careful ,register Run before other functions in the module （ two ）. call register when , The parameters passed to it are decorated functions , for example <function f1 at 0x7ff58ca96310>.
After loading the module ,registry There are two references to decorated functions in ：f1 and f2. These two functions , as well as f3, Only in main Only execute when they are explicitly called .
（2） If you import registration.py modular （ Do not run as script ）, Output is as follows ：
>>> import registration
running register(<function f1 at 0x7f802bba35e0>)
running register(<function f2 at 0x7f802bba3670>)
>>>At this point to see registry Value , The output is as follows ：
>>> registration.registry
[<function f1 at 0x7f802bba35e0>, <function f2 at 0x7f802bba3670>]
>>> explain ： In the above example ,
Decorator functions and decorated functions are defined in the same module . actually , Decorators are usually defined in a module , And then apply it to functions in other modules .
register The function returned by the decorator is the same as the function passed in through the parameter . actually , Most decorators define a function inside , Then return it to . The decorator returns the decorated function intact , But this technology is not useless . quite a lot Python Web The framework uses such decorators to add functions to some kind of central registry , For example, put URL Schema mapping to generation HTTP The registry on the response function . This registration decorator may or may not modify the decorated function .
Using the registration decorator can improve 6.1 Examples of e-commerce promotional discounts in section .
Example 6-6 The main problem is , There is the name of the function in the definition body , however best_promo There are also function names in the list . After adding a new policy function, you may forget to add it to promos In the list , Lead to best_promo Ignore the new strategy , And don't make a mistake , Introduce imperceptible defects to the system .
Example 7-3 promos The values in the list use promotion Decorator filling
promos = [] #1
def promotion(promo_func): #2
promos.append(promo_func)
return promo_func
@promotion #3
def fidelity(order):
""" The integral is 1000 Or more customers 5% discount """
return order.total() * .05 if order.customer.fidelity >= 1000 else 0
@promotion
def bulk_item(order):
""" A single item is 20 When more than one 10% discount """
discount = 0
for item in order.cart:
if item.quantity >= 20:
discount += item.total() * .1
return discount
@promotion
def large_order(order):
""" The different items in the order reach 10 When more than one 7% discount """
distinct_items = {item.product for item in order.cart}
if len(distinct_items) >= 10:
return order.total() * .07
return 0
def best_promo(order): #4
""" Choose the best discount available """
return max(promo(order) for promo in promos)explain ：
#1 promos The list was initially empty .
#2 promotion hold promo_func Add to promos In the list , Then return it intact .
#3 By @promotion Decorative functions will be added to promos In the list .
#4 best_promos There is no need to modify , Because it depends on promos list .
And 6.1 Compared with the scheme given in section , This scheme has several advantages ：
The promotion strategy function does not need to use a special name （ That is, there is no need to _promo ending ）.
@promotion Decorator highlights the function of the decorated function , It is also convenient to disable a promotion strategy ： Just comment out the decorator .
Promotion and discount policies can be defined in other modules , Just use @promotion Just decorate .
Example 7-5 b It's a local variable , Because it is assigned a value in the definition body of the function
>>> def f1(a):
... print(a)
... print(b)
...
>>> b = 6
>>> f1(3)
3
6
>>>
--------------------------------------
>>> def f2(a):
... print(a)
... print(b)
... b = 9
...
>>> f2(3)
3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in f2
UnboundLocalError: local variable 'b' referenced before assignment
>>>
First of all, I output 3, This shows that print(a) Statement executed .
Although there is a global variable b, The second sentence print(b) It can't be carried out , local variable b Is in print(b) After the assignment .
But the truth is ,Python When compiling the definition body of a function , It adjudicates break b It's a local variable , Because it's assigned a value in the function . The generated bytecode confirms this judgment ,Python Will try to get... From the local environment b. Call back f2(3) when , f2 To get and print local variables a Value , But try to get local variables b The value of , Find out b No binding value .
This is the design choice ：Python Variables are not required to be declared , But suppose that the variable assigned in the function definition body is a local variable .
If you want the interpreter to put b As global variables , To use global Statement
>>> b = 6
>>> def f3(a):
... global b
... print(a)
... print(b)
... b = 9
...
>>> f3(3)
3
6
>>>
A closure is a function that extends its scope , It contains references in the function definition body 、 But non global variables that are not defined in the definition body . The point is that it Can access non global variables defined outside the definition body .
Illustrate with examples --> If there's one called avg Function of , Its function is to calculate the mean value of the increasing series of values ;
for example , The average closing price of a commodity throughout history . New prices are added every day , So the average takes into account all the prices so far .
At first ,avg It's used in this way ：
>>> avg(10)
10.0
>>> avg(11)
10.5
>>> avg(12)
11.0
Example 7-8 average0.py： Calculate the class of moving average
class Averager():
def __init__(self):
self.series = []
def __call__(self, new_value):
self.series.append(new_value)
total = sum(self.series)
return total/len(self.series)Averager An instance of is a callable object ：
>>> from average0 import Averager
>>> avg = Averager()
>>> avg(10)
10.0
>>> avg(11)
10.5
>>> avg(12)
11.0
>>>Example 7-9 average.py： Calculate the higher-order function of the moving average
def make_averager():
series = []
def averager(new_value):
series.append(new_value)
total = sum(series)
return total/len(series)
return averagercall make_averager when , Return to one averager Function object . Every time you call averager when , It adds parameters to the series value , Then calculate the current average .
>>> from average import make_averager
>>> avg = make_averager()
>>> avg(10)
10.0
>>> avg(11)
10.5
>>> avg(12)
11.0
>>> explain ： These two examples have something in common ： call Averager() or make_averager() Get a callable object avg, It updates the historical value , Then calculate the current mean . In the example 7-8 in ,avg yes Averager Example ; In the example 7-9
Is an internal function averager.
Compare
Example 7-8 Example 7-9 Common ground ： Get a callable object avg call Averager() call make_averager() Store historical values Averager Class avg stay self.series Instance properties store historical values series yes make_averager A local variable of a function , Because the function is initialized in the definition body , call avg(10) when ,make_averager The function has returned , And its local scope is gone .stay averager Function ,series It's a free variable （free variable）
Free variable ： Refers to variables that are not bound in the local scope
averager The closure of extends beyond the scope of that function , Contains free variables series The binding of
Review the returned averager object , We found that Python stay __code__ attribute （ Represents the compiled function definition body ） Save the names of local variables and free variables in
>>> avg.__code__.co_varnames
('new_value', 'total')
>>> avg.__code__.co_freevars
('series',)
>>>series The binding of is in the returned avg Functional __closure__ Properties of the .avg.__closure__ Each element in corresponds to avg.__code__.co_freevars A name in . These elements are cell object , There is one cell_contents attribute , Save the real value .
>>> avg.__closure__
(<cell at 0x7f802c0ba1c0: list object at 0x7f802bdf3ec0>,)
>>> avg.__closure__[0].cell_contents
[10, 11, 12]
>>>
Sum up , Closures retain the binding of free variables that exist when defining functions , When a function is called like this , Although the definition scope is not available , But you can still use those bindings . Be careful , Only functions nested in other functions may need to deal with external variables that are not in the global scope .
The front implementation make_averager The method of function is not efficient . The better way to do that is , Only the current total value and the number of elements are stored , Then use these two numbers to calculate the mean .
Example 7-13 Calculate the higher-order function of the moving average , Do not save all historical values （ defective ）
>>> def make_averager():
... count = 0
... total = 0
... def averager(new_value):
... count += 1
... total += new_value
... return total / count
... return averager
...
>>> avg = make_averager()
>>> avg(10)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in averager
UnboundLocalError: local variable 'count' referenced before assignment
>>>The problem lies in , When count Is a number or any immutable type ,count += 1 <=> count = count + 1 therefore , We are averager In the body of the definition of count Assigned a value to , This will turn count Become local variable （ And examples 7-5 A truth ）.
Example 7-9 No problem , Because we didn't give series assignment , Just call series.append, And pass it on to sum and len. That is to take advantage of the fact that lists are mutable objects .
But for numbers 、 character string 、 For immutable types such as tuples , Can only read , Can't update .
If you try to rebind , for example count = count + 1, It actually implicitly creates local variables count. such ,count It's not a free variable , So it won't be saved in the closure .
terms of settlement --> nonlocal Statement
nonlocal The purpose of the declaration is to mark the variable as a free variable . If nonlocal The declared variable is given a new value , The binding saved in the closure will be updated .
Example 7-14 Calculate the moving average , Don't save all history （ Use nonlocal correct ）
>>> def make_averager():
... count = 0
... total = 0
... def averager(new_value):
... nonlocal count, total
... count += 1
... total += new_value
... return total / count
... return averager
...
>>> avg = make_averager()
>>> avg(10)
10.0
>>> avg(8)
9.0
>>>
Example 7-15 The running time of the output function 、 The parameters passed in and the result of the call
Define a decorator clockdeco.py
import time
def clock(func):
def clocked(*args): #1
t0 = time.perf_counter()
result = func(*args) #2
elapsed = time.perf_counter() - t0
name = func.__name__
arg_str = ', '.join(repr(arg) for arg in args)
print('[%0.8fs] %s(%s) -> %r' % (elapsed, name, arg_str, result))
return result
return clocked #3 explain ：
#1 Define internal functions clocked, It accepts any positioning parameter .
#2 This line of code is available , Because clocked The closure of contains free variables func.
#3 Returns the internal function , Replace the decorated function .
clockdeco_demo.py
import time
from clockdeco import clock
@clock
def snooze(seconds):
time.sleep(seconds)
@clock
def factorial(n):
return 1 if n < 2 else n*factorial(n-1)
if __name__=='__main__':
print('*' * 40, 'Calling snooze(.123)')
snooze(.123)
print('*' * 40, 'Calling factorial(6)')
print('6! =', factorial(6))Execution results ：
# python3 clockdeco_demo.py
**************************************** Calling snooze(.123)
[0.12405594s] snooze(0.123) -> None
**************************************** Calling factorial(6)
[0.00000079s] factorial(1) -> 1
[0.00001218s] factorial(2) -> 2
[0.00001909s] factorial(3) -> 6
[0.00002585s] factorial(4) -> 24
[0.00003433s] factorial(5) -> 120
[0.00004931s] factorial(6) -> 720
6! = 720
clocked I did the following things roughly
(1) Record the initial time t0.
(2) Call the original factorial function , Save results .
(3) Calculate the elapsed time .
(4) Format the collected data , Then print it out .
(5) Back to page 2 Results saved in step .
This is a Typical behavior of decorators ： Replace the decorated function with a new one , Both accept the same parameters , and （ Usually ） Return the value that the decorated function should have returned , There will also be some additional operations .
Example 7-15 Implemented in clock The decorator has several disadvantages ： Keyword parameters are not supported , And it covers the of the decorated function __name__ and __doc__ attribute . Example 7-17 Use functools.wraps The decorator changes the relevant attributes from func Copied to the clocked in . Besides , This new version can also Handle keyword parameters correctly .
Example 7-17 Improved clock Decorator
clockdeco.py
import time
import functools
def clock(func):
@functools.wraps(func)
def clocked(*args, **kwargs):
t0 = time.time()
result = func(*args, **kwargs)
elapsed = time.time() - t0
name = func.__name__
arg_lst = []
if args:
arg_lst.append(', '.join(repr(arg) for arg in args))
if kwargs:
pairs = ['%s=%r' % (k, w) for k, w in sorted(kwargs.items())]
arg_lst.append(', '.join(pairs))
arg_str = ', '.join(arg_lst)
print('[%0.8fs] %s(%s) -> %r ' % (elapsed, name, arg_str, result))
return result
return clocked-----------------------------
（ The following records are gradually scrawled ）
The two most noteworthy decorators in the standard library are lru_cache And new singledispatch（Python 3.4 newly added ）.
These two ornaments are functools Defined in module .
functools.lru_cache It's a very practical decorator , It implements memos （memoization） function . This is an optimization technique , It saves the results of time-consuming functions , Avoid double calculation when the same parameter is passed in .LRU The three letters are "Least Recently Used" Abbreviation , Indicates that the cache will not grow unlimited , Cache entries that are not used for a period of time are thrown away .
Example 7-19 Use cache to realize , Faster
import functools
from clockdeco import clock
@functools.lru_cache() #1
@clock #2
def fibonacci(n):
if n < 2:
return n
return fibonacci(n-2) + fibonacci(n-1)
if __name__=='__main__':
print(fibonacci(6))Running results ：
# python3 test.py
[0.00000071s] fibonacci(0) -> 0
[0.00000112s] fibonacci(1) -> 1
[0.00009533s] fibonacci(2) -> 1
[0.00000108s] fibonacci(3) -> 2
[0.00010672s] fibonacci(4) -> 3
[0.00000094s] fibonacci(5) -> 5
[0.00011868s] fibonacci(6) -> 8
8
explain ：
#1 Be careful , It must be called like a regular function lru_cache. There is a pair of parentheses in this line ：@functools.lru_cache(),lru_cache Configuration parameters are acceptable
#2 There are decorations stacked here ：@lru_cache() Applied to the @clock On the returned function
hold @d1 and @d2 Two decorators are applied to... In sequence f On the function , The effect is equivalent to f = d1(d2(f)).
The following code ：
@d1
@d2
def f():
print('abcd')Equate to ：
def f():
print('abcd')
f = d1(d2(f))Python Pass the decorated function as the first parameter to the decorator function .
So how to make the decorator accept other parameters ？
The answer is ： Create a decorator factory function , Pass the parameters to it , Returns a decorator , Then apply it to the function to decorate .
For the convenience of enabling or disabling register Function registration function executed , We provide an optional active Parameters , Set to False when , Do not register decorated functions .
Conceptually , This new register Functions are not decorators , It's the decorator factory function . Calling it will return the real decorator , This is the decorator applied to the objective function .
Example 7-23 In order to accept parameters , new register Decorators must be called as functions
registration_param.py
registry = set() #1
def register(active=True): #2
def decorate(func): #3
print('running register(active=%s)->decorate(%s)' % (active, func))
if active: #4
registry.add(func)
else:
registry.discard(func) #5
return func #6
return decorate #7
@register(active=False) #8
def f1():
print('running f1()')
@register() #9
def f2():
print('running f2()')
def f3():
print('running f3()')The results are as follows
>>> import registration_param
running register(active=False)->decorate(<function f1 at 0x7efc9f385ca0>)
running register(active=True)->decorate(<function f2 at 0x7efc9f385dc0>)
>>>
>>> registration_param.registry
{<function f2 at 0x7efc9f385dc0>}
>>>
explain ：
#1 registry Now it's a set object , This makes adding and deleting functions faster .
#2 register Accept an optional keyword parameter .
#3 decorate This internal function is a real decorator ; Be careful , Its argument is a function .
#4 Only active The value of the parameter （ Get... From a closure ） yes True Register only when func.
#5 If active Not true , and func stay registry in , Then delete it .
#6 decorate It's a decorator , Must return a function .
#7 register Is the decorator factory function , Therefore return decorate.
#8 @register Factory functions must be called as functions , And pass in the required parameters .
#9 Even if no parameters are passed in ,register Must also be called as a function （@register()）, That is to return to the real decorator decorate.
The key here is ,register() To return decorate, Then apply it to the decorated function .
by clock Decorator adds a function ： Let the user pass in a format string , Controls the output of the decorated function
Simplicity , Example 7-25 Based on examples 7-15 First implemented in clock
Example 7-25 clockdeco_param.py modular ： A parameterized clock Decorator
clockdeco_param.py
import time
DEFAULT_FMT = '[{elapsed:0.8f}s] {name}({args}) -> {result}'
def clock(fmt=DEFAULT_FMT): #1
def decorate(func): #2
def clocked(*_args): #3
t0 = time.time()
_result = func(*_args) #4
elapsed = time.time() - t0
name = func.__name__
args = ', '.join(repr(arg) for arg in _args) #5
result = repr(_result) #6
print(fmt.format(**locals())) #7
return _result #8
return clocked #9
return decorate #10
if __name__ == '__main__':
@clock()
def snooze(seconds):
time.sleep(seconds)
for i in range(3):
snooze(.123)explain ：
#1 clock Is a parameterized decorator factory function .
#2 decorate It's a real decorator .
#3 clocked Packing decorated functions .
#4 _result Is the real result returned by the decorated function .
#5 _args yes clocked Parameters of ,args Is the string used for display .
#6 result yes _result String representation of , Used for display .
#7 Use here **locals() In order to in fmt I quote clocked Local variables of .(??)
#8 clocked Will replace the decorated function , So it should return the value returned by the decorated function .
#9 decorate return clocked.
#10 clock return decorate.
#11 Test in this module , Call without passing in parameters clock(), Therefore, the applied decorator uses the default format str.
stay shell Running examples in 7-25, The following results will be obtained ：
# python3 clockdeco_param.py
[0.12325597s] snooze(0.123) -> None
[0.12458444s] snooze(0.123) -> None
[0.12462711s] snooze(0.123) -> None
Example 7-26 clockdeco_param_demo1.py Passed in string output format
import time
from clockdeco_param import clock
@clock('{name}: {elapsed}s')
def snooze(seconds):
time.sleep(seconds)
for i in range(3):
snooze(.123)The results are as follows ：
# python3 clockdeco_param_demo1.py
snooze: 0.12509799003601074s
snooze: 0.12530088424682617s
snooze: 0.12441754341125488s
Example 7-26 clockdeco_param_demo2.py Another string output format was passed in
import time
from clockdeco_param import clock
@clock('{name}({args}) dt={elapsed:0.3f}s')
def snooze(seconds):
time.sleep(seconds)
for i in range(3):
snooze(.123)
The results are as follows ：
# python3 clockdeco_param_demo2.py
snooze(0.123) dt=0.125s
snooze(0.123) dt=0.124s
snooze(0.123) dt=0.124s
