Hello, Habitants! Python is a dynamic programming language used in a wide variety of subject areas. While it is easy to write code in Python, it is much more difficult to make the code readable, reusable, and easy to maintain. Third edition of Python. Best Practices and Tools ”will give you the tools to effectively solve any problem of software development and maintenance. The authors begin by talking about the new features in Python 3.7 and the advanced aspects of Python syntax. They continue with advice on the implementation of popular paradigms, including object-oriented, functional and event-driven programming. The authors also talk about the best naming practices, how you can automate the deployment of programs to remote servers. You will learn,how to create useful Python extensions in C, C ++, Cython and CFFI.
Who is this book for
Python, . , Python. , , , Python.
, . , Python. , , . Python 3.7 , Python 2.7 .
- -, , : .
, . , Python. , , . Python 3.7 , Python 2.7 .
- -, , : .
Access patterns for extended attributes
When learning Python, many C ++ and Java programmers are surprised at the lack of the private keyword. The closest concept to it is name mangling. Every time an attribute is prefixed with __, it is dynamically renamed by the interpreter:
class MyClass:
__secret_value = 1
Accessing the __secret_value attribute by its original name will throw an AttributeError exception:
>>> instance_of = MyClass()
>>> instance_of.__secret_value
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'MyClass' object has no attribute '__secret_value'
>>> dir(MyClass)
['_MyClass__secret_value', '__class__', '__delattr__', '__dict__',
'__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__',
'__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__',
'__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
'__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__']
>>> instance_of._MyClass__secret_value
1
This is done specifically in order to avoid name conflict by inheritance, since the attribute is renamed by the class name as a prefix. This is not an exact analogue of private, since the attribute can be accessed through a qualified name. This property can be used to protect access to some attributes, but in practice __ is never used. If the attribute is not public, then it is customary to use the _ prefix. It does not invoke the name decorating algorithm, but documents the attribute as a private element of the class and is the predominant style.
Python has other mechanisms to separate the public from the private part of a class. Descriptors and properties provide a way to tidy up this separation.
Descriptors
The descriptor allows you to customize the action that occurs when you reference an attribute on an object.
Descriptors are at the heart of complex attribute access in Python. They are used to implement properties, methods, class methods, static methods, and supertypes. These are the classes that define how the attributes of another class will be accessed. In other words, a class can delegate control of an attribute to another class.
Descriptor classes are based on three special methods that form the descriptor protocol:
__set __ (self, obj, value) - Called whenever an attribute is set. In the following examples, we will refer to it as “setter”;
__get __ (self, obj, owner = None) - called whenever the attribute is read (hereinafter the getter);
__delete __ (self, object) - Called when del is called by an attribute.
A descriptor that implements __get__ and __set__ is called a data descriptor. If it just implements __get__, it is called a no-data descriptor.
The methods of this protocol are actually called by the __getattribute __ () method (not to be confused with __getattr __ (), which has a different purpose) each time an attribute is looked up. Whenever such a lookup is done using a dot or a direct function call, the __getattribute __ () method is implicitly called, which looks for the attribute in the following order.
- Checks if an attribute is a data descriptor on an object of the instance class.
- If not, it looks to see if the attribute is found in the __dict__ of the instance object.
- Finally, checks if the attribute is a handle without data on the instance class object.
In other words, data descriptors take precedence over __dict__, which in turn takes precedence over non-data descriptors.
For clarity, here's an example from the official Python documentation that shows how descriptors work in real code:
class RevealAccess(object):
""" ,
"""
def __init__(self, initval=None, name='var'):
self.val = initval
self.name = name
def __get__(self, obj, objtype):
print('Retrieving', self.name)
return self.val
def __set__(self, obj, val):
print('Updating', self.name)
self.val = val
class MyClass(object):
x = RevealAccess(10, 'var "x"')
y = 5
Here's an example of using it interactively:
>>> m = MyClass()
>>> m.x
Retrieving var "x"
10
>>> m.x = 20
Updating var "x"
>>> m.x
Retrieving var "x"
20
>>> m.y
5
The example clearly shows that if a class has a data descriptor for that attribute, then the __get __ () method is called to return a value every time an instance attribute is retrieved, and __set __ () is called whenever that attribute is assigned a value. The use of the __del__ method is not shown in the previous example, but it should be obvious: it is called whenever an instance attribute is removed using the del instance.attribute or delattr (instance, 'attribute') statement.
The difference between data descriptors and data descriptors is significant for the reasons we mentioned at the beginning of this subsection. Python uses the descriptor protocol to bind class functions to instances via methods. They also apply to the classmethod and staticmethod decorators. This is because functional objects are essentially also dataless descriptors:
>>> def function(): pass
>>> hasattr(function, '__get__')
True
>>> hasattr(function, '__set__')
False
The same is true for functions created using lambda expressions:
>>> hasattr(lambda: None, '__get__')
True
>>> hasattr(lambda: None, '__set__')
False
Thus, unless __dict__ takes precedence over dataless descriptors, we will not be able to dynamically override specific methods of already instantiated instances at runtime. Fortunately, thanks to the way descriptors work in Python, this is possible; therefore, developers can choose which instances what works without using subclasses.
Real life example: lazy evaluation of attributes. One example of using descriptors is to delay the initialization of a class attribute when it is accessed from an instance. This can be useful if the initialization of such attributes depends on the global application context. Another case is when such initialization is too expensive, and it is not known whether the attribute will be used at all after importing the class. Such a descriptor can be implemented as follows:
class InitOnAccess:
def __init__(self, klass, *args, **kwargs):
self.klass = klass
self.args = args
self.kwargs = kwargs
self._initialized = None
def __get__(self, instance, owner):
if self._initialized is None:
print('initialized!')
self._initialized = self.klass(*self.args, **self.kwargs)
else:
print('cached!')
return self._initialized
Below is an example of use:
>>> class MyClass:
... lazily_initialized = InitOnAccess(list, "argument")
...
>>> m = MyClass()
>>> m.lazily_initialized
initialized!
['a', 'r', 'g', 'u', 'm', 'e', 'n', 't']
>>> m.lazily_initialized
cached!
['a', 'r', 'g', 'u', 'm', 'e', 'n', 't']
The official PyPI OpenGL Python library called PyOpenGL uses a technique like this to implement a lazy_property object that is both a decorator and a data descriptor:
class lazy_property(object):
def __init__(self, function):
self.fget = function
def __get__(self, obj, cls):
value = self.fget(obj)
setattr(obj, self.fget.__name__, value)
return value
This implementation is similar to using the property decorator (we'll talk about it later), but the function that is wrapped in the decorator is executed only once, and then the class attribute is replaced with the value returned by this function property. This method is often useful when two requirements must be met simultaneously:
- an object instance must be saved as a class attribute, which is shared between its instances (to save resources);
- this object cannot be initialized at the time of import, since the process of its creation depends on some global state of the application / context.
In the case of applications written using OpenGL, you will often encounter this situation. For example, creating shaders in OpenGL is expensive because it requires compiling code written in the OpenGL Shading Language (GLSL). It makes sense to create them only once and at the same time keep their description in close proximity to the classes that need them. On the other hand, shader compilations cannot be performed without initializing the OpenGL context, so it is difficult to define and assemble them in the global module namespace at the time of import.
The following example shows a possible use of a modified version of the lazy_property PyOpenGL decorator (here lazy_class_attribute) in some abstract OpenGL application. Changes to the original lazy_property decorator are required to allow the attribute to be shared across different instances of the class:
import OpenGL.GL as gl
from OpenGL.GL import shaders
class lazy_class_attribute(object):
def __init__(self, function):
self.fget = function
def __get__(self, obj, cls):
value = self.fget(obj or cls)
# : -
#
setattr(cls, self.fget.__name__, value)
return value
class ObjectUsingShaderProgram(object):
# -
VERTEX_CODE = """
#version 330 core
layout(location = 0) in vec4 vertexPosition;
void main(){
gl_Position = vertexPosition;
}
"""
# ,
FRAGMENT_CODE = """
#version 330 core
out lowp vec4 out_color;
void main(){
out_color = vec4(1, 1, 1, 1);
}
"""
@lazy_class_attribute
def shader_program(self):
print("compiling!")
return shaders.compileProgram(
shaders.compileShader(
self.VERTEX_CODE, gl.GL_VERTEX_SHADER
),
shaders.compileShader(
self.FRAGMENT_CODE, gl.GL_FRAGMENT_SHADER
)
)
Like all advanced Python syntax features, this one should also be used with care and well documented in code. For inexperienced developers, the changed behavior of a class can be a surprise because the descriptors affect the behavior of the class. Therefore, it is very important to make sure that all members of your team are familiar with descriptors and understand this concept if it plays an important role in the project codebase.
Properties
Properties provide a built-in descriptor type that knows how to associate an attribute with a set of methods. The property takes four optional arguments: fget, fset, fdel, and doc. The latter can be provided to define the docstring associated with the attribute as if it were a method. Below is an example of a Rectangle class that can be manipulated either by directly accessing the attributes that store two corner points, or by using the width and height properties:
class Rectangle:
def __init__(self, x1, y1, x2, y2):
self.x1, self.y1 = x1, y1
self.x2, self.y2 = x2, y2
def _width_get(self):
return self.x2 - self.x1
def _width_set(self, value):
self.x2 = self.x1 + value
def _height_get(self):
return self.y2 - self.y1
def _height_set(self, value):
self.y2 = self.y1 + value
width = property(
_width_get, _width_set,
doc="rectangle width measured from left"
)
height = property(
_height_get, _height_set,
doc="rectangle height measured from top"
)
def __repr__(self):
return "{}({}, {}, {}, {})".format(
self.__class__.__name__,
self.x1, self.y1, self.x2, self.y2
)
The following code snippet provides an example of such properties defined in an interactive session:
>>> rectangle.width, rectangle.height
(15, 24)
>>> rectangle.width = 100
>>> rectangle
Rectangle(10, 10, 110, 34)
>>> rectangle.height = 100
>>> rectangle
Rectangle(10, 10, 110, 110)
>>> help(Rectangle)
Help on class Rectangle in module chapter3:
class Rectangle(builtins.object)
| Methods defined here:
|
| __init__(self, x1, y1, x2, y2)
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| --------------------------------------------------------
| Data descriptors defined here:
| (...)
|
| height
| rectangle height measured from top
|
| width
| rectangle width measured from left
These properties make descriptors easier to write, but should be handled with care when using class inheritance. The attribute is created dynamically using the methods of the current class and will not apply methods that are overridden in derived classes.
The code in the following example will not be able to override the implementation of the fget method from the width property of the parent class (Rectangle):
>>> class MetricRectangle(Rectangle):
... def _width_get(self):
... return "{} meters".format(self.x2 - self.x1)
...
>>> Rectangle(0, 0, 100, 100).width
100
To solve this problem, the entire property should be overwritten in the derived class:
>>> class MetricRectangle(Rectangle):
... def _width_get(self):
... return "{} meters".format(self.x2 - self.x1)
... width = property(_width_get, Rectangle.width.fset)
...
>>> MetricRectangle(0, 0, 100, 100).width
'100 meters'
Unfortunately, the code has some maintainability issues. Confusion can arise if a developer decides to change the parent class but forgets to update the property call. This is why it is not recommended to override only parts of the behavior of properties. Instead of relying on the parent class's implementation, it is a good idea to rewrite all property methods in derived classes if you want to change the way they work. There are usually no other options, since changing the properties of the setter behavior entails a change in the getter's behavior.
The best way to create properties is to use property as a decorator. This will reduce the number of method signatures inside the class and make the code more readable and maintainable:
class Rectangle:
def __init__(self, x1, y1, x2, y2):
self.x1, self.y1 = x1, y1
self.x2, self.y2 = x2, y2
@property
def width(self):
""" """
return self.x2 - self.x1
@width.setter
def width(self, value):
self.x2 = self.x1 + value
@property
def height(self):
""" """
return self.y2 - self.y1
@height.setter
def height(self, value):
self.y2 = self.y1 + value
Slots
An interesting feature that developers rarely use is slots. They allow you to set a static list of attributes for a class using the __slots__ attribute and skip creating a __dict__ dictionary in every instance of the class. They were created to save memory space for classes with few attributes, since __dict__ is not created in every instance.
They can also help in creating classes whose signatures need to be frozen. For example, if you need to restrict the dynamic properties of the language for a specific class, then slots can help:
>>> class Frozen:
... __slots__ = ['ice', 'cream']
...
>>> '__dict__' in dir(Frozen)
False
>>> 'ice' in dir(Frozen)
True
>>> frozen = Frozen()
>>> frozen.ice = True
>>> frozen.cream = None
>>> frozen.icy = True
Traceback (most recent call last): File "<input>", line 1, in <module>
AttributeError: 'Frozen' object has no attribute 'icy'
This feature must be used with caution. When the set of available attributes is limited to slots, it is much more difficult to add something to an object dynamically. Some well-known tricks, such as monkey patching, will not work with instances of classes that have specific slots. Fortunately, new attributes can be added to derived classes if they don't have their own defined slots:
>>> class Unfrozen(Frozen):
... pass
...
>>> unfrozen = Unfrozen()
>>> unfrozen.icy = False
>>> unfrozen.icy
False
About the Authors
Michal Jaworski is a Python programmer with ten years of experience. He has held various positions in various companies: from a regular full-stack developer, then a software architect and, finally, to a vice president of development in a dynamic startup company. Michal is currently a Senior Backend Engineer at Showpad. Has extensive experience in the development of high performance distributed services. In addition, he is an active contributor to many open source Python projects.
Tarek Ziade is a Python developer. Lives in the countryside near Dijon, France. Works at Mozilla, on the services team. Tarek founded the French Python user group (called Afpy) and has written several books on Python in French and English. In his free time from hacking and partying, he is engaged in his favorite hobbies: jogging or playing the trumpet.
You can visit his personal blog (Fetchez le Python) and follow him on Twitter (tarek_ziade).
About the scientific editor
Cody Jackson is a Ph.D., founder of Socius Consulting, an IT and business management consulting firm based in San Antonio, and co-founder of Top Men Technologies. He currently works for CACI International as a Lead Engineer for ICS / SCADA Modeling. In the IT industry since 1994, since his time in the Navy as a nuclear chemist and radio engineer. Prior to CACI, he worked at the university at ECPI as Assistant Professor of Computer Information Systems. Learned Python programming on my own, wrote the books Learning to Program Using Python and Secret Recipes of the Python Ninja.
More details about the book can be found on the website of the publishing house
" Table of Contents
" Excerpt
For Habitants a 25% discount on coupon - Python
Upon payment for the paper version of the book, an e-book is sent to the e-mail.