Now almost every developer is familiar with the concept of "asynchrony" in programming. In an era when information products are so in demand that they are forced to simultaneously process a huge number of requests and also interact in parallel with a large set of other services - without asynchronous programming - nowhere. The need turned out to be so great that a separate language was even created, the main feature of which (in addition to being minimalistic) is a very optimized and convenient work with parallel / concurrent code, namely Golang . Despite the fact that the article is not about him at all, I will often make comparisons and refer to. But here in Python, which will be discussed in this article - there are some problems that I will describe and offer a solution to one of them. If you are interested in this topic - please, under cat.
It just so happens that my favorite language that I use to work, implement pet projects and even rest and relax is Python . I am endlessly captivated by its beauty and simplicity, its obviousness, behind which, with the help of various kinds of syntactic sugar, there are huge opportunities for a laconic description of almost any logic that the human imagination is capable of. I even read somewhere that Python is called an ultra-high-level language, since it can be used to describe abstractions that would be extremely problematic to describe in other languages.
But there is one serious nuance - Pythonvery difficult to fit into modern concepts of the language with the possibility of implementing parallel / concurrent logic. The language, the idea of which originated in the 80s and which is the same age as Java, until a certain time did not imply the execution of any code competitively. If JavaScript initially required concurrency for non-blocking work in a browser, and Golang is a completely fresh language with a real understanding of modern needs, then Python did not face such tasks before.
This, of course, is my personal opinion, but it seems to me that Python is very late with the implementation of asynchrony, since the appearance of the built-in asyncio librarywas, rather, a reaction to the emergence of other implementations of concurrent code execution for Python. Basically, asyncio is built to support existing implementations and contains not only its own event loop implementation, but also a wrapper for other asynchronous libraries, thus offering a common interface for writing asynchronous code. And Python , which was originally created as the most laconic and readable language due to all the factors listed above, when writing asynchronous code becomes a heap of decorators, generators and functions. The situation was slightly corrected by the addition of special directives async and await
I will not list them all and will focus on one that I tried to solve: this is a description of the general logic for asynchronous and synchronous execution. For example, if I want to run a function in parallel in Golang , then I just need to call the function with the go directive :
Parallel execution of function in Golang
package main
import "fmt"
func function(index int) {
fmt.Println("function", index)
}
func main() {
for i := 0; i < 10; i++ {
go function(i)
}
fmt.Println("end")
}
That being said, in Golang, I can run this same function synchronously:
Serial execution of function in Golang
package main
import "fmt"
func function(index int) {
fmt.Println("function", index)
}
func main() {
for i := 0; i < 10; i++ {
function(i)
}
fmt.Println("end")
}
In Python, all coroutines (asynchronous functions) are based on generators and switch between them occurs during the call of blocking functions, returning control to the event loop using the yield directive . To be honest, I don’t know how concurrency / concurrency works in Golang , but I’m not mistaken if I say that it doesn’t work the way it does in Python . Despite the existing differences in the internals of the implementation of the Golang compiler and the CPython interpreter and the inadmissibility of comparing parallelism / concurrency in them, I will still do this and pay attention not to the execution itself, but to the syntax. In PythonI cannot take a function and run it in parallel / concurrently with one operator. For my function to work asynchronously, I must explicitly write it async before its declaration, and after that it is no longer just a function, it is already a coroutine. And I cannot mix their calls in one code without additional actions, because a function and a coroutine in Python are completely different things, despite the similarity in the declaration.
def func1(a, b):
func2(a + b)
await func3(a - b) # , await
My main problem was the need to develop logic that can run both synchronously and asynchronously. A simple example is my library for interaction with Instagram , which I abandoned a long time ago, but now took up it again (which prompted me to search for a solution). I wanted to implement in it the ability to work with the API not only synchronously, but also asynchronously, and this was not just a desire - when collecting data on the Internet, you can send a large number of requests asynchronously and get an answer to all of them faster, but at the same time, massive data collection is not always needed. At the moment, the library implements the following: for working with Instagramthere are 2 classes, one for synchronous work, the other for asynchronous. Each class has the same set of methods, only in the first the methods are synchronous, and in the second they are asynchronous. Each method does the same thing - except for how requests are sent to the Internet. And only because of the differences in one blocking action, I had to almost completely duplicate the logic in each method. It looks like this:
class WebAgent:
def update(self, obj=None, settings=None):
...
response = self.get_request(path=path, **settings)
...
class AsyncWebAgent:
async def update(self, obj=None, settings=None):
...
response = await self.get_request(path=path, **settings)
...
Everything else in the update method and in the update coroutine is absolutely identical. And as many people know, code duplication adds a lot of problems, especially when it comes to fixing bugs and testing.
I wrote my own pySyncAsync library to solve this problem . The idea is as follows - instead of ordinary functions and coroutines, a generator is implemented, in the future I will call it a template. In order to execute a template, you need to generate it as a regular function or as a coroutine. The template, when executed at the moment when it needs to execute asynchronous or synchronous code inside itself, returns a special Call object using yield, which specifies what to call and with what arguments. Depending on how the template will be generated - as a function or as a coroutine - this is how the methods described in the Call object will be executed .
I will show a small example of a template that assumes the ability to make requests to google :
Example google requests using pySyncAsync
import aiohttp
import requests
import pysyncasync as psa
# google
# Call
@psa.register("google_request")
def sync_google_request(query, start):
response = requests.get(
url="https://google.com/search",
params={"q": query, "start": start},
)
return response.status_code, dict(response.headers), response.text
# google
# Call
@psa.register("google_request")
async def async_google_request(query, start):
params = {"q": query, "start": start}
async with aiohttps.ClientSession() as session:
async with session.get(url="https://google.com/search", params=params) as response:
return response.status, dict(response.headers), await response.text()
# 100
def google_search(query):
start = 0
while start < 100:
# Call , google_request
call = Call("google_request", query, start=start)
yield call
status, headers, text = call.result
print(status)
start += 10
if __name__ == "__main__":
#
sync_google_search = psa.generate(google_search, psa.SYNC)
sync_google_search("Python sync")
#
async_google_search = psa.generate(google_search, psa.ASYNC)
loop = asyncio.get_event_loop()
loop.run_until_complete(async_google_search("Python async"))
I'll tell you a little about the internal structure of the library. There is a Manager class , in which functions and coroutines are registered to be called using Call . It is also possible to register templates, but this is optional. The Manager class has methods register , generate, and template . The same methods in the example above were called directly from pysyncasync , only they used a global instance of the Manager class , which was already created in one of the library modules. In fact, you can create your own instance and call the register , generate and template methods from it.thus isolating managers from each other if, for example, a name conflict is possible.
The register method acts as a decorator and allows you to register a function or coroutine for further call from the template. The register decorator accepts as an argument the name under which the function or coroutine is registered in the manager. If the name is not specified, then the function or coroutine is registered under its own name.
The template method allows you to register the generator as a template in the manager. This is necessary in order to be able to get a template by name. Generate
methodallows you to generate a function or coroutine based on a template. It takes two arguments: the first is the name of the template or the template itself, the second is "sync" or "async" - what to generate the template - to a function or to a coroutine. At the output, the generate method gives a ready-made function or coroutine.
I will give an example of generating a template, for example, in a coroutine:
def _async_generate(self, template):
async def wrapper(*args, **kwargs):
...
for call in template(*args, **kwargs):
callback = self._callbacks.get(f"{call.name}:{ASYNC}")
call.result = await callback(*call.args, **call.kwargs)
...
return wrapper
Inside, a coroutine is generated, which simply iterates over the generator and receives objects of the Call class , then takes the previously registered coroutine by name (the name is taken from call ), calls it with arguments (which it also takes from call ) and the result of executing this coroutine also stores in call .
Objects of the Call class are simply containers for storing information about what and how to call and also allow you to store the result in themselves. wrapper can also return the result of the template execution; for this, the template is wrapped in a special Generator class , which is not shown here.
I have omitted some of the nuances, but I hope I have conveyed the essence in general.
To be honest, this article was written by me rather to share my thoughts on solving problems with asynchronous code in Python.and, most importantly, to listen to the opinions of the Khabrav residents. Perhaps I will bump someone into another solution, maybe someone will disagree with this particular implementation and tell you how you can make it better, maybe someone will tell you why such a solution is not needed at all and you should not mix synchronous and asynchronous code, the opinion of each of you is very important to me. Also, I do not pretend to be true of all my reasoning at the beginning of the article. I thought very extensively on the topic of other languages and could be mistaken, plus there is a possibility that I might confuse the concepts, please, if you suddenly notice any inconsistencies - describe in the comments. I will also be glad if there are amendments to the syntax and punctuation.
And thank you for your attention to this issue and to this article in particular!