Async Task Tools Documentation

Release 0.5

David Pineda

Aug 20, 2018

 Table of Contents:

1 What is an async function? 3

1.1 The classic way . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 The asynchronous way . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Recommended Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 How to Create an Asyncio Coroutine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.5 How use the coroutine in a simple way . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.6 Wait a moment, how it’s work? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.7 Do you want more? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.8 Main features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.9 Consider this objects when work with coroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Installing the Module 7

2.1 Developer Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Install with PIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 Why we need an Async While? 9

4 A Big Tool Beyond Responsabilities: The Scheduler 13

4.1 Explanation, please! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

5 The Walking Stick for The Old Scheduler: The Assignator 15

6 Code’s Documentation 17
6.1 The Async While to be a Wheel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
6.2 The Scheduler explained Step by Step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
6.3 The Assignator role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

7 To do 23

8 Indices and tables 25

Index 27

i
ii
 Async Task Tools Documentation, Release 0.5

The Tasktools module allows you to work with the asyncio library from the programming language Python on the
versions above the 3.5.
The features which this module provides is the ability to create async functions that runs in an async while in first
case. The problem is when you have a lot of Task to run at same time (a lot or many) you have to order your mind and
design a system or machine using more resources.
To do that, you have to activate multiple processes and create the tasks for every active process. In that stage the
TaskScheduler and TaskAssignator are useful. Like its names says are specific machines to schedule tasks on the
procesess and activate with the assignator.
Hence, when you have a system that needs to run permanently, check new data, convert and send to other place is
convenient to study this module and implement your functions and classes using the tools from this module.
¡Enjoy!

Table of Contents: 1
Async Task Tools Documentation, Release 0.5

2 Table of Contents:
 CHAPTER 1

What is an async function?

An async function or coroutine is an special kind of function which allows you to work with events that happend
without your control but you need to access to some of the information generated on that.
For example, if you realize a query to a server you can’t control how much time last the work of the computer searching
the data and returning to you, so you have to await for the data. But, what can you do with this awaited time. In the
classic paradigm you have to wait for the returning data, but with the asynchronous paradigm you can do something
else.
With that in mind you can use this tools to message exchange in any case, socket communications (to databases, other
servers, etc) and any system with multiple events happening at same time.
In Python the library async provides an implementation for that purpose, and you have to consider some new concepts
and stages to enable the execution of this coroutines.

1.1 The classic way

The classic way or synchronous programming way, you have a number of actions that the computer runs one after
other, you can’t attend other events.

In python the GIL(Global Interpreter Locking) take charge of that and blocks any other action while the code is
running.

1.2 The asynchronous way

With that programming style we can say the we are near reality, on our programs we depends from things that can
or can’t happend without our control, but we can give some confidence to some systems or the systems that we know

3
Async Task Tools Documentation, Release 0.5

how works (in general), and then give or send, and get or receive data from them.

So, when we detect that something is ready to proceed, we continue with that. The order whem that happen doesn’t
matter.
Better explained:

1.3 Recommended Knowledge

The DavidBeaz site offer you a wide variety of examples, videos and material about the most interesting subjects on
python and computer science. Put in clear a lot of concepts of complex things like the asyncio module.
The WillysCave site explains the main differences betweeon classic and asynchronous programming ways.
The MichaelKuty presentation to explain the evolution of this subjects.

4 Chapter 1. What is an async function?

 Async Task Tools Documentation, Release 0.5

The DavidPineda, yes me, in español with different and useful examples
The Modules consider an interesting list of different uses of asyncio library.

1.4 How to Create an Asyncio Coroutine

Look at this definition:

1 import asyncio
2

3 async def holacoro():

4 for i in range(3):
5 await asyncio.sleep(1)
6 print("Hola %d" % i)

1. Import the module

2. Create a new function with async
3. Use the wait

1.5 How use the coroutine in a simple way

Look at this code:

loop = asyncio.get_event_loop()
#creamos tarea y la asociamos al loop, ejecutandola
loop.run_until_complete(holacoro())

The loop is an object that manage the exuction of the coroutines

1.6 Wait a moment, how it’s work?

Well, step by step:

• Define the coroutine
• Create a Future Task in what te coroutine vill be run

future_task = holacoro()

• Collect other different tasks. Other coroutines will be run.

future_task_2 = othercoro()

• Create a group, set or list

task_list = [future_task, future_task_2]

• Gather the tasks

The tasks are the arguments from the unpacked list of pack to create the gather object.

1.4. How to Create an Asyncio Coroutine 5

Async Task Tools Documentation, Release 0.5

tasks_gathered = asyncio.gather(*tasks)

• Finally, run the gathered tasks.

loop.run_until_complete(tasks_gathered)

1.7 Do you want more?

Bored, but check the PEP’s

1. PEP 492 https://www.python.org/dev/peps/pep-0492/
2. PEP 3156 https://www.python.org/dev/peps/pep-3156/
3. PEP 3153 https://www.python.org/dev/peps/pep-3153/
4. PEP 525 https://www.python.org/dev/peps/pep-0525/
5. PEP 492 https://www.python.org/dev/peps/pep-0492/
6. PEP 530 https://www.python.org/dev/peps/pep-0530/

1.8 Main features

1. The eventloop with many different implementations, the standar and others (like quamash for qt)
2. Abstractions for layers Transport and Protocol, (OSI model?)
3. Support for networking uses: TCP, UDP, UnixSocket, SSL, subprocesses, pipes, etc.
4. Sintaxis based on (async, await) pair.
5. Manage Futures and coroutines
6. Interface to work with threads and multiprocessing

1.9 Consider this objects when work with coroutines

1. Eventloop: the loop manager

2. Coroutines: the special function
3. Futures, Tasks: when you say you will run some coroutine
4. Streams: comunication elements
5. Subprecesses: activate different workes
6. Queues: to share data among processes

6 Chapter 1. What is an async function?

 CHAPTER 2

Installing the Module

To install the module you have different options

2.1 Developer Mode

If you want to work with the last modifications, when the project is updated, you can clone and install in dev mode
(on the terminal).

git clone https://gitlab.com/pineiden/tasktools

python setup.py develop

2.2 Install with PIP

If you want to work with the stable release you can install with pip in your virtualenv .

pip install tasktools

Or download the package from the pipy platform.

wget https://files.pythonhosted.org/packages/05/31/
˓→ee3981f7c4d6ee4968c53e40e0ed77089c50861cbe1960553006d0923a3f/tasktools-0.5.tar.gz

python setup.py install

7
Async Task Tools Documentation, Release 0.5

8 Chapter 2. Installing the Module

 CHAPTER 3

Why we need an Async While?

If you need to run your coroutines and each must be executed independent, like receive stream data from a list of
sources, in which every source have different frecuencies, you need this.
TaskLoop provides a set of functions and asyncio coroutines who can help you on your mission.
You need

fn1 ->->->->->->->-> |
fn2 —>—>—>—> |
.|
.|
.|
fnN –>–>–>–>–>–> |

You can’t do a /for/ and obtain good results. You need free tasks.
So, the model you need is for every source-coroutine is:

9
Async Task Tools Documentation, Release 0.5

The implementation consider:

1. Define coroutine
2. The couroutine need some input?
3. The input is the same ever or change with the result?
4. Create a couroutine mask who await the couroutine and manage the input
5. Call the coro-mask in a task generator, scheduling it.
6. Set a callback when task is done
7. When task is done the callback must be the same task generator
In a schemma, the system is:

Args input
First task scheduled
Call when done
Define
Your Coroutine object Renew task when done Task(k)
Call when done

Coroutine manage your coroutine

The methods or functions avalaible for do this schemma are:

Function Inputs Do
coromask [coro, args, fargs] await coro
renew [task, coro, fargs,*args] renew task
simple_fargs [input, output] return input
simple_fargs_out [input, output] return output

The following code is an example about how to use the taskloop methods, we have two coroutines, define a fargs for

10 Chapter 3. Why we need an Async While?

 Async Task Tools Documentation, Release 0.5

every coroutine, and use the methods on a asyncio event loop.

1 import asyncio
2 import functools
3

4 async def holacoro(v):

5 print("Hola %d" % v)
6 await asyncio.sleep(1)
7 return v+1
8

9 async def sumacoro(*args):

10 c=sum(args)
11 print("La suma es %d" %c)
12 await asyncio.sleep(3)
13 return c
14

15 def fargs_holacoro(args, obtained):

16 return [obtained]
17

18 def fargs_sumacoro(args, obtained):

19 result= [args[-1], obtained]
20 return result
21

22 #every coroutine
23 async def coromask(coro, args, fargs):
24 _in=args
25 obtained=await coro(*args)
26 result=fargs(_in, obtained)
27 return result
28

29 def renew(task, coro, fargs, *args):

30 result=task.result()
31 task=loop.create_task(coromask(coro, result, fargs))
32 task.add_done_callback(functools.partial(renew, task, coro, fargs))
33

34 loop=asyncio.get_event_loop()
35

36 args=[1]
37 task1=loop.create_task(coromask(holacoro,args,fargs_holacoro))
38 task1.add_done_callback(functools.partial(renew, task1, holacoro, fargs_holacoro))
39

40 args2=[1,2]
41 task2=loop.create_task(coromask(sumacoro,args2,fargs_sumacoro))
42 task2.add_done_callback(functools.partial(renew, task2, sumacoro, fargs_sumacoro))
43

44

45 try:
46 loop.run_forever()
47 except KeyboardInterrupt:
48 print('Loop stopped')

Now, those async functions are encapsulated on the Tasktools module inside the taskloop file. Hence, to call them you
have to use the tradicional python system to use modules and functions.

1 from tasktools.taskloop import coromask, renew, simple_fargs

11
Async Task Tools Documentation, Release 0.5

12 Chapter 3. Why we need an Async While?

 CHAPTER 4

A Big Tool Beyond Responsabilities: The Scheduler

The needs to create a system that attends multiples sources at same time produce the implementation of the TaskSched-
uler and TaskAssignator classes that provide the feature of work as a general conceptualization of a big machine with
multiple tasks.
For a specific implementation, for your goal, you have to inherit the Scheduler and implement this funcions and
actions.
You have to create in a memory shared space multiprocesing manager the following basic structures:
• list ipt :: list that manage process assignment
• list ico :: list that manage tasks on the active process
• dicctionary assigned_tasks :: dict that collect assigned tasks
• nproc :: int value with amount of assigned process
• dictionary sta_init :: dict with the flags to control task executions.
The following step is the assignment of your loop function in which you run your desired tasks. There are two cases:
• Unidirectional :: create a dictionary with your run_task async function. That function have to initialice the
classes for every task and then do the unidirectional work.
• Bidirectional :: create a dictionary with you have to separate the different actions. The ‘run_task’ is for initialize
the loop with the actions. The key net2service is the async function that come from network and send to service,
the key service2net is the async function that come from service and send to network,
On the main script you have to put the Scheduler object on the Assignator object, in that way every time you active a
task the assignator produce a message to activate the task on the Scheduler.
The magic on this clasess are the initial settings, in which you define the size of your machine:
• amount of processes
• amount of tasks by process
With that data the machine put in idle the main process task and by every process also in idle the amount of tasks.
When you start to assign the task to do something, the TaskAssignator object start to wake up the idle tasks. So on, is
not simple but works.

13
Async Task Tools Documentation, Release 0.5

4.1 Explanation, please!

I can say this class is the digievolution of the taskloop functionalities. I need to use those tools to collect an amount
of data in real time from a network of stations. If i used only a process the rate to produce a new data is large than the
capacity to process all at the right time. Because of that is necesary the use of more simultaneous processes to work
with all the assigned tasks.
In relation with the amount of stations is necesary to know
• amount of sources
• frequency of new data (f)
• capacity of number of stations or sources by processor in that frequency (cap(f))
We can define
• Amount of workers (procesors) to work with every source (n)
• Amount of tasks availables by every worker (m)
At the running time, the system is an empty machine with n workers in await status, and every worker with m tasks
also in await status.
When the metadata to connect every source arrive to the machine, maybe using an extra worker to attend socket
communication, the data is loaded and start to active the tasks, one source is one task.
And that is the idea. Big, happy, useful machines for everyone.

14 Chapter 4. A Big Tool Beyond Responsabilities: The Scheduler

 CHAPTER 5

The Walking Stick for The Old Scheduler: The Assignator

Here is useful to understand how works the queues and shared dicts or list on memory. Using the manager from
multiprocessing library you can activate this special objects.
The TaskAssignator class works with an instance of TaskScheduler’s subclass implemented specifically for your
goal.
This class handle the list of new data incoming and assign the task to some specifically worker. Register the action to
future control. Simple but effective.
Also you can control if you only need to active a group(for test purposes) or all the list of fata.

1 async def new_process(self, queue_tasks):

2 """
3 This coroutine activate a process with new station task
4 Check every \ts\ seconds the queue_tasks if there are new
5 stations or tasksto add
6

7 Args:
8 :param queue_tasks: a queue to put task ids
9

10 """
11 await asyncio.sleep(self.ts)
12 scheduler = self.scheduler
13 sta_assigned = self.sta_assigned
14 dt_status = self.dt_status
15 dt_group = self.dt_group
16 msg_in = []
17 try:
18 tasks = []
19 W=0
20 if not queue_tasks.empty():
21 for i in range(queue_tasks.qsize()):
22 ids = queue_tasks.get()
23 scheduler.status_tasks[ids] = True
24 scheduler.sta_init[ids] = True
(continues on next page)

15
 Async Task Tools Documentation, Release 0.5

(continued from previous page)

25 if ids in scheduler.stations.keys():
26 q=0
27 for ipt in scheduler.proc_tasks.keys():
28 q+=1
29 if len(scheduler.proc_tasks[ipt])<scheduler.lnproc and \
30 not ids in sta_assigned:
31 if dt_status == 'GROUP':
32 if scheduler.stations[ids]['code'] in dt_group:
33 scheduler.add_task(ids, ipt)
34 scheduler.set_init(ids)
35 sta_assigned.append(ids)
36 ans="TASK %s ADDED TO %s" % (ids, ipt)
37 elif dt_status == 'ALL':
38 scheduler.add_task(ids, ipt)
39 scheduler.set_init(ids)
40 sta_assigned.append(ids)
41 ans = "TASK %s ADDED TO %s" % (ids, ipt)
42 queue_tasks.task_done()
43 except Exception as exec:
44 bprint("Error en asignación de tareas a procesador: %s" %exec)
45 raise exec

16 Chapter 5. The Walking Stick for The Old Scheduler: The Assignator
 CHAPTER 6

Code’s Documentation

These documentation came from the source code.

6.1 The Async While to be a Wheel

Those are the main functions to enable the async while. The core functions to create asynchronous code. Are like the
gears for a big future machine.
tasktools.taskloop.coromask(coro, args, fargs)
A coroutine that mask another coroutine callback with args, and a function callbacks who manage input/output
of corotine callback
Parameters
• coro – is a coroutine object defined by the developer
• args – the list of arguments to run on the corotine coro
• fargs – the function that process the input and create an output related with the coro result
Returns a result, is a list of the elements for future argument
tasktools.taskloop.renew(task, coro, fargs, *args)
A simple function who manages the scheduled task and set the renew of the task
Parameters
• task – is a Future initialized coroutine but not executed yet
• coro – is the corutine to renew when the first is finished
• fargs – the function to process input/output
• args – the unpacked list of extra arguments
tasktools.taskloop.renew_quamash(task, coro, fargs, loop, *args)
A simple function who manages the scheduled task and set the renew of the task. The same as renew but it can
be used with different event loop like quamash for qt

17
Async Task Tools Documentation, Release 0.5

Parameters
• task – is a Future initialized coroutine but not executed yet
• coro – is the corutine to renew when the first is finished
• fargs – the function to process input/output
• loop – a different event loop from asyncio
• args – the unpacked list of extra arguments
tasktools.taskloop.simple_fargs(_in, obtained)
Simple function who can be used in callback on coromask, the inputs are /_in/ and /obtained/ value from the
coroutine executed. Return _in
_in the input list
Parameters obtained – the object that came from the result of coroutine execution
Returns _in
tasktools.taskloop.simple_fargs_out(_in, obtained)
Simple function who can be used in callback on coromask, the inputs are /_in/ and /obtained/ value from the
coroutine executed. Return obtained
Parameters
• _in – the input list
• obtained – the object that came from the result of coroutine execution
Returns obtained

6.2 The Scheduler explained Step by Step

The Scheduler is a like abstract class the enables a set of features that you have to inherit to your own class.
class tasktools.scheduler.TaskScheduler(*args, **kwargs)
This class is a generic tasks scheduler that uses asyncio and multiprocessing For every new process activate M
idle coro tasks until when are assigned by a function that send an id
Parameters
• args – an unpacked list
• kwargs – an unpacked dictionary with at least {ipt dict, ico dict, assigned_task dict, nproc
int, sta_init dict}, all in shared memory manager
add_task(ids, ipt)
Add an ids task to some ipt process
Parameters
• ids – the key of a source
• ipt – the key or identifier of a process
manage_tasks(ipt)
A method to manage the tasks assigned to ipt process
Initialize an event loop, and assign idle tasks for this process
Create the tasks for every source assigned to this process. Check the cases unidirectional and bidirectional.

18 Chapter 6. Code’s Documentation

 Async Task Tools Documentation, Release 0.5

Parameters ipt – the key or identifier of a process

process_sta_manager(ipt)
Manage asignation of station to task inside ipt process
Parameters ipt – the key of the process
Returns a list object with ipt value
process_sta_task(ipt, ico, *args)
Process gather data for ids station
This coroutine is executed for both cases: unidirectional and bidirectional
Parameters
• ipt – is an ipt value to work with the ipt process
• ico – is an ico value to work with the ico object
• args – an unpacked list to execute the run_task
Returns the first two inputs and the unpacked result
process_sta_task_n2s(ipt, ico, *args)
Process channel network to service data for ids station
This coroutine is executed for bidirectional case, on network to service direction
Parameters
• ipt – is an ipt value to work with the ipt process
• ico – is an ico value to work with the ico object
• args – an unpacked list to execute the run_task
Returns the first two inputs and the unpacked result
process_sta_task_s2n(ipt, ico, *args)
Process channel service 2 network for ids station
This coroutine is executed for bidirectional case, on service to network direction
Parameters
• ipt – is an ipt value to work with the ipt process
• ico – is an ico value to work with the ico object
• args – an unpacked list to execute the run_task
Returns the first two inputs and the unpacked result
run_task(*args)
A default coroutine that await .1 secs every time
Parameters args – a unpacked list
Returns the same list
set_ico(uin=4)
Defines a new id for task related to collect data inside a worker, check if exists
Parameters uin – is an optional int to define the length of the keys for ico dict
Returns a new key from ico

6.2. The Scheduler explained Step by Step 19

Async Task Tools Documentation, Release 0.5

set_init_args()
Set the initial list of arguments
In your class you have to rewrite this.
Returns a list of initial arguments
set_ipt(uin=4)
Defines a new id for relation process-collect_task, check if exists
Parameters uin – is an optional int to define the length of the keys for ipt dict
Returns a new key from ipt
set_new_run_task(**coros_callback)
Is a function that you have to call to define the corutines which you need to execute on the wheel (gear)
Is recomended the use at the end of the __init__ method on your new class.
Parameters coros_callback – is a dictionary with the corutines, that depends of what you
need:

• A unidirectional machine: you have to set only the run_task key

• A bidirectional machine: you ahve to set also net2service and service2net keys

set_pst(ids, args)
Set the factory for the wheel’s array
In your class you have to rewrite this.
Parameters
• ids – key of the source
• args – list of arguments
Returns a different list of future arguments

6.3 The Assignator role

This class works fine, you don’t have to implement a subclass of this, you have to use your Scheduler class on it.
class tasktools.assignator.TaskAssignator(scheduler, queue_tasks, sta_assigned, dt_status,
dt_group, *args, **kwargs)
Manage the tasks assigned to a TaskScheduler instance
Parameters
• scheduler – A TaskScheduler instance
• queue_tasks – a queue
• sta_assigned – a dict managed for multiprocessing
• dt_status – a string that select the kind of group {GROUP, ALL}
• dt_group – a list of the group selected
• args – some extra args
• kwargs – some extra keyword args

20 Chapter 6. Code’s Documentation

 Async Task Tools Documentation, Release 0.5

new_process(queue_tasks)
This coroutine activate a process with new station task Check every sseconds the queue_tasks if there are
new stations or tasksto add.
Works on a async wheel
Parameters queue_tasks – a queue to put task ids
new_process_task()
This function allows the system to call the coroutine that add a new_process in an asynchronous loop the
wheel

6.3. The Assignator role 21

Async Task Tools Documentation, Release 0.5

22 Chapter 6. Code’s Documentation

 CHAPTER 7

To do

What more
?
• Examples

23
Async Task Tools Documentation, Release 0.5

24 Chapter 7. To do
 CHAPTER 8

Indices and tables

• genindex
• modindex
• search

25
Async Task Tools Documentation, Release 0.5

26 Chapter 8. Indices and tables

 Index

A set_init_args() (tasktools.scheduler.TaskScheduler
add_task() (tasktools.scheduler.TaskScheduler method), method), 19
18 set_ipt() (tasktools.scheduler.TaskScheduler method), 20
set_new_run_task() (tasktools.scheduler.TaskScheduler
C method), 20
coromask() (in module tasktools.taskloop), 17 set_pst() (tasktools.scheduler.TaskScheduler method), 20
simple_fargs() (in module tasktools.taskloop), 18
M simple_fargs_out() (in module tasktools.taskloop), 18
manage_tasks() (tasktools.scheduler.TaskScheduler
method), 18
T
TaskAssignator (class in tasktools.assignator), 20
N TaskScheduler (class in tasktools.scheduler), 18
new_process() (tasktools.assignator.TaskAssignator
method), 20
new_process_task() (tasktools.assignator.TaskAssignator
method), 21

P
process_sta_manager() (task-
tools.scheduler.TaskScheduler method),
19
process_sta_task() (tasktools.scheduler.TaskScheduler
method), 19
process_sta_task_n2s() (task-
tools.scheduler.TaskScheduler method),
19
process_sta_task_s2n() (task-
tools.scheduler.TaskScheduler method),
19

R
renew() (in module tasktools.taskloop), 17
renew_quamash() (in module tasktools.taskloop), 17
run_task() (tasktools.scheduler.TaskScheduler method),
19

S
set_ico() (tasktools.scheduler.TaskScheduler method), 19

27