python的分布式任務huey如何實現異步化任務講解

 本文我們來分享一個python的輕型的任務隊列程序，他可以讓python的分布式任務huey實現異步化任務，感興趣的朋友可以看看。

   

一個輕型的任務隊列，功能和相關的broker沒有celery強大，重在輕型，而且代碼讀起來也比較的簡單。 


關於huey的介紹:  （比celery輕型，比mrq、rq要好用 ！）

a lightweight alternative.

    written in python

    no deps outside stdlib, except redis (or roll your own backend)

    support for django

supports:

    multi-threaded task execution

    scheduled execution at a given time

    periodic execution, like a crontab

    retrying tasks that fail

    task result storage


安裝:

 代碼如下   Installing
huey can be installed very easily using pip.
 
pip install huey
huey has no dependencies outside the standard library, but currently the only fully-implemented queue backend it ships with requires redis. To use the redis backend, you will need to install the python client.
 
pip install redis
Using git
If you want to run the very latest, feel free to pull down the repo from github and install by hand.
 
git clone https://github.com/coleifer/huey.git
cd huey
python setup.py install
You can run the tests using the test-runner:
 
python setup.py test

關於huey的api，下面有詳細的介紹及參數介紹的。

 代碼如下   from huey import RedisHuey, crontab
 
huey = RedisHuey('my-app', host='redis.myapp.com')
 
@huey.task()
def add_numbers(a, b):
    return a + b
 
@huey.periodic_task(crontab(minute='0', hour='3'))
def nightly_backup():
    sync_all_data()

juey作為woker的時候，一些cli參數。 


常用的是:  

-l                  關於日志文件的執行 。

-w                 workers的數目，-w的數值大了，肯定是增加任務的處理能力

-p --periodic     啟動huey worker的時候，他會從tasks.py裡面找到 需要crontab的任務，會派出幾個線程專門處理這些事情。 

-n                  不啟動關於crontab裡面的預周期執行，只有你觸發的時候，才會執行周期星期的任務。 

--threads   意思你懂的。
1

 代碼如下   # 原文:     
The following table lists the options available for the consumer as well as their default values.
 
-l, --logfile
Path to file used for logging. When a file is specified, by default Huey will use a rotating file handler (1MB / chunk) with a maximum of 3 backups. You can attach your own handler (huey.logger) as well. The default loglevel is INFO.
-v, --verbose
Verbose logging (equates to DEBUG level). If no logfile is specified and verbose is set, then the consumer will log to the console. This is very useful for testing/debugging.
-q, --quiet
Only log errors. The default loglevel for the consumer is INFO.
-w, --workers
Number of worker threads, the default is 1 thread but for applications that have many I/O bound tasks, increasing this number may lead to greater throughput.
-p, --periodic
Indicate that this consumer process should start a thread dedicated to enqueueing “periodic” tasks (crontab-like functionality). This defaults to True, so should not need to be specified in practice.
-n, --no-periodic
Indicate that this consumer process should not enqueue periodic tasks.
-d, --delay
When using a “polling”-type queue backend, the amount of time to wait between polling the backend. Default is 0.1 seconds.
-m, --max-delay
The maximum amount of time to wait between polling, if using weighted backoff. Default is 10 seconds.
-b, --backoff
The amount to back-off when polling for results. Must be greater than one. Default is 1.15.
-u, --utc
Indicates that the consumer should use UTC time for all tasks, crontabs and scheduling. Default is True, so in practice you should not need to specify this option.
--localtime
Indicates that the consumer should use localtime for all tasks, crontabs and scheduling. Default is False.
Examples
 
Running the consumer with 8 threads, a logfile for errors only, and a very short polling interval:
 
huey_consumer.py my.app.huey -l /var/log/app.huey.log -w 8 -b 1.1 -m 1.0

任務隊列huey 是靠著redis來實現queue的任務存儲，所以需要咱們提前先把redis-server和redis-py都裝好。 安裝的方法就不說了，自己搜搜吧。 


我們首先創建下huey的鏈接實例 :

 代碼如下   # config.py
from huey import Huey
from huey.backends.redis_backend import RedisBlockingQueue
 
queue = RedisBlockingQueue('test-queue', host='localhost', port=6379)
huey = Huey(queue)

然後就是關於任務的，也就是你想讓誰到任務隊列這個圈子裡面，和celey、rq，mrq一樣，都是用tasks.py表示的。

 代碼如下   from config import huey # import the huey we instantiated in config.py
 
 
@huey.task()
def count_beans(num):
    print '-- counted %s beans --' % num

再來一個真正去執行的 。  main.py 相當於生產者，tasks.py相當於消費者的關系。  main.py負責喂數據。

 代碼如下   main.py
from config import huey  # import our "huey" object
from tasks import count_beans  # import our task
 
 
if __name__ == '__main__':
    beans = raw_input('How many beans? ')
    count_beans(int(beans))
    print 'Enqueued job to count %s beans' % beans