一文搞懂Python asyncio异步编程的核心原理与实战技巧

Python异步编程

在现代网络应用开发中，高并发处理能力是衡量系统性能的重要指标。Python的asyncio库自3.4版本引入以来，已经成为构建高性能异步应用的标准方案。无论是Web服务器、爬虫系统还是微服务架构，asyncio都能以极低的资源开销实现高效的I/O并发。本文将从事件循环原理出发，结合实际代码示例，帮助你全面掌握Python asyncio异步编程。

Table of Contents

一、asyncio的核心概念：事件循环与协程

asyncio的核心是事件循环（Event Loop），它负责调度和执行所有的异步任务。协程（Coroutine）是使用

async def

定义的函数，调用后不会立即执行，而是返回一个协程对象，需要通过事件循环来驱动运行。



1
2
3
4
5
6
7
8
9
10
11
12
<span class="kn">import</span><span class="w"> </span><span class="nn">asyncio</span>



<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">say_hello</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">delay</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>

    <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">delay</span><span class="p">)</span>

    <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Hello, </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">! (after </span><span class="si">{</span><span class="n">delay</span><span class="si">}</span><span class="s2">s)&quot;</span><span class="p">)</span>

    <span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;greeted </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">&quot;</span>



<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">main</span><span class="p">():</span>

    <span class="n">result</span> <span class="o">=</span> <span class="k">await</span> <span class="n">say_hello</span><span class="p">(</span><span class="s2">&quot;World&quot;</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>

    <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Result: </span><span class="si">{</span><span class="n">result</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>



<span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span>

await

关键字是协程的关键，它将控制权交还给事件循环，让其他协程有机会执行。当await的操作完成后，事件循环会恢复该协程的执行。

二、并发执行多个任务：gather与TaskGroup

实际开发中，我们通常需要同时执行多个异步操作。

1	asyncio.gather()

是最常用的并发执行方式，它会同时启动所有协程并等待全部完成。

代码并发执行



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<span class="kn">import</span><span class="w"> </span><span class="nn">asyncio</span>

<span class="kn">import</span><span class="w"> </span><span class="nn">time</span>



<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">fetch_data</span><span class="p">(</span><span class="n">url</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">delay</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">:</span>

    <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;开始请求: </span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>

    <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">delay</span><span class="p">)</span>

    <span class="k">return</span> <span class="p">{</span><span class="s2">&quot;url&quot;</span><span class="p">:</span> <span class="n">url</span><span class="p">,</span> <span class="s2">&quot;status&quot;</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span> <span class="s2">&quot;delay&quot;</span><span class="p">:</span> <span class="n">delay</span><span class="p">}</span>



<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">main</span><span class="p">():</span>

    <span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>

    <span class="n">urls</span> <span class="o">=</span> <span class="p">[</span>

        <span class="p">(</span><span class="s2">&quot;https://api.example.com/users&quot;</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span>

        <span class="p">(</span><span class="s2">&quot;https://api.example.com/posts&quot;</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>

        <span class="p">(</span><span class="s2">&quot;https://api.example.com/comments&quot;</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span>

    <span class="p">]</span>

    <span class="n">tasks</span> <span class="o">=</span> <span class="p">[</span><span class="n">fetch_data</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">delay</span><span class="p">)</span> <span class="k">for</span> <span class="n">url</span><span class="p">,</span> <span class="n">delay</span> <span class="ow">in</span> <span class="n">urls</span><span class="p">]</span>

    <span class="n">results</span> <span class="o">=</span> <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="n">tasks</span><span class="p">)</span>



    <span class="n">elapsed</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start</span>

    <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;总耗时: </span><span class="si">{</span><span class="n">elapsed</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2">s（而非串行的6s）&quot;</span><span class="p">)</span>

    <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span><span class="p">:</span>

        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;  </span><span class="si">{</span><span class="n">r</span><span class="p">[</span><span class="s1">&#39;url&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2"> -&gt; </span><span class="si">{</span><span class="n">r</span><span class="p">[</span><span class="s1">&#39;status&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>



<span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span>

Python 3.11引入了，它提供了更安全的并发管理，能自动处理异常传播：



1
2
3
4
5
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">main_taskgroup</span><span class="p">():</span>

    <span class="k">async</span> <span class="k">with</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">TaskGroup</span><span class="p">()</span> <span class="k">as</span> <span class="n">tg</span><span class="p">:</span>

        <span class="n">task1</span> <span class="o">=</span> <span class="n">tg</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">fetch_data</span><span class="p">(</span><span class="s2">&quot;https://api.a.com&quot;</span><span class="p">,</span> <span class="mi">2</span><span class="p">))</span>

        <span class="n">task2</span> <span class="o">=</span> <span class="n">tg</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">fetch_data</span><span class="p">(</span><span class="s2">&quot;https://api.b.com&quot;</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>

    <span class="nb">print</span><span class="p">(</span><span class="n">task1</span><span class="o">.</span><span class="n">result</span><span class="p">(),</span> <span class="n">task2</span><span class="o">.</span><span class="n">result</span><span class="p">())</span>

三、异步上下文管理器与异步迭代器

asyncio不仅支持异步函数，还支持异步上下文管理器（

1	async with

）和异步迭代器（

async for

），这在管理数据库连接池、消息队列等资源时非常实用。



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
<span class="kn">import</span><span class="w"> </span><span class="nn">asyncio</span>



<span class="k">class</span><span class="w"> </span><span class="nc">AsyncConnectionPool</span><span class="p">:</span>

    <span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">max_size</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">5</span><span class="p">):</span>

        <span class="bp">self</span><span class="o">.</span><span class="n">max_size</span> <span class="o">=</span> <span class="n">max_size</span>

        <span class="bp">self</span><span class="o">.</span><span class="n">_pool</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">Queue</span><span class="p">(</span><span class="n">maxsize</span><span class="o">=</span><span class="n">max_size</span><span class="p">)</span>



    <span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">connect</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>

        <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">max_size</span><span class="p">):</span>

            <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">_pool</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;conn_</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>



    <span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="fm">__aenter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>

        <span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">_pool</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>

        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;获取连接: </span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>

        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span>



    <span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="fm">__aexit__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">exc_type</span><span class="p">,</span> <span class="n">exc_val</span><span class="p">,</span> <span class="n">exc_tb</span><span class="p">):</span>

        <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">_pool</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="p">)</span>

        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;归还连接: </span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>



<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">main</span><span class="p">():</span>

    <span class="n">pool</span> <span class="o">=</span> <span class="n">AsyncConnectionPool</span><span class="p">(</span><span class="n">max_size</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>

    <span class="k">await</span> <span class="n">pool</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span>

    <span class="k">async</span> <span class="k">with</span> <span class="n">pool</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>

        <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.5</span><span class="p">)</span>

        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;使用 </span><span class="si">{</span><span class="n">conn</span><span class="si">}</span><span class="s2"> 执行查询&quot;</span><span class="p">)</span>



<span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span>

四、异步生成器与流式数据处理

异步生成器使用

async for

逐个产出数据，非常适合处理流式数据、分页API响应等场景：



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
<span class="kn">import</span><span class="w"> </span><span class="nn">asyncio</span>



<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">async_range</span><span class="p">(</span><span class="n">n</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>

    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>

        <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>

        <span class="k">yield</span> <span class="n">i</span>



<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">paginate_api</span><span class="p">(</span><span class="n">page_size</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">10</span><span class="p">,</span> <span class="n">total_pages</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">5</span><span class="p">):</span>

    <span class="k">for</span> <span class="n">page</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">total_pages</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span>

        <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.3</span><span class="p">)</span>

        <span class="n">data</span> <span class="o">=</span> <span class="p">[</span><span class="sa">f</span><span class="s2">&quot;item_</span><span class="si">{</span><span class="n">page</span><span class="si">}</span><span class="s2">_</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">&quot;</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">page_size</span><span class="p">)]</span>

        <span class="k">yield</span> <span class="p">{</span><span class="s2">&quot;page&quot;</span><span class="p">:</span> <span class="n">page</span><span class="p">,</span> <span class="s2">&quot;data&quot;</span><span class="p">:</span> <span class="n">data</span><span class="p">}</span>



<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">main</span><span class="p">():</span>

    <span class="k">async</span> <span class="k">for</span> <span class="n">num</span> <span class="ow">in</span> <span class="n">async_range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>

        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;收到数据: </span><span class="si">{</span><span class="n">num</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>

    <span class="k">async</span> <span class="k">for</span> <span class="n">page_result</span> <span class="ow">in</span> <span class="n">paginate_api</span><span class="p">():</span>

        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;第</span><span class="si">{</span><span class="n">page_result</span><span class="p">[</span><span class="s1">&#39;page&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">页: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">page_result</span><span class="p">[</span><span class="s1">&#39;data&#39;</span><span class="p">])</span><span class="si">}</span><span class="s2">条记录&quot;</span><span class="p">)</span>



<span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span>

五、实战技巧与常见陷阱

在使用asyncio时，有几个重要的实践建议：

1. 避免在协程中使用阻塞操作：如

1	time.sleep()

、

1	requests.get()

等会阻塞整个事件循环。应使用

aiohttp

、

aiofiles

等异步替代库。

2. 使用信号量控制并发数：防止同时打开过多连接导致资源耗尽。



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<span class="kn">import</span><span class="w"> </span><span class="nn">asyncio</span>

<span class="kn">import</span><span class="w"> </span><span class="nn">aiohttp</span>



<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">fetch_with_limit</span><span class="p">(</span><span class="n">sem</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">session</span><span class="p">):</span>

    <span class="k">async</span> <span class="k">with</span> <span class="n">sem</span><span class="p">:</span>

        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;请求: </span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>

        <span class="k">async</span> <span class="k">with</span> <span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="k">as</span> <span class="n">resp</span><span class="p">:</span>

            <span class="k">return</span> <span class="k">await</span> <span class="n">resp</span><span class="o">.</span><span class="n">text</span><span class="p">()</span>



<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">main</span><span class="p">():</span>

    <span class="n">sem</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">Semaphore</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>

    <span class="n">urls</span> <span class="o">=</span> <span class="p">[</span><span class="sa">f</span><span class="s2">&quot;https://httpbin.org/delay/</span><span class="si">{</span><span class="n">i</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="mi">3</span><span class="si">}</span><span class="s2">&quot;</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">20</span><span class="p">)]</span>

    <span class="k">async</span> <span class="k">with</span> <span class="n">aiohttp</span><span class="o">.</span><span class="n">ClientSession</span><span class="p">()</span> <span class="k">as</span> <span class="n">session</span><span class="p">:</span>

        <span class="n">tasks</span> <span class="o">=</span> <span class="p">[</span><span class="n">fetch_with_limit</span><span class="p">(</span><span class="n">sem</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">session</span><span class="p">)</span> <span class="k">for</span> <span class="n">url</span> <span class="ow">in</span> <span class="n">urls</span><span class="p">]</span>

        <span class="n">results</span> <span class="o">=</span> <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="n">tasks</span><span class="p">,</span> <span class="n">return_exceptions</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>

        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;完成 </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">results</span><span class="p">)</span><span class="si">}</span><span class="s2"> 个请求&quot;</span><span class="p">)</span>

3. 超时控制：使用

1	asyncio.wait_for()

或

1	asyncio.timeout()

避免协程无限挂起。



1
2
3
4
5
6
7
8
9
10
11
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">slow_operation</span><span class="p">():</span>

    <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>

    <span class="k">return</span> <span class="s2">&quot;done&quot;</span>



<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">main_timeout</span><span class="p">():</span>

    <span class="k">try</span><span class="p">:</span>

        <span class="n">result</span> <span class="o">=</span> <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">wait_for</span><span class="p">(</span><span class="n">slow_operation</span><span class="p">(),</span> <span class="n">timeout</span><span class="o">=</span><span class="mf">3.0</span><span class="p">)</span>

    <span class="k">except</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">TimeoutError</span><span class="p">:</span>

        <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;操作超时，已取消&quot;</span><span class="p">)</span>



<span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main_timeout</span><span class="p">())</span>

4. 正确处理异常：使用

1	return_exceptions=True

或

TaskGroup

来妥善处理并发任务中的异常，避免单个任务失败导致所有任务被取消。

代码调试

总结

Python asyncio通过事件循环和协程机制，以同步代码的编写风格实现了高效的异步I/O操作。核心要点包括：使用

async def

定义协程、

await

等待异步操作完成、

gather()

或