面试题：复杂场景下Python文件操作的上下文管理优化

Question

在多线程或异步编程环境中，对大量文件进行读写操作时，Python的上下文管理可能会遇到资源竞争等问题。请描述如何优化上下文管理机制以应对这些复杂场景，并给出相应的代码示例或伪代码。

Answer 1

优化方法

1. 使用锁机制：在涉及共享资源（如文件读写）时，通过锁来确保同一时间只有一个线程或异步任务能够访问该资源，避免资源竞争。Python 中 threading 模块提供 Lock 类，asyncio 模块提供 Lock 用于异步编程。
2. 线程池/进程池：使用线程池或进程池来管理并发任务，避免过多线程或进程导致系统资源耗尽。concurrent.futures 模块提供了 ThreadPoolExecutor 和 ProcessPoolExecutor 类。
3. 异步I/O：在异步编程中，使用异步I/O操作（如 aiofiles 库），可以在I/O操作等待时释放控制权，提高效率。

代码示例

多线程环境下使用锁优化文件读写

import threading
import time

lock = threading.Lock()


def write_file(file_path, content):
    with lock:
        with open(file_path, 'w') as f:
            f.write(content)
        time.sleep(1)


threads = []
file_paths = ['file1.txt', 'file2.txt', 'file3.txt']
contents = ['content1', 'content2', 'content3']

for i in range(len(file_paths)):
    t = threading.Thread(target=write_file, args=(file_paths[i], contents[i]))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

异步编程环境下使用异步I/O和锁优化文件读写

import asyncio
import aiofiles


async def async_write_file(file_path, content, lock):
    async with lock:
        async with aiofiles.open(file_path, 'w') as f:
            await f.write(content)


async def main():
    lock = asyncio.Lock()
    tasks = []
    file_paths = ['file1.txt', 'file2.txt', 'file3.txt']
    contents = ['content1', 'content2', 'content3']

    for i in range(len(file_paths)):
        task = asyncio.create_task(async_write_file(file_paths[i], contents[i], lock))
        tasks.append(task)

    await asyncio.gather(*tasks)


if __name__ == "__main__":
    asyncio.run(main())

使用线程池优化多线程文件读写

import concurrent.futures
import time


def write_file(file_path, content):
    with open(file_path, 'w') as f:
        f.write(content)
    time.sleep(1)


file_paths = ['file1.txt', 'file2.txt', 'file3.txt']
contents = ['content1', 'content2', 'content3']

with concurrent.futures.ThreadPoolExecutor() as executor:
    executor.map(write_file, file_paths, contents)