无法启动新线程(Can't start new thread)

 我正在使用多个线程处理目录中的所有文件以并行处理文件。 它一切正常，除了线程似乎保持活着，因此进程的线程数上升，直到它达到1K左右的线程然后它抛出一个thread.error can't start new thread错误。 我知道这个错误是由线程数的操作系统级别限制引起的。 我似乎无法弄清楚错误的位置是保持线程活着。 任何想法？ 这是我的代码的最小版本。 
 
class Worker(Thread):
    def __init__(self, tasks):
        Thread.__init__(self)
        self.tasks = tasks
        self.daemon = True
        self.start()

def run(self):
    while True:
        func, args, kargs = self.tasks.get()
        try:
            func(*args, **kargs)
        except Exception, e: print e
        self.tasks.task_done()


class ThreadPool:
    def __init__(self, num_threads):
        self.tasks = Queue(num_threads)
        for _ in range(num_threads): Worker(self.tasks)

    def add_task(self, func, *args, **kargs):
        self.tasks.put((func, args, kargs))

    def wait_completion(self):
        self.tasks.join()


def foo(filename)
    pool = ThreadPool(32)
    iterable_data = process_file(filename)

    for data in iterable_data:
        pool.add_task(some_function, data)
    pool.wait_completion()

files = os.listdir(directory)
for file in files:
    foo(file)

I'm processing all the files in a directory using multiple threads to process files in parallel. It all works fine, except that threads seem to stay alive and so the thread count of the process goes up until it reaches 1K or so threads and then it throws a thread.error can't start new thread error. I know this error is caused by an OS-level limit on thread count. I can't seem to figure out where the bug is that is keeping the threads alive. Any idea? Here is a minimal version of my code.
 
class Worker(Thread):
    def __init__(self, tasks):
        Thread.__init__(self)
        self.tasks = tasks
        self.daemon = True
        self.start()

def run(self):
    while True:
        func, args, kargs = self.tasks.get()
        try:
            func(*args, **kargs)
        except Exception, e: print e
        self.tasks.task_done()


class ThreadPool:
    def __init__(self, num_threads):
        self.tasks = Queue(num_threads)
        for _ in range(num_threads): Worker(self.tasks)

    def add_task(self, func, *args, **kargs):
        self.tasks.put((func, args, kargs))

    def wait_completion(self):
        self.tasks.join()


def foo(filename)
    pool = ThreadPool(32)
    iterable_data = process_file(filename)

    for data in iterable_data:
        pool.add_task(some_function, data)
    pool.wait_completion()

files = os.listdir(directory)
for file in files:
    foo(file)

原文：https://stackoverflow.com/questions/16309435