使用curl与Python请求(Using curl vs Python requests)

 在进行网站刮擦时，最好是：使用curl，还是使用Python的请求库？ 
 
 我最初计划使用请求并明确指定用户代理。 但是，当我使用它时，我经常会收到“HTTP 429太多请求”错误，而使用curl时，似乎可以避免这种情况。 
 
 我需要更新10,000个标题的元数据信息，我需要一种方法来以并行方式提取每个标题的信息。 
 
 使用每种信息来提取信息的利弊是什么？ 

When doing a scrape of a site, which would be preferable: using curl, or using Python's requests library?
 
I originally planned to use requests and explicitly specify a user agent. However, when I use this I often get an "HTTP 429 too many requests" error, whereas with curl, it seems to avoid that.
 
I need to update metadata information on 10,000 titles, and I need a way to pull down the information for each of the titles in a parallelized fashion.
 
What are the pros and cons of using each for pulling down information?

原文：https://stackoverflow.com/questions/14552088