麻烦用BS4刮痧网站(Trouble Scraping site with BS4)
通常我可以编写一个可以用于抓取的脚本,但是我一直在为这个正在研究的这个研究项目所征集的桌子抓取这个网站有些困难。 我打算在输入目标状态的URL之前验证脚本是否在一个状态下工作。
import requests import bs4 as bs url = ("http://programs.dsireusa.org/system/program/detail/284") dsire_get = requests.get(url) soup = bs.BeautifulSoup(dsire_get.text,'lxml') table = soup.findAll('div', {'data-ng-controller': 'DetailsPageCtrl'}) print(table) #I'm printing "Table" just to ensure that the table information I'm looking for is within this sections
我不确定该网站是否试图阻止人们进行抓取,但是我想要抓取的所有信息都在“&quot”内,如果您看看Table输出的内容。
usually I'm able to write a script that works for scraping, but I've been having some difficulty scraping this site for the table enlisted for this research project I'm working on. I'm planning to verify the script working on one State before entering the URL of my targeted states.
import requests import bs4 as bs url = ("http://programs.dsireusa.org/system/program/detail/284") dsire_get = requests.get(url) soup = bs.BeautifulSoup(dsire_get.text,'lxml') table = soup.findAll('div', {'data-ng-controller': 'DetailsPageCtrl'}) print(table) #I'm printing "Table" just to ensure that the table information I'm looking for is within this sections
I'm not sure if the site is attempting to block people from scraping, but all the info that I'm looking to grab is within """if you look what Table outputs.
原文:https://stackoverflow.com/questions/44954845
最满意答案
你可以这样做:
a = A.find(1) b = B.new # Modify b a.update_attributes(b.attributes.delete_if{ |k,v| %w(id created_at updated_at).include?(k) })
You can do something like this:
a = A.find(1) b = B.new # Modify b a.update_attributes(b.attributes.delete_if{ |k,v| %w(id created_at updated_at).include?(k) })
相关问答
更多-
linux 结构体命名规则[2022-12-20]
record -
createRecord返回记录本身,因为它不需要做网络请求。 要执行网络请求,您需要在新记录上调用.save()并返回一个promise。 这将有效,但不会将您的记录保存到服务器: createNewProject: function() { let project = this.store.createRecord('project', { title: this.get('title'), description: ...
-
INSERT INTO YourTable(val1, val2, val3 ...) VALUES(@val1, @val2, @val3...); SELECT SCOPE_IDENTITY(); 不要在每个语句的末尾忘记分号。 INSERT INTO YourTable(val1, val2, val3 ...) VALUES(@val1, @val2, @val3...); SELECT SCOPE_IDENTITY(); Don't forget the semicolons at the ...
-
问题1我能想到的最简单的解决方案是(使用您的代码作为基础): var est = (from e in MyDB.Detail_Table where e.SRC_System == Location where e.Segment_ID == Segment_ID where e.Transaction_Date >= Start_Date where e.Transaction_Date <= End_Date group e by e.Segment_ID into ...
-
SQLAlchemy - 如何查看新创建记录的主键ID?(SQLAlchemy - How do I see a primary key id for a newly created record?)[2022-07-15]
db.session.flush() #id is a Python builtin... id_= foo.id 发生的事情是,刷新前的原始代码只在程序中,db中没有。 id列可能是在插入时分配的自动生成字段。 插入记录后( flush将更改写入db),SQLAlchemy基本上从执行select id并返回结果(各种数据库使用各种机制)。 现在你已经填充了id。 提交和回滚在性质上与flush不同。 它们会影响数据库中已有的内容。 另外,标记您的评论我需要在DB中添加其他记录时 ... -
好吧,我回答了自己的一部分问题。 对Inserted事件的以下更改将允许我查看新插入的数据。 我有另一个按钮在搜索表单的空模板中添加新记录。 这就是为什么我要更改要插入的模式,因为默认模式是只读。 这将让我插入数据,但插入后根本不显示表格。 不确定为什么插入的事件没有正确地踢。 Protected Sub PatronDS_Inserted(sender As Object, e As SqlDataSourceStatusEventArgs) Handles PatronDS.Inserted ...
-
Sequelize需要将已存在的记录与新创建的记录相关联(Sequelize need to associate an already existing record to newly created ones)[2022-02-24]
findById返回一个promise,就像create一样 - 你必须等待查询完成 function addComputer(inv, pur, phys, mod, sup, manu, stat, purC, acD, useLife, warr, dep, cur, netRe, fair, pic, desc, num, use) { sequelize.Promise.all([ Computer.create({ serial: num, ... -
你可以用清除教义缓存 EntityManager::clear() 见http://doctrine-orm.readthedocs.org/en/latest/reference/working-with-objects.html (搜索clear方法) The solution for this ended up being fairly simple. After trying clear(), refresh() and force clearing Doctrine's cache I was ...
-
在Sugar 7中执行此操作的一种方法是覆盖CreateView。 这是一个CustomCreateView的示例,它在成功创建新帐户之后,但在Sugar对创建的记录作出反应之前,在警报消息中输出新id。 custom/modules/Accounts/clients/base/views/create/create.js : ({ extendsFrom: 'CreateView', // This initialize function override does nothing ex ...
-
你可以这样做: a = A.find(1) b = B.new # Modify b a.update_attributes(b.attributes.delete_if{ |k,v| %w(id created_at updated_at).include?(k) }) You can do something like this: a = A.find(1) b = B.new # Modify b a.update_attributes(b.attributes.delete_if{ |k, ...