I want to scrape some data from a page which is in a table. So I am only bothered about the data in the table. Earlier I was using Mechanize, but I found sometimes some of the data are missing, especially in the bottom of the table. Googling, I found out that it may be due to mechanize not handling Jquery/Ajax.
So I switched to Selenium today. How do I wait for one and only one table to load completely and then extract all links from that table using selenium and python? If I wait for complete page to load, it is taking some time. I want to ensure that only data in the table is loaded. My current code:
driver = webdriver.Firefox() for page in range(1, 2): driver.get("http://somesite.com/page/"+str(page)) table = driver.find_element_by_css_selector('div.datatable') links = table.find_elements_by_tag_name('a') for link in links: print link.text
Best Answer
Use
WebDriverWait
to wait until the table is located:This would be an explicit wait.
Alternatively, you can make the driver wait implicitly: