Posts

Showing posts with the label nonetype

NoneType error during multi-page scrape

NoneType error during multi-page scrape I'm working on a web scraper and am close to getting what I need, but I can't figure out why I'm getting a NoneType error all of a sudden after it finishes scraping the fourth page (of 204). Here's my code: script_path = os.path.dirname(os.path.realpath(__file__)) driver = webdriver.PhantomJS(executable_path="/usr/local/bin/bin/phantomjs", service_args=['--ignore-ssl-errors=true', '--ssl-protocol=any']) case_list = #this function launches the headless browser and gets us to the first page of results, which we'll scrape using main def search(): driver.get('https://www.courts.mo.gov/casenet/cases/nameSearch.do') if 'Service Unavailable' in driver.page_source: log('Casenet website seems to be down. Receiving "service unavailable"') driver.quit() gc.collect() return False time.sleep(2) court = Select(driver.find_element_by_i...