Clicking 'Show More' button on Google Scholar Results

white and black pyramid under blue sky during night time

I’m trying to scrape a google scholar page, but I can only get the first twenty results that are shown. I’m trying to use selenium to click ‘show more’ so that I can get the rest of the results. Here is what I have, however, it isn’t working (I have the URL stored in a variable):

driver = webdriver.Chrome(executable_path ="/Applications/chromedriver84")
element = driver.find_element_by_tag_name('button')

Any suggestions? Thanks in advance.

asked 3 hours ago

51 bronze badge

2 Answers

The element ‘show more’ in the page has id = ‘gsc_bpf_more’, since you know that, you may use Selenium expected_conditions to wait until the button is loaded on the page and then click it

from selenium import webdriver
from import By
from import WebDriverWait
from import expected_conditions as EC

driver = webdriver.Chrome(executable_path="/Applications/chromedriver84")

    #Wait up to 10s until the element is loaded on the page
    element = WebDriverWait(driver, 10).until(
        #Locate element by id
        EC.presence_of_element_located((By.ID, 'gsc_bpf_more'))

If you want to check what more Selenium EC can do, check this

answered 2 hours ago
Voz bonitaVoz bonita

161 bronze badge

Importing ActionChains you can call .click() function to click on elements at the screen:

    from selenium import webdriver
    from bs4 import BeautifulSoup
    from import ChromeDriverManager
    from selenium.webdriver import ActionChains
    from selenium.webdriver.common.keys import Keys
    import time

    driver = webdriver.Chrome(ChromeDriverManager().install())
    more = driver.find_element_by_class_name('gs_btnPD')
    for _ in range(0,5):

answered 2 hours ago
Luan TorresLuan Torres

978 bronze badges

Latest posts