Python – How to Run Headless Chrome with Selenium

google-chromegoogle-chrome-headlesspythonselenium-chromedriverselenium-webdriver

I'm trying some stuff out with selenium, and I really want my script to run quickly.

I thought that running my script with headless Chrome would make it faster.

First, is that assumption correct, or does it not matter if I run my script with a headless driver?

I want headless Chrome to work, but somehow it isn't working correctly. I tried different things, and most suggested that it would work as said here in the October update:

How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?

But when I tried that, I saw weird console output, and it still doesn't seem to work.

Any tips appreciated.

Best Answer

To run chrome-headless just add --headless via chrome_options.add_argument, e.g.:

from selenium import webdriver 
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
# chrome_options.add_argument("--disable-extensions")
# chrome_options.add_argument("--disable-gpu")
# chrome_options.add_argument("--no-sandbox") # linux only
chrome_options.add_argument("--headless=new") # for Chrome >= 109
# chrome_options.add_argument("--headless")
# chrome_options.headless = True # also works
driver = webdriver.Chrome(options=chrome_options)
start_url = "https://duckgo.com"
driver.get(start_url)
print(driver.page_source.encode("utf-8"))
# b'<!DOCTYPE html><html xmlns="http://www....
driver.quit()

So my thought is that running it with headless chrome would make my script faster.

Try using chrome options like --disable-extensions or --disable-gpu and benchmark it, but I wouldn't count with substantial improvement.

References: headless-chrome

Related Solutions

Selenium – Working with Chrome but Not Headless Chrome

I had a same experience like you, and solved it by using xvfb and pyvirtualdisplay.

I use chromedrive=v2.3.1, chrome-browser=v60 and Selenium=3.4.3

In Headless chrome, some of script seems not to work as expected.

Please refer to vpassapera's comment in https://gist.github.com/addyosmani/5336747.

How about try it like below,

from pyvirtualdisplay import Display

display = Display(visible=0, size=(800, 600))
display.start()

# Do Not use headless chrome option
# options.add_argument('headless')

url = 'https://10.11.227.21/tmui/'
driver.get(url + "login.jsp")

html_source = driver.page_source
print(html_source)

blocStatus = WebDriverWait(driver,    TIMEOUT).until(EC.presence_of_element_located((By.ID, "username")))
inputElement = driver.find_element_by_id("username")
inputElement.send_keys('actualLogin')
inputElement = driver.find_element_by_id("passwd")
inputElement.send_keys('actualPassword')
inputElement.submit()

display.stop()

xvfb is required to use "pyvortualdisplay"

$ sudo apt-get install -y xvfb

Selenium ChromeDriver – How to Configure ChromeDriver for Headless Mode

It should look like this:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')  # Last I checked this was necessary.
driver = webdriver.Chrome(CHROMEDRIVER_PATH, chrome_options=options)

This works for me using Python 3.6, I'm sure it'll work for 2.7 too.

Update 2018-10-26: These days you can just do this:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True
driver = webdriver.Chrome(CHROMEDRIVER_PATH, options=options)

Update 2023-05-22: Chrome’s Headless mode got an upgrade and now both headless and headful modes have been unified under the same implementation. See https://developer.chrome.com/articles/new-headless/

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('--headless=new')
driver = webdriver.Chrome(CHROMEDRIVER_PATH, options=options)

To try the new Headless mode, pass the --headless=new command-line flag:

chrome --headless=new

For now, the old Headless mode is still available via:

chrome --headless=old

Currently, passing the --headless command line flag without an explicit value still activates the old Headless mode — but we plan to change this default to new Headless over time.

Best Answer

Related Solutions

Selenium – Working with Chrome but Not Headless Chrome

Selenium ChromeDriver – How to Configure ChromeDriver for Headless Mode

Related Question