How to Run Multiple Browsers in Parallel#

Which Methods Can Boost the Performance?#

You can run Browserist as a normal, linear script or with various methods for concurrent processing:

• Asynchronous
• Multi-threading
• Multi-processing

Which Method Is Faster?#

Multi-processing and multi-threading are the fastest methods, sometimes twice as fast as running the same job in linear or asynchronous mode. For instance, measuring execution time of the code examples below yield the results like this in seconds:

Find code examples of the tests below.

Basic Code Example#

Imagine that you want to scrape a website with multiple browser types: Chrome, Edge, Firefox. A simplified example of this:

1. Open browser instance
2. Load website example.com and do something
3. Close browser instance

How the code could look:

1
2
3
4
5
6
7

from browserist import Browser

with Browser() as browser:
    print("1. Opening X browser")
    browser.open.url("https://example.com")
    print("2. Page loaded with X browser")
    print("3. Closing X browser")

This will print the following to the terminal:

1. Opening X browser
2. Page loaded with X browser
3. Closing X browser

Test Code Examples#

Let's try this with four different methods from linear to concurrent processing and run the tests with three different browsers (Chrome, Edge, Firefox):

1. Linear
2. Asynchronous
3. Multi-threading
4. Multi-processing

Even Faster with Headless and Disable Images#

Gain even more performance by running the browsers in headless mode and with images disabled, including the added benefit that headless mode allows you to run the job as a background task while doing something else.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

from browserist import Browser, BrowserSettings, BrowserType

settings = BrowserSettings(
    type=BrowserType.CHROME,
    headless=True,
    disable_images=True)

browser = Browser(settings)

with Browser() as browser:
    print("1. Opening X browser")
    browser.open.url("https://example.com")
    print("2. Page loaded with X browser")
    print("3. Closing X browser")

Results in seconds and compared to previous method: