Google SERP Scraping With Python

last-update:2023-08-07 03:30:16

Here are two codes for Google SERP scraping with Python, so you can extract SERP data easily.

Google SERP Scraping With Python Code 1

import requests
from bs4 import BeautifulSoup

# Search keywords
search_term = "scrapingbypass"

# Request headers
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36"}

# Search pages of results
num_pages = 5

for page in range(0,num_pages):
    # Request URL
    if page == 0:
        url = f"{search_term}"
        url = f"{search_term}&start={page*10}"

    # Request
    response = requests.get(url, headers=headers)

    # Parse HTML
    soup = BeautifulSoup(response.content, "html.parser")

    # Extract search reult
    search_results =".yuRUbf")

    # Print title and link
    for result in search_results:
        title = result.select_one("h3").text
        link = result.select_one("a")["href"]
        print(f"{no}: {title}: {link}")

Google SERP Scraping With Python Code 2

# SERP scraper
import time
from selenium import webdriver
from import By
from selenium.webdriver.common.keys import Keys
from import WebDriverWait
from import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
# Set up WebDriver
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(options=options)
# Load Google search page
url = ''
# Search for keyword
search_box = driver.find_element(By.NAME, 'q')
search_term = 'scrapingbypass'
num = 1
# Scrape multiple pages
for page in range(1, 6):  # Scrape the first 5 pages of results
    # Wait for the search results page to load
        element_present = EC.presence_of_element_located((By.CSS_SELECTOR, '.g'))
        WebDriverWait(driver, 10).until(element_present)
    except TimeoutException:
        print("Timed out waiting for page to load")
    # Parse the search results
    search_results = driver.find_elements(By.CSS_SELECTOR, '.g')
    for result in search_results:
        link = result.find_element(By.CSS_SELECTOR, 'a').get_attribute('href')
        title = result.find_element(By.CSS_SELECTOR, 'h3').text
        num = num + 1
    # Click on the next page
        next_button = driver.find_element(By.CSS_SELECTOR, '#pnnext')
# Close the WebDriver

Code generator

last-update:2023-08-04 05:50:19

ScrapingBypass offers a code generator for users to generate code snippets online for commands such as cURLJavaScriptTypeScriptJavaPython to send HTTP requests.

You have to enter the request type, and URL, select the API version, programming language, API or proxy mode, and some optional settings, including proxy settings, query parameters, request headers, and request body.

You can then copy the code to run it in the corresponding application, or run it directly on the web page.

Create an account

last-update:2023-08-07 06:23:39

Register an account

Enter the register web page

Click "Register" on the navigation bar

Enter your account information

Enter your email, password, click "Send Captcha".

You will receive an email with a code like the following image:

Then enter the verification code received from your email.

Finally, click "Sign up".

Log in

Enter the login web page

On the register web page, click "Sign in".

Enter account information

Enter your email and password, then click "Sign in"

Change password

After login, on the "My Account" panel, you can see the "Account information" block.

Click "Change password"

ScrapingBypass change password

Or click you account email on the top right of the web page, then click "Change password".

Enter your current password and new password, then click "Submit".

Then you can get a message shows "Password changed".

Sign out

Click the account email on the top right of the web page, then click "Sign out".

ScrapingBypass sign out

Forget password

Enter the login web page, then click "Forget password?".

Enter your registered email, new password, then click "Send Captcha".

You will receive an email with a verification code like the following image:

Enter the verification code received from your email.

Finally, click "Reset Password".

Then you can get a message shows "Reset password successfully".

Buy a plan

last-update:2023-08-04 05:31:23

After login, on "My account" panel, click "Buy a plan".

Select a plan

Select a payment

Digital currency

Now ScrapingBypass only supports Tether - USDT and USD Coin - USDC.

Note: Please use the Ethereum mainnet to initiate the transfer, we do not support any other Ethereum side chains, such as: BSC, Polygon, etc.

Then click “Submit"


Scan QR code

Copy the address


Select "Stripe(AliPay)", click "Submit".

Click "To pay".

Scan the code to pay.

API key

last-update:2023-08-04 05:34:17

After login, enter "My Account" panel, there is an API token (API key) block.

Get API key

Click "Get apikey"

Enter your password, and click "Confirm"

Then your API key will appear.

Change your API key

Click "Refresh apikey"

Enter your password, and click "Confirm".

Your current API key will be shown here.