phpunit selenium- scraping through python selenium or beautifulsoup

Question

I want to scrape the url below https://turo.com/search?airportCode=EWR&customDelivery=true&defaultZoomLevel=11&endDate=04%2F05%2F2019&endTime=11%3A00&international=true&isMapSearch=false&itemsPerPage=200&location=EWR&locationType=Airport&maximumDistanceInMiles=30&sortType=RELEVANCE&startDate=03%2F05%2F2019&startTime=10%3A00

I want all the href link of the cars listed so that i can scrape forward but i am unable to get it

Kindly  help with this!

Answer 1

You can pull in the json response with the query and work with that:

 import requests

from pandas.io.json import json_normalize

base_url = 'https://turo.com'

url = 'https://turo.com/api/search?'

headers = {

'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',

'Accept': '*/*',

'Accept-Encoding': 'gzip, deflate, br',

'Accept-Language': 'en-US,en;q=0.9',

'Connection': 'keep-alive',

'Host': 'turo.com',

'Referer': 'https://turo.com/search'}

params = {

'airportCode': 'EWR',

'customDelivery': 'true',

'defaultZoomLevel': '11',

'endDate': '04/05/2019',

'endTime': '11:00',

'international': 'true',

'isMapSearch': 'false',

'itemsPerPage': '200',

'location': 'EWR',

'locationType': 'Airport',

'maximumDistanceInMiles': '30',

'sortType': 'RELEVANCE',

'startDate': '03/05/2019',

'startTime': '10:00'}

response = requests.get(url, headers=headers, params=params)

data = response.json() 

search_id = data['searchId']

print (search_id)

for ele in data['list']:

    link = ele['vehicle']['url']

    print (base_url + link) 

links_list = [ base_url + ele['vehicle']['url'] for ele in data['list'] ]

# If you want to Manipulate a table of the data

df = json_normalize(data['list'])