A social media marketplace: scraping vehicle category

To find best offers, car dealers (eScraper clients) used to spend much time browsing the listings manually and copying them for further processing. With the scraping technology, they could automate this task. eScraper extracts vehicle listings (under the “Vehicle” category) from a social media marketplace into a file. This lets our clients work with structured data and, therefore, speeds up their work and the work of their agents.

Web scraping a social media marketplace for cars: task overview

The clients interested in scraping car data, may have a number of requirements: extracting data for specific countries, locations, or areas; receiving fresh files regularly; getting dealership and private party listings separately or excludimg dealership listings; extracting specific fields; formatting the file for import to a CRM or other software.

The steps to scrape vehicle listings from a social media marketplace are as follows:

1. Describe what data to extract and how it should be organized.
2. Order free scrape.
3. Review the sample file and request modifications (if any).
4. Place an order and get car listings in a CSV file.

Solutions

Below we’ll describe the actions that we take to provide the perfect result file for the eScraper clients.

Searching for necessary data. Usually, a social media allows you to select the location of a company by city, neighborhood, or ZIP code and specify a radius to search within. So, to outline the area, a client provides zip code + radius, or a list of target cities, or cities + radius as in the example below:

Locations to search for cars to scrape from a social media marketplace

We then search by this location on a social media marketplace and extract car offers within the outlined area:

Searching for cars in the specified location

Scraping data on a schedule. Fresh data is provided once or twice a week according to the agreement between a client and eScraper. A client receives an updated file by email or by the link in Google Sheets with a timestamp:

File sample with vehicles scraped from a social media marketplace

Preparing a file. Following the client’s requirements, we’re providing such columns in the file:

• Listing URL
• The vehicle year and name For Example “ 2011 Ford Focus”
• The vehicle name and year separated into two fields (one for the year “2011”, one for the make and model like “Ford Focus”)
• The price
• The State/Region/
• The City
• The Seller Name
• The description provided by the seller
• Image URL

To present data in that format, we use formulas to split the records from one column into two. This allows clients to sort and filter data in the file according to their needs.

eScraper can extract all data available on the ad’s page. Thus the list of fields may differ and is requested by a client. For instance, we can also extract Mileage “Driven XXX km”, Transmission, Exterior Color, Fuel Type, and other available details.

Challenges, and how we overcame them

To prevent scraping, social media websites can block the robot with errors and captchas, especially when there are thousands of pages to be scraped. As we are providing listings with 15-17K lines twice a week, we experienced being blocked by the marketplace too. After tuning the scraping bot, we managed to proceed and extract the data.

Extracting additional fields like Seller URL, Name, and seller Description required us to click every ad in the search results.

To extract all car details the scraper clicks every ad

Loading each car page takes more time, so we had to adjust the start of data extraction to be able to provide a new file on time.

Result

Currently, we extract car listings from social media marketplaces for clients from the United States, Canada, Columbia, UK. Mostly, data is provided on a regular basis, as new offers appear every day and car dealers need to process them asap.

If you have any specific case, like scraping a social media marketplace for cars, vans, or trucks for sale by the owner and/or dealer, contact us. We’ll discuss your task via email or arrange a phone call. Then we’ll do a free test scrape, and prepare a file that will be perfect for you.