When the number of rows is not equal to the number of products it means that the products we scraped had variations.

Thus, if there is 1 parent product with 4 variations (for instance, sizes XS, X, M, L) we will scrape them as separate products and there will be 5 lines in the file.

When scraping variations, it takes us more time to configure the scraper (to find all possible variations), to adjust columns accordingly, and quite often, to reload the page to extract images for variations. Also, if products have variations, the result file grows instantly, and the larger the file, the more complicated it is to configure it per your needs.

That’s why we count product variations as separate products and charge per row.

Extracting data from websites with extensive listings has some specific features:

• After the test scrape, we can provide an approximate estimate of the number of products and the cost of extracting them. The exact estimate will be provided for each file sent (see the next paragraph).
• We scrape data by chunks and send them to you asap. That is done to avoid being banned and to avoid issues with too large files (you will not be able even to open a CSV file with one million rows).
• The faster you need the whole data, the more it will cost to scrape it. We are using proxies to avoid being banned. Therefore, to extract the whole data faster we will be using more servers at a time to provide the required data to you.

If data extraction requires additional tweaking that increases the cost of work we will provide a quote beforehand, get your approval, and do the work.

This is a very broad topic, so we’ll answer the most common questions.

• Can anyone just scrape all my work from my website and paste it into theirs?

This is a very important topic and you should discuss it with your legal representative to make sure you protect your work (including Copyright marks, legal policies, watermarks, robots.txt settings). We cannot provide you legal consultation on this matter.

Overall if you Google a little bit, most likely you will already find your data copied by third-party websites, like comparisons, review sites, etc. You can keep track of your unique content with Google Alerts (using brand names). As soon your content is indexed by Google and it is not legally used (a legal use might be your partners, affiliates), you can request its removal via the Google Search Console – https://www.google.com/webmasters/tools/dmca-dashboard



• In what cases automated data collection is considered legal?

You can find a number of articles explaining the legal side of data scraping and crawling.

General advice is to ask the data provider for permission to scrape data. For example, if you sell products from a manufacturer who does not provide you all information you need in the appropriate format and you have permission to scrape it, you can use data scraping.

Also if you need to migrate from a custom-written CRM and you have no options for the data export or the data is not import-ready, you can use scraping too.

Overall, this topic is very sensitive. You can read the latest case of Google in Australia. Yes, Google also crawls and scrapes data and it is considered illegal by the Australian court: Google says it will remove search function in Australia if media code becomes law



• How to use scraped data legally?

Obey the copyright laws and Terms of Service (ToS) of the website that you want to scrape.

Unless you have permission to use the content as is, don’t pass off scraped content as your own. It does not only break the copyright but also bad for SEO. At best, copied content won’t rank high, at worst, your website can be penalized for plagiarism. Read more about duplicate and scraped content usage:
https://developers.google.com/search/docs/advanced/guidelines/scraped-content https://developers.google.com/search/docs/advanced/guidelines/duplicate-content