What is price intelligence?

Price Intelligence is leveraging web data to make better pricing, marketing, and business decisions. Basically, it is all about making use of the available data to optimize your pricing strategy, making it more competitive, increasing profitability, and ultimately, improving your business performance.

From competitor monitoring to dynamic pricing and MAP monitoring, web extracted pricing data has endless uses. Brands and e-commerce companies use pricing data to monitor an overall view of the market. Dynamic pricing can be used to make automatic pricing decisions based on competitor’s data combined with internal data so that you always remain profitable. MAP or Minimum Advertising Price monitoring is a technique that uses web extracted data to ensure the resellers and partners are maintaining the pricing according to the company policies.

During our webinar on “Fuelling Price Intelligence With Web Data Extraction” in June 2020, we got a lot of questions related to the processes and challenges of pricing data extraction. We cover a few important questions here:

Q: While monitoring prices, how do you account for promotional codes? How do you automatically apply additional promotions codes that can be applied to the selling price as seen on the listing page?

A: It varies from website to website, but the general idea is to find the pages where such promotion codes are available and build the logic of looking up code and applying it (clicking a button or sending an AJAX request) into your extraction code.

Q: How do you detect and act on the erroneous pricing that some websites intentionally showcase to confuse price intelligence efforts?

A: Websites showcase erroneous pricing data when they detect you scraping regularly. This especially happens when you are trying to scale - i.e scrape a lot of products very frequently. Erroneous pricing is not easily recognizable, but comparing the prices or other data fields with previously extracted data and manually checking if there is a big difference in the extracted data can help.

The long-term solution for this would be to be smarter about how you scale and be more thoughtful about the proxy solutions you use.

Q: How do you make sure that the data you scrape is accurate?

A: Scraping accurate data is all about having a reliable quality assurance process. The first step towards this process is to have a well-defined JSON schema. Your QA process needs to be a balanced combination of automated ways of testing the data as well as manual ways. This blog post gives a detailed description of data validation techniques.

Q: What are the challenges in scraping prices from sites that are javascript heavy?

A: For javascript heavy sites, the simplest way would be to inspect the website and see if it uses any hidden APIs that have the data in JSON or other simple formats. This way, you can get the data without executing the javascript.

However, sometimes that may not work. In that case, you will need to execute the javascript using a headless browser like Splash or Selenium or Puppeteer. The challenge here is that it will consume more resources making the process more expensive.

Q: How do you match products across websites?

A: There are many ways to conduct product matching. The main idea would be to gather as many product-specific parameters as you can about the product that you want to match and then compare those parameters.

Eg: For a TV, the product-specific parameters would be resolution, weight, sound, etc. If in comparison, 90% of the parameters of any two products are the same, there is a high chance that it is the same product across two websites.

You can build models on this concept to identify product duplication.

Price intelligence webinar

Want to know more about how you can fuel your price intelligence decisions with web extracted data? Watch this webinar where our Technology Evangelist, Attila Toth, takes you on a deep dive through the main challenges affecting price intelligence projects from both a business and technical perspective, and more importantly, how you can solve them.

If you have any more questions or queries on Price Intelligence data extraction, feel free to leave a comment below and we will try our best to answer them.