Web scraping finds stores guilty of price inflation

This is the last post ending a series of articles that have traced the prices of some of the top gifts, gadgets, and gizmos from Black Friday 2015 through to January 2016.

Check out Part 1 comparing Black Friday and Cyber Monday, Part 2 comparing Black Friday, Cyber Monday, and Green Monday, and Part 3 comparing Christmas Eve and New Year’s Eve.

And so we reach the end…

We’re all looking to be savvy shoppers who can buy nice things on a budget. So get ready to block out your calendar and wait in line at the right store.

It’s been a little over two months since we first started tracking the prices of popular products, starting on Black Friday and covering all the major discount days during the winter of 2015. We’ve finally reached the culmination of this project and are ready to declare the best days to shop for presents and deals this year.

Plus, we're calling out those guilty of price inflation, so you're going to want to keep reading.

Methodology

For this specific post, we compared all prices from Black Friday, Cyber Monday, Green Monday, Christmas Eve, and New Year’s Eve to those taken from January 13, 2016. We chose this date as our baseline because the traditional sales period is over, but it is still in a relevant time frame.

Tech and Trivia

While we covered the specifics of our methodology in the first post, we would like to share several tips and tricks that we used to get past some of the challenges of scraping so many different websites and products.

Some websites are really friendly toward web crawlers, providing machine-readable data for the content that you see in your browser. During this project, we got lucky. One of the websites we scraped for this series provided Microdata for the relevant information we needed about its products:

This was a perfect opportunity to use Extruct, a Python library we developed to extract Microdata from HTML content. Take a look at how Extruct works in this Scrapy shell session:

As you can see, it’s a piece of cake to scrape websites that use semantic markup. With Extruct there’s no need to write specific XPath or CSS selectors for the website. So, if the website you are trying to scrape has some itemprop properties inside the HTML, it’s your lucky day. The site is using Microdata and you can likely extract all the data you need with Extruct.

The Results

If this article doesn’t hammer home the completely arbitrary nature of prices, I’m not sure what will. Between price inflation and false sales, there were actually very few examples of products being fully discounted across the board on any particular day. Instead, getting a good deal was much more contingent on visiting the right store at the right time.

Apple TV

The Apple TV was one of the most interesting examples of this:

Apple TV-1

On Black Friday 2015, only Target and Walmart offered sales for this item (both $51.75). On Cyber Monday, Target shot up the price to $69, about $5 more than Best Buy and Sam’s Club.

Target continued this overpriced trend until Christmas Eve before discounting to $59. They then returned the price to the inflated $69 for New Year’s Eve and kept this price constant through to January 13, 2016.

Walmart, on the other hand, brought the price up to $63 on Cyber Monday before holding steady at $64.99, the same price as Best Buy, for the rest of the study. Target was clearly the best bet for a cheaper price as long as you purchased at the right time (Black Friday and Christmas Eve), but they are clearly guilty of inflating prices to capitalize on shopper anxiety.

Continuously Discounted

Out of all of the products, there were only 3 that were continuously discounted throughout the winter season compared to January 13, 2016. These were:

Roku Streaming Stick at Walmart
iPhone 6S from a seller on Amazon
Wii U from Walmart

Products Cheapest on Black Friday

When compared to every other day, Black Friday was by far the cheapest day to purchase these 7 products:

Amazon Fire TV Stick on Amazon
Apple TV at Target
Samsung Galaxy Note 5 at Target
Apple iPad Air 2 at Sam’s Club
Fitbit Charge HR at Target
Fitbit Surge at Best Buy
Jawbone UP 3 at Best Buy

The Jawbone UP 3 warrants closer attention because while Black Friday was the cheapest day to purchase this at Best Buy, the second cheapest time to buy it was January 13, 2016. For the rest of the winter season, Best Buy’s prices for the Jawbone UP 3 were higher than both Amazon and Target (see the Wearables graphs below).

Products Most Expensive on Black Friday

When compared to every other day, Black Friday was the most expensive day to purchase these products:

iPhone 6S Plus from an Amazon seller
PS4 Bundle from an Amazon seller

Products Cheapest on January 13, 2016

January 13, 2016 was the cheapest day for these products:

Google Nexus 6P from an Amazon seller
Apple iPad Air 2 from an Amazon seller
Apple iPad Mini 4 from an Amazon seller
Apple iPad Pro 128GB Silver from an Amazon seller
Dell Venue 8 7000 from an Amazon seller
Samsung Galaxy Tab S2 8.0'' 32GB at Amazon
LeapFrog LeapTV from an Amazon seller and at Target and Walmart
Mega Bloks Minions Supervillain Jet at Walmart
Star Wars BB-8 Droid R/C from an Amazon seller
Star Wars BB-8 Droid Sphero at Amazon and Best Buy (although it should be noted that this is a fairly negligible discount of about $1 for each of these retailers)
Nintendo 3DS XL at Amazon

Inflated Prices: Categories and Retailer

The worst culprits for inflated pricing were products in the Toys (4 products) and Tablets (5 products) categories (not counting the same product that was inflated in different stores). Amazon was the worst place to shop since it had 8 of the inflated items in these categories.

Unaffected Products

These were the products that were completely unaffected throughout the duration of this project:

Apple TV at Best Buy
Google Chromecast at Target
Samsung Galaxy S6 Edge at Sam’s Club
Amazon Kindle Voyage at Best Buy and Amazon
Apple iPad Air 2 at Walmart
Apple iPad Mini 4 at Sam’s Club, Target, and Walmart
Apple iPad Pro at Best Buy
Microsoft Surface 4 at Best Buy and Walmart
LeapFrog Epic Tablet at Walmart
LeapFrog LeapTV at Best Buy
Nintendo 3DS XL at Sam’s Club
Fitbit Surge at Amazon and Walmart
Jawbone UP 3 at Sam’s Club

Price Inflation vs Actual Deals

Let's take a closer look at the guilty parties who snuck in inflated prices among the sales.

Amazon was by far and away the worst offender when it came to price inflation. On every single one of the 5 days, there were about 13 to 18 products that were more expensive than the prices listed on January 13, 2016. This was far in excess of the 8 to 11 deals offered.

The Good, the Bad, and the Ugly

Tablets

Tablets were most discounted on Green Monday (7 deals), followed by Black Friday (6 deals).
New Year’s Eve was the worst day to buy a tablet with only 3 deals offered.
Best Buy was the best place to shop with 4 deals on Green Monday and 3 deals on Black Friday.
Walmart was the worst place to shop with 0 deals across all the days.

Wearables

Wearables were most discounted on Black Friday (7 deals), followed by Cyber Monday.
Green Monday was the worst day to buy a wearable with only 2 deals offered.
Best Buy was the best place to shop on Black Friday (4 deals).
Amazon was the best place to shop on Cyber Monday (2 deals).
Sam’s Club was the worst place to shop with 0 deals across all the days.

Home Theater

Home Theater products were most discounted on Black Friday (8 deals), followed by Christmas Eve (7 deals). Green Monday was the worst day to shop for these products with only 3 deals offered.
Amazon, Best Buy, and Target all tied for the best places to shop on Black Friday (2 deals each).
Best Buy, Target, and Walmart tied for the best places to shop on Christmas Eve (2 deals each).
Sam’s Club was the worst place to shop with minor sales (less than $5) on two days and 0 on the rest.

Video Game Consoles

Video games and consoles were the most discounted on Black Friday and Green Monday with 13 deals each (minimal discounts are included in this figure).
New Year’s Eve was the worst time to buy video games and consoles with 2 deals only.
Target was the best place to shop on Black Friday and Green Monday with 4 deals offered on each day.
Sam’s Club was the worse place to shop with either the lowest number of deals or tied for the lowest number of deals on each of the days.

Phones

Phones were the most discounted on Black Friday and Christmas Eve with 8 deals each.
Cyber Monday was the worst day to buy Phones with only 6 deals.
On Black Friday, Target was the best place to shop for phones (4 deals).
On Christmas Eve, Best Buy was the best place to shop (3 deals).
Sam’s Club was the worst place to shop with 0 deals across all the days.

Toys

Green Monday and New Year’s Eve were the two most discounted days for toys with 3 deals each.
Black Friday, Cyber Monday, and Christmas Eve all tied for the worst days to shop for toys with only 1 deal each. In fact, this was the only category that featured at least one item on every single day that was more expensive than January 13, 2016.
Target was the best place to shop on Green Monday with 2 deals.
Amazon had 2 deals on New Year's Eve, but one was discounted by a mere $0.61, so I'm going to consider this a tie with Best Buy (1 deal).
Amazon was the worst place to shop for toys; 3 to 5 items were more expensive than January 13, 2016 on each of the days. Walmart was the second worst for toys; 2 items were more expensive than January 13, 2016 on each of the days.

Winner: Black Friday

I’m going to go ahead and place the winners:

Black Friday is the winner with a total of 19 deals.
Green Monday is in second place with 19 deals as well, but one product had less than a $5 discount.
Cyber Monday is third with 17 deals, but one product had less than a $5 discount.
Christmas Eve is in fourth place with 17 deals as well, but two products had less than a $5 discount.
New Year’s Eve is in fifth place with 14 deals, but three products had less than a $5 discount.

Wrap Up

Now, there is no guarantee that these same patterns will hold this year. But based on 2015, I highly recommend that you plan around the Toys situation. Every single toy suffered from price inflation with every single store inflating the price of at least one toy. If you are in a bind, though, then I’d wait for Green Monday and avoid Amazon.

If a Fitbit goes on sale on Black Friday, take the deal. It’s not likely to get any cheaper. I’m still congratulating myself on getting my Fitbit Charge HR for $30 cheaper on Black Friday!

Do your homework ahead of time and don’t worry if you hesitate and miss a sale, there’s usually another retailer and another day.

For more articles like this, follow us on Twitter, Facebook, and LinkedIn, and don’t forget to subscribe to our blog.