The importance of web scraping in data journalism

Scraping news sites is a great way to keep up to date with trends and developments in any industry, but for data journalists, a web scraper tool can streamline the process of researching a topic thoroughly, allowing them to produce more reliable, data-driven articles as a result.

A good data extraction service such as that offered by Zyte can generate a structured database of information that can be analyzed and transformed into news.

This not only makes data journalism faster and easier but can also help journalists to make more sense of the data, as the structured data can be filtered, sorted, and processed in various ways, without the need for the journalist to have a college degree in math.

What is data journalism?

Data journalism is a discipline that involves taking large sets of data and interpreting the information within. It's a specialism that requires the journalist to identify what the data really means and report that to the reader.

In many cases, data journalism is accompanied by a graphical representation of the data being reported. That could be as simple as a table or chart of the data, or it could be something like an infographic that attempts to display the data in a more accessible, reader-friendly format.

Data journalists need a unique set of skills:

• Ability to compile and understand large quantities of data
• Ability to represent that data in words, graphs, tables, and pictures
• Ability to report findings in a coherent, professional news article

Scraping news sites is one way to streamline the process, allowing journalists to focus on their core skill — reporting — and removing a lot of the time and statistical expertise from the research stage.

Data journalism examples

Historically, data journalists would spend weeks manually compiling information and statistics from different sources, but now much of this work can be automated by scraping data from the internet.

Some examples of the types of data that journalists might use:

• A comparison of a large number of prices to measure increases in the cost of living
• Collating and comparing feedback and reviews to detect poorly performing brands
• Measuring crime data across different jurisdictions to identify high-risk areas

Much of this journalist data can be collected by scraping news sites, for example, by looking for reports of crimes or outbreaks of disease, but you can extract data from websites of all kinds to create the basis for a data journalism article.

How can web scraping help data journalists?

We've already mentioned some of the advantages of web scraping for data journalism. But here are a few things that web scraping eases for a journalist.

• Generate a large database, without having to manually enter that data
• Monitor certain keywords to stay on top of a story
• Access data in seconds

While web scraping is hugely beneficial, it is not without its challenges. Not all websites are structured and formatted properly. Some websites change regularly and present challenges for bots to extract data from their websites. So to make the most of the benefits provided by web scraping, it is best to partner with a professional data extraction service like Zyte. That way, you can concentrate on your findings and the story rather than the technicalities of data extraction.

Benefits of outsourcing your web scraping

Clean data

The difference between bad and good scraped data is not just structure, but also how clean the data is. The last thing you need is fragments of HTML scraped along with the usable data.

Zyte produces clean data, ready to use. No need to manually work through erasing the formatting from each entry - just a database of structured, clean, usable data ready to go.

Legal compliance

At Zyte, it’s critical that our services respect the rights of the websites and companies whose data we scrape. Zyte will conduct a legal assessment before commencing a data extraction project to ensure the scraped data is compliant.

Structured data

We've mentioned structured data above but it's worth recognizing just how valuable this is for journalists working data journalism that deals with vast, complex data sets.

Structured data means everything is in its place. You can see at a glance which fields are numerical, which are text, which are factual, and which are more open-ended, such as user reviews — cutting through the complexity of analyzing all that information so your article can have a solid statistical basis, with the human interest of individual opinions too.

Find out more

To find out more about any of the above, or to enquire about Zyte's data extraction services for journalists, please contact us today and we will be happy to discuss how we can help leverage data journalism so you write that Pulitzer piece you've been planning for years.