Where Does All This Data Come From?

It doesn’t seem possible that anyone can make a comment in any of the recruiting Facebook groups without being asked for the source and validity of the data. This is understandable, given that so much data is shared in a debate, often without reference to where that number has come from. Recruiting, in particular, has also suffered from urban myth syndrome. Popular truths, that said enough times by enough people become accepted law, even though they don’t necessarily stand up to a data sniff test. I often find myself asking: “Do you think that, or do you know that?”

Increasingly, talent acquisition leaders are being asked to become an expert voice within their organisations. I once heard a half joking quote in a conversation at a conference, when we had those, that has stuck with me for a number of years since:

“In God we trust, everyone else bring data!”

Google tells me that this can first be attributed to American Management Theorist W. Edwards Deming.

Whilst there is some dispute in who said the exact quote first, I think you get the point. This was quoted in a book in 1986 when we were first being asked to show some proof behind our proposals and statements. Jump forward to today, and we have a very different challenge. In a world where every click, task and action is digitally recorded, we often have the problem of too much data to navigate. Developing an “expert voice” around your area of interest is less about bringing data, and much more about defining the right data, applying context and being able to define and communicate meaning. This is a critical skill in establishing a credible expert voice in an organisation. The art of “big data analytics” is really being able to turn lots of numbers into meaning, and communicating why this is important, to someone who is probably not an expert in the discussion topic, and be sufficiently credible to get their support.

Over the last few months I have been speaking to the users of Horsefly Analytics, to understand how and why they use the labor market data available. One of the questions I always ask is what makes them confident enough to present Horsefly’s data to support their expert voice. They are, after all, staking their reputation and in some cases, careers, positively and negatively on actions resulting from their recommendations. The source data has to be right.

The answer that I’m continually getting is that although there are sometimes surprises (and that’s healthy), the numbers, trends and conclusions feel right to someone working in the market day to day, and come from a broad enough range of sources to present real trends, whilst picking up on variances globally.

My challenge was that I was hearing about trillions of data sources, and that felt too big to imagine. Kind of like trying to picture infinity. I asked Horsefly CEO Will Crandle to break down for me how they acquire so many data points, extract what they need and derive meaning in a fast time frame. Answers to queries need to come quickly. This is Crandle’s answer to my question on why I should trust the data:

“Data Points: We collate and aggregate data from hundreds of thousands of data sources:

  • Government demographics on a town by town basis (125k Towns/Cities mapped)
  • 400m merged and de-duplicated social profiles from across the social web
  • Millions of de-duplicated job board adverts collected per month in real-time from both mainstream and niche sources globally
  • Salary aggregation sources
  • Gender demographics globally
  • Constantly evolving 550,000 taxonomy keywords translated across 38 countries
  • All data is rigorously benchmarked across government data sources. Overlaying multiple data sources ensures the highest accuracy
  • We employ data scientists with a background in statistics for algorithm building
  • Humans! When data is sparse or unavailable, we crowd source data from targeted, skilled sources.”

    In particular, the focus on labor markets and the robust taxonomy stands out to me. If you have a burning labor market question you want to ask, message me or leave it in comments. We will run the search in Horsefly Analytics, and publish the findings. Put it to the test.


Leave a Comment