Big Data Jobs

in CompSci

The Biggest Misconception about Data Scientists

4 min read

LinkedIn reports that “Data Mining,” in relation to data science, was the number one in-demand skill in 2014.

Sitting atop a mountainous treasure trove of data, most all businesses are thirsty for people who can take a massive set of data and turn it into something meaningful. Whether it’s pinpointing new sources of revenue or predicting the next, best product feature, businesses are depending on data scientists to derive valuable and potentially lucrative insights. In other words, Big Data means big money! At least that’s the general consensus.

In fact, data scientists’ salaries are among the highest in tech right now, nationally averaging about $100,000, according to Glassdoor. And if you’re living in Seattle, that base salary shoots up to $150,000, Mashable reports. Plus, a 2014 Gartner study says that 73% of employers plan to spend more money on data scientists in the next two years.

Data scientists are truly valuable in extrapolating, analyzing and finding patterns in existing data using statistics and machine learning. But there’s often a big gap between expectations and reality of what data science can do for business. After speaking with a few highly sought-after data scientists, here’s what we found:

Biggest Misconception: Data Scientists Have Magic Formulas to Extract Meaningful Data

There’s no one, magic formula data scientists use to dig up the right patterns from any given massive dataset. The key to leveraging Big Data is the creativity and insight necessary to ask the right questions that’s unique to each business model. “Data scientists understand the process of analyzing and finding commonalities in data, but they don’t have the necessary business context to determine which commonalities or anomalies can actually help the business,” says David Giannetto, author of Big Social Mobile. At least not all of them. Any ordinary data mining expert can pull data, but the top-tiered data scientists should be able to spot trends to boost the business model.

Data mining alone is simply not enough to move the business forward. It’s a means to an end. “There’s an expectation that collecting large quantities of data will make an organization smarter,” says Mark Schwarz, VP of Data Science at Square Root. “In reality, that often leads to decision paralysis and the inability to actually move the needle within the business.”

Brian Lange, data scientist at Datascope Analytics, would agree and says while they can, in many cases, make better predictions with more data at their disposal than every before, data scientists aren’t immune to the inherent pitfalls that comes with making predictions. “Overfitting (the statistics equivalent of missing the forest for the trees), false assumptions, sample bias- these are all things that aren’t solved by the quantity of data you throw at the problem,” he says.

So what should companies do instead? “Companies should be focused on uncovering the right information,” Schwarz says. Average data experts can pull data based on a number of factors handed to them, but the truly valuable data scientists have strong business acumen to proactively focus on data that will actually change behavior.

What Gets Top Data Scientists Excited?

While every data scientist is different, Lange says that most data scientists aren’t easily impressed by fancy, expensive enterprise software. “We love having fast computers and blank checks to use on cloud computing bills, but when it comes to software, many of us prefer working with languages and tools that are completely free, like Python, R, PostgreSQL, and ElasticSearch,” Lange says.

Data Scientist vs. Data Engineer vs. Business Analyst

Many roles in Big Data have overlapping qualities, but it’s crucial to be able to differentiate between the data scientist, data engineer and business or data analyst. They each have individual values and are complimentary to one another, making up a strong Big Data team.

Here’s a high level visual that differentiate the roles by a few of their core functions, inspired by a great post from Kevin SchmidtBiz.

Demystifying Data Science

Business Analyst: 

Business analysts’ strengths lie in their business acumen. They can communicate well with both the data scientist and C-suite to help drive data-driven decisions faster. They typically work across sales and marketing teams to make data-driven decisions. The best business analysts also have skills in statistics to be able to glean interesting insights from past behavior.

Data Engineer:
While data scientists dig into the research and visualization of data, data engineers ensure the data is powered and flows correctly through the pipeline. They’re typically software engineers who can engineer a strong foundation for data scientists or analysts to think critically about the data.

Data Scientist:
Data science is largely rooted in statistics, data modeling, analytics and algorithms. They focus on conducting research, asking open-ended questions and optimizing data to help companies get better at what they do. For instance, top-tiered data scientists are the minds behind recommended products on Amazon. Data mining (the most in-demand skill on LinkedIn) is a subset of data science as the means to the end of extracting value from data using techniques, like pattern recognition, algorithm design and clustering, to name a few, to better predict future behavior.

Each organization may define these roles differently, but this is a general guideline on the distinctions and similarities of Big Data skills necessary to move the needle for your business goals. Before you build out your Big Data team, prioritize your Big Data needs in a way that makes sense for your business. And keep in mind that, while Big Data can be pivotal in boosting business, it’s crucial to empower your data scientist and analysts with business context.


Follow us @HackerRank4Work and LinkedIn.

Image credit: Philip Kromer on Flickr, network graph of people on Twitter talking about Big Data. 


Share your thoughts