What’s Behind the Data Scientist Shortage?

If you’ve checked your LinkedIn feed over the last few years, you’ve probably noticed the tech industry’s wild roller coaster ride. First, the pandemic led to a huge wave of tech hirings—promptly followed by the Great Resignation—and then huge rounds of layoffs.

In all, there were a staggering 1,405 rounds of tech layoffs which affected nearly 220,000 individuals. Even tech titans such as Amazon and Meta were seen cutting thousands of jobs and Twitter was reported to have made late night cuts to their data scientist teams.

At the same time, the U.S. Bureau of Labor Statistics (BLS) reports that there will be a 36 percent job growth rate over the next decade for data scientists. This rate is immensely higher than most other roles they track—where the average growth rate is five percent—and is even higher than the 31.4 percent growth they predicted for data scientists only back in 2020.

The BLS also predicts a growth rate of almost 28 percent by 2026 for the number of roles requiring data science skills. Yet, most companies have been struggling to fill these roles. 

So, is the current shortage of data scientists simply a classic Economics 101 case study of low supply and high demand, is there an upcoming AI Winter—or is something deeper brewing below the surface? There seem to be some differing opinions, so let’s dive deeper into three factors that could be influencing the data scientist shortage. 

Are Job Titles for Data Scientists Being Downgraded?

Comedians used to joke about people inflating their job titles to feel more important or to get a salary bump. These days—in the midst of a great global recession—the theory has arisen that companies are actually deflating job titles to fight economic uncertainty. 

A recent Interview Query analysis focusing on job posting data notes a curious trend for “data science” roles: 

• There was a 26 percent drop from October 2021 to October 2022 regarding the number of data scientist job openings 
• At the same time, the number of data engineers and data analyst job openings increased

Apps such as Blind—which provide an anonymous forum and community for verified employees to discuss issues—echo these sentiments. Many posts ruminate over cases of “data science teams being downgraded to analysts” and SQL retrievers and verify the theory that the data scientist role is getting split up into different names, such as data analyst or machine learning engineer.

This may be bad news for experienced data scientists, but it’s good news for employers trying to save a buck and for young analysts trying to break into the field. Glassdoor confirms that—in the U.S. job market—data analysts now make about 30 percent less than data scientists. 

The analysis makes three final notes:

• While data scientists roles may be diminishing, ML engineers, software engineers, and AI researchers will be in high demand
  • Also, many firms are now clearly separating data analyst and data engineer roles from data scientist roles 
• Many data scientist interviews have simply not been happening—companies may list the job, but aren’t actually reviewing resumes 
• The data scientist role has been seen as a “luxury” for many firms and the recession has caused them to tighten their belts and lay off many data scientists

Are Data Scientists Too Theoretical to Add Value and Help the Business?

Data Science is an extremely difficult and interdisciplinary field—and some worry that all of their academic prowess does not always translate well into real business value and ROI. Also, the data science skill set takes years to acquire and is long and complex:

• Knowing statistics, math—especially statistics and probability—and programming, in addition to knowledge of the business domain, computer science and software development, and presentation skills is essential. 
• Acquiring, processing, cleaning, integrating, and storing data is fundamental to data science. You have to work well with SQL and NoSQL databases. 
• Investigating and exploring data analysis and selecting the best algorithms and models and actually putting them into production is vital.
• Reiterating constantly and delivering value and tangible results to stakeholders is necessary—not to mention keeping up with compliance and AI ethics.

The real challenge is finding a competent data scientist who can clean and augment the data

and then put models into production. A lot of data science knowledge is built on carefully planned and well-organized courses. Specific knowledge of a business problem, however, can’t be taught in the classroom. 

Professionals who truly understand how to query and connect to databases, implement an object store and containerize models, and transform them into APIs and embed them into edge devices are in limited supply. In short, enterprises need people who can apply practical applications to their datasets.

It’s one reason why Gartner has pinpointed AI engineering as a top strategic technology trend—to put more focus on operationalizing AI models. Universities are improving, but it would still help if they would give their budding data scientists more practical work to put their intricate theories into practice.

In addition, once data scientists get started, they often feel siloed, especially with the increase in remote work. Senior data scientist Daliana Liu, who worked as a senior data scientist and senior machine learning instructor for Amazon Web Services (AWS), offered these comments on the reality of working in the field: 

“I felt there’s a gap between what I learned in school, and what I actually do, and I also feel very insecure sometimes,” she said. “I didn’t know a lot of other data scientists who worked in the industry, so I wished I could have a community and talk to them.”

Data Scientists: Help Us Help You with Better Jobs that Add Value to the Business

So far you’ve only heard the macroeconomic and employer perspectives, respectively. But what do many data scientists actually think? Many believe that the data scientist shortage doesn’t fall on their shoulders—but rather on enterprises that fall short in the following areas:

• Hiring Processes—Focus on candidates’ strengths and treat them respectfully. Check sites such as Glassdoor to see what candidates are saying about your hiring and interview processes.   
• Lining Up Priorities—Make sure you have things lined up correctly and that you start with clear business objectives. Data scientists shouldn’t be given shiny new toys in search of value.
• Aligning Strategies—Your talent, data, and business strategies should align to bring value to the business. If top leadership knows what top data science talent needs, they’ll find it easier to justify the costs of a data science team and the hiring process will run smoother.
• Meeting Job Expectations—Many data scientists finally get their dream job—and rather than exciting AI or ML projects, it’s just basic analytics using Excel/SQL and some visualization tools. There are many areas of data science—so perhaps  say job titles should be more specific: NLP data scientist, analytical data scientist, audio data scientist, etc.             

• Investing in Data Engineering and Tooling—In the world of Big Data, it’s popular to say, “Garbage in, garbage out.” Yet, in addition to data quality, a recent Anaconda survey put solid investment in data engineering and tooling as the top priority—even above the data science talent shortage. 
• Delivering a Good Package—If you want to keep top data science talent—you have to pay them well. There is a good reason why most data scientists spend less than two years at the same job and less than two percent have stayed at the same job for more than five years. Pay up and provide good perks and benefits—or spend more resources trying to replace a trusted data scientist and their invaluable institutional knowledge.
• Keeping the Know-How In House—As data scientists do change jobs frequently—enterprises often lose their practical know-how and institutional knowledge when they leave. With ML Studio, companies can protect their ML projects and ensure that they are easily understood by any member of the team—old or brand new.

ML Studio makes AI accessible for organizations that need to solve real-world business problems by providing clients with a robust and ready-to-use End-to-End AI platform that doesn’t go off budget.

Contact us for a personal demo.