Saturday, February 27, 2021
  • Setup menu at Appearance » Menus and assign menu to Top Bar Navigation
Advertisement
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
No Result
View All Result
Home Machine Learning

Challenges and Changes Facing the Profession

May 4, 2020
in Machine Learning
Challenges and Changes Facing the Profession
587
SHARES
3.3k
VIEWS
Share on FacebookShare on Twitter

Last year, the fastest-growing job title in the world was that of the machine learning (ML) engineer, and this looks set to continue for the foreseeable future. According to Indeed, the average base salary of an ML engineer in the US is $146,085, and the number of machine learning engineer openings grew by 344% between 2015 and 2018. Machine learning engineers dominate the job postings around artificial intelligence (A.I.), with 94% of job advertisements that contain AI or ML terminology targeting machine learning engineers specifically.

This demonstrates that organizations understand how profound an effect machine learning promises to have on businesses and society. AI and ML are predicted to drive a “Fourth Industrial Revolution” that will see vast improvements in global productivity and open up new avenues for innovation; by 2030, it’s predicted that the global economy will be $15.7 trillion richer solely because of developments from these technologies.

You might also like

MindMed Closes Acquisition of HealthMode, a Leading Machine Learning Digital Medicine Company

Providence exec explains the differences, their healthcare applications

An Epic cognitive computing platform primer

The scale of demand for machine learning engineers is also unsurprising given how complex the role is. The goal of machine learning engineers is to deploy and manage machine learning models that process and learn from the patterns and structures in vast quantities of data, into applications running in production, to unlock real business value while ensuring compliance with corporate governance standards.

To do this, machine learning engineers have to sit at the intersection of three complex disciplines. The first discipline is data science, which is where the theoretical models that inform machine learning are created; the second discipline is DevOps, which focuses on the infrastructure and processes for scaling the operationalization of applications; and the third is software engineering, which is needed to make scalable and reliable code to run machine learning programs. 

It’s the fact that machine learning engineers have to be at ease in the language of data science, software engineering, and DevOps that makes them so scarce—and their value to organizations so great. A machine learning engineer has to have a deep skill-set; they must know multiple programming languages, have a very strong grasp of mathematics, and be able to understand and apply theoretical topics in computer science and statistics. They have to be comfortable with taking state-of-the-art models, which may only work in a specialized environment, and converting them into robust and scalable systems that are fit for a business environment. 

As a burgeoning occupation, the role of a machine learning engineer is constantly evolving. The tools and capabilities that these engineers have in 2020 are radically different from those they had available in 2015, and this is set to continue evolve as the specialism matures. One of the best ways to understand what the role of a machine learning engineer means to an organization is to look at the challenges they face in practice, and how they evolve over time. 

Four major challenges that every machine learning engineer has to deal with are data provenance, good data, reproducibility, and model monitoring.

Challenge 1: Data Provenance

Across a model’s development and deployment lifecycle, there’s interaction between a variety of systems and teams. This results in a highly complex chain of data from a variety of sources. At the same time, there is a greater demand than ever for data to be audited, and there to be a clear lineage of its organizational uses. This is increasingly a priority for regulators, with financial regulators now demanding that all machine learning data be stored for seven years for auditing purposes.

This does not only make the data and metadata used in models more complex, but it also makes the interactions between the constituent pieces of data far more complex. This means machine learning engineers need to put the right infrastructure in place to ensure the right data and metadata is accessible, all while making sure it is properly organized. 

Sign Up Today

Membership has its benefits. Sign up for a free Dice profile, add your
resume, discover great career insights and set your tech career in motion.
Register now

Challenge 2: Good Data 

In 2016, it was estimated that the US alone lost $3.1 trillion to “bad data”—data that’s improperly formatted, duplicated, or incomplete. People and businesses across all sectors lose time and money because of this, but in a job that requires building and running accurate models reliant on input data, these issues can seriously jeopardize projects. 

IBM estimates that around 80 percent of a data scientist’s time is spent finding, cleaning up, and organizing the data they put into their models. Over time, however, increasingly sophisticated error and anomaly detection programs will likely be used to comb through datasets and screen out information that is incomplete or inaccurate. 

This means that, as time goes on and machine learning capabilities continue to develop, we’ll see machine learning engineers have more tools in their belt to clean up the information their programs use, and thus be able to focus more time spent on putting together ML programs themselves.

Challenge 3: Reproducibility

Reproducibility is often defined as the ability to be able to keep a snapshot of the state of a specific machine learning model, and being able to reproduce the same experiment with the exact same results regardless of the time and location. This involves a great level of complexity, given that machine learning requires reproducibility of three components: 1) code, 2) artifacts, and 3) data. If one of these change, then the result will change. 

To add to this complexity, it’s also necessary to keep reproducibility of entire pipelines that may consist of two or more of these atomic steps, which introduces an exponential level of complexity. For machine learning, reproducibility is important because it lets engineers and data scientists know that the results of a model can be relied upon when they are deployed live, as they will be the same if they are run today as if they were run in two years.

Designing infrastructure for machine learning that is reproducible is a huge challenge. It will continue to be a thorn in the side of machine learning engineers for many years to come. One thing that may make this easier in coming years is the rise of universally accepted frameworks for machine learning test environments, which will provide a consistent barometer for engineers to measure their efforts against.

Challenge 4: Monitoring

It’s easy to forget that the lifecycle of a machine learning model only begins when it’s deployed to production. Consequently, a machine learning engineer not only needs to do the work of coding, testing, and deploying a model, but they’ll have to also develop the right tools to monitor it. 

The production environment of a model can often throw up scenarios the machine learning engineer didn’t anticipate when they were creating it. Without monitoring and intervention after deployment, it’s likely that a model can end up being rendered dysfunctional or produce skewed results by unexpected data. Without accurate monitoring, results can often slowly ‘drift’ away from what is expected due to input data becoming misaligned with the data a model was “trained” with, producing less and less effective or logical results. 

Adversarial attacks on models, often far more sophisticated than tweets and a chatbot, are of increasing concern, and it is clear that monitoring by machine learning engineers is needed to stop a model being rendered counterproductive by unexpected data. As more machine learning models are deployed, and as more economic output becomes dependent upon these models, this challenge is only going to grow in prominence for machine learning engineers going forward.

Machine Learning Engineer: An Evolving Role

One of the most exciting things about the role of the machine learning engineer is that it’s a job that’s still being defined, and still faces so many open problems. That means machine learning engineers get the thrill of working in a constantly changing field that deals with cutting-edge problems. 

Challenges such as data quality may be problems we can make major progress towards in the coming years. Other challenges, such monitoring, look set to become more pressing in the more immediate future. Given the constant flux of machine learning engineering as an occupation, it’s of little wonder that curiosity and an innovative mindset are essential qualities for this relatively new profession.

Alex Housley is CEO of Seldon.

Credit: Google News

Previous Post

How to Display Mode-Specific Images

Next Post

PlanetScale scales out MySQL | ZDNet

Related Posts

MindMed Closes Acquisition of HealthMode, a Leading Machine Learning Digital Medicine Company
Machine Learning

MindMed Closes Acquisition of HealthMode, a Leading Machine Learning Digital Medicine Company

February 27, 2021
Healthcare leaders debunk 3 myths about machine learning
Machine Learning

Providence exec explains the differences, their healthcare applications

February 27, 2021
An Epic cognitive computing platform primer
Machine Learning

An Epic cognitive computing platform primer

February 27, 2021
AI and machine learning to help global battle with cancer
Machine Learning

AI and machine learning to help global battle with cancer

February 26, 2021
How Artificial Intelligence, Machine Learning will further advance Ed-tech sector?
Machine Learning

How Artificial Intelligence, Machine Learning will further advance Ed-tech sector?

February 26, 2021
Next Post
PlanetScale scales out MySQL | ZDNet

PlanetScale scales out MySQL | ZDNet

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recommended

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

January 6, 2019
Microsoft, Google Use Artificial Intelligence to Fight Hackers

Microsoft, Google Use Artificial Intelligence to Fight Hackers

January 6, 2019

Categories

  • Artificial Intelligence
  • Big Data
  • Blockchain
  • Crypto News
  • Data Science
  • Digital Marketing
  • Internet Privacy
  • Internet Security
  • Learn to Code
  • Machine Learning
  • Marketing Technology
  • Neural Networks
  • Technology Companies

Don't miss it

Fastest VPN in 2021 | ZDNet
Internet Security

Fastest VPN in 2021 | ZDNet

February 27, 2021
Increasing Adoption of Informatics will Promote Growth of Data Analytics Outsourcing Market
Data Science

Increasing Adoption of Informatics will Promote Growth of Data Analytics Outsourcing Market

February 27, 2021
MindMed Closes Acquisition of HealthMode, a Leading Machine Learning Digital Medicine Company
Machine Learning

MindMed Closes Acquisition of HealthMode, a Leading Machine Learning Digital Medicine Company

February 27, 2021
How AI Can Be Used in Agriculture Sector for Higher Productivity? | by ANOLYTICS
Neural Networks

How AI Can Be Used in Agriculture Sector for Higher Productivity? | by ANOLYTICS

February 27, 2021
Berlin resident jailed for threatening to bomb NHS hospital unless Bitcoin ransom was paid
Internet Security

Berlin resident jailed for threatening to bomb NHS hospital unless Bitcoin ransom was paid

February 27, 2021
The Ethereum Virtual Machine (EVM)
Data Science

The Ethereum Virtual Machine (EVM)

February 27, 2021
NikolaNews

NikolaNews.com is an online News Portal which aims to share news about blockchain, AI, Big Data, and Data Privacy and more!

What’s New Here?

  • Fastest VPN in 2021 | ZDNet February 27, 2021
  • Increasing Adoption of Informatics will Promote Growth of Data Analytics Outsourcing Market February 27, 2021
  • MindMed Closes Acquisition of HealthMode, a Leading Machine Learning Digital Medicine Company February 27, 2021
  • How AI Can Be Used in Agriculture Sector for Higher Productivity? | by ANOLYTICS February 27, 2021

Subscribe to get more!

© 2019 NikolaNews.com - Global Tech Updates

No Result
View All Result
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News

© 2019 NikolaNews.com - Global Tech Updates