Friday, March 5, 2021
  • Setup menu at Appearance » Menus and assign menu to Top Bar Navigation
Advertisement
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
No Result
View All Result
Home Data Science

Visualizing the Performance of COVID Models

June 5, 2020
in Data Science
Visualizing the Performance of COVID Models
587
SHARES
3.3k
VIEWS
Share on FacebookShare on Twitter

by Chris McLean & Peter Bruce

You might also like

A Plethora of Machine Learning Articles: Part 2

The Effect IoT Has Had on Software Testing

Why Cloud Data Discovery Matters for Your Business

Never before have statistical models received the attention they are getting now in the midst of the Coronavirus pandemic.  It is hard to read a news feed today without encountering either:

  • New predictions from models such as the IHME model and others, or
  • Critiques of older predictions.

So – how have older predictions turned out? 

If you’ve been paying attention to the news, you’ll know the answer is “not so well.”  Consider this prediction from April 16 for the IHME (University of Washington “Murray”) model:

Figure 1:  Deaths per IHME predictions

Actual deaths after the prediction was made – the orange up and down line – not only exceeded the line estimate (the smooth downward-sloping curve) but eventually exceeded and stayed above the upper bounds of the uncertainty interval.  And the uncertainty interval itself is curious – why is it so large right after the final actual data point, when uncertainty is at its least, and so much smaller later, when uncertainty would be greatest?

It is easy to take pot shots at specific models and predictions at given points in time; the only predictions that are correct 100% of the time are those that are made after the fact.  And it is important to note that the IHME model changed dramatically over time as new data became available.  Initially, it was based exclusively on the Wuhan data from China, data no longer considered reliable.  Then data from Italy and Spain were added in.  More recently, improvements have been made to the way uncertainty levels are calculated so that they better reflect the real-world scenario.

A Tool For Visualizing Prior Forecasts

Here we present a tool that allows users to go back and visualize changes in model projections over time and compare the projections with what actually happened.  We have collected data and examined models from the IHME and Los Alamos National Lab as these organizations maintained a history of their projections.  We compared them to data retrieved from covidtracker.com (pulled from the Johns Hopkins University aggregation site).   We put the projections and the data together in an interactive visualization tool; the plot above is the result of selecting one model at one point in time.  You can try the COVID Model Visualization Tool here. [note – you need to change “CI” to “UI” in the visualization tool legend]

These models were scored for R2 and MAPE, coefficient of determination and Mean Absolute Percent Error, respectively; both are measures of how well the model predicted the data [1].  Generally, the Los Alamos model outperformed the IHME model which had a temporary very large variance in its predictions for April 5th’s projection.  Both struggled in some areas and both had strong points.

For example, the Los Alamos model generally has a wider variance in the 95% Uncertainty Interval (UI) than the IHME model, but its MAPE and R2 are better.  And, as noted above, the IHME projected 95% UI does not cover actual recorded data beginning on its projection of April 16th.  On the other hand, the IHME projection seems to pick up on the weekly ‘seasonality’ of the data and corrects for the fact that recorded deaths seem to follow a trend throughout the week.  See our blog on this point.

Conclusion

Many critical decisions are being made on the basis of ever-changing projections – projections that the public increasingly are finding surprisingly volatile. There are several ways people have been coping with this volatility:

  • Believe nothing
  • Believe the latest prediction/analysis in your news-feed
  • Cherry-pick and choose the analysis that suits your predilections
  • Form your own picture based on continuous assessment and re-assessment

The fourth option is the hardest one to execute, but it is the proper choice for those who want to find the truth from data.  The visualization tool introduced here should help.

Acknowledgment

The authors would like to thank Sam Ballerini and Andrew Stewart for their work in helping to create the visualization tool introduced here.  Reader comments invited using the comments section below.


Credit: Data Science Central By: Paul Derstine

Previous Post

Dow Futures Stumble Ahead of Ominous Nonfarm Payrolls Report

Next Post

China, Iran, and Russia worked together to call out US hypocrisy on BLM protests

Related Posts

A Plethora of Machine Learning Articles: Part 2
Data Science

A Plethora of Machine Learning Articles: Part 2

March 4, 2021
The Effect IoT Has Had on Software Testing
Data Science

The Effect IoT Has Had on Software Testing

March 3, 2021
Why Cloud Data Discovery Matters for Your Business
Data Science

Why Cloud Data Discovery Matters for Your Business

March 2, 2021
DSC Weekly Digest 01 March 2021
Data Science

DSC Weekly Digest 01 March 2021

March 2, 2021
Companies in the Global Data Science Platforms Resorting to Product Innovation to Stay Ahead in the Game
Data Science

Companies in the Global Data Science Platforms Resorting to Product Innovation to Stay Ahead in the Game

March 2, 2021
Next Post
China, Iran, and Russia worked together to call out US hypocrisy on BLM protests

China, Iran, and Russia worked together to call out US hypocrisy on BLM protests

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recommended

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

January 6, 2019
Microsoft, Google Use Artificial Intelligence to Fight Hackers

Microsoft, Google Use Artificial Intelligence to Fight Hackers

January 6, 2019

Categories

  • Artificial Intelligence
  • Big Data
  • Blockchain
  • Crypto News
  • Data Science
  • Digital Marketing
  • Internet Privacy
  • Internet Security
  • Learn to Code
  • Machine Learning
  • Marketing Technology
  • Neural Networks
  • Technology Companies

Don't miss it

With its acquisition of Auth0, Okta goes all in on CIAM
Internet Security

With its acquisition of Auth0, Okta goes all in on CIAM

March 5, 2021
Survey Finds Many Companies Do Little or No Management of Cloud Spending  
Artificial Intelligence

Survey Finds Many Companies Do Little or No Management of Cloud Spending  

March 5, 2021
UVA doctors give us a glimpse into the future of artificial intelligence
Machine Learning

UVA doctors give us a glimpse into the future of artificial intelligence

March 5, 2021
Labeling Case Study — Agriculture— Pigs’ Productivity, Behavior, and Welfare Image Labeling | by ByteBridge | Feb, 2021
Neural Networks

Labeling Case Study — Agriculture— Pigs’ Productivity, Behavior, and Welfare Image Labeling | by ByteBridge | Feb, 2021

March 5, 2021
Brand Positioning and Competitors’ Positioning
Marketing Technology

Brand Positioning and Competitors’ Positioning

March 5, 2021
Singapore Airlines frequent flyer members hit in third-party data security breach
Internet Security

Singapore Airlines frequent flyer members hit in third-party data security breach

March 5, 2021
NikolaNews

NikolaNews.com is an online News Portal which aims to share news about blockchain, AI, Big Data, and Data Privacy and more!

What’s New Here?

  • With its acquisition of Auth0, Okta goes all in on CIAM March 5, 2021
  • Survey Finds Many Companies Do Little or No Management of Cloud Spending   March 5, 2021
  • UVA doctors give us a glimpse into the future of artificial intelligence March 5, 2021
  • Labeling Case Study — Agriculture— Pigs’ Productivity, Behavior, and Welfare Image Labeling | by ByteBridge | Feb, 2021 March 5, 2021

Subscribe to get more!

© 2019 NikolaNews.com - Global Tech Updates

No Result
View All Result
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News

© 2019 NikolaNews.com - Global Tech Updates