Saturday, April 10, 2021
  • Setup menu at Appearance » Menus and assign menu to Top Bar Navigation
Advertisement
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
No Result
View All Result
Home Neural Networks

Exploding And Vanishing Gradient Problem: Math Behind The Truth

July 7, 2020
in Neural Networks
Exploding And Vanishing Gradient Problem: Math Behind The Truth
590
SHARES
3.3k
VIEWS
Share on FacebookShare on Twitter

The sigmoid’(z1),sigmoid’(z2).. etc are less than 1/4. Because derivative of sigmoid function is less than 1/4. See below. The weight matrices w1,w2,w3,w4 are initialized using gaussian method to have a mean of 0 and standard deviation of 1. Hence ||w(i)|| is less than 1. Therefore, in derivative we multiply such terms which are less than 1 and 1/4. Hence on multiplying such small terms for a huge number of times we get very small gradient which makes the model to almost stop learning.

The reason that if we have deeper models than starting hidden layers will have low speed of learning is: we move deeper as we reach the starting hidden layers during backprop and hence more such terms are involved which makes the gradient small.

You might also like

A Primer of 29 Interactions for AI

Univariate Linear Regression: Explained with Examples | by WeiQin Chuah | Apr, 2021

Disentangling AI, Machine Learning, and Deep Learning | by James Montantes | Apr, 2021

Read it!

Similar is the case with exploding gradient, If we initialize our weight matrices with very large values, then the derivative will be very large and hence the model will have highly unstable training.

Thanks for Reading..guys.

Credit: BecomingHuman By: Manik Soni

Previous Post

Home router warning: They're riddled with known flaws and run ancient, unpatched Linux

Next Post

Mangalore University College organizes webinar on Artificial Intelligence & Machine Learning

Related Posts

A Primer of 29 Interactions for AI
Neural Networks

A Primer of 29 Interactions for AI

April 10, 2021
Univariate Linear Regression: Explained with Examples | by WeiQin Chuah | Apr, 2021
Neural Networks

Univariate Linear Regression: Explained with Examples | by WeiQin Chuah | Apr, 2021

April 10, 2021
Disentangling AI, Machine Learning, and Deep Learning | by James Montantes | Apr, 2021
Neural Networks

Disentangling AI, Machine Learning, and Deep Learning | by James Montantes | Apr, 2021

April 9, 2021
Artificial Intelligence Courses, books, and programs for entrepreneurs | by Farhad Rahbarnia | Apr, 2021
Neural Networks

Artificial Intelligence Courses, books, and programs for entrepreneurs | by Farhad Rahbarnia | Apr, 2021

April 9, 2021
Co-founder Guide: Time and Goal Management | by Farhad Rahbarnia | Apr, 2021
Neural Networks

Co-founder Guide: Time and Goal Management | by Farhad Rahbarnia | Apr, 2021

April 9, 2021
Next Post
Mangalore University College organizes webinar on Artificial Intelligence & Machine Learning

Mangalore University College organizes webinar on Artificial Intelligence & Machine Learning

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recommended

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

January 6, 2019
Microsoft, Google Use Artificial Intelligence to Fight Hackers

Microsoft, Google Use Artificial Intelligence to Fight Hackers

January 6, 2019

Categories

  • Artificial Intelligence
  • Big Data
  • Blockchain
  • Crypto News
  • Data Science
  • Digital Marketing
  • Internet Privacy
  • Internet Security
  • Learn to Code
  • Machine Learning
  • Marketing Technology
  • Neural Networks
  • Technology Companies

Don't miss it

Vue.js vs AngularJS Development in 2021: Side-by-Side Comparison
Data Science

Vue.js vs AngularJS Development in 2021: Side-by-Side Comparison

April 10, 2021
IBM releases Qiskit modules that use quantum computers to improve machine learning
Machine Learning

IBM releases Qiskit modules that use quantum computers to improve machine learning

April 10, 2021
Hackers Tampered With APKPure Store to Distribute Malware Apps
Internet Privacy

Hackers Tampered With APKPure Store to Distribute Malware Apps

April 10, 2021
5 Dominating IoT Trends Positively Impacting Telecom Sector in 2021
Data Science

5 Dominating IoT Trends Positively Impacting Telecom Sector in 2021

April 10, 2021
One-stop machine learning platform turns health care data into insights | MIT News
Machine Learning

One-stop machine learning platform turns health care data into insights | MIT News

April 10, 2021
Machine learning: is there a limit to technological patents in Brazil?
Machine Learning

Disclosing AI Inventions – Part I: Identifying the Unique Disclosure Issues

April 10, 2021
NikolaNews

NikolaNews.com is an online News Portal which aims to share news about blockchain, AI, Big Data, and Data Privacy and more!

What’s New Here?

  • Vue.js vs AngularJS Development in 2021: Side-by-Side Comparison April 10, 2021
  • IBM releases Qiskit modules that use quantum computers to improve machine learning April 10, 2021
  • Hackers Tampered With APKPure Store to Distribute Malware Apps April 10, 2021
  • 5 Dominating IoT Trends Positively Impacting Telecom Sector in 2021 April 10, 2021

Subscribe to get more!

© 2019 NikolaNews.com - Global Tech Updates

No Result
View All Result
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News

© 2019 NikolaNews.com - Global Tech Updates