Saturday, March 6, 2021
  • Setup menu at Appearance » Menus and assign menu to Top Bar Navigation
Advertisement
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
No Result
View All Result
Home Neural Networks

Basics of Networking: The Neural Way

April 1, 2020
in Neural Networks
Basics of Networking: The Neural Way
587
SHARES
3.3k
VIEWS
Share on FacebookShare on Twitter
Source

Inspired by neuroscience, it is no longer a mystery for ML engineers how a perceptron takes in inputs, applies a linear combination, and if that linear combination is greater or lesser than some threshold value, produces an output of 1 or 0 respectively. In the language of mathematics, it looks like this:

In this formula, the Greek letter Sigma ∑ refers to the summation, while i means to iterate over input (x) and weight (w) pairings.

You might also like

Labeling Service Case Study — Video Annotation — License Plate Recognition | by ByteBridge | Feb, 2021

5 Tech Trends Redefining the Home Buying Experience in 2021 | by Iflexion | Mar, 2021

Labeling Case Study — Agriculture— Pigs’ Productivity, Behavior, and Welfare Image Labeling | by ByteBridge | Feb, 2021

For the sake of simplicity, let’s replace the threshold with what’s known as the neuron’s bias. Now we can rewrite the equation as:

In this case, bias could be represented as — threshold. In other words, bias denotes how easy it is to get the neuron to output a 1. While a big bias would mean that it is easy for the neuron to output a 1, in the case of a negative bias it would be more difficult.

There are a couple of reasons to make this small change. We randomly assign numbers for weights (we don’t know the right ones beforehand), and as the neural network trains, it makes incremental changes to those weights to produce more accurate outputs.

A function that transforms the values or states the conditions for the decision of the output neuron is known as an activation function. This is just one of the various activation functions called as ‘Heaviside step function’.

Other activation functions include the sigmoid, tanh, and softmax functions, and they each have their own purpose.

Even when dealing with absolutes (1s and 0s; yes and no’s), it would be beneficial to have the output give an intermediary value. It is a bit like answering “maybe” when asked a yes-or-no question you have no idea about, instead of guessing. This is essentially the advantage of the sigmoid function over the Heaviside.

Like the Heaviside step function, the sigmoid has its values between 0 and 1. Yet this time, there is no step; it has been smoothed out to create a continuous line. Thus, the output can be considered as a probability of being a success (a 1), or a yes. Having this feature is particularly important for the network’s learning capabilities.

So far, we have explored the architecture of the perceptron, the simplest neural network model, and seen two activation functions: the Heaviside step function and the sigmoid function.

Yet, if we were to make these networks more complicated, any number of layers in between these two are known as hidden layers. The more the number of layers, the more nuanced the decision-making can get.

The networks often go by different names: deep feedforward networks, feedforward neural networks, or multi-layer perceptrons (MLP). They are called feedforward networks because the information flows in one general (forward) direction, where mathematical functions are applied at each stage. In fact, they are called ‘networks’ because of this chain of functions (the output of the first layer function is the input to the second layer and that output is the input to the third layer and so on). The length of that chain gives the depth of the model, and this is actually where the term ‘deep’ in deep learning comes from.

Adding hidden layers can allow the neural network to make more complex decisions, but more on that, and how neural networks learn through a process known as backpropagation.

Can you imagine what a daunting task thinking would be if we were to start our thinking process from scratch every time. For instance, while reading a book we process each word based on our understanding of previous words rather than start thinking from scratch again as our thoughts are persistent.

From the perspective of a traditional neural network, it would not be so clear how to use the reasoning about previous events in a film to inform later scenes (Koenker et al, 2001).

Another way to think of this is that the network contains “memory” nodes that depend on all previous memory. For a network with 3 layers (input, hidden, output), and 2 activation functions (f & g), a standard feed-forward network could be represented as (Fig. 1):

hidden = f(input)

output = g(hidden)

Figure 1.Overview of Feed Forward Network

These Neural Networks receive an input (a single vector), and transform it through a series of hidden layers. Each hidden layer is made up of a set of neurons, where each neuron is fully connected to all neurons in the previous layer, and where there is no connection among neurons in a single layer function. The last fully-connected layer is called the “output layer” and in classification settings it represents the class scores (Koenker et al, 2001).

Convolutional Networks

In the past, neural networks for complex tasks were too inefficient to be used in practice (Hopfield, 1982). A more practical method was devised by Yann LeCun and his team of researchers in the AT&T labs by applying a convolutional neural network to a handwritten digit recognition task (Hopfield, 1982). They came up with the idea that instead of having the n-number of weights (nodes in the layers after the input layer) process images fully, one could go over image to discover related features. Not only does it drastically reduce the computation requirements, but it also makes the weights generalizable (i.e. once a feature is learned it can be applied to the same image in a different location and also in different images) (Koenker et al, 2001).

We can think of images as two-dimensional functions. In the case of mage transformations convolutions occur in such a way so that the image function is convolved with a very small, local function called a “kernel.”

The kernel slides to every position of the image and computes a new pixel as a weighted sum of the pixels it floats over.

Figure 2.Overview of an Kernel

For example, by taking the average of a 3×3 box of pixels, an image can be blurred as the kernel takes the value 1/91/9 on each pixel in the box. If side by side pixels are similar, approximately zero will be returned. In the case of edges, adjacent pixels are very different in the direction perpendicular to the edge.

So, what is the relationship of convolution to convolutional neural networks?

Consider a 1-dimensional convolutional layer with inputs {xn} and outputs {yn}.

yn=A(xn,xn+1,…)yn=A(xn,xn+1,…)

Generally, A would be multiple neurons. But suppose it is a single neuron for a moment. Recall that a typical neuron in a neural network is described by:

x0x0, x1x1… are the inputs while the weights, w0w0, w1w1, … describe how the neuron connects to its inputs.

If the weight is negative an input inhibits the neuron from firing,

If it is positive, it encourages.

It’s this wiring of neurons, describing all the weights and which ones are identical, that are the main purpose of convolution.

The aim is to have a weight matrix, WW:

For example, as seen in this formula, Each row of the matrix describes the weights connecting a neuron to its inputs.

As there are multiple copies of the same neuron, many weights appear in multiple positions, which corresponds to the equations:

So while, normally, a weight matrix connects every input to every neuron with different weights:

It should be noted that while he same weights appear in various positions, neurons don’t connect to many possible inputs, therefore there are lots of zeros.

Multiplying by the above matrix would be equal to convolving with […0,w1,w0,0…][…0,w1,w0,0…]. Yet, what about two-dimensional convolutional layers?

As seen in Figure 3, the wiring of a two dimensional convolutional layer corresponds to a two-dimensional convolution.

Let’s consider our example of using a convolution to detect edges in an image, above, by sliding a kernel around and applying it to every patch. In this case, a convolutional layer will apply a neuron to every patch of the image.

Figure 3. Example for Convolutional Neural Networks (github)

One of the biggest advantages provided by CNN is that the input is recognized as an image by having neurons arranged in 3 dimensions: width, height, depth. The neurons in a layer will only be connected to a small section of the layer before it, rather than all neurons being fully-connected. The following figure provides a visualization:

Figure 4. Overview of a CNN (github)

In Fig. 5, on the left hand side there is a regular 3-layer Neural Network. On the right hand side, a ConvNet arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. The red input layer holds the image, furthermore its width and height would be the dimensions of the image, while the depth would be 3 (Red, Green, Blue channels).

Main type of layers to build ConvNet architectures are:

● Convolutional Layer,

● Pooling Layer, and

● Fully-Connected Layer.

Below is an example for a simple ConvNet for CIFAR-10 classification which has the architecture [INPUT — CONV — RELU — POOL — FC]. In more detail (Eliasmith, 2013):

● INPUT [32x32x3] will hold the raw pixel values of the image, in this case an image of width 32, height 32, and with three color channels R,G,B.

● CONV layer will deliver the output of neurons connected to local regions which may result in volume such as [32x32x12] if the decision is to use 12 filters.

● RELU layer applies the max(0,x) thresholding at zero. The volume remains unchanged ([32x32x12]).

● POOL layer will perform a downsampling operation along the spatial dimensions (width, height), resulting in volume such as [16x16x12].

● FC (i.e. fully-connected) layer will compute the class scores, resulting in volume of size [1x1x10], where each of the 10 numbers correspond to a class score, such as among the 10 categories of CIFAR-10.

In this way, a transformation of the original image takes place layer by layer as CNN moves the original pixel values to the final class scores. While some layers contain parameters others don’t. It should be taken into account that the transformations performed by CONV/FC layers are not only a function of the activations in the input volume, but also of the parameters (neurons’ biased and weights) (Koenker et al, 2001).

Figure 5.The activations of an example ConvNet architecture (github)

In Figure 5, while the initial volume stores the raw image pixels (left) the last volume stores the class scores (right). Each column refers to the volume of activations along the processing path. Given the difficulty of visualizing 3D volumes, each volume’s slices are laid out in rows. The last layer volume refers to the scores for each class.

This may sound a bit confusing, yet who said that neural networking would be easy. Yet, once these basics are grasped in detail, it should also be not a rocket science!

Credit: BecomingHuman By: The AI LAB

Previous Post

What Is Marketing Orchestration? | Infographic

Next Post

Tipalti Introduces Tipalti Pi — Artificial Intelligence for Accounts Payable

Related Posts

Labeling Service Case Study — Video Annotation — License Plate Recognition | by ByteBridge | Feb, 2021
Neural Networks

Labeling Service Case Study — Video Annotation — License Plate Recognition | by ByteBridge | Feb, 2021

March 6, 2021
5 Tech Trends Redefining the Home Buying Experience in 2021 | by Iflexion | Mar, 2021
Neural Networks

5 Tech Trends Redefining the Home Buying Experience in 2021 | by Iflexion | Mar, 2021

March 6, 2021
Labeling Case Study — Agriculture— Pigs’ Productivity, Behavior, and Welfare Image Labeling | by ByteBridge | Feb, 2021
Neural Networks

Labeling Case Study — Agriculture— Pigs’ Productivity, Behavior, and Welfare Image Labeling | by ByteBridge | Feb, 2021

March 5, 2021
8 concepts you must know in the field of Artificial Intelligence | by Diana Diaz Castro | Feb, 2021
Neural Networks

8 concepts you must know in the field of Artificial Intelligence | by Diana Diaz Castro | Feb, 2021

March 5, 2021
The Examples and Benefits of AI in Healthcare: From accurate diagnosis to remote patient monitoring | by ITRex Group | Mar, 2021
Neural Networks

The Examples and Benefits of AI in Healthcare: From accurate diagnosis to remote patient monitoring | by ITRex Group | Mar, 2021

March 4, 2021
Next Post
Tipalti Introduces Tipalti Pi — Artificial Intelligence for Accounts Payable

Tipalti Introduces Tipalti Pi — Artificial Intelligence for Accounts Payable

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recommended

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

January 6, 2019
Microsoft, Google Use Artificial Intelligence to Fight Hackers

Microsoft, Google Use Artificial Intelligence to Fight Hackers

January 6, 2019

Categories

  • Artificial Intelligence
  • Big Data
  • Blockchain
  • Crypto News
  • Data Science
  • Digital Marketing
  • Internet Privacy
  • Internet Security
  • Learn to Code
  • Machine Learning
  • Marketing Technology
  • Neural Networks
  • Technology Companies

Don't miss it

These two unusual versions of ransomware tell us a lot about how attacks are evolving
Internet Security

These two unusual versions of ransomware tell us a lot about how attacks are evolving

March 6, 2021
Researchers Find 3 New Malware Strains Used by SolarWinds Hackers
Internet Privacy

Researchers Find 3 New Malware Strains Used by SolarWinds Hackers

March 6, 2021
Analysis: The increasing scope of UK cryptocurrency regulation
Blockchain

Analysis: The increasing scope of UK cryptocurrency regulation

March 6, 2021
Will AI Replace Lawyers & Other Myths: Legal AI Mythbusters | Onit
Machine Learning

Will AI Replace Lawyers & Other Myths: Legal AI Mythbusters | Onit

March 6, 2021
Labeling Service Case Study — Video Annotation — License Plate Recognition | by ByteBridge | Feb, 2021
Neural Networks

Labeling Service Case Study — Video Annotation — License Plate Recognition | by ByteBridge | Feb, 2021

March 6, 2021
Microsoft: We’ve found three more pieces of malware used by the SolarWinds attackers
Internet Security

Microsoft: We’ve found three more pieces of malware used by the SolarWinds attackers

March 6, 2021
NikolaNews

NikolaNews.com is an online News Portal which aims to share news about blockchain, AI, Big Data, and Data Privacy and more!

What’s New Here?

  • These two unusual versions of ransomware tell us a lot about how attacks are evolving March 6, 2021
  • Researchers Find 3 New Malware Strains Used by SolarWinds Hackers March 6, 2021
  • Analysis: The increasing scope of UK cryptocurrency regulation March 6, 2021
  • Will AI Replace Lawyers & Other Myths: Legal AI Mythbusters | Onit March 6, 2021

Subscribe to get more!

© 2019 NikolaNews.com - Global Tech Updates

No Result
View All Result
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News

© 2019 NikolaNews.com - Global Tech Updates