How to evaluate the Machine Learning models? — Part 3 | by RAVI SHEKHAR TIWARI | Oct, 2020

FN (False Negative) = Predicted value is Falsely predicted and the actual value is positive but the model predicted negative value . It is also known as Type 2 error.

FP (False Positive) = Predicted value is Falsely predicted and the actual value is negative but the model predicted positive value. It is also known as Type 1 error.

TN (True Negative) = Predicted value matches the actual value. The model predicted the negative value and the actual value is also negative.

TP (True Positive)= Predicted value matches the actual value. The model predicted the positive value and the actual value is also positive.

Why we need the Confusion metric?

Suppose we have dataset which has 90 dog images and 10 cat images and a model is trained to predict dog, where a trained model predicted 80 correct sample and 20 incorrect sample.

Jupyter Notebook: Link

2.1. Accuracy

Accuracy is defined as the no of correctly predicted sample( TN+ TP) to the total no. of samples (TN+TP+FN+FP).

1. How to automatically deskew (straighten) a text image using OpenCV

2. Explanation of YOLO V4 a one stage detector

3. 5 Best Artificial Intelligence Online Courses for Beginners in 2020

4. A Non Mathematical guide to the mathematics behind Machine Learning

In above image, we can see accuracy is giving wrong data about the result i.e. model is saying it will predict dog 80% of the time, actually it is doing opposite. We saw that, the accuracy of the model is very good 80% but dataset is imbalance i.e. one class has more no of sample than other which makes model biased towards one class and hence in real world application it will not be able to work according to our expectation. So we need other metrics also which support our model more.

Jupyter Notebook: Link

2.2. TPR

TPR is known as True Positive rate, which is ration of TP: ( TP+FN). Value of TPR is directly proportional to goodness of model i.e. TPR increases model becomes better. Alternative to accuracy metric.

Jupyter Notebook: Link

2.3. FNR

FNR is known as False Negative rate, which is ration of FN: ( TP+FN). Value of FNR is indirectly proportional to goodness of model i.e. FNR decreases model becomes better. Alternative to accuracy metric.

Jupyter Notebook: Link

2.4. TNR

TNR is known as True Negative rate, which is ration of TN: ( FP+TN). Value of TNR is directly proportional to goodness of model i.e. FPR increases model becomes better. Alternative to accuracy metric.

Jupyter Notebook: Link

2.5. FPR

FPR is known as False Positive rate, which is ration of FP: ( FP+TN). Value of FNR is indirectly proportional to goodness of model i.e. FPR decreases model becomes better. Alternative to accuracy metric.

Jupyter Notebook: Link

2.6. Precision

Precision tell us how many of the samples were predicted correctly to the all positive predicted samples i.e. TP:(TP+FP). We cannot afford to have more FP FP because it will degrade our model. It is very useful when FP is more important than the FN especially in terms of healthcare.

Jupyter Notebook: Link

2.7. Recall

Recall tell us how many of the samples were predicted correctly to the all the actual positive predicted samples i.e. TP:(TP+FN). We cannot afford to have more FN because it will degrade our model. It is very useful when FN is more important than the FP especially in terms of security of places.

High precision we will have low recall and vice versa. It is trade of between these two metric similar to Bias-Variance Tradeoff, so it depends upon the use case which metric we can select.

Jupyter Notebook: Link

2.8. F1-score

As we have discussed above we precision and recall are inversely related and in some cases where we are unable to decide which metric to use so we procced with F1-score which is combination of precision and recall. F1-score is harmonic mean of precision and recall has high value when precision and recall are equal. It is always used with other evaluation metric because it lacks interpretation.

Jupyter Notebook: Link

2.9. F-Beta Sore.

We have discussed earlier that F-score uses geometric mean instead of the arithmetic mean because Harmonic mean punishes penalize bigger value more.

If P=0 recall=1 then arithmetic mean=0.5, it is clear it is output of dumb model- ignore the input feature and predict one of the classes only. Now if we take Harmonic mean of Precision and recall, we will get 0. Some researchers can give more importance/ weight to precision or recall, so they can attach the adjustable parameter known as beta-square which will give beta times importance to precision or recall respectively.

Jupyter Notebook: Link

2.10. Thresholding

Thresholding is a technique in which is applied to the probability of the prediction of the models. Suppose you have used “softmax” as the activation layer in the output layer — gives probability of the prediction. In thresholding we can select a particular value so that if the predicted value is higher to the particular value we can classify into one class/label and if the predicted value is lower that the particular value we can classify into another class/label. This particular value is known as threshold and the process is called thresholding.

The threshold is determined by considering TP, TN, FP, and FN as well as the use case or business scenario.

Jupyter Notebook: Link

2.11. AUC — ROC

AUC is referred as Area Under the Curve i.e. the the amount of area which are under the line( linear or non linear). ROC means Receiver Operating Characteristic, initially it was used for differentiating noise from not noise but in recent years it is used very frequently in binary classification.

ROC gives trade-off between TP and FP where x- axis represent FPR and y-axis represent TPR. Total area of ROC is 1 unit because TPR and FPR value has range[0,1]. It can be clearly analyzed that more the area under the curve better the model as it can clearly differentiate between positive and negative class but AUC in ROC should be greater than 0.5 unit.

AUC = 1, then the binary classification model is able to perfectly differentiate between all the Positive and the Negative class points correctly.

AUC =0, then the classifier would be predicting all Negatives as Positives, and all Positives as Negatives.

0.5<AUC<1, there is a high chance that the binary classification model will be able to differentiate between the positive class values from the negative class values. This is so because the classifier is able to detect more numbers of True positives and True negatives than False negatives and False positives.

AUC=0.5, then the classifier is not able to distinguish between Positive and Negative class points. Meaning either the classifier is predicting random class for all the dataset.

Few more terms:

TPR is also known as Sensivity.

Specificity = 1- FPR

So, don’t get confused when you will see the combination of these words in any other blog/articles.

Steps to build AUC-ROC:

1. Arrange the dataset on the basis on the predictions of probability in decreasing order.
2. Set the threshold as the ith row of the prediction (for 1st take 1 prediction for second take 2nd as threshold).
3. Calculate the value of TPR and FPR.
4. Iterate through all the data point in dataset and repeat the step 1, 2 and 3.
5. Plot the value of TPR (Sensivity) and FPR (Specificity = 1- FPR).
6. Select the model with maximum AUC.

Jupyter Notebook: Link

2.11. AUC — PRC

AUC is referred as Area Under the Curve i.e. the the amount of area which are under the line( linear or non linear). PRC means Precision Recall Curve, it is used very frequently in binary classification.

PRC gives trade-off between Precision and Recall where x- axis represent Precision and y-axis represent Recall. Total area of PRC is 1 unit because Precision and Recall value has range[0,1]. It can be clearly analyzed that more the area under the curve better the model as it can clearly differentiate between positive and negative class but AUC in PRC should be greater than 0.5 unit because of imbalance dataset as small change in prediction can show drastic change in AUC-PRC curve.

A binary classification model with correct predictions is depicted as a point at a coordinate of (1,1). A skillful model is represented by a curve that bows towards a coordinate of (1,1). A dumb binary classification model will be a horizontal line on the plot with a precision that is proportional to the number of positive examples in the dataset. For a balanced dataset this will be 0.5.The focus of the PR curve on the minority class makes it an effective diagnostic for imbalanced binary classification models. Precision and recall make it possible to assess the performance of a classifier on the minority class. Precision-recall curves (PR curves) are recommended for highly skewed domains where ROC curves may provide an excessively optimistic view of the performance

Steps to build AUC-PRC:

1. Arrange the dataset on the basis on the predictions of probability in decreasing order.
2. Set the threshold as the ith row of the prediction (for 1st take 1 prediction for second take 2nd as threshold).
3. Calculate the value of Precision and Recall .
4. Iterate through all the data point in dataset and repeat the step 1, 2 and 3.
5. Plot the value of Precision and Recall.
6. Select the model with maximum AUC.

Jupyter Notebook: Link

Micro average is the precision or recall of f1-score of classes.