CBSE Textbook: Evaluating Models

 Page 1


                   
Introduction 
Till now we have learnt about the 4 stages of AI project cycle, viz. Problem scoping, Data 
acquisition, Data exploration and modelling. While in modelling we can make different types 
of models, how do we check if one’s better than the other? That’s where Evaluation comes 
into play. In the Evaluation stage, we will explore different methods of evaluating an AI model. 
Model Evaluation is an integral part of the model development process. It helps to find the 
best model that represents our data and how well the chosen model will work in the future 
 
3.1: Importance of Model Evaluation 
What is evaluation? 
 
? Model evaluation is the process of using different 
evaluation metrics to understand a machine learning 
model’s performance 
? An AI model gets better with constructive feedback 
? You build a model, get feedback from metrics, make 
improvements and continue until you achieve a 
desirable accuracy 
 
 
 
 
 
 
• It’s like the report card of your school 
• There are many parameters like grades, percentage, 
percentiles, ranks 
• Your academic performance gets evaluated and you know 
where to work more to get better 
Page 2


                   
Introduction 
Till now we have learnt about the 4 stages of AI project cycle, viz. Problem scoping, Data 
acquisition, Data exploration and modelling. While in modelling we can make different types 
of models, how do we check if one’s better than the other? That’s where Evaluation comes 
into play. In the Evaluation stage, we will explore different methods of evaluating an AI model. 
Model Evaluation is an integral part of the model development process. It helps to find the 
best model that represents our data and how well the chosen model will work in the future 
 
3.1: Importance of Model Evaluation 
What is evaluation? 
 
? Model evaluation is the process of using different 
evaluation metrics to understand a machine learning 
model’s performance 
? An AI model gets better with constructive feedback 
? You build a model, get feedback from metrics, make 
improvements and continue until you achieve a 
desirable accuracy 
 
 
 
 
 
 
• It’s like the report card of your school 
• There are many parameters like grades, percentage, 
percentiles, ranks 
• Your academic performance gets evaluated and you know 
where to work more to get better 
                  
 
Need of model evaluation 
In essence, model evaluation is like giving your AI model a report card. It helps you understand its 
strengths, weaknesses, and suitability for the task at hand. This feedback loop is essential for 
building trustworthy and reliable AI systems. 
After understanding the need for Model Evaluation, let’s know how to begin with the process. 
There can be different Evaluation techniques, depending of the type and purpose of the model. 
 
 
3.2: Splitting the training set data for Evaluation 
Train-test split 
? The train-test split is a technique for evaluating the performance of a machine learning 
algorithm 
? It can be used for any supervised learning algorithm 
? The procedure involves taking a dataset and dividing it into two subsets: The training 
dataset and the testing dataset 
? The train-test procedure is appropriate when there is a sufficiently large dataset available 
 
 
 
Page 3


                   
Introduction 
Till now we have learnt about the 4 stages of AI project cycle, viz. Problem scoping, Data 
acquisition, Data exploration and modelling. While in modelling we can make different types 
of models, how do we check if one’s better than the other? That’s where Evaluation comes 
into play. In the Evaluation stage, we will explore different methods of evaluating an AI model. 
Model Evaluation is an integral part of the model development process. It helps to find the 
best model that represents our data and how well the chosen model will work in the future 
 
3.1: Importance of Model Evaluation 
What is evaluation? 
 
? Model evaluation is the process of using different 
evaluation metrics to understand a machine learning 
model’s performance 
? An AI model gets better with constructive feedback 
? You build a model, get feedback from metrics, make 
improvements and continue until you achieve a 
desirable accuracy 
 
 
 
 
 
 
• It’s like the report card of your school 
• There are many parameters like grades, percentage, 
percentiles, ranks 
• Your academic performance gets evaluated and you know 
where to work more to get better 
                  
 
Need of model evaluation 
In essence, model evaluation is like giving your AI model a report card. It helps you understand its 
strengths, weaknesses, and suitability for the task at hand. This feedback loop is essential for 
building trustworthy and reliable AI systems. 
After understanding the need for Model Evaluation, let’s know how to begin with the process. 
There can be different Evaluation techniques, depending of the type and purpose of the model. 
 
 
3.2: Splitting the training set data for Evaluation 
Train-test split 
? The train-test split is a technique for evaluating the performance of a machine learning 
algorithm 
? It can be used for any supervised learning algorithm 
? The procedure involves taking a dataset and dividing it into two subsets: The training 
dataset and the testing dataset 
? The train-test procedure is appropriate when there is a sufficiently large dataset available 
 
 
 
                  
 
Need of Train-test split 
? The train dataset is used to make the model learn 
? The input elements of the test dataset are provided to the trained model. The model makes 
predictions, and the predicted values are compared to the expected values 
? The objective is to estimate the performance of the machine learning model on new data: 
data not used to train the model 
 
This is how we expect to use the model in practice. Namely, to fit it on available data with known 
inputs and outputs, then make predictions on new examples in the future where we do not 
have the expected output or target values. 
 
Remember that It’s not recommended to use the data we used to build the model to evaluate 
it. This is because our model will simply remember the whole training set, and will therefore 
always predict the correct label for any point in the training set. This is known as overfitting. 
 
 
3.3: Accuracy and Error 
? Bob and Billy went to a concert 
? Bob brought Rs 300 and Billy brought Rs 550 as 
the entry fee for that 
? The entry fee per person was Rs 500 
? Can you tell: 
? Who is more accurate? Bob or Billy? 
? How much is the error for both Bob and Billy in estimating the concert entry fee? 
 
 
You will learn more about the concepts including train test split and cross validation in 
higher classes. 
Page 4


                   
Introduction 
Till now we have learnt about the 4 stages of AI project cycle, viz. Problem scoping, Data 
acquisition, Data exploration and modelling. While in modelling we can make different types 
of models, how do we check if one’s better than the other? That’s where Evaluation comes 
into play. In the Evaluation stage, we will explore different methods of evaluating an AI model. 
Model Evaluation is an integral part of the model development process. It helps to find the 
best model that represents our data and how well the chosen model will work in the future 
 
3.1: Importance of Model Evaluation 
What is evaluation? 
 
? Model evaluation is the process of using different 
evaluation metrics to understand a machine learning 
model’s performance 
? An AI model gets better with constructive feedback 
? You build a model, get feedback from metrics, make 
improvements and continue until you achieve a 
desirable accuracy 
 
 
 
 
 
 
• It’s like the report card of your school 
• There are many parameters like grades, percentage, 
percentiles, ranks 
• Your academic performance gets evaluated and you know 
where to work more to get better 
                  
 
Need of model evaluation 
In essence, model evaluation is like giving your AI model a report card. It helps you understand its 
strengths, weaknesses, and suitability for the task at hand. This feedback loop is essential for 
building trustworthy and reliable AI systems. 
After understanding the need for Model Evaluation, let’s know how to begin with the process. 
There can be different Evaluation techniques, depending of the type and purpose of the model. 
 
 
3.2: Splitting the training set data for Evaluation 
Train-test split 
? The train-test split is a technique for evaluating the performance of a machine learning 
algorithm 
? It can be used for any supervised learning algorithm 
? The procedure involves taking a dataset and dividing it into two subsets: The training 
dataset and the testing dataset 
? The train-test procedure is appropriate when there is a sufficiently large dataset available 
 
 
 
                  
 
Need of Train-test split 
? The train dataset is used to make the model learn 
? The input elements of the test dataset are provided to the trained model. The model makes 
predictions, and the predicted values are compared to the expected values 
? The objective is to estimate the performance of the machine learning model on new data: 
data not used to train the model 
 
This is how we expect to use the model in practice. Namely, to fit it on available data with known 
inputs and outputs, then make predictions on new examples in the future where we do not 
have the expected output or target values. 
 
Remember that It’s not recommended to use the data we used to build the model to evaluate 
it. This is because our model will simply remember the whole training set, and will therefore 
always predict the correct label for any point in the training set. This is known as overfitting. 
 
 
3.3: Accuracy and Error 
? Bob and Billy went to a concert 
? Bob brought Rs 300 and Billy brought Rs 550 as 
the entry fee for that 
? The entry fee per person was Rs 500 
? Can you tell: 
? Who is more accurate? Bob or Billy? 
? How much is the error for both Bob and Billy in estimating the concert entry fee? 
 
 
You will learn more about the concepts including train test split and cross validation in 
higher classes. 
                  
 
Accuracy 
? Accuracy is an evaluation metric that allows you to measure the total number of 
predictions a model gets right. 
? The accuracy of the model and performance of the model is directly proportional, and 
hence better the performance of the model, the more accurate are the predictions. 
Error 
? Error can be described as an action that is inaccurate or wrong. 
? In Machine Learning, the error is used to see how accurately our model can predict data it 
uses to learn new, unseen data. 
? Based on our error, we choose the machine learning model which performs best for a 
particular dataset. 
 
 
Error refers to the difference between a model's prediction and the actual outcome. It quantifies how often 
the model makes mistakes. 
 
 
Imagine you're training a model to predict if you have a certain disease (classification task). 
• Error: If the model predicts you don’t have a disease but you actually have a disease, that's 
an error. The error quantifies how far off the prediction was from reality. 
• Accuracy: If the model correctly predicts disease or no disease for a particular period, it  
has 100% accuracy for that period. 
 
 
Page 5


                   
Introduction 
Till now we have learnt about the 4 stages of AI project cycle, viz. Problem scoping, Data 
acquisition, Data exploration and modelling. While in modelling we can make different types 
of models, how do we check if one’s better than the other? That’s where Evaluation comes 
into play. In the Evaluation stage, we will explore different methods of evaluating an AI model. 
Model Evaluation is an integral part of the model development process. It helps to find the 
best model that represents our data and how well the chosen model will work in the future 
 
3.1: Importance of Model Evaluation 
What is evaluation? 
 
? Model evaluation is the process of using different 
evaluation metrics to understand a machine learning 
model’s performance 
? An AI model gets better with constructive feedback 
? You build a model, get feedback from metrics, make 
improvements and continue until you achieve a 
desirable accuracy 
 
 
 
 
 
 
• It’s like the report card of your school 
• There are many parameters like grades, percentage, 
percentiles, ranks 
• Your academic performance gets evaluated and you know 
where to work more to get better 
                  
 
Need of model evaluation 
In essence, model evaluation is like giving your AI model a report card. It helps you understand its 
strengths, weaknesses, and suitability for the task at hand. This feedback loop is essential for 
building trustworthy and reliable AI systems. 
After understanding the need for Model Evaluation, let’s know how to begin with the process. 
There can be different Evaluation techniques, depending of the type and purpose of the model. 
 
 
3.2: Splitting the training set data for Evaluation 
Train-test split 
? The train-test split is a technique for evaluating the performance of a machine learning 
algorithm 
? It can be used for any supervised learning algorithm 
? The procedure involves taking a dataset and dividing it into two subsets: The training 
dataset and the testing dataset 
? The train-test procedure is appropriate when there is a sufficiently large dataset available 
 
 
 
                  
 
Need of Train-test split 
? The train dataset is used to make the model learn 
? The input elements of the test dataset are provided to the trained model. The model makes 
predictions, and the predicted values are compared to the expected values 
? The objective is to estimate the performance of the machine learning model on new data: 
data not used to train the model 
 
This is how we expect to use the model in practice. Namely, to fit it on available data with known 
inputs and outputs, then make predictions on new examples in the future where we do not 
have the expected output or target values. 
 
Remember that It’s not recommended to use the data we used to build the model to evaluate 
it. This is because our model will simply remember the whole training set, and will therefore 
always predict the correct label for any point in the training set. This is known as overfitting. 
 
 
3.3: Accuracy and Error 
? Bob and Billy went to a concert 
? Bob brought Rs 300 and Billy brought Rs 550 as 
the entry fee for that 
? The entry fee per person was Rs 500 
? Can you tell: 
? Who is more accurate? Bob or Billy? 
? How much is the error for both Bob and Billy in estimating the concert entry fee? 
 
 
You will learn more about the concepts including train test split and cross validation in 
higher classes. 
                  
 
Accuracy 
? Accuracy is an evaluation metric that allows you to measure the total number of 
predictions a model gets right. 
? The accuracy of the model and performance of the model is directly proportional, and 
hence better the performance of the model, the more accurate are the predictions. 
Error 
? Error can be described as an action that is inaccurate or wrong. 
? In Machine Learning, the error is used to see how accurately our model can predict data it 
uses to learn new, unseen data. 
? Based on our error, we choose the machine learning model which performs best for a 
particular dataset. 
 
 
Error refers to the difference between a model's prediction and the actual outcome. It quantifies how often 
the model makes mistakes. 
 
 
Imagine you're training a model to predict if you have a certain disease (classification task). 
• Error: If the model predicts you don’t have a disease but you actually have a disease, that's 
an error. The error quantifies how far off the prediction was from reality. 
• Accuracy: If the model correctly predicts disease or no disease for a particular period, it  
has 100% accuracy for that period. 
 
 
                  
 
Key Points: 
• Here the goal is to minimize error and maximize accuracy. 
• Real-world data can be messy, and even the best models make mistakes. 
• Sometimes, focusing solely on accuracy might not be ideal. For instance, in medical 
diagnosis, a model with slightly lower accuracy but a strong focus on avoiding incorrectly 
identifying a healthy person as sick might be preferable. 
• Choosing the right error or accuracy metric depends on the specific task and its 
requirements. 
Understanding both error and accuracy is crucial for effectively evaluating and improving AI 
models. 
Activity 1: Find the accuracy of the AI model 
 
 
Calculate the accuracy of the House Price prediction AI model 
• Read the instructions and fill in the blank cells in the table. 
• The formula for finding error and accuracy is shown in the table 
• Accuracy of the AI model is the mean accuracy of all five samples 
• Percentage accuracy can be seen by multiplying the accuracy with 100 
 
Predicted House 
Price (USD) 
Actual House 
Price (USD) 
Error Abs 
(Actual-Predicted) 
Error Rate 
(Error/Actual) 
Accuracy 
(1-Error rate) 
Accuracy% 
(Accuracy*100)% 
391k 402k Abs (402k-391k)= 
11k 
11k/402k=0.027 1-0.027= 0.973 0.973*100%= 97.3% 
453k 488k 
    
125k 97k 
    
871k 907k 
    
322k 425k 
    
 
 
*Abs means the absolute value, which means only the magnitude of the difference without any 
negative sign (if any)  
The Model Evaluation stands on the two pillars of accuracy and error. Let’s understand some 
more metrics standing on these two pillars. 
 
 
 
 
Purpose: To understand how to calculate the error and the accuracy. 
Say: “The youth will understand the concept of accuracy and error and practice it 
mathematically.”