Worksheet Solutions: Evaluating Models | Artificial Intelligence for Class 10 PDF Download

Multiple Choice Questions (MCQs)

Q.1 What is the primary purpose of the evaluation stage in the AI project cycle?

a. To collect data for the model
b. To assess the reliability of an AI model by comparing predictions with actual outcomes
c. To preprocess the data for modeling
d. To deploy the model in a real-world scenario

Ans: b. To assess the reliability of an AI model by comparing predictions with actual outcomes

The evaluation stage is crucial for determining how well an AI model performs by comparing its predictions with actual outcomes, ensuring reliability.

Q.2 Why is it not recommended to use the training dataset for evaluating an AI model?

a. It reduces the accuracy of the model
b. The model may overfit and simply recall the training data
c. It increases the computation time
d. It makes the model incompatible with test data

Ans: b. The model may overfit and simply recall the training data

Using the training dataset for evaluation can lead to overfitting, where the model memorizes the training data instead of generalizing to new data.

Q.3 In the context of the forest fire prediction model, what does a False Negative (FN) represent?

a. Predicting a fire when there is none
b. Correctly predicting a fire
c. Predicting no fire when a fire has occurred
d. Correctly predicting no fire

Ans: c. Predicting no fire when a fire has occurred

A False Negative in this context indicates a failure to detect a fire, which is critical for timely response and prevention.

Q.4 Which evaluation metric measures the percentage of correct predictions out of all observations?

a. Precision
b. Recall
c. Accuracy
d. F1 Score

Ans: c. Accuracy

Accuracy is defined as the ratio of correct predictions to total observations, providing a straightforward measure of model performance.

Q.5 What does the F1 Score represent in model evaluation?

a. The percentage of true positive cases
b. The balance between precision and recall
c. The total number of false negatives
d. The ratio of true negatives to false positives

Ans: b. The balance between precision and recall

The F1 Score is a harmonic mean of precision and recall, providing a single metric that balances both aspects of model performance.

Fill in the Blanks

Ans: Evaluation

The process referred to is known as evaluation, which assesses how well a model performs by comparing predicted results against actual data.

Q7: The __________ matrix is used to record the comparison between prediction and reality in a model evaluation.

Ans: Confusion

The confusion matrix is a tool used in machine learning that summarizes the performance of a classification algorithm by showing the counts of true versus predicted classifications.

Q8: In a confusion matrix, the case where the model correctly predicts a positive outcome is called __________.

Ans: True Positive

A true positive occurs when the model correctly identifies a positive instance, indicating successful prediction.

Q9: The evaluation metric that considers both True Positives and False Positives is called __________.

Ans: Precision

Precision measures the accuracy of positive predictions, calculated as the ratio of true positives to the total predicted positives.

Q10: The formula for the F1 Score is a measure of balance between __________ and __________.

Ans: Precision, Recall

The F1 Score is a metric that combines precision and recall to provide a single score that reflects the balance between them in model evaluation.

True or False

Ans: False

The confusion matrix provides a visual representation of the performance of a classification model by summarizing the prediction outcomes, but it does not serve as a direct performance metric.

Q.12: Accuracy is calculated as the ratio of True Positives and True Negatives to the total number of observations.

Ans: True

Accuracy is defined as the ratio of the sum of True Positives and True Negatives to the total number of observations, reflecting the overall correctness of a model's predictions.

Q.13: High precision in a model indicates a high number of False Positives.

Ans: False

High precision actually signifies a low number of False Positives, as precision measures the ratio of True Positives to the total predicted positives (True Positives + False Positives).

Q.14: A False Positive in the forest fire scenario means the model predicts a fire when no fire has occurred.

Ans: True

A False Positive indicates that the model incorrectly identifies the presence of a fire when, in reality, there is none, which can lead to unnecessary alarm and resource allocation.

Q.15: The F1 Score is useful when both precision and recall are important for evaluating a model.

Ans: True

The F1 Score combines precision and recall into a single metric, making it especially valuable when the balance between these two measures is critical for model evaluation.

Short Answer Questions

Q.16: Define model evaluation in one sentence.
Ans: Model evaluation is the process of assessing the reliability and performance of an AI model by comparing its predictions with actual outcomes using a test dataset.  

Q.17: Name two outcomes in the confusion matrix where the model’s prediction matches reality.
Ans: True Positive (TP) and True Negative (TN).  

Q.18: Explain the difference between Precision and Accuracy in the context of model evaluation.
Ans: Precision measures the percentage of true positive predictions out of all positive predictions (TP / (TP + FP)), focusing on the correctness of positive predictions, while Accuracy measures the percentage of all correct predictions (TP + TN) out of total observations (TP + TN + FP + FN), reflecting overall model correctness.  

Q.19: What is a False Negative, and why might it be costly in the forest fire prediction scenario?
Ans: A False Negative occurs when the model predicts no fire when a fire has actually occurred, which is costly in the forest fire scenario because failing to detect a fire could lead to delayed response, resulting in significant damage, loss of life, or environmental harm.  

Q.20: Why is the F1 Score considered a better evaluation metric than accuracy in some cases?
Ans: The F1 Score is better than accuracy in cases where there is an imbalance between classes or when both False Positives and False Negatives are critical, as it balances Precision and Recall, providing a more comprehensive measure of model performance compared to accuracy, which can be misleading in imbalanced datasets.  

Long Answer Questions

Q.21: Explain the role of the confusion matrix in evaluating an AI model, using the forest fire prediction scenario as an example.
Ans: The confusion matrix is a tool that records the comparison between an AI model’s predictions and actual outcomes, categorizing results into True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). In the forest fire prediction scenario, where the model predicts whether a fire has occurred, the confusion matrix maps predictions against reality. For example, a True Positive occurs when the model correctly predicts a fire (Reality: Yes, Prediction: Yes), a True Negative when it correctly predicts no fire (Reality: No, Prediction: No), a False Positive when it predicts a fire that didn’t occur (Reality: No, Prediction: Yes), and a False Negative when it misses a fire (Reality: Yes, Prediction: No). This matrix helps evaluate model performance by providing the data needed to calculate metrics like Accuracy, Precision, Recall, and F1 Score, enabling identification of specific errors, such as frequent False Negatives, which are critical in this scenario due to the high cost of missing a fire.  

Q.22: Describe the steps to calculate Accuracy, Precision, Recall, and F1 Score for an AI model, using the confusion matrix components (TP, TN, FP, FN).
Ans: To calculate the evaluation metrics using the confusion matrix components (True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN)), follow these steps:  

• Step 1: Accuracy
  Accuracy is the percentage of correct predictions out of all observations.
  Formula: Accuracy = (TP + TN) / (TP + TN + FP + FN) * 100%
  Example: If TP = 50, TN = 40, FP = 5, FN = 5, then Accuracy = (50 + 40) / (50 + 40 + 5 + 5) * 100% = 90 / 100 * 100% = 90%.  
• Step 2: Precision
  Precision is the percentage of true positive predictions out of all positive predictions.
  Formula: Precision = TP / (TP + FP)
  Example: Using TP = 50, FP = 5, Precision = 50 / (50 + 5) = 50 / 55 ≈ 0.909 or 90.9%.  
• Step 3: Recall
  Recall is the percentage of true positive predictions out of all actual positive cases.
  Formula: Recall = TP / (TP + FN)
  Example: Using TP = 50, FN = 5, Recall = 50 / (50 + 5) = 50 / 55 ≈ 0.909 or 90.9%.  
• Step 4: F1 Score
  F1 Score is the harmonic mean of Precision and Recall, balancing both metrics.
  Formula: F1 Score = 2 * (Precision * Recall) / (Precision + Recall)
  Example: Using Precision = 0.909, Recall = 0.909, F1 Score = 2 * (0.909 * 0.909) / (0.909 + 0.909) = 2 * 0.826 / 1.818 ≈ 0.909 or 90.9%.

Q.23: Discuss why a high accuracy might not indicate good model performance in the forest fire scenario, and suggest an alternative metric.
Ans: In the forest fire scenario, a high accuracy might not indicate good model performance because of class imbalance, where the occurrence of fires (positive cases) is rare compared to no-fire cases (negative cases). For example, if fires occur only 2% of the time, a model that always predicts “no fire” achieves 98% accuracy by correctly predicting the 98% no-fire cases (True Negatives) but fails to detect any fires (True Positives = 0), missing all critical fire events (False Negatives). This high accuracy is misleading as it overlooks the model’s inability to detect fires, which is the primary objective. An alternative metric like the F1 Score is better because it balances Precision (correct fire predictions out of all predicted fires) and Recall (correct fire predictions out of all actual fires), ensuring the model is evaluated on its ability to detect fires while minimizing false positives and false negatives, which are both critical in this high-stakes scenario.  

Q.24: Provide an example of a scenario where a high False Negative cost is critical, and explain why minimizing False Negatives is important in that context.
Ans: An example of a scenario with a high False Negative cost is a medical diagnosis system for detecting a life-threatening disease, such as cancer. In this context, a False Negative occurs when the model predicts a patient does not have cancer when they actually do. Minimizing False Negatives is critical because failing to detect the disease could delay treatment, potentially leading to severe health deterioration or death. For instance, if a patient with early-stage cancer is incorrectly classified as healthy, they might miss critical early intervention, worsening their prognosis. Thus, a high Recall (TP / (TP + FN)) is prioritized to ensure most actual positive cases are detected, even if it means accepting some False Positives, as the cost of missing a diagnosis outweighs the cost of additional testing triggered by false alarms.