We have already talked about it in this post. Just want to add few more things after finishing a course. This post is just an extension of above with some practical considerations.

We are claiming that accuracy may not be a good measure always. When you are building automated machine learning you must trust it.

Case Study

• You want to show positive reviews on your website.
• Say in your dataset 90% reviews are negative.
• A classifier can achieve 90% accuracy by predicting all of them as negative.
• But what you are interested in is finding out remaining 10% and display it on your website.

 

Precision = Did I show something negative?

Recall = How good I am at finding positive reviews?

 

Analogy with Optimist and Pessimist

• Optimist assigns every/most review as positive
  • Very good recall, but less precision
• Pessimist assigns every/most review with negative
  • Bad recall, good precision

 

Trade-off

• Trade-off comes while scoring, not while training
• We can assign labels based on probabilities
• Decision tree gives probability by no of positive and negative samples at leaf node
• Logistic regression of-course gives probability
• We can change threshold to trade off between precision and recall
• Positive when prob > 1 => Pessimist
• Positive when prob > 0 => Optimist

 

Single no not always useful

• Single numbers like F1 score and AUC are something I am not great fan of
• You can not always choose classifier just by AUC, ROC curve might intersesct
  • This intersection means that one classifier is better at some range of precision
  • But if they don’t intersect we choose the one with higher AUC
• From business perspective we are should be clear whether we want more precision or recall
• Another practical metric they talked about was precision at k
  • Say I want to display 5 reviews on my website
  • What is the precision after 5 values I have chosen

 

When to use different metrics ?

Pick the metric that matches what the model’s output is consumed by.

A retrieval layer emits a ranked list under a budget → only the ordering matters → AUC / recall@k. 

Topic classification emits a hard label → you’re evaluating cells of a confusion matrix → F1.

CTR emits a probability that gets multiplied by something downstream (bid = pCTR × value) → being off by 2x in the probability directly costs money even if the ranking is perfect → you need a calibration-sensitive metric → NE.

1. Threshold / confusion-matrix metrics (output is a decision)

• Accuracy — nearly useless under imbalance
• Precision, Recall, Specificity (TNR), FPR
• F1; F-beta (β>1 favors recall, β<1 favors precision)
• Macro vs micro vs weighted F1 — macro treats every class equally so rare topics actually show up; micro is dominated by head classes. For topic classification this choice usually matters more than F1 vs anything else.

2. Ranking / threshold-free metrics (output is an ordering)

• ROC-AUC — P(random positive scores above random negative). Invariant to any monotone rescaling, which is exactly why it can’t catch a miscalibrated CTR model.
• PR-AUC / Average Precision — the one you want under heavy imbalance. ROC-AUC’s baseline is 0.5 regardless of prevalence; PR-AUC’s baseline is the prevalence, so it doesn’t flatter a model on a 0.1% positive rate.
• Gini = 2·AUC − 1; KS statistic (max gap between the two score CDFs) — both standard in credit risk
• NDCG, MRR, MAP when relevance is graded rather than binary

3. Probabilistic / calibration metrics (output is a number you do arithmetic on)

• Log loss — proper scoring rule, penalizes both discrimination and miscalibration
• NE = log loss ÷ entropy of the empirical base rate. The normalization is what makes it comparable across surfaces, countries, and time periods with different base CTRs; 1.0 = no better than always predicting the base rate.
• Calibration ratio / COPC (sum of actuals ÷ sum of predictions) — the bias check that NE alone won’t give you cleanly