Precision/Recall Trade-off
·
To understand this trade-off,
ü let’s look at how the SGDClassifier
makes its classification
decisions.
ü For each instance, it computes
a score based on a decision function.
ü If that score is greater than a threshold,
•
it assigns the instance to the positive class;
•
otherwise it assigns it to the negative class.
Figure 3-3. In this precision/recall trade-off, images are ranked by
their classifier score, and those above the chosen decision threshold are
considered positive; the higher the threshold, the lower the recall, but (in
general) the higher the precision.
•
Figure 3 shows a few digits positioned from the lowest score on the left to the highest score on the right.
•
Suppose the decision threshold is positioned at the central arrow (between the two 5s):
ü
you will find
ü 4 true positives (actual 5s) on the right
of that threshold, and
ü 1 false positive (actually a 6).
ü
Therefore, with that threshold, the precision is 80% (4 out of 5).
•
But out of 6 actual 5s, the classifier
only detects
4,
ü so the
recall is 67% (4 out of 6).
•
If you raise the threshold (move it to the arrow on
the right),
ü the false positive (the 6) becomes a true negative,
ü thereby increasing the precision (up
to 100% in this case),
ü but
one true positive becomes a false negative, decreasing recall down
to 50%.
•
Conversely, lowering the threshold
increases recall and reduces precision.
·
Scikit-Learn does not let you set the threshold
directly,
ü but it
does give you access to the decision scores that it uses to make predictions.
·
Instead of calling the classifier’s predict() method,
ü you
can call its decision_function() method,
·
which returns a score for each
instance, and
·
then use any threshold you
want to make predictions based on those scores:
y_scores = sgd_clf.decision_function([some_digit])
y_scores
array([2412.53175101])
threshold = 0
y_some_digit_pred = (y_scores > threshold)
array([ True])
·
The SGDClassifier uses a threshold equal to 0, so the previous code returns the same result as the predict()
method (i.e., True).
·
Let’s raise the threshold:
threshold = 8000
y_some_digit_pred = (y_scores > threshold)
y_some_digit_pred
array([False])
•
This confirms that raising the threshold decreases recall.
•
The image actually represents a 5, and
ü the classifier detects it when the threshold is 0,
•
but it misses it when the threshold is increased to 8,000.
How do you decide which threshold to use?
·
First, use the cross_val_predict() function to get the scores of all instances in the training set,
ü but this time specify that
you want to return decision scores instead of predictions:
y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3, method = "decision_function")
With these scores,
•
use the precision_recall_curve() function to compute
ü precision and
ü recall for all possible thresholds:
from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve
(y_train_5, y_scores)
•
Finally, use Matplotlib to plot precision and recall as functions of the threshold value (Figure 4):
def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):
plt.plot(thresholds, precisions[:-1], "b--", label="Precision")
plt.plot(thresholds, recalls[:-1], "g-", label="Recall")
[...] # highlight the
threshold and add the legend, axis label, and grid
plot_precision_recall_vs_threshold(precisions, recalls, thresholds)
plt.show()
Figure 4.
Precision and recall versus the decision threshold
·
You may wonder why the precision curve is bumpier than the recall curve in Figure 4.
·
The reason is that precision may sometimes go down when you raise the threshold.
·
To understand why,
ü look back at Figure 3 and
ü notice what happens when
you start from the central threshold and
ü move it just one digit to
the right:
·
precision goes from 4/5 (80%) down to 3/4 (75%).
·
On the other hand,
ü recall can only go down when the threshold
is increased,
ü which explains why its curve looks smooth.
•
Another way to select a good
precision/recall trade-off is to
•
plot precision directly against recall, as
shown in Figure 5 (the same threshold as earlier is highlighted).
ü
You can see that precision really starts to fall sharply
around 80% recall.
ü
You will probably want to select a precision/recall trade-off just
before that drop—for example, at around 60% recall.
ü
But of course, the choice depends on
your project.
Figure 5. Precision versus recall
·
Suppose you decide to aim for 90% precision.
·
You look up the first plot and
ü find that you need to use a threshold of about 8,000.
·
To be more precise you can search for the lowest threshold that gives you at least 90%
precision
·
np.argmax() will give you the first index of the maximum value,
ü which in this case means
the first True value:
threshold_90_precision = thresholds [np.argmax ( precisions >= 0.90 ) ] # ~7816
•
To make predictions (on the training set for now),
ü instead of calling the classifier’s predict()
method,
ü
you can run this code:
y_train_pred_90 = (y_scores >= threshold_90_precision)
•
Let’s check these predictions’ precision and recall:
precision_score(y_train_5, y_train_pred_90)
0.900038008361839
recall_score(y_train_5, y_train_pred_90)
0.4368197749492714
·
Great, you have a 90% precision classifier!
·
As you can see, it is fairly easy to create a classifier with virtually any
precision you want:
ü just set a high enough threshold.
·
But wait, not so fast.
·
A high-precision classifier is not very useful if its recall is too low!
