---
jupyter:
jupytext:
text_representation:
extension: .md
format_name: markdown
format_version: '1.1'
jupytext_version: 1.1.0-rc0
kernelspec:
display_name: Python 3
language: python
name: python3
---
```raw_mimetype="text/restructuredtext"
.. meta::
:description: Topic: Measuring classification accuracy, Difficulty: Easy, Category: Practice Problem
:keywords: numpy, vectorization, practice, machine learning, classifier
```
# Measuring the Accuracy of a Classification Model
Suppose that we are working on a project in which we have some model that can process an image and classify its content. For example, my `cat_dog_goose_other` function tries to classify whether a picture is of a cat (class 0), a dog (class 1), a goose (class 2), or something else (class 3). We want to measure the *accuracy* of our classifier. That is, we want to feed it a series of images whose contents are known and tally the number of times the model's prediction matches the true content of an image. The accuracy is the fraction of images that the model classifies correctly.
For each image we feed the `cat_dog_goose_other` model, it will produce four **scores** - one score for each class. The model was designed such that the class with the highest score corresponds to its prediction. There are no constraints on the values the scores can take. For example, if the model processes one image it will return a shape-$(1, 4)$ score-array:
```python
>>> scores = cat_dog_goose_other(image)
# processing one image produces a 1x4 array of classification scores
>>> scores
array([[-10, 33, 580, 100]])
```
Here, our model has predicted that this is a picture of a goose, since the score associate with class 2 (`scores[2]`) is the largest value. In general, if we pass `cat_dog_goose_other` an array of $N$ images, it will return a shape-$(N, 4)$ array of classification scores - each of the $N$ images has $4$ scores associated with it.
Because we are measuring our model's accuracy, we have curated a set of images whose contents are known. That is, we have a true **label** for each image, which is encoded as a class-ID. For example, a picture of a cat would have the label `0` associated with it, a picture of a dog would have the label `1` and so on. Thus, a stack of $N$ images would have associated with it a shape-$(N,)$ array of integer labels, each label is within $[0, 4)$.
Suppose we have passed our model five images, and it produced the following scores:
```python
# Classification scores produced by `cat_dog_goose_other`
# on five images. A shape-(5, 4) array.
>>> import numpy as np
>>> scores = np.array([[ 30, 1, 10, 80], # prediction: other
... [-10, 20, 0, -5], # prediction: dog
... [ 27, 50, 9, 30], # prediction: dog
... [ -1, 0, 84, 3], # prediction: goose
... [ 5, 2, 10, 0]]) # prediction: goose
```
And suppose that the true labels for these five images are:
```python
# truth: cat, dog, dog, goose, other
>>> labels = np.array([0, 1, 1, 2, 3])
```
Our model classified three out of five images correctly; thus, our accuracy function should return 0.6:
```python
>>> classification_accuracy(scores, labels)
0.6
```
To generalize this problem, assume that your classifier is dealing with $K$ classes (instead of $4$). Complete the following function.
**Tip:** You will find it useful to leverage [numpy's argmax function](https://docs.scipy.org/doc/numpy/reference/generated/numpy.argmax.html#numpy.argmax)`f
```python
def classification_accuracy(classification_scores, true_labels):
"""
Returns the fractional classification accuracy for a batch of N predictions.
Parameters
----------
classification_scores : numpy.ndarray, shape=(N, K)
The scores for K classes, for a batch of N pieces of data
(e.g. images).
true_labels : numpy.ndarray, shape=(N,)
The true label for each datum in the batch: each label is an
integer in the domain [0, K).
Returns
-------
float
(num_correct) / N
"""
# YOUR CODE HERE
pass
```
### Unvectorized Solution
A simple approach to this problem is to first loop over the rows of our classification scores. We know that each such row stores the scores for each class for a particular data point, and that the *index* of the highest score in that row gives us the predicted label for that data point (e.g. image in our hypothetical use-case). We can then directly compare these predicted labels with the true labels to compute the accuracy.
We can use the function `numpy.argmax` to get the index of the highest score, and thus the predicted class-ID, for each data point. Recall that NumPy arrays use [row-major traversal ordering](https://www.pythonlikeyoumeanit.com/Module3_IntroducingNumpy/ArrayTraversal.html#How-to-Traverse-an-Array:-Row-major-%28C%29-vs-Column-major-%28F%29-Traversal-Ordering), so performing a for-loop over `classification_scores` will yield one row of the array at a time.
```python
pred_labels = [] # Will store the N predicted class-IDs
for row in classification_scores:
# store the index associated with the highest score for each datum
pred_labels.append(np.argmax(row))
```
Next, we need to count the fraction of predicted class-IDs that match the true labels classification matches the true classification.
```python
num_correct = 0
for i in range(len(pred_labels)):
if pred_labels[i] == true_labels[i]:
num_correct += 1
```
Or we can make use of [a generator comprehension](https://www.pythonlikeyoumeanit.com/Module2_EssentialsOfPython/Generators_and_Comprehensions.html#Creating-your-own-generator:-generator-comprehensions) and [itertools](https://www.pythonlikeyoumeanit.com/Module2_EssentialsOfPython/Itertools.html) to be much more succinct:
```python
# recall: int(True) -> 1, int(False) -> 0
num_correct = sum(p == t for p, t in zip(pred_labels, true_labels))
```
We can formally write this out into the following function:
```python
def unvectorized_accuracy(classification_scores, true_labels):
"""
Returns the fractional classification accuracy for a batch of N predictions.
Parameters
----------
classification_scores : numpy.ndarray, shape=(N, K)
The scores for K classes, for a batch of N pieces of data
(e.g. images).
true_labels : numpy.ndarray, shape=(N,)
The true label for each datum in the batch: each label is an
integer in the domain [0, K).
Returns
-------
float
(num_correct) / N
"""
pred_labels = [] # Will store the N predicted class-IDs
for row in classification_scores:
pred_labels.append(np.argmax(row))
num_correct = 0
for i in range(len(pred_labels)):
if pred_labels[i] == true_labels[i]:
num_correct += 1
return num_correct / len(true_labels)
```
Testing against our example from above:
```python
>>> unvectorized_accuracy(scores, labels)
0.6
```
Horray! We have a working accuracy function! However, this function can be greatly simplified and optimized by [vectorizing](https://www.pythonlikeyoumeanit.com/Module3_IntroducingNumpy/VectorizedOperations.html) it.
### Vectorized Solution
`numpy.argmax` is one of NumPy's [vectorized sequential functions](https://www.pythonlikeyoumeanit.com/Module3_IntroducingNumpy/VectorizedOperations.html#Sequential-Functions). As such, it accepts [axis as a keyword argument](https://www.pythonlikeyoumeanit.com/Module3_IntroducingNumpy/VectorizedOperations.html#Specifying-the-axis-Keyword-Argument-in-Sequential-NumPy-Functions). This means that, instead of calling `np.argmax` on each row of `classification_scores` in a for-loop, we can simply instruct `np.argmax` to operate *across the columns of each row of the array* by specifying `axis=1`.
```python
# returns the column-index of the max value
# within each row of `classification_scores`
pred_labels = np.argmax(classification_scores, axis=1)
```
This simple expression eliminates our first for-loop entirely.
Next, we can use NumPy's *vectorized logical operations*, specifically `==`, to get a boolean-valued array that stores `True` wherever the predicted labels match the true labels and `False` everywhere else. Recall that `True` behaves like `1` and `False` like `0`. Thus, we can call `np.mean` on our resulting boolean-valued array to compute the number of correct predictions divided by the total number of predictions. We can thus vectorize our second for-loop with:
```python
# computes the fraction of correctly predicted labels
frac_correct = np.mean(pred_labels == true_labels)
```
All together, making keen use of vectorization allows us to write our classification accuracy function *in a single line of code*.
```python
def classification_accuracy(classification_scores, true_labels):
"""
Returns the fractional classification accuracy for a batch of N predictions.
Parameters
----------
classification_scores : numpy.ndarray, shape=(N, K)
The scores for K classes, for a batch of N pieces of data
(e.g. images).
true_labels : numpy.ndarray, shape=(N,)
The true label for each datum in the batch: each label is an
integer in the domain [0, K).
Returns
-------
float
(num_correct) / N
"""
return np.mean(np.argmax(classification_scores, axis=1) == true_labels)
```
Not only is this cleaner to look at, but it was also simpler and less error-prone to write. Moreover, it is much faster than our unvectorized solution - given $N=10,000$ data points and $K=100$ classes, our vectorized solution is roughly $40\times$ faster
(The following "time-it" code blocks must be run in independent cells in a Jupyter notebook or IPython console - `%%timeit` must be the topmost command in the cell)
```python
>>> N = 10000
>>> K = 100
>>> scores = np.random.rand(N, K)
>>> labels = np.random.randint(low=0, high=K, size=N)
```
```python
>>> %%timeit
... unvectorized_accuracy(scores, labels)
39.5 ms ± 1.2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```
```python
>>> %%timeit
... classification_accuracy(scores, labels)
1.6 ms ± 7.04 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```