.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import pandas as pd\n",
    "from matplotlib import pyplot as plt\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# weird OSX error fix\n",
    "os.environ['KMP_DUPLICATE_LIB_OK']='True'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning using Neural Networks (Part 2)\n",
    "\n",
    "Welcome to the tutorial on Machine Learning using Neural Networks! In this part of the tutorial, we will use neural networks to recognize handwritten digits. Will will start by outlining why neural networks are good for recognizing images, and then train a neural network to recognize handwritten digits. We will use a famous dataset for this task, which is called MNIST.\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image Recognition using Neural Networks\n",
    "\n",
    "How can computer programs recognize objects in a photograph? This has been one of the central questions in the computer vision research. It is very simple for humans to recognize objects in the photographs, just like we do in real world. But for computers, this task becomes notoriously difficult! Although with a little help from neural networks, it becomes rather easy because neural networks behave in a slightly intuitive way. Here is how a neural network would recognize whether there is a face in an image or not:\n",
    "\n",
    "<img src=\"\n",
    "./mnist/imgrecog_1.png\"/>\n",
    "\n",
    "What you see in the image above essentially tell us that neural networks try to recognize different 'features' of the image first, and based on appearance of those features, they decide whether the image contains the signal of interest or not (a human face in this case).\n",
    "\n",
    "The problem of detecting a face in a photograph is a binary classification problem, but for our example, we are going to turn to a multi-class classification problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Recognizing Handwritten Digits using Neural Networks\n",
    "\n",
    "To explore how to recognize images, we will try train a neural network model that can tell us which handwritten digit is there in the image we give to it. We are going to use a database of images that was originally developed in 1994 called the MNIST dataset. It is based on samples of high school students and U. S. Census Bureau employees' writing- https://en.wikipedia.org/wiki/MNIST_database\n",
    "\n",
    "<img src=\"mnist/mnist_1.png\"/>\n",
    "\n",
    "You can see that there are many different ways of wriring 0s, 1s, 2s, and so on. How is neural network going to detect which digit is there in the image? It turns out that neural network will try to break down the problem in parts and work it out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Representing Images as Data\n",
    "\n",
    "In order to use a neural network to recognize images, we must first convert images into numeric values. Each image is translated into a matrix of values. We will first read in the MNIST data and then plot it to demonstrate what inputs are required in neural networks.\n",
    "\n",
    "The MNIST dataset contains many thousand training examples for handwritten digits. Each training example is a 28x28 pixel image. This means that we have 28x28 = 784 features for each of our training example. As the output, we have a numeric value from 0 to 9, which tells the actual digit that is stored in those 784 pixels. We are going to use this information to train and evalulate our model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we jump further in, you might ask, what do these 784 pixels contain? They contain what is called pixel intensity value that indicates how dark the pixel is. If we print these pixel intensity values, we get to see the actual image. Here is an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load up the training and test datasets from keras\n",
    "(x_train, y_train),(x_test, y_test) = keras.datasets.mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create index dataframes for selecting arrays from vector of images\n",
    "y_train_df = pd.DataFrame(y_train)\n",
    "x_test_df = pd.DataFrame(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert pixel values to be between 0 and 1\n",
    "x_train = x_train/255.0\n",
    "x_test = x_test/255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Draw a heatmap with the pixel intensity in each cell using training example 555\n",
    "f, ax = plt.subplots(figsize=(15, 15))\n",
    "sns.heatmap(x_train[555], annot=True, linewidths=.5, ax=ax, cmap=\"Greys\", cbar_kws={'label': 'Pixel Values'})\n",
    "plt.title(\"MNIST Plot of a {} image with pixel intensities\".format(y_train[555]), size =\"large\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conceptually, How Could an Algorithm Detect a Zero from Many Different Examples?\n",
    "\n",
    "Let us say the neural network is trying to detect whether there is 0 in the image or not. It will proceed like this:\n",
    "\n",
    "- Detect handwriting strokes in different parts of the image\n",
    "- Put these strokes together to figure out what digit is there in the image\n",
    "\n",
    "For a 0, we might have these strokes in the image:\n",
    "\n",
    "<img src=\"mnist/mnist_2.png\" width=350/>\n",
    "\n",
    "where the final image looks like this:\n",
    "\n",
    "<img src=\"mnist/mnist_3.png\" width=105/>\n",
    "\n",
    "And remember, there can be many different types of 0s! But most of them will have circular strokes in all four sides of the image.\n",
    "\n",
    "The code below plots images that were labled (the training data) as zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number = 0\n",
    "zeros = x_train[y_train_df[y_train_df[0]==number].index]\n",
    "fig = plt.figure(figsize=(20, 20))\n",
    "plt.title(\"100 different handwritten {}s from MNIST dataset\".format(number), fontsize = 25)\n",
    "for x in range(100):\n",
    "    ax = fig.add_subplot(10, 10, x+1)\n",
    "    ax.matshow(zeros[x],cmap = matplotlib.cm.binary)\n",
    "    plt.xticks(np.array([]))\n",
    "    plt.yticks(np.array([]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detecting 0 to 9\n",
    "\n",
    "So you now have a rough idea about how can a neural network detect 0 in an image. However, we need to detect all the digits, from 0 to 9. How will we do this using using a neural network?\n",
    "\n",
    "So far, we have only talked about binary outcomes where a neural network is detecting whether there is 'some thing' present in its input or not. This requires us to have one neuron in the output layer. But if we want to detect multiple things, we need to have multiple neurons in the output layer, one for each thing we need to detect. In our case, we need to detect all the arabic numerals, so there will be 10 output nodes. Here is what our neural network will look like:\n",
    "\n",
    "<img src=\"mnist/mnist_4.png\"/>\n",
    "\n",
    "Let us find out where those 784 input neurons are coming from."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to feed a lot of these images to our neural network, and it will learn characteristics of different digits for us. Then we can use this trained model to make predictions.\n",
    "\n",
    "### Model Training\n",
    "\n",
    "Let us start the model training by looking at the the dataset. We will use <b>Tensorflow</b> (https://tensorflow.rstudio.com/) package for our machine learning task via the library keras (https://keras.rstudio.com/). Here are some rows from our dataset:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our training set will have 60,000 training samples and 10,000 validation examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will set up the structure of our network model. Each line in hte square brackets is a layer in the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up model\n",
    "model = keras.Sequential([\n",
    "    keras.layers.Flatten(input_shape=(28, 28)),\n",
    "    keras.layers.Dense(16, activation=tf.nn.sigmoid),\n",
    "    keras.layers.Dense(10, activation=tf.nn.softmax)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to configure our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note the optimizer sets several hyperparameters like the learning rate. \n",
    "# See info here: https://www.tensorflow.org/api_docs/python/tf/keras/optimizers\n",
    "model.compile(optimizer='adam', \n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we fit the model to the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(x_train, y_train, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now that our model is trained, we will generate some predictions and assess the accuracy. <b>Remember, always assess accuracy using test data!</b>\n",
    "\n",
    "### Model Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save the predictions of the test data into a list of arrays. Next, let's look at the first array and see the output values from the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the output values in the first array [0]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to see the neuron with the highest activation value, we can get the max from the output layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.argmax(predictions[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also plot the output neuron values to see what they look like for one of this array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = list(range(10))\n",
    "y = predictions[18]\n",
    "\n",
    "plt.figure(\"Outputs\")\n",
    "plt.bar(x,y)\n",
    "plt.xticks(x)\n",
    "plt.title(\"Outputs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at both the images and predicitons together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_image(i, predictions_array, true_label, img):\n",
    "  predictions_array, true_label, img = predictions_array[i], true_label[i], img[i]\n",
    "  plt.grid(False)\n",
    "  plt.xticks([])\n",
    "  plt.yticks([])\n",
    "  \n",
    "  plt.imshow(img, cmap=plt.cm.binary)\n",
    "\n",
    "  predicted_label = np.argmax(predictions_array)\n",
    "  if predicted_label == true_label:\n",
    "    color = 'blue'\n",
    "  else:\n",
    "    color = 'red'\n",
    "  \n",
    "  plt.xlabel(\"pred:{} value:{:2.0f}% label:{}\".format(predicted_label,\n",
    "                                100*np.max(predictions_array),\n",
    "                                true_label),\n",
    "                                color=color)\n",
    "\n",
    "def plot_value_array(i, predictions_array, true_label):\n",
    "  predictions_array, true_label = predictions_array[i], true_label[i]\n",
    "  plt.grid(False)\n",
    "  plt.xticks(list(range(10)))\n",
    "  plt.yticks([])\n",
    "  thisplot = plt.bar(list(range(10)), predictions_array, color=\"#777777\")\n",
    "  plt.ylim([0, 1]) \n",
    "  predicted_label = np.argmax(predictions_array)\n",
    " \n",
    "  thisplot[predicted_label].set_color('red')\n",
    "  thisplot[true_label].set_color('blue')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a random image to see what the model predictied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# i is the indec of the image we want to look at.\n",
    "i = 44\n",
    "plot_image(i, predictions, y_test, x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at another image with the bar plot plot of the output values and color code the bars with red representing the wrong guess and blue the correct guess."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 8\n",
    "plt.figure(figsize=(6,3))\n",
    "plt.subplot(1,2,1)\n",
    "plot_image(i, predictions, y_test, x_test)\n",
    "plt.subplot(1,2,2)\n",
    "plot_value_array(i, predictions,  y_test)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at several different images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the first X test images, their predicted label, and the true label\n",
    "# Color correct predictions in blue, incorrect predictions in red\n",
    "num_rows = 5\n",
    "num_cols = 5\n",
    "num_images = num_rows*num_cols\n",
    "plt.figure(figsize=(2*2*num_cols, 2*num_rows))\n",
    "for i in range(num_images):\n",
    "  plt.subplot(num_rows, 2*num_cols, 2*i+1)\n",
    "  plot_image(i, predictions, y_test, x_test)\n",
    "  plt.subplot(num_rows, 2*num_cols, 2*i+2)\n",
    "  plot_value_array(i, predictions, y_test)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's formally evaluate our predictions against the true labled values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_loss, test_acc = model.evaluate(x_test, y_test)\n",
    "\n",
    "print('Test accuracy:', test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A confusion matrix is a great way to visualize the predictions versus the actual labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# take the array of predictions for each image and just look at the max value.\n",
    "preds = [np.argmax(i) for i in predictions]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(preds, y_test)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the missed predictions.  Do you see a pattern?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How can we improve this model? There are different ways to improve predictions of a neural network:\n",
    "\n",
    "- <b>Change architecture</b>\n",
    "- <b>Try different hyperparameters</b>\n",
    "- <b>Try advanced methods like convolution</b>\n",
    "\n",
    "Trying these different things is out of the scope of this tutorial, but if you want to explore these options, you can do so in <a href=\"https://playground.tensorflow.org\">TensorFlow Playground</a>.\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "In this part of the tutorial, we saw how neural networks are trained using software packages. Don't worry if this was too complicated, you always get help about doing complex things when working in teams. Websites like [StackOverflow](https://stackoverflow.com/) can also help in case you get stuck.\n",
    "\n",
    "We hope that you found both part 1 and part 2 of the tutorial useful.\n",
    "\n",
    "## References\n",
    "\n",
    "We would like to thank Michael Nielson for his excellent online textbook on neural networks. We extensively borrowed part 1 materials from his textbook which can be found <a href=\"http://neuralnetworksanddeeplearning.com\">here</a>."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}