{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e5d62ccb",
   "metadata": {},
   "source": [
    "<font size=\"5\"><b>Lecture 11: Deep Learning DL1 (exercises)</b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2152cca6",
   "metadata": {},
   "source": [
    "**HOW TO RUN THIS NOTEBOOK**:<br>\n",
    "1) if TF installed locally<br>\n",
    "    activate the conda environment in which you installed Tensor Flow (e.g., \"tf\"), and launch jupyter & this notebook from it:<br>\n",
    "    ```shell\n",
    "    $ conda activate tf\n",
    "    $ jupyter notebook\n",
    "    ```<br>\n",
    "2) if using GoogleColab<br>\n",
    "    simply open this notebook<br>\n",
    "    NB: see GoogleDrive (valade@igeofisica.unam.mx) > Colab Notebooks > CV4GS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TF Machine Learning [Glossary](https://developers.google.com/machine-learning/glossary#logits)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initialize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f5abe018",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tf.__version__)\n",
    "print(tf.keras.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MLP (MNIST fashion dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    We follow one of the Tensor Flow's <a href=\"https://www.tensorflow.org/tutorials/keras/classification\" target=\"_blank\">tutorials</a> and train a very simple network to recognize objects in 28x28 pixel images (MNIST fashion dataset). Because this involves lots of framework functions, it will be more of a \"guided copy and paste\" than actual programming.<br>\n",
    "    <br>\n",
    "    <u>Note</u>: the network and variable names has slightly been changed with respect to the online tutorial, following Aurélien Géron's book \"Hands-on Machine Learning with Scikit-Learn, Keras & TensorFlow\" 2nd edition.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    Load the MNIST data, and explore the dataset (print the shape of the arrays, plot a few images, etc).\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# --- access the Fashion MNIST directly from TensorFlow\n",
    "fashion_mnist = tf.keras.datasets.fashion_mnist\n",
    "\n",
    "(X_train_full, y_train_full), (X_test, y_test) = fashion_mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# --- set class names\n",
    "class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',\n",
    "               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## preprocess data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    Preprocess the data:<br>\n",
    "    1. split the full training dataset (images and labels) to have create a validation dataset of 5000 instances<br>\n",
    "    2. scale the image values to range between [0, 1]\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## build model (using the Sequential API)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    Build the model by configuring the layers.<br>\n",
    "    Print the model using the \"summary()\" method. Explore the model layers once build.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    tf.keras.layers.Dense(300, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(100, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## compile model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    Before the model is ready for training, it needs a few more settings. These are added during the model's compile step:\n",
    "    <ul>\n",
    "    <li>Loss function —This measures how accurate the model is during training. You want to minimize this function to \"steer\" the model in the right direction.</li>\n",
    "    <li>Optimizer —This is how the model is updated based on the data it sees and its loss function.</li>\n",
    "    <li>Metrics —Used to monitor the training and testing steps. The following example uses accuracy, the fraction of the images that are correctly classified.</li>\n",
    "    </ul>\n",
    "    <br>\n",
    "    Use the compile() method of your model, and set the following:<br>\n",
    "    - loss=\"sparse_categorical_crossentropy\"<br>\n",
    "    - optimizer=\"sgd\"<br>\n",
    "    - metrics=[\"accuracy\"]<br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=\"sgd\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## train model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    Time to actually train your model! \n",
    "    To do so, simply call its fit() method, and pass it:\n",
    "    <ul>\n",
    "        <li>the input features (X_train) and the target classes (y_train)</li>\n",
    "        <li>the number of epochs to train (or else it would default to just 1, which would definitely not be enough to converge to a good solution)</li>\n",
    "        <li>the validation set (this is optional)</li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(X_train, y_train, epochs=30, validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## view training history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    We will use the pandas library to plot the \"history\" of the training.<br>\n",
    "    <br>\n",
    "    <u>Note</u>: it is very useful to track the training progression in real time. The additional package \"TensorBoard\" allows this (and more): we will install and start using it next week.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(history.history).plot(figsize=(8, 5))\n",
    "plt.grid(True)\n",
    "plt.gca().set_ylim(0, 1) # set the vertical range to [0-1]\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## evaluate model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    After training is completed you should evaluate it, by comparing how it performs on the test dataset.<br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_loss, test_acc = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    Finally, you can use the trained model to make predictions about images!<br>\n",
    "    Use the predict() method and pass a few images. Is this what you expected? Are the predictions correct according to you?\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "308.6px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}