{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "[Master]Fine-Tune-BERT-for-Text-Classification-with-TensorFlow.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zGCJYkQj_Uu2"
      },
      "source": [
        "<h2 align=center> Fine-Tune BERT for Text Classification with TensorFlow</h2>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4y2m1S6e12il"
      },
      "source": [
        "<div align=\"center\">\n",
        "    <img width=\"512px\" src='https://drive.google.com/uc?id=1fnJTeJs5HUpz7nix-F9E6EZdgUflqyEu' />\n",
        "    <p style=\"text-align: center;color:gray\">Figure 1: BERT Classification Model</p>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eYYYWqWr_WCC"
      },
      "source": [
        "In this project, you will learn how to fine-tune a BERT model for text classification using TensorFlow and TF-Hub."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yQG5PCO_WFx"
      },
      "source": [
        "The pretrained BERT model used in this project is [available](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2) on [TensorFlow Hub](https://tfhub.dev/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7pKNS21u_WJo"
      },
      "source": [
        "### Learning Objectives"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_3NHSMXv_WMv"
      },
      "source": [
        "By the time you complete this project, you will be able to:\n",
        "\n",
        "- Build TensorFlow Input Pipelines for Text Data with the [`tf.data`](https://www.tensorflow.org/api_docs/python/tf/data) API\n",
        "- Tokenize and Preprocess Text for BERT\n",
        "- Fine-tune BERT for text classification with TensorFlow 2 and [TF Hub](https://tfhub.dev)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o6BEe-3-AVRQ"
      },
      "source": [
        "### Prerequisites"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sc9f-8rLAVUS"
      },
      "source": [
        "In order to be successful with this project, it is assumed you are:\n",
        "\n",
        "- Competent in the Python programming language\n",
        "- Familiar with deep learning for Natural Language Processing (NLP)\n",
        "- Familiar with TensorFlow, and its Keras API"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MYXXV5n3Ab-4"
      },
      "source": [
        "### Contents"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XhK-SYGyAjxe"
      },
      "source": [
        "This project/notebook consists of several Tasks.\n",
        "\n",
        "- **[Task 1]()**: Introduction to the Project.\n",
        "- **[Task 2]()**: Setup your TensorFlow and Colab Runtime\n",
        "- **[Task 3]()**: Download and Import the Quora Insincere Questions Dataset\n",
        "- **[Task 4]()**: Create tf.data.Datasets for Training and Evaluation\n",
        "- **[Task 5]()**: Download a Pre-trained BERT Model from TensorFlow Hub\n",
        "- **[Task 6]()**: Tokenize and Preprocess Text for BERT\n",
        "- **[Task 7]()**: Wrap a Python Function into a TensorFlow op for Eager Execution\n",
        "- **[Task 8]()**: Create a TensorFlow Input Pipeline with `tf.data`\n",
        "- **[Task 9]()**: Add a Classification Head to the BERT `hub.KerasLayer`\n",
        "- **[Task 10]()**: Fine-Tune BERT for Text Classification\n",
        "- **[Task 11]()**: Evaluate the BERT Text Classification Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IaArqXjRAcBa"
      },
      "source": [
        "## Task 2: Setup your TensorFlow and Colab Runtime."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GDDhjzZ5A4Q_"
      },
      "source": [
        "You will only be able to use the Colab Notebook after you save it to your Google Drive folder. Click on the File menu and select “Save a copy in Drive…\n",
        "\n",
        "![Copy to Drive](https://drive.google.com/uc?id=1CH3eDmuJL8WR0AP1r3UE6sOPuqq8_Wl7)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mpe6GhLuBJWB"
      },
      "source": [
        "### Check GPU Availability\n",
        "\n",
        "Check if your Colab notebook is configured to use Graphical Processing Units (GPUs). If zero GPUs are available, check if the Colab notebook is configured to use GPUs (Menu > Runtime > Change Runtime Type).\n",
        "\n",
        "![Hardware Accelerator Settings](https://drive.google.com/uc?id=1qrihuuMtvzXJHiRV8M7RngbxFYipXKQx)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8V9c8vzSL3aj"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Obch3rAuBVf0"
      },
      "source": [
        "### Install TensorFlow and TensorFlow Model Garden"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bUQEY3dFB0jX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "a4e2f5bd-d19b-4565-cf97-900f98767939"
      },
      "source": [
        "import tensorflow as tf\n",
        "print(tf.version.VERSION)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2.3.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aU3YLZ1TYKUt"
      },
      "source": [
        "!pip install -q tensorflow==2.3.0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AFRTC-zwUy6D"
      },
      "source": [
        "!git clone --depth 1 -b v2.3.0 https://github.com/tensorflow/models.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3H2G0571zLLs"
      },
      "source": [
        "# install requirements to use tensorflow/models repository\n",
        "!pip install -Uqr models/official/requirements.txt\n",
        "# you may have to restart the runtime afterwards"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GVjksk4yCXur"
      },
      "source": [
        "## Restart the Runtime\n",
        "\n",
        "**Note** \n",
        "After installing the required Python packages, you'll need to restart the Colab Runtime Engine (Menu > Runtime > Restart runtime...)\n",
        "\n",
        "![Restart of the Colab Runtime Engine](https://drive.google.com/uc?id=1xnjAy2sxIymKhydkqb0RKzgVK9rh3teH)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IMsEoT3Fg4Wg"
      },
      "source": [
        "## Task 3: Download and Import the Quora Insincere Questions Dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GmqEylyFYTdP"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "import sys\n",
        "sys.path.append('models')\n",
        "from official.nlp.data import classifier_data_lib\n",
        "from official.nlp.bert import tokenization\n",
        "from official.nlp import optimization"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZuX1lB8pPJ-W",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        },
        "outputId": "65894a32-cb22-4201-b767-b2e5a49a5624"
      },
      "source": [
        "print(\"TF Version: \", tf.__version__)\n",
        "print(\"Eager mode: \", tf.executing_eagerly())\n",
        "print(\"Hub version: \", hub.__version__)\n",
        "print(\"GPU is\", \"available\" if tf.config.experimental.list_physical_devices(\"GPU\") else \"NOT AVAILABLE\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TF Version:  2.3.0\n",
            "Eager mode:  True\n",
            "Hub version:  0.9.0\n",
            "GPU is available\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QtbwpWgyEZg7"
      },
      "source": [
        "A downloadable copy of the [Quora Insincere Questions Classification data](https://www.kaggle.com/c/quora-insincere-questions-classification/data) can be found [https://archive.org/download/fine-tune-bert-tensorflow-train.csv/train.csv.zip](https://archive.org/download/fine-tune-bert-tensorflow-train.csv/train.csv.zip). Decompress and read the data into a pandas DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0nI-9itVwCCQ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "137234df-54aa-4d93-963c-70cd78cdd1cd"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "df = pd.read_csv('https://archive.org/download/fine-tune-bert-tensorflow-train.csv/train.csv.zip',\n",
        "                 compression='zip', low_memory=False)\n",
        "df.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1306122, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yeHE98KiMvDd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 677
        },
        "outputId": "4b6cb84c-01d4-479e-9293-604ce2a6282d"
      },
      "source": [
        "df.tail(20)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>qid</th>\n",
              "      <th>question_text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1306102</th>\n",
              "      <td>ffff3778790af9baae76</td>\n",
              "      <td>What steps can I take to live a normal life if...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306103</th>\n",
              "      <td>ffff3f0a2449ffe4b9ff</td>\n",
              "      <td>Isn't Trump right after all? Why should the US...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306104</th>\n",
              "      <td>ffff41393389d4206066</td>\n",
              "      <td>Is 33 too late for a career in creative advert...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306105</th>\n",
              "      <td>ffff42493fc203cd9532</td>\n",
              "      <td>What is difference between the filteration wor...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306106</th>\n",
              "      <td>ffff48dd47bee89fff79</td>\n",
              "      <td>If the universe \"popped\" into existence from n...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306107</th>\n",
              "      <td>ffff5fd051a032f32a39</td>\n",
              "      <td>How does a shared service technology team meas...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306108</th>\n",
              "      <td>ffff6d528040d3888b93</td>\n",
              "      <td>How is DSATM civil engineering?</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306109</th>\n",
              "      <td>ffff8776cd30cdc8d7f8</td>\n",
              "      <td>Do you know any problem that depends solely on...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306110</th>\n",
              "      <td>ffff94d427ade3716cd1</td>\n",
              "      <td>What are some comic ideas for you Tube videos ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306111</th>\n",
              "      <td>ffffa382c58368071dc9</td>\n",
              "      <td>If you had $10 million of Bitcoin, could you s...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306112</th>\n",
              "      <td>ffffa5b0fa76431c063f</td>\n",
              "      <td>Are you ashamed of being an Indian?</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306113</th>\n",
              "      <td>ffffae5dbda3dc9e9771</td>\n",
              "      <td>What are the methods to determine fossil ages ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306114</th>\n",
              "      <td>ffffba7c4888798571c1</td>\n",
              "      <td>What is your story today?</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306115</th>\n",
              "      <td>ffffc0c7158658a06fd9</td>\n",
              "      <td>How do I consume 150 gms protein daily both ve...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306116</th>\n",
              "      <td>ffffc404da586ac5a08f</td>\n",
              "      <td>What are the good career options for a msc che...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306117</th>\n",
              "      <td>ffffcc4e2331aaf1e41e</td>\n",
              "      <td>What other technical skills do you need as a c...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306118</th>\n",
              "      <td>ffffd431801e5a2f4861</td>\n",
              "      <td>Does MS in ECE have good job prospects in USA ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306119</th>\n",
              "      <td>ffffd48fb36b63db010c</td>\n",
              "      <td>Is foam insulation toxic?</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306120</th>\n",
              "      <td>ffffec519fa37cf60c78</td>\n",
              "      <td>How can one start a research project based on ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306121</th>\n",
              "      <td>ffffed09fedb5088744a</td>\n",
              "      <td>Who wins in a battle between a Wolverine and a...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                          qid  ... target\n",
              "1306102  ffff3778790af9baae76  ...      0\n",
              "1306103  ffff3f0a2449ffe4b9ff  ...      1\n",
              "1306104  ffff41393389d4206066  ...      0\n",
              "1306105  ffff42493fc203cd9532  ...      0\n",
              "1306106  ffff48dd47bee89fff79  ...      0\n",
              "1306107  ffff5fd051a032f32a39  ...      0\n",
              "1306108  ffff6d528040d3888b93  ...      0\n",
              "1306109  ffff8776cd30cdc8d7f8  ...      0\n",
              "1306110  ffff94d427ade3716cd1  ...      0\n",
              "1306111  ffffa382c58368071dc9  ...      0\n",
              "1306112  ffffa5b0fa76431c063f  ...      1\n",
              "1306113  ffffae5dbda3dc9e9771  ...      0\n",
              "1306114  ffffba7c4888798571c1  ...      0\n",
              "1306115  ffffc0c7158658a06fd9  ...      0\n",
              "1306116  ffffc404da586ac5a08f  ...      0\n",
              "1306117  ffffcc4e2331aaf1e41e  ...      0\n",
              "1306118  ffffd431801e5a2f4861  ...      0\n",
              "1306119  ffffd48fb36b63db010c  ...      0\n",
              "1306120  ffffec519fa37cf60c78  ...      0\n",
              "1306121  ffffed09fedb5088744a  ...      0\n",
              "\n",
              "[20 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "leRFRWJMocVa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "8de6f91c-57c6-44cb-c4c7-1199d1e820e0"
      },
      "source": [
        "df.target.plot(kind='hist', title='Target distribution');"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "[encoded data, 7864 characters]\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ELjswHcFHfp3"
      },
      "source": [
        "## Task 4: Create tf.data.Datasets for Training and Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fScULIGPwuWk",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "45769c51-82d2-4347-ece7-738af314777e"
      },
      "source": [
        "train_df, remaining = train_test_split(df, random_state=42, train_size=0.0075, stratify=df.target.values)\n",
        "valid_df, _ = train_test_split(remaining, random_state=42, train_size=0.00075, stratify=remaining.target.values)\n",
        "train_df.shape, valid_df.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((9795, 3), (972, 3))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qQYMGT5_qLPX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "65e133ba-89fe-41b0-f666-db96b50c2f68"
      },
      "source": [
        "with tf.device('/cpu:0'):\n",
        "  train_data = tf.data.Dataset.from_tensor_slices((train_df.question_text.values, train_df.target.values))\n",
        "  valid_data = tf.data.Dataset.from_tensor_slices((valid_df.question_text.values, valid_df.target.values))\n",
        "\n",
        "  for text, label in train_data.take(1):\n",
        "    print(text)\n",
        "    print(label)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tf.Tensor(b'Why are unhealthy relationships so desirable?', shape=(), dtype=string)\n",
            "tf.Tensor(0, shape=(), dtype=int64)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e2-ReN88Hvy_"
      },
      "source": [
        "## Task 5: Download a Pre-trained BERT Model from TensorFlow Hub"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EMb5M86b4-BU"
      },
      "source": [
        "\"\"\"\n",
        "Each line of the dataset is composed of the review text and its label\n",
        "- Data preprocessing consists of transforming text to BERT input features:\n",
        "input_word_ids, input_mask, segment_ids\n",
        "- In the process, tokenizing the text is done with the provided BERT model tokenizer\n",
        "\"\"\"\n",
        "\n",
        "label_list = [0, 1] # Label categories\n",
        "max_seq_length = 128 # maximum length of (token) input sequences\n",
        "train_batch_size = 32\n",
        "\n",
        "# Get BERT layer and tokenizer:\n",
        "# More details here: https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2\n",
        "bert_layer = hub.KerasLayer(\"https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2\",\n",
        "                            trainable=True)\n",
        "vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()\n",
        "do_lower_case = bert_layer.resolved_object.do_lower_case.numpy()\n",
        "tokenizer = tokenization.FullTokenizer(vocab_file, do_lower_case)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wEUezMK-zkkI",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "2deee318-ff99-4a1c-f285-7cf3258f8a93"
      },
      "source": [
        "tokenizer.wordpiece_tokenizer.tokenize('hi, how are you doing?')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['hi', '##,', 'how', 'are', 'you', 'doing', '##?']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5AFsmTO5JSmc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "15409374-b640-4dc9-a36a-f38aac5e2b29"
      },
      "source": [
        "tokenizer.convert_tokens_to_ids(tokenizer.wordpiece_tokenizer.tokenize('hi, how are you doing?'))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[7632, 29623, 2129, 2024, 2017, 2725, 29632]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9QinzNq6OsP1"
      },
      "source": [
        "## Task 6: Tokenize and Preprocess Text for BERT"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3FTqJ698zZ1e"
      },
      "source": [
        "<div align=\"center\">\n",
        "    <img width=\"512px\" src='https://drive.google.com/uc?id=1-SpKFELnEvBMBqO7h3iypo8q9uUUo96P' />\n",
        "    <p style=\"text-align: center;color:gray\">Figure 2: BERT Tokenizer</p>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cWYkggYe6HZc"
      },
      "source": [
        "We'll need to transform our data into a format BERT understands. This involves two steps. First, we create InputExamples using `classifier_data_lib`'s constructor `InputExample` provided in the BERT library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m-21A5aNJM0W"
      },
      "source": [
        "# This provides a function to convert row to input features and label\n",
        "\n",
        "def to_feature(text, label, label_list=label_list, max_seq_length=max_seq_length, tokenizer=tokenizer):\n",
        "  example = classifier_data_lib.InputExample(guid = None,\n",
        "                                            text_a = text.numpy(), \n",
        "                                            text_b = None, \n",
        "                                            label = label.numpy())\n",
        "  feature = classifier_data_lib.convert_single_example(0, example, label_list,\n",
        "                                    max_seq_length, tokenizer)\n",
        "  \n",
        "  return (feature.input_ids, feature.input_mask, feature.segment_ids, feature.label_id)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A_HQSsHwWCsK"
      },
      "source": [
        "You want to use [`Dataset.map`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map) to apply this function to each element of the dataset. [`Dataset.map`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map) runs in graph mode.\n",
        "\n",
        "- Graph tensors do not have a value.\n",
        "- In graph mode you can only use TensorFlow Ops and functions.\n",
        "\n",
        "So you can't `.map` this function directly: You need to wrap it in a [`tf.py_function`](https://www.tensorflow.org/api_docs/python/tf/py_function). The [`tf.py_function`](https://www.tensorflow.org/api_docs/python/tf/py_function) will pass regular tensors (with a value and a `.numpy()` method to access it), to the wrapped python function."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zaNlkKVfWX0Q"
      },
      "source": [
        "## Task 7: Wrap a Python Function into a TensorFlow op for Eager Execution"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AGACBcfCWC2O"
      },
      "source": [
        "def to_feature_map(text, label):\n",
        "  input_ids, input_mask, segment_ids, label_id = tf.py_function(to_feature, inp=[text, label], \n",
        "                                Tout=[tf.int32, tf.int32, tf.int32, tf.int32])\n",
        "\n",
        "  # py_func doesn't set the shape of the returned tensors.\n",
        "  input_ids.set_shape([max_seq_length])\n",
        "  input_mask.set_shape([max_seq_length])\n",
        "  segment_ids.set_shape([max_seq_length])\n",
        "  label_id.set_shape([])\n",
        "\n",
        "  x = {\n",
        "        'input_word_ids': input_ids,\n",
        "        'input_mask': input_mask,\n",
        "        'input_type_ids': segment_ids\n",
        "    }\n",
        "  return (x, label_id)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dhdO6MjTbtn1"
      },
      "source": [
        "## Task 8: Create a TensorFlow Input Pipeline with `tf.data`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LHRdiO3dnPNr"
      },
      "source": [
        "with tf.device('/cpu:0'):\n",
        "  # train\n",
        "  train_data = (train_data.map(to_feature_map,\n",
        "                              num_parallel_calls=tf.data.experimental.AUTOTUNE)\n",
        "                          #.cache()\n",
        "                          .shuffle(1000)\n",
        "                          .batch(32, drop_remainder=True)\n",
        "                          .prefetch(tf.data.experimental.AUTOTUNE))\n",
        "\n",
        "  # valid\n",
        "  valid_data = (valid_data.map(to_feature_map,\n",
        "                            num_parallel_calls=tf.data.experimental.AUTOTUNE)\n",
        "                          .batch(32, drop_remainder=True)\n",
        "                          .prefetch(tf.data.experimental.AUTOTUNE)) "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KLUWnfx-YDi2"
      },
      "source": [
        "The resulting `tf.data.Datasets` return `(features, labels)` pairs, as expected by [`keras.Model.fit`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B0Z2cy9GHQ8x",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        },
        "outputId": "0cb5b236-fb1f-48ac-a23e-81c9c2b228f6"
      },
      "source": [
        "# data spec\n",
        "train_data.element_spec"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "({'input_mask': TensorSpec(shape=(32, 128), dtype=tf.int32, name=None),\n",
              "  'input_type_ids': TensorSpec(shape=(32, 128), dtype=tf.int32, name=None),\n",
              "  'input_word_ids': TensorSpec(shape=(32, 128), dtype=tf.int32, name=None)},\n",
              " TensorSpec(shape=(32,), dtype=tf.int32, name=None))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DGAH-ycYOmao",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        },
        "outputId": "4030e117-9959-4ef2-be4b-125ac7d2882a"
      },
      "source": [
        "# data spec\n",
        "valid_data.element_spec"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "({'input_mask': TensorSpec(shape=(32, 128), dtype=tf.int32, name=None),\n",
              "  'input_type_ids': TensorSpec(shape=(32, 128), dtype=tf.int32, name=None),\n",
              "  'input_word_ids': TensorSpec(shape=(32, 128), dtype=tf.int32, name=None)},\n",
              " TensorSpec(shape=(32,), dtype=tf.int32, name=None))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZxe-7yhPyQe"
      },
      "source": [
        "## Task 9: Add a Classification Head to the BERT Layer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9THH5V0Dw2HO"
      },
      "source": [
        "<div align=\"center\">\n",
        "    <img width=\"512px\" src='https://drive.google.com/uc?id=1fnJTeJs5HUpz7nix-F9E6EZdgUflqyEu' />\n",
        "    <p style=\"text-align: center;color:gray\">Figure 3: BERT Layer</p>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G9il4gtlADcp"
      },
      "source": [
        "# Building the model\n",
        "def create_model():\n",
        "  input_word_ids = tf.keras.layers.Input(shape=(max_seq_length,), dtype=tf.int32,\n",
        "                                      name=\"input_word_ids\")\n",
        "  input_mask = tf.keras.layers.Input(shape=(max_seq_length,), dtype=tf.int32,\n",
        "                                  name=\"input_mask\")\n",
        "  input_type_ids = tf.keras.layers.Input(shape=(max_seq_length,), dtype=tf.int32,\n",
        "                                  name=\"input_type_ids\")\n",
        "\n",
        "  pooled_output, sequence_output = bert_layer([input_word_ids, input_mask, input_type_ids])\n",
        "\n",
        "  drop = tf.keras.layers.Dropout(0.4)(pooled_output)\n",
        "  output = tf.keras.layers.Dense(1, activation=\"sigmoid\", name=\"output\")(drop)\n",
        "\n",
        "  model = tf.keras.Model(\n",
        "    inputs={\n",
        "        'input_word_ids': input_word_ids,\n",
        "        'input_mask': input_mask,\n",
        "        'input_type_ids': input_type_ids\n",
        "    },\n",
        "    outputs=output)\n",
        "  return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S6maM-vr7YaJ"
      },
      "source": [
        "## Task 10: Fine-Tune BERT for Text Classification"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ptCtiiONsBgo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 417
        },
        "outputId": "e16ceba2-bea2-416f-fc8f-165cf9393cc7"
      },
      "source": [
        "model = create_model()\n",
        "model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=2e-5),\n",
        "              loss=tf.keras.losses.BinaryCrossentropy(),\n",
        "              metrics=[tf.keras.metrics.BinaryAccuracy()])\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"functional_1\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_word_ids (InputLayer)     [(None, 128)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_mask (InputLayer)         [(None, 128)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_type_ids (InputLayer)     [(None, 128)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "keras_layer (KerasLayer)        [(None, 768), (None, 109482241   input_word_ids[0][0]             \n",
            "                                                                 input_mask[0][0]                 \n",
            "                                                                 input_type_ids[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "dropout (Dropout)               (None, 768)          0           keras_layer[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "output (Dense)                  (None, 1)            769         dropout[0][0]                    \n",
            "==================================================================================================\n",
            "Total params: 109,483,010\n",
            "Trainable params: 109,483,009\n",
            "Non-trainable params: 1\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6GJaFnkbMtPL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 338
        },
        "outputId": "3ae08dbc-ccca-403f-bda8-8cf83d19825c"
      },
      "source": [
        "tf.keras.utils.plot_model(model=model, show_shapes=True, dpi=76, )"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "image/png": "[encoded data, 39720 characters]\n",
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OcREcgPUHr9O",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 146
        },
        "outputId": "0a193357-3361-4c8f-9268-a21d106abf62"
      },
      "source": [
        "# Train model\n",
        "epochs = 4\n",
        "history = model.fit(train_data,\n",
        "                    validation_data=valid_data,\n",
        "                    epochs=epochs,\n",
        "                    verbose=1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/2\n",
            "306/306 [==============================] - ETA: 0s - loss: 0.1679 - binary_accuracy: 0.9391WARNING:tensorflow:Callbacks method `on_test_batch_end` is slow compared to the batch time (batch time: 0.0122s vs `on_test_batch_end` time: 0.1396s). Check your callbacks.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Callbacks method `on_test_batch_end` is slow compared to the batch time (batch time: 0.0122s vs `on_test_batch_end` time: 0.1396s). Check your callbacks.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r306/306 [==============================] - 147s 480ms/step - loss: 0.1679 - binary_accuracy: 0.9391 - val_loss: 0.1348 - val_binary_accuracy: 0.9531\n",
            "Epoch 2/2\n",
            "306/306 [==============================] - 146s 478ms/step - loss: 0.1040 - binary_accuracy: 0.9608 - val_loss: 0.1600 - val_binary_accuracy: 0.9563\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kNZl1lx_cA5Y"
      },
      "source": [
        "## Task 11: Evaluate the BERT Text Classification Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dCjgrUYH_IsE"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def plot_graphs(history, metric):\n",
        "  plt.plot(history.history[metric])\n",
        "  plt.plot(history.history['val_'+metric], '')\n",
        "  plt.xlabel(\"Epochs\")\n",
        "  plt.ylabel(metric)\n",
        "  plt.legend([metric, 'val_'+metric])\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v6lrFRra_KmA",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "dd4cfcf2-f6f1-4e92-c17c-e50291e0bb09"
      },
      "source": [
        "plot_graphs(history, 'binary_accuracy')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "[encoded data, 19200 characters]\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "opu9neBA_98R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 280
        },
        "outputId": "b62c34d5-566f-4bcc-b153-574a5ad33465"
      },
      "source": [
        "plot_graphs(history, 'loss')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "[encoded data, 18624 characters]\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hkhtCCgnUbY6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 110
        },
        "outputId": "9a82488a-2427-433e-a5eb-5741ccfc3bcf"
      },
      "source": [
        "model.evaluate(valid_data, verbose=1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " 2/30 [=>............................] - ETA: 4s - loss: 0.1791 - binary_accuracy: 0.9375WARNING:tensorflow:Callbacks method `on_test_batch_end` is slow compared to the batch time (batch time: 0.0145s vs `on_test_batch_end` time: 0.1386s). Check your callbacks.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Callbacks method `on_test_batch_end` is slow compared to the batch time (batch time: 0.0145s vs `on_test_batch_end` time: 0.1386s). Check your callbacks.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "30/30 [==============================] - 5s 154ms/step - loss: 0.1600 - binary_accuracy: 0.9563\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.1600438952445984, 0.956250011920929]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K4B8NQBLd9rN"
      },
      "source": [
        "sample_example = [\" \",\\\n",
        "                  \" \",\\\n",
        "                  \" \",\\\n",
        "                  \" \",\\\n",
        "                  \" \",\\\n",
        "                  \" \"]\n",
        "test_data = tf.data.Dataset.from_tensor_slices((sample_example, [0]*len(sample_example)))\n",
        "test_data = (test_data.map(to_feature_map).batch(1))\n",
        "preds = model.predict(test_data)\n",
        "#['Toxic' if pred >=0.5 else 'Sincere' for pred in preds]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FeVNOGfFJT9O"
      },
      "source": [
        "preds"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I_YWudFRJT__"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hENB__IlJUCk"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wkYpiGrhJUFK"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iYqbQZJnJUHw"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aiKuBGgfJUKv"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}