tf data input pipeline

Author: codebasics

44_tf_data_pipeline/Exercise/reviews/negative/neg_1.txt

@@ -0,0 +1 @@
+Basically there's a family where a little boy (Jake) thinks there's a zombie in his closet & his parents are fighting all the time.<br /><br />This movie is slower than a soap opera... and suddenly, Jake decides to become Rambo and kill the zombie.<br /><br />OK, first of all when you're going to make a film you must Decide if its a thriller or a drama! As a drama the movie is watchable. Parents are divorcing & arguing like in real life. And then we have Jake with his closet which totally ruins all the film! I expected to see a BOOGEYMAN similar movie, and instead i watched a drama with some meaningless thriller spots.<br /><br />3 out of 10 just for the well playing parents & descent dialogs. As for the shots with Jake: just ignore them.

44_tf_data_pipeline/Exercise/reviews/negative/neg_2.txt

@@ -0,0 +1 @@
+This show was an amazing, fresh & innovative idea in the 70's when it first aired. The first 7 or 8 years were brilliant, but things dropped off after that. By 1990, the show was not really funny anymore, and it's continued its decline further to the complete waste of time it is today.<br /><br />It's truly disgraceful how far this show has fallen. The writing is painfully bad, the performances are almost as bad - if not for the mildly entertaining respite of the guest-hosts, this show probably wouldn't still be on the air. I find it so hard to believe that the same creator that hand-selected the original cast also chose the band of hacks that followed. How can one recognize such brilliance and then see fit to replace it with such mediocrity? I felt I must give 2 stars out of respect for the original cast that made this show such a huge success. As it is now, the show is just awful. I can't believe it's still on the air.

44_tf_data_pipeline/Exercise/reviews/negative/neg_3.txt

@@ -0,0 +1 @@
+One of the other reviewers has mentioned that after watching just 1 Oz episode you'll be hooked. They are right, as this is exactly what happened with me.<br /><br />The first thing that struck me about Oz was its brutality and unflinching scenes of violence, which set in right from the word GO. Trust me, this is not a show for the faint hearted or timid. This show pulls no punches with regards to drugs, sex or violence. Its is hardcore, in the classic use of the word.<br /><br />It is called OZ as that is the nickname given to the Oswald Maximum Security State Penitentary. It focuses mainly on Emerald City, an experimental section of the prison where all the cells have glass fronts and face inwards, so privacy is not high on the agenda. Em City is home to many..Aryans, Muslims, gangstas, Latinos, Christians, Italians, Irish and more....so scuffles, death stares, dodgy dealings and shady agreements are never far away.<br /><br />I would say the main appeal of the show is due to the fact that it goes where other shows wouldn't dare. Forget pretty pictures painted for mainstream audiences, forget charm, forget romance...OZ doesn't mess around. The first episode I ever saw struck me as so nasty it was surreal, I couldn't say I was ready for it, but as I watched more, I developed a taste for Oz, and got accustomed to the high levels of graphic violence. Not just violence, but injustice (crooked guards who'll be sold out for a nickel, inmates who'll kill on order and get away with it, well mannered, middle class inmates being turned into prison bitches due to their lack of street skills or prison experience) Watching Oz, you may become comfortable with what is uncomfortable viewing....thats if you can get in touch with your darker side.

44_tf_data_pipeline/Exercise/reviews/positive/pos_1.txt

@@ -0,0 +1 @@
+A wonderful little production. <br /><br />The filming technique is very unassuming- very old-time-BBC fashion and gives a comforting, and sometimes discomforting, sense of realism to the entire piece. <br /><br />The actors are extremely well chosen- Michael Sheen not only "has got all the polari" but he has all the voices down pat too! You can truly see the seamless editing guided by the references to Williams' diary entries, not only is it well worth the watching but it is a terrificly written and performed piece. A masterful production about one of the great master's of comedy and his life. <br /><br />The realism really comes home with the little things: the fantasy of the guard which, rather than use the traditional 'dream' techniques remains solid then disappears. It plays on our knowledge and our senses, particularly with the scenes concerning Orton and Halliwell and the sets (particularly of their flat with Halliwell's murals decorating every surface) are terribly well done.

44_tf_data_pipeline/Exercise/reviews/positive/pos_2.txt

@@ -0,0 +1,229 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<h3  align=\"center\" style='color:blue'>TF Data Input Pipeline: Exercise Solution</h3>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Moview reviews are present as individual text file (one file per review) in review folder. \n",
+    "\n",
+    "Folder structure looks like this,\n",
+    "\n",
+    "reviews\n",
+    "\n",
+    "    |__ positive\n",
+    "        |__pos_1.txt\n",
+    "        |__pos_2.txt\n",
+    "        |__pos_3.txt\n",
+    "    |__ negative\n",
+    "        |__neg_1.txt\n",
+    "        |__neg_2.txt\n",
+    "        |__neg_3.txt\n",
+    "   \n",
+    "You need to read these reviews using tf.data.Dataset and perform following transformations,\n",
+    "\n",
+    "(1) Read text review and generate a label from folder name. your dataset should have review text and label as a tuple\n",
+    "\n",
+    "(2) Filter blank text review. Two files are blank in this dataset\n",
+    "\n",
+    "(3) Do all of the above transformations in single line of code. Also shuffle all the reviews"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 168,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import tensorflow as tf"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<h3 style='color:purple'>Retrieve review file paths in a tensorflow dataset</h3>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 177,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "reviews_ds = tf.data.Dataset.list_files('reviews/*/*', shuffle=False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 178,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "b'reviews\\\\negative\\\\neg_1.txt'\n",
+      "b'reviews\\\\negative\\\\neg_2.txt'\n",
+      "b'reviews\\\\negative\\\\neg_3.txt'\n",
+      "b'reviews\\\\positive\\\\pos_1.txt'\n",
+      "b'reviews\\\\positive\\\\pos_2.txt'\n",
+      "b'reviews\\\\positive\\\\pos_3.txt'\n"
+     ]
+    }
+   ],
+   "source": [
+    "for file in reviews_ds:\n",
+    "    print(file.numpy())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<h3 style='color:purple'>Extract review text from these files. Extract label from folder name</h3>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 179,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "def extract_review_and_label(file_path):\n",
+    "    return tf.io.read_file(file_path), tf.strings.split(file_path, os.path.sep)[-2]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 180,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Review:  b\"Basically there's a family where a little boy (Jak\"\n",
+      "Label:  b'negative'\n",
+      "Review:  b'This show was an amazing, fresh & innovative idea '\n",
+      "Label:  b'negative'\n",
+      "Review:  b''\n",
+      "Label:  b'negative'\n",
+      "Review:  b'One of the other reviewers has mentioned that afte'\n",
+      "Label:  b'positive'\n",
+      "Review:  b'A wonderful little production. <br /><br />The fil'\n",
+      "Label:  b'positive'\n",
+      "Review:  b''\n",
+      "Label:  b'positive'\n"
+     ]
+    }
+   ],
+   "source": [
+    "reviews_ds_1 = reviews_ds.map(extract_review_and_label)\n",
+    "for review, label in reviews_ds_1:\n",
+    "    print(\"Review: \",review.numpy()[:50])\n",
+    "    print(\"Label: \",label.numpy())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<h3 style='color:purple'>Filter blank reviews</h3>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 181,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Review:  b\"Basically there's a family where a little boy (Jak\"\n",
+      "Label:  b'negative'\n",
+      "Review:  b'This show was an amazing, fresh & innovative idea '\n",
+      "Label:  b'negative'\n",
+      "Review:  b'One of the other reviewers has mentioned that afte'\n",
+      "Label:  b'positive'\n",
+      "Review:  b'A wonderful little production. <br /><br />The fil'\n",
+      "Label:  b'positive'\n"
+     ]
+    }
+   ],
+   "source": [
+    "reviews_ds_2 = reviews_ds_1.filter(lambda review, label: review!=\"\")\n",
+    "for review, label in reviews_ds_2.as_numpy_iterator():\n",
+    "    print(\"Review: \",review[:50])\n",
+    "    print(\"Label: \",label)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<h3 style='color:purple'>Perform map, filter and shuffle all in single line of code</h3>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 182,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Review: b'This show was an amazing, fresh & innovative idea '\n",
+      "Label: b'negative'\n",
+      "Review: b\"Basically there's a family where a little boy (Jak\"\n",
+      "Label: b'negative'\n",
+      "Review: b'A wonderful little production. <br /><br />The fil'\n",
+      "Label: b'positive'\n",
+      "Review: b'One of the other reviewers has mentioned that afte'\n",
+      "Label: b'positive'\n"
+     ]
+    }
+   ],
+   "source": [
+    "final_ds = reviews_ds.map(extract_review_and_label).filter(lambda review, label: review!=\"\").shuffle(3)\n",
+    "for review, label in final_ds.as_numpy_iterator():\n",
+    "    print(\"Review:\",review[:50])\n",
+    "    print(\"Label:\",label)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}