Udacity Deep Learning Nano Degree

Added Weights Initialization

Author: TarrySingh

deep-learning/udacity-deeplearning/README.md

@@ -0,0 +1,46 @@
+# Deep Learning Nanodegree Foundation
+
+This repository contains material related to Udacity's [Deep Learning Nanodegree Foundation](https://www.udacity.com/course/deep-learning-nanodegree-foundation--nd101) program. It consists of a bunch of tutorial notebooks for various deep learning topics. In most cases, the notebooks lead you through implementing models such as convolutional networks, recurrent networks, and GANs. There are other topics covered such as weight intialization and batch normalization.
+
+There are also notebooks used as projects for the Nanodegree program. In the program itself, the projects are reviewed by Udacity experts, but they are available here as well.
+
+## Table Of Contents
+
+### Tutorials
+
+* [Sentiment Analysis with Numpy](https://github.com/udacity/deep-learning/tree/master/sentiment-network): [Andrew Trask](http://iamtrask.github.io/) leads you through building a sentiment analysis model, predicting if some text is positive or negative.
+* [Intro to TensorFlow](https://github.com/udacity/deep-learning/tree/master/intro-to-tensorflow): Starting building neural networks with Tensorflow.
+* [Weight Intialization](https://github.com/udacity/deep-learning/tree/master/weight-initialization): Explore how initializing network weights affects performance.
+* [Autoencoders](https://github.com/udacity/deep-learning/tree/master/autoencoder): Build models for image compression and denoising, using feed-forward and convolution networks in TensorFlow.
+* [Transfer Learning (ConvNet)](https://github.com/udacity/deep-learning/tree/master/transfer-learning). In practice, most people don't train their own large networkd on huge datasets, but use pretrained networks such as VGGnet. Here you'll use VGGnet to classify images of flowers without training a network on the images themselves.
+* [Intro to Recurrent Networks (Character-wise RNN)](https://github.com/udacity/deep-learning/tree/master/intro-to-rnns): Recurrent neural networks are able to use information about the sequence of data, such as the sequence of characters in text.
+* [Embeddings (Word2Vec)](https://github.com/udacity/deep-learning/tree/master/embeddings): Implement the Word2Vec model to find semantic representations of words for use in natural language processing.
+* [Sentiment Analysis RNN](https://github.com/udacity/deep-learning/tree/master/sentiment-rnn): Implement a recurrent neural network that can predict if a text sample is positive or negative.
+* [Tensorboard](https://github.com/udacity/deep-learning/tree/master/tensorboard): Use TensorBoard to visualize the network graph, as well as how parameters change through training.
+* [Reinforcement Learning (Q-Learning)](https://github.com/udacity/deep-learning/tree/master/reinforcement): Implement a deep Q-learning network to play a simple game from OpenAI Gym.
+* [Sequence to sequence](https://github.com/udacity/deep-learning/tree/master/seq2seq): Implement a sequence-to-sequence recurrent network.
+* [Batch normalization](https://github.com/udacity/deep-learning/tree/master/batch-norm): Learn how to improve training rates and network stability with batch normalizations.
+* [Generative Adversatial Network on MNIST](https://github.com/udacity/deep-learning/tree/master/gan_mnist): Train a simple generative adversarial network on the MNIST dataset.
+* [Deep Convolutional GAN (DCGAN)](https://github.com/udacity/deep-learning/tree/master/dcgan-svhn): Implement a DCGAN to generate new images based on the Street View House Numbers (SVHN) dataset.
+* [Intro to TFLearn](https://github.com/udacity/deep-learning/tree/master/intro-to-tflearn): A couple introductions to a high-level library for building neural networks.
+
+### Projects
+
+* [Your First Neural Network](https://github.com/udacity/deep-learning/tree/master/first-neural-network): Implement a neural network in Numpy to predict bike rentals.
+* [Image classification](https://github.com/udacity/deep-learning/tree/master/image-classification): Build a convolutional neural network with TensorFlow to classify CIFAR-10 images.
+* [Text Generation](https://github.com/udacity/deep-learning/tree/master/tv-script-generation): Train a recurrent neural network on scripts from The Simpson's (copyright Fox) to generate new scripts.
+* [Machine Translation](https://github.com/udacity/deep-learning/tree/master/language-translation): Train a sequence to sequence network for English to French translation (on a simple dataset)
+* [Face Generation](https://github.com/udacity/deep-learning/tree/master/face_generation): Use a DCGAN on the CelebA dataset to generate images of novel and realistic human faces.
+
+
+## Dependencies
+
+Each directory has a `requirements.txt` describing the minimal dependencies required to run the notebooks in that directory.
+
+### pip
+
+To install these dependencies with pip, you can issue `pip3 install -r requirements.txt`.
+
+### Conda Environments
+
+You can find Conda environment files for the Deep Learning program in the `environments` folder. Note that environment files are platform dependent. Versions with `tensorflow-gpu` are labeled in the filename with "GPU".

deep-learning/udacity-deeplearning/weight-initialization/.gitignore

@@ -0,0 +1 @@
+MNIST_data/

deep-learning/udacity-deeplearning/weight-initialization/helper.py

@@ -0,0 +1,116 @@
+import numpy as np
+import matplotlib.pyplot as plt
+import tensorflow as tf
+
+
+def hist_dist(title, distribution_tensor, hist_range=(-4, 4)):
+    """
+    Display histogram of a TF distribution
+    """
+    with tf.Session() as sess:
+        values = sess.run(distribution_tensor)
+
+    plt.title(title)
+    plt.hist(values, np.linspace(*hist_range, num=len(values)/2))
+    plt.show()
+
+
+def _get_loss_acc(dataset, weights):
+    """
+    Get losses and validation accuracy of example neural network
+    """
+    batch_size = 128
+    epochs = 2
+    learning_rate = 0.001
+
+    features = tf.placeholder(tf.float32)
+    labels = tf.placeholder(tf.float32)
+    learn_rate = tf.placeholder(tf.float32)
+
+    biases = [
+        tf.Variable(tf.zeros([256])),
+        tf.Variable(tf.zeros([128])),
+        tf.Variable(tf.zeros([dataset.train.labels.shape[1]]))
+    ]
+
+    # Layers
+    layer_1 = tf.nn.relu(tf.matmul(features, weights[0]) + biases[0])
+    layer_2 = tf.nn.relu(tf.matmul(layer_1, weights[1]) + biases[1])
+    logits = tf.matmul(layer_2, weights[2]) + biases[2]
+
+    # Training loss
+    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=labels))
+
+    # Optimizer
+    optimizer = tf.train.AdamOptimizer(learn_rate).minimize(loss)
+
+    # Accuracy
+    correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(labels, 1))
+    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
+
+    # Measurements use for graphing loss
+    loss_batch = []
+
+    with tf.Session() as session:
+        session.run(tf.global_variables_initializer())
+        batch_count = int((dataset.train.num_examples / batch_size))
+
+        # The training cycle
+        for epoch_i in range(epochs):
+            for batch_i in range(batch_count):
+                batch_features, batch_labels = dataset.train.next_batch(batch_size)
+
+                # Run optimizer and get loss
+                session.run(
+                    optimizer,
+                    feed_dict={features: batch_features, labels: batch_labels, learn_rate: learning_rate})
+                l = session.run(
+                    loss,
+                    feed_dict={features: batch_features, labels: batch_labels, learn_rate: learning_rate})
+                loss_batch.append(l)
+
+        valid_acc = session.run(
+            accuracy,
+            feed_dict={features: dataset.validation.images, labels: dataset.validation.labels, learn_rate: 1.0})
+
+    # Hack to Reset batches
+    dataset.train._index_in_epoch = 0
+    dataset.train._epochs_completed = 0
+
+    return loss_batch, valid_acc
+
+
+def compare_init_weights(
+        dataset,
+        title,
+        weight_init_list,
+        plot_n_batches=100):
+    """
+    Plot loss and print stats of weights using an example neural network
+    """
+    colors = ['r', 'b', 'g', 'c', 'y', 'k']
+    label_accs = []
+    label_loss = []
+
+    assert len(weight_init_list) <= len(colors), 'Too many inital weights to plot'
+
+    for i, (weights, label) in enumerate(weight_init_list):
+        loss, val_acc = _get_loss_acc(dataset, weights)
+
+        plt.plot(loss[:plot_n_batches], colors[i], label=label)
+        label_accs.append((label, val_acc))
+        label_loss.append((label, loss[-1]))
+
+    plt.title(title)
+    plt.xlabel('Batches')
+    plt.ylabel('Loss')
+    plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
+    plt.show()
+
+    print('After 858 Batches (2 Epochs):')
+    print('Validation Accuracy')
+    for label, val_acc in label_accs:
+        print('  {:7.3f}% -- {}'.format(val_acc*100, label))
+    print('Loss')
+    for label, loss in label_loss:
+        print('  {:7.3f}  -- {}'.format(loss, label))

deep-learning/udacity-deeplearning/weight-initialization/images/neural_network.png

@@ -0,0 +1,47 @@
+appdirs==1.4.3
+appnope==0.1.0
+bleach==2.0.0
+cycler==0.10.0
+decorator==4.0.11
+entrypoints==0.2.2
+html5lib==0.999999999
+ipykernel==4.5.2
+ipython==5.3.0
+ipython-genutils==0.2.0
+ipywidgets==6.0.0
+Jinja2==2.9.5
+jsonschema==2.6.0
+jupyter==1.0.0
+jupyter-client==5.0.0
+jupyter-console==5.1.0
+jupyter-core==4.3.0
+MarkupSafe==1.0
+matplotlib==2.0.0
+mistune==0.7.4
+nbconvert==5.1.1
+nbformat==4.3.0
+notebook==4.4.1
+numpy==1.12.1
+packaging==16.8
+pandocfilters==1.4.1
+pexpect==4.2.1
+pickleshare==0.7.4
+prompt-toolkit==1.0.14
+protobuf==3.2.0
+ptyprocess==0.5.1
+Pygments==2.2.0
+pyparsing==2.2.0
+python-dateutil==2.6.0
+pytz==2017.2
+pyzmq==16.0.2
+qtconsole==4.3.0
+simplegeneric==0.8.1
+six==1.10.0
+tensorflow==1.0.0
+terminado==0.6
+testpath==0.3
+tornado==4.4.3
+traitlets==4.3.2
+wcwidth==0.1.7
+webencodings==0.5
+widgetsnbextension==2.0.0