# How popular are neural networks over the years ?

The following graphics presents the number of entries in Google Scholar :

Deep learning becomes more popular while neural networks exhibits a downward slope.

In many books or blogs, you’ll read that neural netwoks have known a golden age in the 60’s and a dark age (AI winter) in the 1970. Not so sure according the the constantly increasing number of research papers published in during these decades:

# TensorFlow Word2vec nsync_cv_h: no such file or directory

-I  YOUR_TENSOR_FLOW_DIR/lib/python3.5/site-packages/external/nsync/public/


-I indicates the path to the header files.

You should have something like this:

g++ -std=c++11 -shared word2vec_ops.cc word2vec_kernels.cc -o word2vec_ops.so -fPIC -I $TF_INC -I /home/philippe/tensorflow/lib/python3.5/site-packages/external/nsync/public/ -O2 -D_GLIBCXX_USE_CXX11_ABI=0  # Quick and simple TensorFlow installation guide for Ubuntu 16.04 (virtualenv) ## Install Since Python 3 is natively installed on Ubuntu 16.04, TensorFLow installation can be done quickly with the following lines: $ sudo apt-get install python3-pip python3-dev python-virtualenv
$virtualenv --system-site-packages -p python3 ~/tensorflow$ source ~/tensorflow/bin/activate
(tensorflow)$pip3 install --upgrade tensorflow  Note that TensorFlow is installed in a virtual environment isolated from other Python development. ## Test your installation Download or create the file tf_test.py containing : # Python import tensorflow as tf hello = tf.constant('Hello, TensorFlow!') sess = tf.Session() print(sess.run(hello))  In the TensorFlow virtual environnement, run the previous script: (tensorflow)$ python3 tf_test.py


It should display something like:

'Hello, TensorFlow!'


## Virtual environnement

### Activate

To activate the TensorFlow virtual environnement:



# Perceptron update rule demonstration

## Perceptron

Let’s consider the following simple perceptron:

## Transfert function

The transfert function is given by:

$y= f(w_1.x_1 + w_2.x_2 + ... + w_N.x_N) = f(\sum\limits_{i=1}^N w_i.x_i)$

Let’s define the sum $S(w_i,x_i)$:

$S(w_i,x_i)= \sum\limits_{i=1}^N w_i.x_i)$

Let’s rewrite $y$:

$y= f(\sum\limits_{i=1}^N w_i.x_i)=f(S(w_i,x_i))$

## Error (or loss)

In artificial neural networks, the error we want to minimize is:

$E=(y$

with:

• $E$ the error
• $y$ the expected output (from training data set)
• $y$ the real output of the network (from network)

In practice and to simplify the maths, this error is divided by two:

$E=\frac{1}{2}(y$

The algorithm (gradient descent) used to train the network (i.e. updating the weights) is given by:

$w_i$

where:

• $w_i$ the weight before update
• $w_i$ the weight after update
• $\eta$ the learning rate

## Derivating the error

Let’s derivate the error:

$\frac{dE}{dw_i} = \frac{1}{2}\frac{d}{dw_i}(y$

Thanks to the chain rule [ $(f \circ g)$ ] the previous equation can be rewritten:

$\frac{dE}{dw_i} = \frac{2}{2}(y$

Let’s calculate the derivative of $y$:

$\frac{dy}{dw_i} = \frac{df(S(w_i,x_i))}{dw_i}$

Thanks to the chain rule [ $(f \circ g)$ ] the previous equation can be rewritten:

$\frac{df(S)}{dw_i} = \frac{df(S)}{dS}\frac{dS}{dw_i} = x_i\frac{df(S)}{dS}$

The derivative of the error becomes:

$\frac{dE}{dw_i} = -x_i(y$

## Updating the weights

The weights can be updated with the following formula:

$w_i$

In conclusion :

$w_i$

# Simplest perceptron update rules demonstration

## Simple perceptron

Let’s consider the following simple perceptron with a transfert function given by $f(x)=x$ to keep the maths simple:

## Transfert function

The transfert function is given by:

$y= w_1.x_1 + w_2.x_2 + ... + w_N.x_N = \sum\limits_{i=1}^N w_i.x_i$

## Error (or loss)

In artificial neural networks, the error we want to minimize is:

$E=(y$

with:

• $E$ the error
• $y$ the expected output (from training data set)
• $y$ the real output of the network (from networt)

In practice and to simplify the maths, this error is divided by two:

$E=\frac{1}{2}(y$

The algorithm (gradient descent) used to train the network (i.e. updating the weights) is given by:

$w_i$

where:

• $w_i$ the weight before update
• $w_i$ the weight after update
• $\eta$ the learning rate

## Derivating the error

Let’s derivate the error:

$\frac{dE}{dw_i} = \frac{1}{2}\frac{d}{dw_i}(y$

Thanks to the chain rule [ $(f \circ g)$ ] the previous equation can be rewritten:

$\frac{dE}{dw_i} = \frac{2}{2}(y$

As $y= w_1.x_1 + w_2.x_2 + ... + w_N.x_N$ :

$\frac{dE}{dw_i} = -(y$

## Updating the weights

The weights can be updated with the following formula:

$w_i$

In conclusion :

$w_i$