Category Archives: Artificial Intelligence

How popular are neural networks over the years ?

The following graphics presents the number of entries in Google Scholar :

Deep learning becomes more popular while neural networks exhibits a downward slope.

In many books or blogs, you’ll read that neural netwoks have known a golden age in the 60’s and a dark age (AI winter) in the 1970. Not so sure according the the constantly increasing number of research papers published in during these decades:


TensorFlow Word2vec nsync_cv_h: no such file or directory

Add the path to header files to the compiler, add the following line to g++ :

-I  YOUR_TENSOR_FLOW_DIR/lib/python3.5/site-packages/external/nsync/public/

-I indicates the path to the header files.

You should have something like this:

g++ -std=c++11 -shared word2vec_ops.cc word2vec_kernels.cc -o word2vec_ops.so -fPIC -I $TF_INC -I /home/philippe/tensorflow/lib/python3.5/site-packages/external/nsync/public/ -O2 -D_GLIBCXX_USE_CXX11_ABI=0

Quick and simple TensorFlow installation guide for Ubuntu 16.04 (virtualenv)

Install

Since Python 3 is natively installed on Ubuntu 16.04, TensorFLow installation can be done quickly with the following lines:

$ sudo apt-get install python3-pip python3-dev python-virtualenv
$ virtualenv --system-site-packages -p python3 ~/tensorflow
$ source ~/tensorflow/bin/activate
(tensorflow)$ pip3 install --upgrade tensorflow

Note that TensorFlow is installed in a virtual environment isolated from other Python development.

Test your installation

Download or create the file tf_test.py containing :

# Python
import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(sess.run(hello))

In the TensorFlow virtual environnement, run the previous script:

(tensorflow)$ python3 tf_test.py 

It should display something like:

'Hello, TensorFlow!'

Virtual environnement

Activate

To activate the TensorFlow virtual environnement:

$ source ~/tensorflow/bin/activate

Deactivate

To deactivate the TensorFlow virtual environnement:

(tensorflow)$ deactivate

Download

Perceptron update rule demonstration

Updated version of this page on neural-networks.io.

Perceptron

Let’s consider the following simple perceptron:

Transfert function

The transfert function is given by:

 y= f(w_1.x_1 + w_2.x_2 + ... + w_N.x_N) = f(\sum\limits_{i=1}^N w_i.x_i)

Let’s define the sum S(w_i,x_i):

 S(w_i,x_i)= \sum\limits_{i=1}^N w_i.x_i)

Let’s rewrite y:

 y= f(\sum\limits_{i=1}^N w_i.x_i)=f(S(w_i,x_i))

Error (or loss)

In artificial neural networks, the error we want to minimize is:

 E=(y

with:

  • E the error
  • y the expected output (from training data set)
  • y the real output of the network (from network)

In practice and to simplify the maths, this error is divided by two:

 E=\frac{1}{2}(y

Gradient descent

The algorithm (gradient descent) used to train the network (i.e. updating the weights) is given by:

 w_i

where:

  • w_i the weight before update
  • w_i the weight after update
  • \eta the learning rate

Derivating the error

Let’s derivate the error:

 \frac{dE}{dw_i} = \frac{1}{2}\frac{d}{dw_i}(y

Thanks to the chain rule [  (f \circ g) ] the previous equation can be rewritten:

 \frac{dE}{dw_i} = \frac{2}{2}(y

Let’s calculate the derivative of y:

 \frac{dy}{dw_i} = \frac{df(S(w_i,x_i))}{dw_i}

Thanks to the chain rule [  (f \circ g) ] the previous equation can be rewritten:

 \frac{df(S)}{dw_i} = \frac{df(S)}{dS}\frac{dS}{dw_i} = x_i\frac{df(S)}{dS}

The derivative of the error becomes:

 \frac{dE}{dw_i} = -x_i(y

Updating the weights

The weights can be updated with the following formula:

 w_i

In conclusion :

 w_i

Simplest perceptron update rules demonstration

Updated version of this page on neural-networks.io.

Simple perceptron

Let’s consider the following simple perceptron with a transfert function given by f(x)=x to keep the maths simple:

Transfert function

The transfert function is given by:

 y= w_1.x_1 + w_2.x_2 + ... + w_N.x_N = \sum\limits_{i=1}^N w_i.x_i

Error (or loss)

In artificial neural networks, the error we want to minimize is:

 E=(y

with:

  • E the error
  • y the expected output (from training data set)
  • y the real output of the network (from networt)

In practice and to simplify the maths, this error is divided by two:

 E=\frac{1}{2}(y

Gradient descent

The algorithm (gradient descent) used to train the network (i.e. updating the weights) is given by:

 w_i

where:

  • w_i the weight before update
  • w_i the weight after update
  • \eta the learning rate

Derivating the error

Let’s derivate the error:

 \frac{dE}{dw_i} = \frac{1}{2}\frac{d}{dw_i}(y

Thanks to the chain rule [  (f \circ g) ] the previous equation can be rewritten:

 \frac{dE}{dw_i} = \frac{2}{2}(y

As  y= w_1.x_1 + w_2.x_2 + ... + w_N.x_N :

 \frac{dE}{dw_i} = -(y

Updating the weights

The weights can be updated with the following formula:

 w_i

In conclusion :

 w_i