Back

Explore Courses Blog Tutorials Interview Questions
0 votes
1 view
in Machine Learning by (19k points)

For learning purposes, I have implemented a simple neural network framework which only supports multi-layer perceptrons and simple backpropagation. It works okay-ish for linear classification, and the usual XOR problem, but for sine function approximation the results are not that satisfying.

 

I'm basically trying to approximate one period of the sine function with one hidden layer consisting of 6-10 neurons. The network uses hyperbolic tangent as an activation function for the hidden layer and a linear function for the output. The result remains a quite rough estimate of the sine wave and takes long to calculate.

 

I looked at encog for reference and but even with that, I fail to get it work with simple backpropagation (by switching to resilient propagation it starts to get better but is still way worse than the super slick R script provided in this similar question). So am I actually trying to do something that's not possible? Is it not possible to approximate sine with simple backpropagation (no momentum, no dynamic learning rate)? What is the actual method used by the neural network library in R?

I know that it is definitely possible to find a good-enough approximation even with simple backpropagation (if you are incredibly lucky with your initial weights) but I actually was more interested to know if this is a feasible approach. The R script I linked to just seems to converge so incredibly fast and robustly (in 40 epochs with only few learning samples) compared to my implementation or even encog's resilient propagation. I'm just wondering if there's something I can do to improve my backpropagation algorithm to get that same performance or do I have to look into some more advanced learning method?

1 Answer

0 votes
by (33.1k points)

You should use TensorFlow to implement this. It is the most commonly used deep learning library to implement neural networks.

For example:

A two-layer neural network that uses 100 neurons per layer to train in seconds.

image

Code:

import tensorflow as tf

import numpy as np

 

with tf.name_scope('placeholders'):

    x = tf.placeholder('float', [None, 1])

    y = tf.placeholder('float', [None, 1])

 

with tf.name_scope('neural_network'):

    x1 = tf.contrib.layers.fully_connected(x, 100)

    x2 = tf.contrib.layers.fully_connected(x1, 100)

    result = tf.contrib.layers.fully_connected(x2, 1,

                                               activation_fn=None)

 

    loss = tf.nn.l2_loss(result - y)

 

with tf.name_scope('optimizer'):

    train_op = tf.train.AdamOptimizer().minimize(loss)

 

with tf.Session() as sess:

    sess.run(tf.global_variables_initializer())

 

    # Train the network

    for i in range(10000):

        xpts = np.random.rand(100) * 10

        ypts = np.sin(xpts)

 

        _, loss_result = sess.run([train_op, loss],

                                  feed_dict={x: xpts[:, None],

                                             y: ypts[:, None]})

 

        print('iteration {}, loss={}'.format(i, loss_result))

For more details and a variable amount of insights, study Machine Learning Course. Also, one can study Neural Network Tutorial to master the domain. 

Welcome to Intellipaat Community. Get your technical queries answered by top developers!

28.4k questions

29.7k answers

500 comments

94.1k users

Browse Categories

...