tag:blogger.com,1999:blog-13382978.post240434057985700537..comments2015-12-18T09:56:07.631-08:00Comments on Evolving Stuff: Learning to Generate Mona Lisa, AnimatedTom Lahorehttps://plus.google.com/112705120028060356042noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-13382978.post-32600183018887680762013-01-08T22:14:44.270-08:002013-01-08T22:14:44.270-08:00Just posted the source code on GitHub: https://git...Just posted the source code on GitHub: https://github.com/evolvingstuff/MonaLisaThomashttps://www.blogger.com/profile/03351482918298747254noreply@blogger.comtag:blogger.com,1999:blog-13382978.post-75805193059442836502013-01-03T09:36:33.200-08:002013-01-03T09:36:33.200-08:00Hi Edwin,
Here are a few suggestions I can think ...Hi Edwin,<br /><br />Here are a few suggestions I can think of:<br /><br />(1) My implementation of the rectified linear neuron allows for a slight slope, so that if x >= 0 then f(x) = x, else f(x) = 0.01*x. This makes it so the derivative never goes completely to zero. Thus derivative is: if x >= 0 then f'(x) = 1, else f'(x) = 0.01.<br /><br />(2) I found it very important to experiment with initial random link weights. I'm generating them using Gaussian with std dev of 0.3/sqrt(INSTAR_SIZE). So if a given neuron has 100 inputs feeding into it, the std dev would be 0.03.<br /><br />(3) I'm using a fixed learning rate param of 0.1. Vanilla backprop, no momentum or anything fancy.<br /><br />(4) Very important to randomize the order in which the pixels are trained. I initially did them in scan order but got very bad results.<br /><br />(5) I'm using linear outputs, not sigmoid.<br /><br />(6) Experiment on smaller images first to see if you are getting good results.<br /><br />Hope that helps! I'm also going to be posting my code for this in the next week or two. My github profile is: https://github.com/evolvingstuffThomashttps://www.blogger.com/profile/03351482918298747254noreply@blogger.comtag:blogger.com,1999:blog-13382978.post-62103562195280760442013-01-03T08:33:17.228-08:002013-01-03T08:33:17.228-08:00Nice stuff. It inspired me to try the same, howeve...Nice stuff. It inspired me to try the same, however I have a hard time trying to reproduce your results.<br /><br />I am currently trying to implement this in Java using Encog 3. For this I am using the ResilientPropagation trainer. My network layout is as follows: I use three hidden layers using my own implementation for the rectified linear activation function max(0,x), with 300 neurons each. In the output layer I use the sigmoid activation function. Hidden and output layers use biases.<br /><br />Input and target values are all within [0,1] range.<br /><br />Training with 300 neurons in each hidden layer takes forever on my laptop. The network produces saturated colors and does not really seem to converge to the image. Also when I reduce the number of neurons to around 30 in each layer I get mostly saturated colors.<br /><br />Any hints on what I am doing wrong?Edwinhttp://fileunderpopular.comnoreply@blogger.com