Using threads in a neural network

Question

Hello,
I've written a neural network and I'm wondering how I could use threads in it's execution to 1) increase (more precisely achieve!) learning speed and 2) print out the current error value for the network so that I can see how it is working without using the de-bugger. Basically, i've read the Concurrency tutorial but I'm having trouble getting my head around how I can apply it to my network (must be one of those days!)
I'll give a brief explanation of how i've implemented the NN to see if anybody can shed any light on how I should proceed (i.e. whether it can be threaded, what parts to thread etc.)
The network consists of classes:
Neuron - stores input values to be put into the network and performs the activation functions (just a mathematical operation)
WeightMatrix - contains random weights in a 2-D array with methods for accessing and changing those weights based on output error
Layer - simply an array that stores a collection of neurons
InputPattern - stores the values in an array and target value of a pattern (e.g. for logical AND i would store in pattern[0] = 1; pattern [1] = 1; target = 1;)
PatternSet - set of InputPatterns stored so that they can be input into the network for learning
NeuralNetwork - the main class that I want to thread. This class contains multiple layers and multiple WeightMatrices (that connects the neurons in each layer). The learn algorithm then uses the methods of the previous classes to generate neuron inputs and ouputs and error values given a specific input. It uses a loop that iterates through as follows:
    public float learn(PatternSet p)
        InputPattern currentPattern = null;
        double netError=0f;
        float previousError=0f;
        float outputValue = 0f;
        float sum=0f;
        float wcv=0f;
        float output1=0f;
        float output2=0f;
        float currentError=0f;
        float multiply=0f;
        float outputError = 0f;
        float weight = 0f;
        int count;
        int setPosition=0;
        int setSize = p.getSetSize();
        Neuron outputNeuron = layers[getNumberOfLayers()-1].getNeuron(0);
        //execute learning loop and repeat until an acceptable error value is obtained
        do
             //set input layer neuron values to pattern values
            currentPattern = p.getPattern(setPosition);
            for (int i=0; i<currentPattern.getPatternSize(); i++)
                layers[0].getNeuron(i).setNeuronInput(currentPattern.getValue(i));
            currentError = layers[getNumberOfLayers()-1].getNeuron(0).getOutputError();
            for (int a=0; a<layers[getNumberOfLayers()-1].getNumberOfNeurons(); a++)
                //set target value of output neuron
                layers[getNumberOfLayers()-1].getNeuron(a).setTarget(currentPattern.getTarget());
            //iterates between weight layers - i.e. there will be a weight matrix between each layer of the NN
            for (int i=0; i<getNumberOfLayers()-1; i++)
                for (int j=0; j<layers[i+1].getNumberOfNeurons(); j++)
                    sum =0f;
                    count=0;
                    for (int k=0; k<layers.getNumberOfNeurons(); k++)
weight = weights[i].getWeight(k,j);
outputValue = layers[i].getNeuron(count).getOutput();
multiply = layers[i].getNeuron(count).getOutput() * (weights[i].getWeight(k,j));
//add values
sum = sum + multiply;
count++;
//check that all weighted neuron outputs have been completed
if (count == layers[i].getNumberOfNeurons())
//pass results to neuron
layers[i+1].getNeuron(j).setNeuronInput(sum);
//activate neuron
layers[i+1].getNeuron(j).neuronActivation();
//calculate output error of neuron for given input
layers[i+1].getNeuron(j).calculateOutputError();
//check that output layer has been reached and all neurons have been summed together
if (i == getNumberOfLayers()-2 && count == layers[i].getNumberOfNeurons())
outputError = layers[i+1].getNeuron(j).getOutputError();
netError = layers[i+1].getNeuron(j).getNetError();
for (int a=getNumberOfLayers()-1; a>0; a--)
for (int b=0; b<layers[a-1].getNumberOfNeurons(); b++)
for (int c=0; c<layers[a].getNumberOfNeurons(); c++)
output1 = layers[a-1].getNeuron(b).getOutput();
output2 = layers[a].getNeuron(c).getOutput();
wcv = learningRate * (outputError) * output1 * output2 * (1-output2);
weights[a-1].changeWeight(wcv, b, c);
learningCycle++;
if (setPosition < setSize-1)
setPosition++;
else
setPosition=0;
while (netError > acceptableError && learningCycle < 1000000000);
return currentError;
}At the moment the net doesn't seem to learn to an acceptable degree of accuracy, so I was looking to use threads to monitor it's error value change while I left it running just to ensure that it is working as intended (which it seems to be based on NetBeans debugger output). For the moment, all I'm aiming for is an output of the netError value of the NN at a particular time - would this be possible given my current implementation?
Thanks for the help,
Nick                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       

Answer 1

For huge NN and a really multi-core CPU (reporting to OS as a multiple CPU's) one may benefit from having:
- an example pump
- a separate threads for calculation of forward and backwards propagation with in/out queues.
Example pump pumps one forward example to each forward processing thread. It waits for them to complete. Then it reads their output and finds errors to backpropagate. It pumps errors to back-propagation threads. They finds weights correction but does not update weight matrix only are pushing them to the output temporary arrays. The example pump takes those corrections, combines them an updates weights.
Redo from start.
The rule of thumb for high-performance is - avoid locks. If must access data which are changing, make a copy of them in bulk operation, prepare bulk result and read-write in bulk operations.
In this example a whole bunch of weight matrixes and states of neurons are such a kind of data. Each thread should use separate copy and the teaching pump should combine them together. This makes one to split data in two blocks - non-changing, common for all threads (the geometry of NN and weights) and changing, separate for each thread (weight correction, in/out of neurons).
Avoid "new", "clone" and etc. for the preference of System.arraycopy on existing data.
Regards,
Tomasz Sztejka.