Improved Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution

In this paper, we propose an Improved Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (IKFMPAM-WD). This model is based on the conventional Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (KFMPAM-WD). The proposed model can realize probabilistic association for the training set including one-to-many relations. Moreover, this model has enough robustness for noisy input and damaged neurons. We carried out a series of computer experiments and confirmed the effectiveness of the proposed model.


Introduction
Recently, neural networks are drawing much attention as a method to realize flexible information processing.Neural networks consider neuron groups of the brain in the creature, and imitate these neurons technologically.Neural networks have some features, especially one of the important features is that the networks can learn to acquire the ability of information processing.
In the field of neural network, many models have been proposed such as the Back Propagation algorithm [1], the Kohonen Feature Map (KFM) [2], the Hopfield network [3], and the Bidirectional Associative Memory [4].In these models, the learning process and the recall process are divided, and therefore they need all information to learn in advance.
However, in the real world, it is very difficult to get all information to learn in advance, so we need the model whose learning process and recall process are not divided.As such model, Grossberg and Carpenter proposed the ART (Adaptive Resonance Theory) [5].However, the ART is based on the local representation, and therefore it is not robust for damaged neurons in the Map Layer.While in the field of associative memories, some models have been proposed [6 -8].Since these models are based on the distributed representation, they have the robustness for damaged neurons.However, their storage capacities are small because their learning algorithm is based on the Hebbian learning.
On the other hand, the Kohonen Feature Map (KFM) associative memory [9] has been proposed.Although the KFM associative memory is based on the local representation as similar as the ART [5], it can learn new patterns successively [10], and its storage capacity is larger than that of models in refs.[6 -8].It can deal with auto and hetero associations and the asso-ciations for plural sequential patterns including common terms [11,12].Moreover, the KFM associative memory with area representation [13] has been proposed.In the model, the area representation [14] was introduced to the KFM associative memory, and it has robustness for damaged neurons.However, it can not deal with one-to-many associations, and associations of analog patterns.As the model which can deal with analog patterns and one-to-many associations, the Kohonen Feature Map Associative Memory with Refractoriness based on Area Representation [15] has been proposed.In the model, one-to-many associations are realized by refractoriness of neurons.Moreover, by improvement of the calculation of the internal states of the neurons in the Map Layer, it has enough robustness for damaged neurons when analog patterns are memorized.However, all these models can not realize probabilistic association for the training set including one-to-many relations.As the model which can realize probabilistic association for the training set including oneto-many relations, the Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (KFMPAM-WD) [16] has been proposed.However, in this model, the weights are updated only in the area corresponding to the input pattern, so the learning considering the neighborhood is not carried out.
In this paper, we propose an Improved Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (IKFMPAM-WD).This model is based on the conventional Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution [16].The proposed model can realize probabilistic association for the training set including one-to-many relations.Moreover, this model has enough robustness for noisy input and damaged neurons.And, the learning considering the neighborhood can be realized.

KFM Probabilistic Associative Memory based on Weights Distribution
Here, we explain the conventional Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (KFMPAM-WD)(16).

Structure
Figure 1 shows the structure of the conventional KFMPAM-WD.As shown in Fig. 1, this model has two layers; (1) Input/Output Layer and (2) Map Layer, and the Input/Output Layer is divided into some parts.

Learning process
In the learning algorithm of the conventional KFMPAM-WD, the connection weights are learned as follows: 1.The initial values of weights are chosen randomly.

2.
The Euclidian distance between the learning vector X (p) and the connection weights vector W i , d(X (p) , W i ) is calculated. (p) , W i ) θ t is satisfied for all neurons, the input pattern X (p) is regarded as an unknown pattern.If the input pattern is regarded as a known pattern, go to (8).

4.
The neuron which is the center of the learning area r is determined as follows: where F is the set of the neurons whose connection weights are fixed.d iz is the distance between the neuron i and the neuron z whose connection weights are fixed.In Eq.( 1), D ij is the radius of the ellipse area whose center is the neuron i for the direction to the neuron j, and is given by where a i is the long radius of the ellipse area whose center is the neuron i and b i is the short radius of the ellipse area whose center is the neuron i.In the KFMPAM-WD, a i and b i can be set for each training pattern.m ij is the slope of the line through the neurons i and j.In Eq.( 1), the neuron whose Euclidian distance between its connection weights and the learning vector is minimum in the neurons which can be take areas without overlaps to the areas corresponding to the patterns which are already trained.In Eq.( 1), a i and b i are used as the size of the area for the learning vector. (p) , W r )> θ t is satisfied, the connection weights of the neurons in the ellipse whose center is the neuron r are updated as follows:

If d(X
where α(t) is the learning rate and is given by Here, α 0 is the initial value of α(t) and T is the upper limit of the learning iterations.

7.
The connection weights of the neuron r W r are fixed.

Recall process
In the recall process of the KFMPAM-WD, when the pattern X is given to the Input/Output Layer, the output of the neuron i in the Map Layer, x i map is calculated by where r is selected randomly from the neurons which satisfy 1 where θ map is the threshold of the neuron in the Map Layer, and g( ⋅ ) is given by In the KFMPAM-WD, one of the neurons whose connection weights are similar to the input pattern are selected randomly as the winner neuron.So, the probabilistic association can be realized based on the weights distribution.
When the binary pattern X is given to the Input/Output Layer, the output of the neuron k in the Input/Output Layer x k io is given by where θ b io is the threshold of the neurons in the Input/Output Layer.
When the analog pattern X is given to the Input/Output Layer, the output of the neuron k in the Input/Output Layer x k io is given by

Improved KFM Probabilistic Associative Memory based on Weights Distribution
Here, we explain the proposed Improved Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (IKFMPAM-WD).The proposed model is based on the conventional Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (KFMPAM-WD) [16] described in 2.

Structure
Figure 2 shows the structure of the proposed IKFMPAM-WD.As shown in Fig. 2, the proposed model has two layers; (1) Input/Output Layer and (2) Map Layer, and the Input/ Output Layer is divided into some parts as similar as the conventional KFMPAM-WD.

Learning process
In the learning algorithm of the proposed IKFMPAM-WD, the connection weights are learned as follows: 1.The initial values of weights are chosen randomly.

2.
The Euclidian distance between the learning vector X (p) and the connection weights vector W i , d(X (p) , W i ), is calculated. (p) , W i ) θ t is satisfied for all neurons, the input pattern X (p) is regarded as an unknown pattern.If the input pattern is regarded as a known pattern, go to (8).

4.
The neuron which is the center of the learning area r is determined by Eq.( 1).In Eq.( 1), the neuron whose Euclid distance between its connection weights and the learning vector is minimum in the neurons which can be take areas without overlaps to the areas corresponding to the patterns which are already trained.In Eq.( 1), a i and b i are used as the size of the area for the learning vector. (p) , W r ) θ t is satisfied, the connection weights of the neurons in the ellipse whose center is the neuron r are updated as follows:

If d(X
where θ 1 learn are thresholds.H (d ri ¯) and H (d i * i ¯) are given by Eq.( 11) and these are semifixed function.Especially, H (d ri ¯) behaves as the neighborhood function.Here, i * shows the nearest weight-fixed neuron from the neuron i.
where d ij ¯ shows the normalized radius of the ellipse area whose center is the neuron i for the direction to the neuron j, and is given by In Eq.( 11), D (1 D) is the constant to decide the neighborhood area size and is the steepness parameter.If there is no weight-fixed neuron, is used.

7.
The connection weights of the neuron r W r are fixed.

Recall process
The recall process of the proposed IKFMPAM-WD is same as that of the conventional KFMPAM-WD described in 2.3.

Computer experiment results
Here, we show the computer experiment results to demonstrate the effectiveness of the proposed IKFMPAM-WD.

Experimental conditions
Table 1 shows the experimental conditions used in the experiments of 4.2 ∼ 4.6.

Binary patterns
In this experiment, the binary patterns including one-to-many relations shown in Fig. 3 were memorized in the network composed of 800 neurons in the Input/Output Layer and 400 neurons in the Map Layer. Figure 4 shows a part of the association result when "crow" was given to the Input/Output Layer.As shown in Fig. 4, when "crow" was given to the net-work, "mouse" (t=1), "monkey" (t=2) and "lion" (t=4) were recalled.Figure 5 shows a part of the association result when "duck" was given to the Input/Output Layer.In this case, "dog" (t=251), "cat" (t=252) and "penguin" (t=255) were recalled.From these results, we can confirmed that the proposed model can recall binary patterns including one-to-many relations.

Parameters for Learning
Threshold for Learning θ t learn 10 -4 Neighborhood Area Size     Figure 6 shows the Map Layer after the pattern pairs shown in Fig. 3 were memorized.In Fig. 6, red neurons show the center neuron in each area, blue neurons show the neurons in areas for the patterns including "crow", green neurons show the neurons in areas for the patterns including "duck".As shown in Fig. 6, the proposed model can learn each learning pattern with various size area.Moreover, since the connection weights are updated not only in the area but also in the neighborhood area in the proposed model, areas corresponding to the pattern pairs including "crow"/"duck" are arranged in near area each other.

Artificial Neural Networks -Architectures and Applications
Table 3 shows the recall times of each pattern in the trial of Fig. 4 (t=1∼250) and Fig. 5 (t=251∼ 500).In this table, normalized values are also shown in ( ).From these results, we can confirmed that the proposed model can realize probabilistic associations based on the weight distributions.

Analog patterns
In this experiment, the analog patterns including one-to-many relations shown in Fig. 7 were memorized in the network composed of 800 neurons in the Input/Output Layer and 400 neurons in the Map Layer. Figure 8 shows a part of the association result when "bear" was given to the Input/Output Layer.As shown in Fig. 8, when "bear" was given to the network, "lion" (t=1), "raccoon dog" (t=2) and "penguin" (t=3) were recalled.Figure 9 shows a part of the association result when "mouse" was given to the Input/Output Layer.In this case, "monkey" (t=251), "hen" (t=252) and "chick" (t=253) were recalled.From these results, we can confirmed that the proposed model can recall analog patterns including oneto-many relations.
Figure 10 shows the Map Layer after the pattern pairs shown in Fig. 7 were memorized.In Fig. 10, red neurons show the center neuron in each area, blue neurons show the neurons in the areas for the patterns including "bear", green neurons show the neurons in the areas for the patterns including "mouse".As shown in Fig. 10, the proposed model can learn each learning pattern with various size area.
Table 5 shows the recall times of each pattern in the trial of Fig. 8 (t=1∼ 250) and Fig. 9 (t=251∼ 500).In this table, normalized values are also shown in ( ).From these results, we can confirmed that the proposed model can realize probabilistic associations based on the weight distributions.

Storage capacity
Here, we examined the storage capacity of the proposed model.Figures 11 and 12 show the storage capacity of the proposed model.In this experiment, we used the network composed of 800 neurons in the Input/Output Layer and 400/900 neurons in the Map Layer, and 1-to-P (P=2,3,4) random pattern pairs were memorized as the area (a i =2.5 and b i =1.5).Figures 11 and 12 show the average of 100 trials, and the storage capacities of the conventional model( 16) are also shown for reference in Figs. 13 and 14.From these results, we can confirm that the storage capacity of the proposed model is almost same as that of the conventional model (16).As shown in Figs.11 and 12, the storage capacity of the proposed model does not depend on binary or analog pattern.And it does not depend on P in one-to-P relations.It depends on the number of neurons in the Map Layer.

Association result for noisy input
Figure 15 shows a part of the association result of the proposed model when the pattern "cat" with 20% noise was given during t=1∼ 500.Figure 16 shows a part of the association result of the propsoed model when the pattern "crow" with 20% noise was given t=501∼ 1000.As shown in these figures, the proposed model can recall correct patterns even when the noisy input was given.Figure 19 shows a part of the association result of the proposed model when the pattern "bear" was given during t=1∼ 500.Figure 20 shows a part of the association result of the proposed model when the pattern "mouse" was given t=501∼ 1000.In these experiments, the network whose 20% of neurons in the Map Layer are damaged were used.As shown in these figures, the proposed model can recall correct patterns even when the some neurons in the Map Layer are damaged.

Learning speed
Here, we examined the learning speed of the proposed model.In this experiment, 10 random patterns were memorized in the network composed of 800 neurons in the Input/ Output Layer and 900 neurons in the Map Layer.Table 6 shows the learning time of the proposed model and the conventional model (16).These results are average of 100 trials on the Personal Computer (Intel Pentium 4 (3.2GHz),FreeBSD 4.11, gcc 2.95.3).As shown in Table 6, the learning time of the proposed model is shorter than that of the conventional model.

Conclusions
In

Figure 4 .
Figure 4. One-to-Many Associations for Binary Patterns (When "crow" was Given).

Figure 5 .
Figure 5. One-to-Many Associations for Binary Patterns (When "duck" was Given).

Figure 6 .
Figure 6.Area Representation for Learning Pattern in Fig. 3.

Figure 8 :
Figure 8: One-to-Many Associations for Analog Patterns (When "bear" was Given).

Figure 9 .
Figure 9. One-to-Many Associations for Analog Patterns (When "mouse" was Given).

Figure 10 .
Figure 10.Area Representation for Learning Pattern in Fig. 7.

4. 4 . 2 .
Figures 17 and 18 show the robustness for noisy input of the proposed model.In this experiment, 10 randam patterns in one-to-one relations were memorized in the network composed of 800 neurons in the Input/Output Layer and 900 neurons in the Map Layer.Figures 17 and 18 are the average of 100 trials.As shown in these figures, the proposed model has robustness for noisy input as similar as the conventional model(16).

4. 5 .
Robustness for damaged neurons 4.5.1.Association result when some neurons in map layer are damaged

Figures
Figures 21 and 22 show the robustness when the winner neurons are damaged in the proposed model.In this experiment, 1∼ 10 random patterns in one-to-one relations were memorized in the network composed of 800 neurons in the Input/Output Layer and 900 neurons in the Map Layer.Figures 21 and 22 are the average of 100 trials.As shown in these figures, the proposed model has robustness when the winner neurons are damaged as similar as the conventional model [16].

Figures
Figures 23 and 24 show the robustness for damaged neurons in the proposed model.In this experiment, 10 random patterns in one-to-one relations were memorized in the network composed of 800 neurons in the Input/Output Layer and 900 neurons in the Map Layer.Figures 23 and 24 are the average of 100 trials.As shown in these figures, the proposed model has robustness for damaged neurons as similar as the conventional model [16].
this paper, we have proposed the Improved Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution.This model is based on the conventional Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution.The proposed model can realize probabilistic association for the training set including one-tomany relations.Moreover, this model has enough robustness for noisy input and damaged neurons.We carried out a series of computer experiments and confirmed the effectiveness of the proposed model.