Example: memory networks in Cicada

Cicada originally aimed to be a neural network engine, missed by a mile, and ended up as a scripting language. So let’s use it to implement an associative neural network: a type of pattern-learning algorithm which will learn to associate inputs with desired outputs.

Algorithm: We’ll use a training algorithm for symmetric networks (J. R. Movellan, Contrastive Hebbian learning in interactive networks, 1990) which trains in two phases: clamped, and free. In the clamped phase, set the input neuron activities to the network input, and set the output neuron activities to the pattern we are trying to learn. Then repeatedly adjust the other ‘hidden’ neurons’ activities x_i using the formula:

   x_i  <--  1 / [ 1 + exp(-∑_j w_ij x_j - b_i) ].

Here w_ij is the connection weight from neuron j to neuron i, and b_i is a bias. Once the network settles into a steady state, apply the following training rule to the weights and biases:

   w_ij  <--  w_ij + η · x_i x_j

    b_i  <--  b_i + η · x_i

Next, repeat the process for the free phase of operation, where that the activities of the output neurons update freely using the same rule as the hidden neurons. After reaching steady state, a negative weight/bias adjustment is applied. The idea is to reinforce correlated firing patterns on the difference between what the network should vs. does currently compute. As the saying goes, neurons that fire together, wire together.

To use the network to recover a memory, run it in its ‘free’ mode without applying any training.

• C implementation
• Putting the C in Cicada
• Writing and debugging a Cicada wrapper
• The Anagrambler

Last update: November 12, 2025