• Primitive functions implement the elementary concepts of an unified framework. There are primitive functions for allocating units, defining connections and performing elementary operations on the units or the connections.
  
• Connection functions are implemented in the file "sn28/lib/connect.sn" . This library implement the most common connection schemes: full connections, local connections, shared weight connections, time delay connections,...
  
• Simulation functions are implemented in the file "sn28/lib/netenv.sn" . This library calls the sequence of primitive functions needed for training or testing a multilayer network using various optimisation strategies. The user can program training and test epochs, plot curves and adjust the various parameters of the training algorithms.
  
• Example libraries take advantage of the Netenv mechanism for implementing several neural network models. Since the sequencement is performed in lisp, it is possible to build different algorithms by writing new sequencement functions using the same primitives.
  

Curious readers might find Netenv based code for the following algorithms:

• Kmeans and LVQ in directory "sn28/examples/netlvq/" ,
  
• RBF networks in directory "sn28/examples/netrbf/" ,
  
• Hopfield networks, Kanerva memories, Adalines and Rosenblatt's perceptrons in directory "sn28/examples/netold/" .
  

• Section 2 describes primitive functions directly implemented in C language. These functions implement the elementary concepts of neural networks. There are primitives for allocating units, defining connections and performing elementary computations on units and connections.
  
• Section 3 introduces high level functions for training and testing neural networks. These functions are implemented in file "sn28/lib/netenv.sn" .
  
• Section 4 describes high level functions for creating standard connection patterns between layers. These functions, located in file "sn28/lib/connect.sn" , can handle full connections, local connections, shared weight connections and time delay connections.
  

Most of these functions, however, are seldom used directly: users call instead high level functions which actually use the low level functions. Primitive functions are the basic elements of such higher level functions You may use them if you want to define a new algorithm and thus write a new library.

Understanding these basic computation functions of the simulator, however, is a very efficient way to use the full capacity of SN2.8.

The special unit 0 is often referred to as the bias unit. It is always allocated and its initial state is set to 1 . Establishing connections from this unit is a simple way to introduce an adaptable bias parameters.

This function returns the number of bytes allocated.

; create a set of 10 units and store the list of their indices in the variable my-layer
? (setq my-layer (newneurons 10))
= (5 6 7 8 9 10 11 12 13 14)

See: (alloc-net units links [ weights ])

The optional argument val specifies an (initial) weight value for this connection. With kernels sn28new and sn28itenew, the optional argument eps specifies the learning rate of the connection.

Like function connect , function dup-connection handles the optional arguments val for setting the weight value and eps for setting the learning rate.

Cutting a connection does not free the corresponding weight slot. The weight value however is cleared when you cut the last connection sharing this weight. This property ensures that the weight vector generated by BPtool after pruning a network can be used by the regular C code generated by Nettool.

Internal variables associated with the units are called ``nfields'', an acronym for ``neuron fields''. Internal variables associated with the connections are called ``sfields'', an acronym for ``synapse fields''.

Fields are highly specialized: in order to increase speed, many computational functions are working on implicit fields. In addition, all possible fields are not available on all versions of SN2.8. There is no reason indeed to slow down a regular network by updating the fields required by a shared weigths network.

Fields marked with a star (*) do not exist in sn28new and sn28itenew.Accessing these fields however is simulated by writing to a reading from the equivalent sfields s-epsilon and s-sigma of the incoming connections.

A group of functions have been defined for directly accessing most fields.

With kernels sn28ite and sn28itenew, this function sets the learning rate of all the connections to cell n and returns the mean of those learning rates.

This function is defined by kernels sn28new and sn28itenew only.

This function is defined by kernels sn28new and sn28itenew only.

This function is defined by kernels sn28new and sn28itenew only.

Kernels sn28ite and sn28itenew implement shared weight networks by setting up connections which physically share certain sfields: they point to the same piece of memory. A star (*) in the above table indicates those sfields which are shared when you set up a shared weight connection with function dup-connection .

This function is defined by kernels sn28new and sn28itenew only.

This function is defined by kernels sn28ite and sn28itenew only.

This function is defined by kernels sn28new and sn28itenew only.

This function is defined by kernels sn28new and sn28itenew only.

This function is defined by kernels sn28ite and sn28itenew only.

This function is defined by kernels sn28ite and sn28itenew only.

(set-nfield input n-val 0)
; clears all the states of layer input.

If argument l2 is not defined, list l1 is assumed. Execution is slightly faster in this case.

See: (nfield n f [ v ])

f1 = g2*f2 + g3*f3

f1 = f2*f3

Function mul-nfield returns the sum of all the computed terms i.e. the dot product of fields f2 and f3 .

f1 = g / f2

Function invert-nfield returns the sum of all the computed terms.

sf1 = g2 * sf2 + g3 * sf3

f(x) = 0

f(x) = A tanh (B x) + C x + D

f(x) = A B x + C x + D

f(x) = A g(Bx) + Cx + D

where

g(x) = -1  if x < -1
g(x) =  1  if x > +1
g(x) =  x  otherwise

f(x) = A g(Bx) + Cx + D

where

g(x) =  1  if x > 0
g(x) = -1  otherwise

f(x) = 0  if x < -1
f(x) = 0  if x > +1
f(x) = (1 - x**2)**3  otherwise

See: DZ

df(x) = AB (1 + tanh2 (Bx)) + C

df(x) = 0  if x < -1
df(x) = 0  if x > +1
df(x) = -6 x (1 - x**2)**2 otherwise

(nlf-df-all (nlf-spline X Y))

(nlf-ddf-all (nlf-spline X Y))

nsum(i) = Sum on j ( sval(j,i) nval(j) )

The optional argument theta specifies the standard deviation of a gaussian noise added to the weighted sum.

nval(i) = NLF [nsum(i)]

See: (update-weighted-sum [ theta ]... layers ...)
See: (update-state-only nlf ... layers ...)

Cost = 0.5 *  Sum on i (nval(output i) - nval(desired i))**2

with respect to the weighted sum and state of units in layer output:

- Gradient of Cost on nval(outputi)
  =  nbacksum(output i)  =  nval(desired i) - nval(output i)
- Gradient of Cost on nsum(output i))
  =  ngrad(output i)  =  nbacksum(output i)  DNLF[nsum(output i)]

In addition, kernels sn28new and sn28itenew set the second derivative to 1:

nsqbacksum(output i) = 1

The desired values for a network are usually stored in a pseudo layer desired. The function init-grad-lms compares these desired values with the states of the output layer output and initializes the computation of the gradients. Function init-grad-lms returns the value of the cost function.

nbacksum(output i)
  = 0 if nval(output i) > threshold and nval(desired i) > threshold,
  = 0 if nval(output i) < threshold and nval(desired i) < threshold,
  = nval(desired i) - nval(output i)  otherwise.
- Gradient of Cost on nsum(outputi)
  = ngrad(output i) = nbacksum(output i) * DNLF[nsum(output i)]

This cost function is useful for implementing Rosenblatt's perceptron like algorithms. These algorithms however are often unstable when the minimal cost is not zero.

- Gradient of Cost on nsum(i)
= ngrad(i) = nbacksum(i) * DNLF[nsum(i)]

The NLF object dnlf must be the derivative of the NLF object given as argument to the function update-state-only called on that layer.

sdelta(i, j) = alpha * sdelta(i, j) + nval(i) * ngrad(j) * nepsilon(j)
sval(i, j) = ( 1 - decay * nepsilon(j) ) * sval(i, j) + sdelta(i, j)

Kernels sn28new and sn28itenew use sepsilon(i,j) instead of nepsilon(j) .

When layers l are specified, the weight update is only performed on the incoming connections to layers l . Otherwise, the weight update happens in all the network.

When arguments alpha and/or decay are zero, a faster optimized code is used.

With kernels sn28ite and sn28itenew, the possible presence of shared weights makes the computation more complex and slower. Function update-weight then calls the functions clear-acc , update-acc , update-delta and update-wght-only described in the next section. Three differences with the non iterative version are then important:

• The execution is 40% slower,
  
• Argument decay is no longer scaled by the value of field n-epsilon:
  

sval(i, j) = (1 - decay) * sval(i, j) + sdelta(i, j)

• Restricting the update to specific layers with an the optional argument l has an ambiguous meaning. Selected connections may shared weights with unselected connections. Since the right computation is very slow, it is considerably more efficient to use directly clear-acc and update-acc as discussed in the sequencement section.
  

The sfield s-acc is also used for accumulating the contributions of several connections to the gradient of shared weights. In fact, shared weights and batch gradient use the same underlying mechanisms. Four functions then replace the usual call to update-weight :

sacc = 0

sacc(i, j) = sacc(i, j) + nval(i) * ngrad(j) * nepsilon(j)

Kernels sn28new and sn28itenew use sepsilon(i,j) instead of nepsilon(j) .

sdelta = alpha * sdelta + sacc

sval = (1 - decay) * sval + sdelta

The curvature is described by the matrix of the second derivatives of the cost, the Hessian matrix. Since this matrix has a huge number of coefficients, we only compute the diagonal coefficients.

The weight update equation then becomes

W =  W  -  Gradient of Cost on W * Inverse( Hessian of Cost on W )

This simple approach, however, is plagued by several problems:

• The cost actually is the average of the squared differences over the training set. We want however to adapt the weights after each pattern (online) rather than after each swep through the training set (batch).

  We add therefore two learning rates: sfield s-epsilon controls the update term, like the usual learning rate and gamma controls the exponential average of the second derivatives for each pattern. This exponential average is stored in the s-sigma fields.
  

• The second derivative may be null or even negative in certain area of the cost function. Therefore, a third parameter, mu , is introduced. The update equation becomes:
  

W =  W - sepsilon * Gradient of Cost on W / (mu + |ssigma|)

We have replaced our learning rate problem by three new parameters. These new parameters however are much more robust: we always use the same values although more refined values may prove more efficient:

sepsilon = 0.0005
gamma = mu = 0.05

This scheme is further complexified by two important additional factors:

• The contributions to the second derivatives of shared connections are accumulated in a new sfield s-hess .
  
• It is empirically more efficient to use the Levemberg-Marquard approximation of the second derivatives. This approximation ensures that the value of field s-sigma is greater or equal to zero!
  

These algorithms are implemented by kernels sn28new and sn28itenew. Two sets of new functions are required for computing the derivatives and for updating the weights.

For each unit i in layers:
  nsqbacksum(i) =  Sum on k ([sval(i, k)]**2 * nggrad(k))
  nggrad(i) = nbacksum(i) * DDNLF(nsum(i)) + nsqbacksum(i) * [ DNLF(nsum(i)) ]**2
  ssigma(j, i) = gamma * nggrad(i) * [nval(j)]**2 + (1 - gamma) * ssigma (j, i)

This computation requires that the fields n-ggrad of all downstream units are correct. All the fields n-ggrad and s-sigma thus are computed during a single backward reccurrence similar to the usual back-propagation.

Remark:

• If a unit has no downstream connection, field n-sqbacksum is left unchanged. Applying update-ggradient to the output layer thus leaves the value 1 stored by init-grad in that field and still computes fields n-ggrad and s-sigma .

  
  

The computation is similar to that of update-ggrad except that the nfield n-ggrad is computed according to the following formula:

nggrad(i) = nsqbacksum(i) * [DNLF(nsum(i))]2

shess = 0

For each connection:
shess = shess + ssigma

For all weights:
sacc = sacc / (mu + |shess|)

 (clear-acc)
 (clear-hess)
 (update-acc...layers...)
 (update-hess...layers...)
 (hessian-scale mu)
 (update-delta alpha)
 (update-wghtonly decay)

The performance remarks discussed with function update-weight function are even more valid. With kernel sn28itenew, function update-w-newton becomes very slow if you specify argument layers. It is much better to use the basic functions themselves.

See: (update-weight alpha decay [... l ...])

This state-of-the-art implementation of the conjugate gradient method rely on partial line searches based on the analytical computation of the cost function curvature. Such partial line searches are allowed by a self-restarting Hestenes-Stiefel formula for computing the conjugate directions.

For each cell i in layers layers, function update-deltastate updates the gradient field n-grad according to the following formula:

ngrad(i) = DNLF(nsum(i)) isu(j;; sdelta(j,i) nval(j) + sval(j,i) ngrad(j))

This function updates the weight vector according to the following formula:

sval = sval + f(sdelta . sacc;curvature) sdelta

f( Sum on i (wi xi) )

euclidian unit compute a function of the distance between the vector xi and the weights wi :

f( Sum on i (wi - xi)**2 )

Euclidian units are useful for implementing several algorithms, like Kohonen's map, Learning Vector Quantization, Radial Basis Functions, K means and Nearest Neighbour. This implementation in Netenv provides an alternative to KNNenv.

A few additional functions whose name contains "nn" implement these units in SN2.8. These functions just replace their standard counterparts for quasi-linear units. An euclidian unit in SN is the same object than a quasi-linear unit. The only difference is the use of a different set of functions for computing their states, gradients and weights.

nsum (i) =  Sum on j ( sval(j, i) - nval(j) )**2

Argument theta is the standard deviation of an optional gaussian noise added during the computation.

nbacksum(i) = 2 * Sum on k ( ngrad(k) * [nval(i) - sval(i,k)] )

Units in layers do not need to be actually euclidian units. Their downstream units however have to be euclidian units.

• Units in layers do not need to be euclidian units. Their downstream units however have to be euclidian units.
  
• Argument dnlf is the derivative of the NLF used by update-state-only for layers layers . It is not the derivative of the NLF associated to the euclidian units which actually do not belong to layers layers .

  
  

For each unit i in layers,
 For each unit  j connected to i:
   sdelta (i,j) = alpha * sdelta(j,i) + (1 - alpha) * ngrad(i) * (sval(j,i) - nval(j)) * nepsilon(i)
   sval(i,j) = (1 - nepsilon(i) * decay) * sval(j, i) + sdelta(j, i)

With kernels sn28ite or sn28itenew, this function has the same limitations as function update-weight . It is then better to use explicitely clear-acc , update-nn-acc , update-delta and update-wghtonly .

See: (update-weight alpha decay [... l ...])

sacc(j,i) = sacc(j, i) + ngrad(i) * ( sval(j, i) - nval(j) ) * nepsilon(i)

nsqbacksum(i) = 2 * Sum on k ( 2 * (sval(i, k) - nval(i))**2 * nggrad(k) - ngrad(k) )
nggrad(i) = nsqbacksum(i) * [DNLF(nsum(i))]2 - nbacksum(i) * DDNLF(nsum(i))

The sfields s-sigma of each connection are then updated:

ssigma(i,j) = (1 - gamma) * ssigma(i, j) 
    + gamma[ 2 * nggrad(j) * (sval(i, j) - nval(i))**2 - ngrad(j) ]

nsqbacksum(i) = 2 * Sum on k ( 2 * (sval(i, k) - nval(i))**2 * nggrad(k) - ngrad(k) )
nggrad(i) = nsqbacksum(i)  [DNLF(nsum(i))]**2 
ssigma(i,j) = (1 - gamma) * ssigma(i, j) + gamma[ 2 * nggrad(j) * (sval(i, j) - nval(i))**2 - ngrad(j) ]

 (clear-acc)
 (clear-hess)
 (update-nn-acc...layers...)
 (update-hess...layers...)
 (hessian-scale mu)
 (update-delta alpha)
 (update-wghtonly decay)

Like update-weight this function is much slower than calling the seven above functions.

See: (update-w-newton alpha decay mu [... layers ...])
See: (update-weight alpha decay [... l ...])

layer1  Input
layer2  Quasi Linear (nlf2, dnlf2, ddnlf2)
  
layer3  Euclidian    (nlf3, dnlf3, ddnlf3)
  
layer4  Quasi Linear (nlf4, dnlf4, ddnlf4)
  

Let us consider an additional layer <des> which contains desired states for
layer <layer4>.
The standard backpropagation algorithm thus is

;present a pattern into layer layer1
;forward-prop
 (update-state nlf2 layer2)
 (update-nn-state nlf3 layer3)
 (update-state nlf4)
; present a desired output into layer des
; backward-prop
 (init-grad-lms dnlf4 layer4 des)
 (update-gradient dnlf3 layer3)
 (update-nn-gradient dnlf2 layer2)
; update-weights
 (update-weight 0 0 layer4 layer2)
 (update-nn-weight 0 0 layer3)

With kernels sn28ite and sn28itenew, functions update-weight and update-nn-weight are very inefficient when they do not operate on the whole network. This is the case here. The last two lines thus are better replaced by:

; update-weights (sn28ite or sn28itenew)
 (clear-acc)
 (update-acc layer4 layer2)
 (update-nn-acc layer3)
 (update-delta 0)
 (update-wghtonly 0)

And finally, if we use the Levemberg-Marquardt algorithm, these lines become:

; second backward pass (newton)
 (update-lmggradient dnlf4 0.05 layer4)
 (update-lmggradient dnlf3 0.05 layer3)
 (update-nn-lmggradient dnlf2 0.05 layer2)
; weight update (newton)
 (clear-acc)
 (clear-hess)
 (update-acc layer4 layer2)
 (update-nn-acc layer3)
 (update-hess)
 (hessian-scale)
 (update-delta 0)
 (update-wghtonly 0)

• Functions for creating a network ( build-net , build-net-nobias ) and creating the internal data structures used by the library ( define-net ).
  
• Functions for initializing the weights ( forget-XXX ) and setting the parameters ( epsi-XXX , nlf-XXX )).
  
• Functions for managing a data base of examples, loading these examples into the network ( present-pattern , present-desired ) and accessing nfields layer per layer ( state , weighted-sum , etc...).
  
• Functions for running the simulation ( run , trun , learn ) and testing the performance ( perf ).

  These simulation functions are organised around a few key functions:
  

• Function test-pattern computes the output of the network for a specific pattern and sets a couple of global variables ( local-error , good-answer ) according to the result.
  
• Four functions implement the various training algorithms:

  - Function basic-iteration implements the `stochastic gradient back-propagation algorithm.

  - Function batch-iteration implements the batch gradient back-propagation algorithm.

  - Function basic-newton-iteration implements both Newton's and Levemberg-Marquardt's stochastic back-propagation algorithms.

  - Function batch-newton-iteration implements both Newton's and Levemberg-Marquardt's batch back-propagation algorithms.
  

These four functions in turn make heavy use of eight simpler functions:

 forward-prop  do-compute-err
 backward-prop  backward-2-prop
 do-update-weight  do-update-weight-newton
 do-update-acc  acc-update-hess

These functions are automatically defined when function define-net is called, either manually by the user or automatically during the network definition (the usual network definition function, build-net , calls this function).

By changing the definition of these functions, you can create new connectionist models. For instance, both the kMeans and LVQ algorithms have been implemented by redefining these functions (see file "sn28/examples/netlvq/lvq.sn" ).

• The first argument lunits defines the units. Argument lunits must be a list with the following structure:

  ( (name-1 num-1) (name-2 num-2).....(name-N num-N) )

  Each pair (name num) in list lunits defines a layer named name with num units. More precisely, build-net stores in symbol name the list returned by (newneurons num) .

  The order of the layers in argument lunits defines the order of the computation during the forward pass. The first pair thus is always the input layer and the last pair is always the output layer.

  Function build-net also creates an additional layer desired-layer which has the same size as the last layer in lunits . This additional layer is used to store the desired outputs of the network.
  

• The second argument llinks is a possibly empty list of the form:

  ( (from1 to1) (from2 to2)....(fromn ton) )

  For each pair (from to) , function build-net creates a full connection from layer from to layer to . This capability is handy for creating simple networks. Complex connection patterns however are usually created by calling directly the functions of library "connect.sn" once build-net has returned.

  In addition, function build-net always connects the bias unit (unit 0 ) to all units in all layers but the input layer.

  Function build-net generates several global variables. The global variables are assigned with functions make-desired , make-output and make-input . Finally, function build-net calls define-net for defining the eight forward and backward propagation functions.

  Example: Creating a ``4-2-4 decoder''.
  

(alloc-net 50 100)
= 4400
? (build-net
  ; create the cells
   '((input 4)    ; input layer
     (hidden 2)    ; a hidden layer
     (output 4) )  ; output layer
  ; make connections
   '((input hidden)  ; connect input  layer to the hidden layer
     (hidden output))) ; connect hidden layer to the output layer
= ( ( ) ( ) )
?  input
= ( 1 2 3 4 )
? input-layer
= ( 1 2 3 4 )
? hidden
= ( 5 6 )
? nnum  ; number of units
= 15
? snum  ; number of connections
= 22

A variable net-struc is set for backward compatibility purposes. It contains some additional information about the structure of the network.

Remark: A limited support is offered by build-net , build-net-nobias and define-net for defining specific activation functions per layer. This feature is described in section 3.5.3.

The default functions present-pattern and present-desired just store the n -th row of matrices pattern-matrix or desired-matrix into the input layer or into the desired layer.

You should call function ensemble instead of modifying directly these variables.

Examples in the training set are used for learning; examples in the test set are only used for evaluating the performance of the neural network on fresh data. This measure gives an idea of the generalization ability of the network.

Function test-set also prints a pessimistic evaluation of a 90% confidence interval on the accuracy of the performance measure. This evaluation is computed with function hoeffding according to the value of variable confidence.

The bounds of the test set are stored in the global variables tpatt-min and tpatt-max .

See: tpatt-min tpatt-max

The bound is computed according to Hoeffding's formula:

P ( | mean - 1/n * Sum on i (Xi) | ) > epsilon )
  <  2 * exp( -2 * n * e**2)  = 1 - eta 

This function is often redefined by the user. The default function just loads the n -th row of the 2-dimensionnal matrix pattern-matrix into layer l using function get-pattern and adds a gaussian noise of standard deviation input-noise .

This function is often redefined by the user. The default function just loads the n -th vector of the 2-dimensionnal matrix desired-matrix into layer l using function get-pattern .

Internal variables associated with the units are called ``nfields'', an acronym for ``neuron fields''. Internal variables associated with the connections are called ``sfields'', an acronym for ``synapse fields''.

Fields are highly specialized: in order to increase speed, many computational functions are working on implicit fields. In addition, all possible fields are not available on all versions of SN2.8. There is no reason indeed to slow down a regular network by updating the fields required by a shared weigths network.

• If list arg is the empty list, function f is called on all units.
  
• If list arg contains a single number, function f is called on all units, except the bias unit, with the single number as a second argument.
  
• If list arg contains a single list, function f is called on all units in this list.
  
• If list arg contains a list and a number, function f is called on all units in the list with the number as a second argument.
  
• If list arg contains two lists of equal sizes, function f is called with two arguments extracted from both lists.

  This clumsy function looks quite complicated but is not.

  If we define:
  

(de state l (mapncar 'nval l))
= state

we can easily set or get the state of a list of units:

? (setq foo (newneurons 4))  ; allocate 4 units
= ( 1 2 3 4 ) 
? (state)                      ; get their state (with the bias unit state)
= ( 1 0 0 0 0 ) 
? (state '(2 3) 0.5)           ; set the state of units 2 and 3 to 0.5
= 0.5 
? (state foo)                ; display the states of layer foo
= ( 1 0 0.5 0.5 0 ) 
? (state 0.33)                 ; set the state of all the units to 0.33
= 0.33 
? (state)                      ; display the network state
= ( 1 0.33 0.33 0.33 0.33 )

And in fact, the following functions are already defined:

effective_epsilon = epsilon / (mu + | sigma |)

where epsilon is the learning rate attached to the unit (or the weight) and sigma is the mean second derivative of the cost function with respect to the weights of the unit extracted from the sfield s-hess .

• Functions get-pattern and get-pattern-2 are used for loading the network input with new data.
  
• Functions matrix-to-weight and weight-to-matrix are used for saving or loading weights.

  
  

(dim my-patterns 10 5)     ; define a matrix with 10 vectors of dimension 5
(initialize-my-patterns)   ; put interesting data into the matrix
                           ; transfer elements (7 x) to units 1, 2, 3, 11, 12
(get-pattern my-pattern 7 '(1 2 3) '(11 12)) 

Matrix arr is transfered entirely into the units defined by l1 ... ln . It may have any number of dimensions. The sum of the length of the lists l1 ... ln should be equal to the number of elements in the matrix. This function is mainly designed for dealing with submatrices.

Function get-pattern-2 is slightly slower than get-pattern. However, it implements a more general way to transfer data from a matrix to a layer in the network.

; get pattern is now defined as 
(de get-pattern(arr n . l)
   (apply get-pattern-2 (cons (submatrix arr n ()) l)) )

; save the weight configuration into variable a-good-solution
? (setq a-good-solution (dump-weights))
=...

Caution: This function does not create connections but just sets the weight values. The network structure must be exactly identical to the structure used when dump-weights has been called.

; restore the weights stored in variable a-good-solution
? (rest-weights a-good-solution)
= ()

First, it is possible to store them under ascii or binary formats.

Secondly, it is possible to store them with (resp. without) information related to the neural net architecture, leading to a explanative (resp. blind) format.

Function load-net and merge-net recognize all these formats.

Function save-net stores the weights into file str with a binary explanative format. The first 12 bytes of the file are:

• A ``magic'' number on 4 bytes for identifying the file type.
  
• The number of units in list l , as a 4 bytes integer.
  
• The number of training iterations achieved so far, as a 4 bytes integer. There is then one record per connection. Each record is composed of 12 bytes.
  
• A 4 bytes integer gives the rank in list l of the upstream unit.
  
• A 4 bytes integer gives the rank in list l of the downstream unit.
  
• A 4 bytes floating point number gives the value of sfield s-val for this connection.

  
  

The first line of the file is a file header. It is composed of the letters ".WEI" , of the numbers of cells involved in the file and of the age of the network. Each connection is described on one line, consisting in the number of the upstream cell, the number of the downstream cell and the weight value.

Note that the celle 0 is always the threshold cell. After 160 iterations, a three layers XOR network might be saved as:

        .WEI    6 160
           0    3 -1.7688258
           0    4  2.2176821
           0    5  1.5779345
           1    3  2.3539493
           1    4  2.0589452
           2    3  0.8007546
           2    4  1.9538255
           3    5  1.5308707
           4    5 -1.4315528

In this file, unit 0 is the bias, units 1 and 2 are the input units, units 3 and 4 are the hidden units and unit 5 is the output unit.

When argument l is omitted, this function saves the weights in file str as a regular mono-dimensional TLisp matrix using function save-matrix . This matrix just contains the weights by their order of creation.

When argument l is omitted, this function saves the weights in file str using blind format which means a regular mono-dimensional TLisp matrix using function save-ascii-matrix . This matrix just contains the weights by their order of creation.

See: (merge-net str [ l ])

When argument l is omitted, function merge-net returns the number of units involved by the weight file str . When argument l is omitted and when str is not an explanative format weight file, function merge-net returns () . This feature is used for sensing the file format (explanative vs. blind).

Function merge-net is usually called with the same list of units than the corresponding call to save-net . Using a different list lets you load the weights of a network into a part of another network and vice-versa.

Function merge-net never creates a connection. Connections have to be created first.

;; The following function is defined in the file <"netenv.sn">. 
;; It loads a weight file created by save-net  without arguments 
 
(de load-net (str)
  (merge-net str (range 1 (1-num))) )

Function mask-epsi sets the learning rate for each unit to s divided by the square root of the number of incoming connections and by the square root of the sharing count of the incoming connections.

Most other learning parameters are constant for all the network. They are stored in a couple of global variables.

X(i) = f(A(i) + N(q))

Where A(i) is the total input to unit i and N(q) is a zero mean gaussian random variable with standard deviation q . This computation is disabled when theta is set to 0 . The simulation is slightly faster in this case. The default value is 0 . A non zero value can be used for:

• limiting the bandwidth of the units
  
• getting a more robust solution

  
  

increment W(i,j)  :=  increment W(i,j)  -  epsilon * gradient of Cost on W(i,j)

This computation is disabled when alpha is set to 0 . The simulation is slightly faster in this case. The default value is 0 .

Effective epsilon(i,j)  =  epsilon(i,j) / (mu + | sigma(i,j) |)

sigma(i) := (1 - gamma) * sigma(i) + gamma * (d2 E / d W 2) * X(i)**2
where X(i)**2  is the square of the state of unit i.

See: (update-state-only nlf ... layers ...)
See: (update-weight alpha decay [... l ...])
See: (update-ggradient dnlf ddnlf gamma ... layers ...)
See: (update-w-newton alpha decay mu [... layers ...])

Since NLF objects have a functional behavior, writing (nlf x) then returns the value of the default NLF in x , writing (dnlf x) returns the value of its derivative in x and writing (ddnlf x) returns the value of its second derivative in x .

f(x)  = scale * tanh( mx * x ) + offset + dmin * x

Since the hyperbolic tangent is an odd function taking values in range ]-1,1[, argument scale determines the amplitude, argument dmin the minimum value of the derivative (if dmin is not 0 the asymptotes are slanted), argument mx defines the gain or the inverse of the ``temperature'' and argument offset shifts the curve up or down.

Argument offset default to 0 . If argument scale is omitted, a scale is computed to ensure that f(1) = 1 and f(-1) = -1 .

The most common settings are the following:

• mx=0.5 , dmin=0 , scale=0.5 , offset=0.5 corresponds to the standard logistic function going from 0 to 1 , with f(0)=0.5 .
  
• mx=2/3 , dmin=0 gives a function which goes from -1.715905 to +1.715905 . This function fulfils the conditions f(1)=1 and f(-1)=-1 . Moreover, the maximum of the second derivative of this function is at 1 . This is the default activation function of SN2.8.
  

The default non linear function for SN2.8 and its derivatives:

(nlf-tanh 0.666 0)

Although this function is not differentiable, the computation of its derivative causes no problem except for two points. SN2.8 assumes that the derivative at these two boundaries are those of the central part. A typical piecewise non linear function: and its derivatives:

(nlf-lin  0.1  1.1)

f(x) = (1-x**2)**3  if -1<x<1
f(x) = 0  otherwise

? (nlf-tanh 0.6666 0)                  ; sets a standard function
= 1.14393
? (let* ((x (range -4 4 0.05))
         (y (all ((i x)) (nlf i))))
    (nlf-spline x y) )                 ; copy it with a cubic spline
= 400

(build-net
 '((input 4)
   (hidden 2)
   (output 4 (nlf-lin 1 1)))
 '((input hidden)
   (hidden output)) )

In that example, layer hidden uses the default NLF, defined by variables nlf , dnlf , ddnlf . This default NLF can be changed by subsequent calls to a function nlf-XXX . Layer output however uses a fixed linear function, defined by (nlf-lin 1 1) . Subsequent use of functions nlf-XXX will never affect the activation function of layer output.

Function test-pattern is defined as follows:

(de test-pattern (n)
 ; transfers pattern n into the input-layer
 (present-pattern input-layer n)
 ; propagates the states forward: computes the outputs
 (forward-prop net-struc)
 
 ; gets the desired outputs
 (present-desired desired-layer n)
 
 ; computes the error for this pattern,
 ; and stores it into local-error
 (do-compute-err output-layer desired-layer)
 
 ; decides if the result in good-answer
 ; and stores the result in good-answer
 (setq good-answer (classify n)
 
 ; call displaying functions
 (process-pending-events)
 (disp-perf-iteration n)
 
 ; returns the error
 local-error)

Functions forward-prop and do-compute-error are defined at network creation time by the function define-net . These functions are described in the next section.

Function classify is defined by the set-class-XXX functions. It defines how to measure the quality of the systems. The default classify function returns t if the outputs and the desired outputs have the same sign. This is too restrictive for most applications: you should always call one of the set-class-XXX functions in order to define your classification criterion.

Function perf prints the average error over the examples and over the outputs as well as the percentage of good recognition (as defined by the function classify). This function also calls the function save-perf wich saves the result of the measure in the file whose name is contained in the global variable perf-file .

This function computes the average error over patterns between n1 and n2 and stores the value into global variable global-error . It also computes the percentage of good answers and stores it into global variable good-an-percent . The value of global-error is returned.

This file is used for storing information about the main neural function calls and results.

• It logs all calls to functions perf .
  
• Furthermore it records explanatory comments lines (prefixed with a double semi-column ";;" ) when initialization or learning step control occurs.

  The following functions use perf-file :
  

perf,
epsi-layer, epsi-sqrt-layer, forget-layer, forget-sqrt-layer,
epsi, epsi-sqrt, mu, forget, forget-sqrt, ensemble, test-set.

Example:

;;; ============ NEW PERFORMANCE
;;; ============ NEW NETWORK
;;; (forget-sqrt 1)
;;; (epsi-sqrt 0.1)
;;; (ensemble 0 319)
;;; (test-set 320 479)
;;; trun-action using function 'learn'...
     0 0.53568070   9.69 ;{0-319}
     0 0.53332978  10.00 ;{320-479}
   320 0.07470918  93.75 ;{0-319}
   320 0.08033946  91.25 ;{320-479}
   640 0.05402946  97.50 ;{0-319}
   640 0.06082892  95.62 ;{320-479}
   960 0.04978141  98.75 ;{0-319}
   960 0.05853230  96.88 ;{320-479}
  1280 0.04746896  99.06 ;{0-319}
  1280 0.05897417  97.50 ;{320-479}
;;; (epsi-sqrt 0.05)
  1600 0.03919556 100.00 ;{0-319}
  1600 0.05168496  97.50 ;{320-479}
  1920 0.03774762 100.00 ;{0-319}
  1920 0.05145141  97.50 ;{320-479}
;;; (epsi-sqrt 0.025)
  2240 0.03479438 100.00 ;{0-319}
  2240 0.04916938  98.12 ;{320-479}
  2560 0.03418482 100.00 ;{0-319}
  2560 0.04903031  98.12 ;{320-479}
  2880 0.03369421 100.00 ;{0-319}
  2880 0.04896870  98.12 ;{320-479}
  3200 0.03326655 100.00 ;{0-319}
  3200 0.04892774  98.12 ;{320-479}
;;; break

• If variable perf-file contains the empty list or an empty string, function save-perf does nothing.
  
• If variable perf-file contains the name of an existing file, the values of n1 , n2 , etc... are appended on a single line at the end of the file.
  

Function save-perf is called by function perf .

The output is considered correct if all the states of the units in output-layer are in the same region as the states of units in desired-layer . This lisp function is slower than class-quadrant . Argument pn is not used.

Learning functions learn and run use the function next-pattern to choose the next pattern to present to the network. The user can redefine next-pattern to fit his needs or select a predefined method.

The same pattern is kept until the network returns the right answer (until function classify returns a positive result). The next pattern is then chosen using function next-choice .

This function is used by the conjugate gradient algorithms.

(de do-update-weight (ns)
 (clear-acc)
 (do-update-acc ns)
 (update-delta alpha)
 (update-wght only decay))
 (do-update-acc ns)

With kernels sn28new and sn28itenew only, this function loops over the connections and adds their contribution to the gradient in the field s-acc of the weights. Usually defined as a single call to update-acc , this function may be redefined as a sequence of calls to update-acc .

With kernel sn28itenew however, this function might be redefined as:

(de do-update-weight (ns)
 (clear-acc)
 (clear-hess)
 (do-update-acc ns)
 (do-update-hess ns)
 (hessian-scale mu)
 (update-delta alpha)
 (update-wghtonly decay))
 (do-update-hess ns)

Accumulates the contributions of each connection to the second derivatives (in fields s-sigma ) into the field s-hess of the weights. This is usually done by a single call to function update-hess .

C =  0.5 * | DesiredOutput - ActualOutput |**2

Function init-grad then calls init-grad-lms .

If the actual output is larger than the desired output and the desired output is larger than threshold or if the actual output is smaller than the desired output and the desired output is smaller than threshold then the error is set to 0 and no gradient is propagated. Otherwise the error is the classical mean-square error.

Function init-grad then calls init-grad-thlms.

The gradients then are propagated backward using function backward-prop and the weights are updated with function do-update-weight .

Function basic-iteration also is in charge of incrementing the variable age and calling a few display functions. It is defined as follows:

(de basic-iteration (n)
 ; get a pattern 
 (present-pattern input-layer n)
 ; propagates the states
 (forward-prop net-struc)
 ; get the desired output
 (present-desired desired-layer n)
 ; computes local-error and initializes the gradients
 (do-compute-err output-layer desired-layer)
 ; propagates the gradients
 (backward-prop net-struc)
 ; update the weights
 (do-update-weight net-struc)
 ; increments "age"
 (incr age)
 ; computes "good answer"
 (setq good-answer (classify n))
 ; display functions
 (process-pending-events)
 (disp-basic-iteration)
  local-error)

The following learning function provides convenient interfaces for calling function basic-iteration .

• When flag lev-mar is the empty list, the Quasi Newton algorithm is applied.
  
• When flag lev-mar is t , the Quasi Levemberg Marquardt algorithm is applied.

  
  

The batch version accumulates the gradients over a set of patterns before performing a weight update. The batch version is much slower than the on-line version (it usually takes much more forward and backward passes to learn a given problem). It is sometimes useful for analyzing the error surface and the behavior of an algorithm.

Function run-batch performs cyc-length batch learning iterations. Each iteration consist in a sweep through the entire training set (i.e. patterns between patt-min and patt-max ). Then it measures the performance on this training set. This entire sequence is repeated cyc-num times.

Function trun-batch performs cyc-length batch learning iterations. Each iteration consist in a sweep through the entire training set (i.e. patterns between patt-min and patt-max ). Then it measures the performance on this training set and on the test set (i.e. patterns between tpatt-min and tpatt-max ). This entire sequence is repeated cyc-num times.

Function batch-iteration accumulates the gradients and the errors of the training set patterns and then performs a weight update. The global variable global-error is set to the average output error over the set of patterns and its value is returned. The global variable age is incremented after each pattern presentation in order to remain consistent with the stochastic version.

The optional arguments grad1 , grad2 and gradc are three matrices used for continuing the conjugate gradient algorithm without loosing the information accumulated during the previous epochs. Function learn-batch-cg indeed returns a list (it grad1 grad2 gradc) that you can use as argument list for the next call to function learn-batch-cg .

;; WRONG: Restarts the conjugate gradient every other epoch!
(repeat n 
 (learn-batch-cg 2) 
 (perf) ) 
;; RIGHT: Runs the conjugate gradient during 2*n epochs!
(let ((cgs (list 2)))
 (repeat n
 (setq cgs (apply learn-batch-cg cgs))
 (perf) ) )

• Performing cyc-length epochs of batch conjugate gradient. Each epoch consist in a sweep through the entire training set,
  
• Measuring and displaying the performance on the training set.
  

This function is simular to function run usually used for the online gradient algorithm.

• Performing cyc-length epochs of batch conjugate gradient. Each epoch consist in a sweep through the entire training set,
  
• Measuring and displaying the performance on the training set,
  
• Measuring and displaying the performance on the test set.
  

This function is similar to function trun used for the online gradient algorithm.

This method sometimes improves the generalization performances by adjusting the effective number of parameters. The simulation speed is also improved. The optimised runtime speed however may remain identical because modern computers spend more time to perform a test than to perform a multiplication by zero.

Using pruning methods is quite a long process because the mathematics of these method usually require that the weight saliencies are computed on an optimum of the cost function. The best results are obtained after training the network during a long time before applying OBD.

Two library functions of "netenv.sn" implement OBD. These functions require kernels sn28new or sn28itenew because the computation of the saliency criterion requires the evaluation of the second derivatives of the cost function.

A typical use of OBD consists in the following steps:

• 1 Training thoroughly a network until it reaches an optimum.
  
• 2 Computing the curvature information using obd-compute-curvature .
  
• 3 Pruning a small proportion of weights (such as 10%) using obd-prune .
  
• 4 Retraining the network until it reaches a new minimum.
  
• 5 Repeating steps 2, 3 and 4 until enough weights have been removed.
  

An example file is provided in directory "sn28/examples/obd.sn" .

When pruning a shared weight network (using sn28itenew) you must remember that the saliency criterion is a characteristic of the weight and not of the connection. All the connections sharing a same weight will be either kept or removed as a whole.

See: (basic-iteration n )

See: (test-pattern n )

(de disp-everything (patt-num)
  (print-layer output-layer)
  (printf "age=%d  pattern=%d  error=%9.5f  %s\n" 
      age patt-num local-error 
      (if good-answer "  ok  " "**arrgh**"))
  (plot-error age local-error)
  (draw-net net-struc patt-num) )

(de disp-error (patt-num)
  (plot-error age local-error) )

(de disp-net (patt-num)
  (draw-net net-struc patt-num) )

(de disp-text (patt-num)
  (printf "age=%d  pattern=%d  error=%9.5f  %s\n" 
     age patt-num local-error 
     (if good-answer "  ok  " "**arrgh**")) )

Function draw-net is called by certain functions disp-XXX . The first argument is now obsolete. The second argument is the index of the current pattern.

It is easy to write a display function using function draw-list . Here is an example of how to define a draw-net function for a small network:

; Define the network as in the "getting started" 4-2-4 encoder example
; then define a display function for the 4-2-4 encoder
(de draw-net (a b)       ; a and b are dummy arguments, we won't use them
  (draw-list 100 100 (state output) 4 30 50 48)
  (draw-list 150 170 (state hid)    2 30 50 48)
  (draw-list 100 240 (state input)  4 30 50 48) )
(new-window)            ; now open a graphic window
(set-disp-net)          ; enable display
(learn 16)              ; do a couple iterations, look at the graphic screen.

The network editor ``Nettool'' generates automatically draw-net functions. These functions create an interactive display window. By clicking outside the network, the window is refreshed and displays the states of the network. By clicking on a neuron, the weights of the connections leading to the target neuron are displayed and the other neurons are turned to color gray (corresponding to weight 0).

As soon as these plot ports exist, functions run and trun plot an error curve in this window.

As soon as these plot ports are defined, functions run and trun plot a performance curve in this window.

Remark: You can destroy these error and performance windows, either by using your window manager or by typing:

(delete perf-plotting-window)
(delete error-plotting-window)

Plotting is then cancelled.

These functions have been designed for both efficiency and simplicity. They only use the two low-level functions connect and dup-connection .

See: Connections in Netenv.

Function local-1d-connect complains if n1 != step*n2+size-step .

(r1c1 r1c2 r1c3...r1cN r2c1 r2c2...rMcN).

Cells of layer layer2 are connected to a window of xsize*ysize cells from layer1 , stepped by xstep and ystep cells. Function local-2d-connect complains if row1 != ystep*row2+ysize-ystep or col1 != xstep*col2+xsize-xstep .

However, side effects are handled in a different way: when using the function local-2d-connect , cells in the periphery of layer1 have less downstream connections than the central cells. Input data thus are best located in the center of the input layer.

In certain cases, it is easier to use the function local-toric-connect whose sliding window wraps around layer1 .

Layer layer1 always has col1*xstep columns and row1*ystep rows. The input window of the rightmost cells of layer2 wraps thus on the left of layer1 . The input window of the bottommost cells similarly wraps on the top.

However, side effects are handled like function local-toric-connect .

Example:

; divide the resolution of a 16x16 layer 'layer1' by 2 into a 8x8 layer 'layer2'
(equal-mask-connect layer1 16 16 
     layer2  8  8
     2  2  2  2 )

(t1f1 t2f1 t3f1...t1f2 t2f2...).

Each frame of layer2 is connected to a sliding window in layer1 of size frames, stepped by step frames, always using the same set of weights.

Function mask-2d-reverse complains if row2 != ystep*row1+ysize-ystep or col2 != xstep*col1+xsize-xstep .

Function build-net , however, always creates a separate bias per unit. The solution consist in using function build-net-nobias instead of function build-net . It uses the same arguments, but no biases are created.

The following functions let the user create biases by hand.

• NetTool is a graphical network editor. Using NetTool you can build a network using all the connection patterns defined in the library "connect.sn" . The program then generates lisp functions for creating and drawing the network.
  
• BPTool is a graphical interface for controlling the training process. BPTool allows you to perform the most common actions allowed by the library "netenv.sn" . It provides sensible default values for all the convergence parameters.
  
• A few other tools ``StatTool``, ``NormTool`` and ``ImageTool`` are useful for browsing and preprocessing the data. These tools are briefly discussed in the last chapter.
  

This manual describes all the functionalities of these tools.

The NetTool window appears when function NetTool is runned. If a function NetTool-net is defined, the corresponding network is loaded into the editor.

The NetTool window includes a menu bar and a workspace controlled by two scrollbars.

Using NetTool you can build a network using all the connection patterns defined in the library "connect.sn" . NetTool deals with two kind of objects: layers and layer-to-layer connections.

• Layers are just a collection of units or neurons. Both 1 dimensional and 2 dimensional layers are supported. As a convention, the leftmost layer is always the network input layer. Similarly, the rightmost layer is always the output layer.
  
• Layer-to-layer connections define the connection pattern between the units of two layers. These connection patterns are defined using the library "connect.sn" . Connections always link layers from left to right. It is thus not possible to edit recurrent networks with NetTool.
  

The NetTool editor generates three lisp functions:

• Function create-net creates a network with the specified architecture. This function calls functions defined in the libraries "netenv.sn" and "connect.sn" .
  
• Function draw-net opens a graphic window and displays the activation state of each layer. This function is used in conjunction with the set-disp-net mode of library Netenv.
  
• Function NetTool-net encodes the structure of the network for the next runs of NetTool. You should not call this function yourself.
  

These functions can be used by the interpreter or saved into a file.

Item "New Network" clears the workspace and initializes the network editor. If the current network has not been saved into a file, a confirmation dialog is displayed.

Item "Load Network" pops up a dialog for loading a network description file. A mouse click on button "Load Network" of the requester reads the specified network description file. A network description file is just a file containing three SN functions: NetTool-net , create-net and draw-net . NetTool extracts the network architecture from the first function, NetTool-net .

Item "Save or Create Network" pops up a dialog for saving a network description file or creating a network.

• The network description may be either saved into a file or used for creating a network.

  When item "Create network in Memory" is selected, the network editor creates the description functions in the interpreter memory. Then it calls function create-net in order to actually create the network.

  When item "Save network in File" is selected, the network editor saves the description functions in the specified file.
  

• Network descriptions are composed of three functions.

  Function NetTool-net encodes the structure of the network for subsequent runs of the network editor. It is always generated.

  Function create-net builds the network using the Netenv functions. It is generated only When item Generate create-net is checked.

  Function draw-net draws the network in connection with the Netenv library. It is generated only When item Generate draw-net is checked.
  

• Function create-net usually allocates memory for the exact number of units or connections required by the network. The user can however ask for additionnal space using the fields "Allocate more cells" and "Allocate more connections" .
  
• A mouse click on button "Compile & Save" creates a network description file.
  

Item "Quit" closes the network editor window. If unsaved changes have been performed on the current network, a confirmation dialog is displayed.

• Item "New" . Selecting this item or typing n in the workspace pops up a dialog for creating a new layer with various characteristics.
  
• Item "Edit" . Selecting this item or typing e in the workspace pops up a dialog for modifying the characteristics of the selected layer.
  
• Item "Duplicate" Selecting this item or typing d in the workspace make copies of the selected layers and of their connections.
  
• Item "Delete" Deletes the selected layers.
  

Several parameters can be specified.

The name of the layer is controled by an editstring. A default unique name is provided.

The size of the layer is controled by an editstring.

• A single number defines a 1-dimensional layer. For instance a size of "40" indicates a one dimension layer with 40 units.
  
• Two numbers define a 2-dimensional layer. For instance a size of "6x10" defines a matrix of 160 units with 16 rows and 10 columns.
  

The biases of the units in the layer is controled by four exclusive buttons.

• Item "None" specifies that no units in this layer are connected to the bias unit.
  
• Item "Full" Every unit is connected to the bias unit with a different weight.
  
• Item "Shared" specifies that every unit is connected to the bias unit with the same weight. This function generates a network with shared weight. Only the kernels sn28ite and sn28itenew can simulate such networks.
  
• Item "Column shared" specifies that every unit in a column of the layer is connected to the bias unit with the same weight. There is a different weight per column. This option is useful for defining time-delay networks. This function generates a network with shared weight. Only the kernels sn28ite and sn28itenew can simulate such networks.
  

The display mode of the layer is controled by three exclusive buttons This mode is only useful for generating function draw-net.

• Item "None" specifies that function draw-net will not display this layer.
  
• Item "Hinton" specifies that function draw-net will display this layer as small squares using function draw-list . This display style has been popularized by several famous papers written by Geoffrey Hinton.
  
• Item "Gray Levels" specifies that function draw-net will display this layer as gray levels using function gray-draw-list .
  

The new layer appears in the workspace when you click button "Ok" .

Four zones can be activated:

• The layer label is a selection zone. A first click in the layer label selects the layer. A second click on the layer label calls the layer edition popup requester. A click outside a layer deselects all layers. The label of selected layers is rendered in white on a black background.
  
• The gray triangle on the right side of the drag area is a connection handle. It is used for creating a new connection.
  
• You can resize a layer using the resize box. The form factor of the layer is constrained by the size and the display mode of the layer.
  
• You can press the mouse on the gray rectangle and move the layer around. You will notice that an invisible grid helps you to align the layers nicely.

  
  

This dialog can be poped up and handle a given layer when the layer to edit has been selected by clicking on its label and either item Edit of menu Layer is called or the label is clicked on again.

See: Creating Layers in NetTool.

The name of the new layers is derived from the name of the original layers. The parameters of the new layer are the same as the original layers. New connections are created similarily with the connections with the original layers.

After the initial click in the connection handle, while the mouse is moved but its button is not released yet, a dashed line is drawn between the connection handle and the mouse position.

When the mouse button is released on the downstream layer, the connection edition dialog appears and lets you specify a pattern of connections among those offered by the functions of library "connect.sn" .

Menu Full / Local proposes full and local connection patterns.