[under construction]

One of Lush's most interesting features is its Lisp-to-C compiler and the ability to freely intermix Lisp and C code in a single function. Unlike many other interpreted/compiled languages, and contrary to many well established principles of language design, the two dialects that the Lush interpreter and the Lush compiler implement are two very different languages, though they share the same syntax. The compiled dialect is strongly typed, lexically bound, and has no garbage collector, while the interpreted dialect has loose typing, is dynamically bound, and is garbage collected.

In fact, the Lush compiler is not designed to replace the interpreter, but rather to complement it. Lush applications are generally a combination of compiled and interpreted code. The compiled part is often itself a combination of Lush and C code. The compiled code will generally contain the "expensive" and heavily numerical parts of an application, where performance is the main requirement, while the interpreted part will contain the high-level logic, the user interface, the memory management, etc, where flexibility is more important than pure performance.

Numerous examples of compiled functions with and without in-line C-code are available in the libraries (in directories lsh and packages).

Users should just follow the following four rules:

• Every Lush file should use the function libload to load all required files (i.e. files defining functions or classes required by this lush file).
  
• The compilation of Lush function and classes should be performed by inserting one or several calls to dhc-make at the end of the lush file that defines those functions and classes.
  
• Users should identify the main Lush file of the project, that is to say the file that directly or indirectly loads all the Lush files of your project. This is usually the file with the higher level functions. It is sometimes convenient to create a Lush file for the sole purpose of being the main Lush file.
  
• When you want to load the most recent version of all your project files, you just need to load the main Lush file using either libload or the macro-character |^L| . This will load all modified files, reload all files that depends on modified files, and recompile all functions and classes that need recompilation, either because they have been modified, or because they depend on modified classes or functions.
  

Function dhc-make controls the compilation of classes and functions specified by the fspecs arguments.

Although this function can be used from the command line, it mostly is used within a Lush file to control the compilation of classes and functions defined in this file. We will assume now that dhc-make is invoked from a Lush file.

The first argument fname is either the empty list or a string representing filenames of the C and object files output by the Lush compiler.

• Passing the empty list instructs dhc-make to generate a suitable filename derived from the name of the Lush file containing the dhc-make command. Generated C files will be created in a subdirectory "C" of the directory contaning the Lush file. Generated object files will be created in a subdirectory of "C" whose platform dependent name is provided by expression (getconf "host") .
  
• Passing a string specifies a particular base name. Function dhc-make adds the suitable suffixes for C files or object files, replacing any other suffix possibly present in fname . Argument fname may specify a directory for both the C and object files. Otherwise the C and object files will be generated as explained above.
  

        (dhc-make () 
                  foo 
                  (rectangle rectangle area)
                  bar )

Before starting the compilation, function dhc-make tests whether the target files already exist and are up-to-date:

The C file will be generated if any of the following conditions is true.

• no such C file exists,
  
• the Lush file is more recent than the existing C file,
  
• the Lush file depends on another Lush file which is more recent than the existing C file. These dependencies are those collected by function libload .
  

The object file will be generated if any of the following conditions is true:

• no such object file exists,
  
• the C file is more recent than the existing object file, as happen, for instance, when the C file was regenerated during this invocation of dhc-make .
  

Finally dhc-make performs a mod-load of the object file and checks that the compiled functions are now executable. It returns the name of the object file.

Note: Function dhc-make can also be invoked from the command line. When this is the case, it always generates fresh C and object files, and derives their names from the last fspec argument.

• Argument libs may be a list of library names. Function dhc-make-with-libs works like dhc-make but also loads all the specified libraries after loading the compiled functions, and before checking whether the compiled functions are executable.
  
• Argument libs may be the symbol t . Function dhc-make-with-libs then works like dhc-make but does not cause an error if the compiled functions are not executable because they contain unresolved references. The caller is expected to load additional modules using mod-load in order to satisfy these unresolved references.
  

This function is similar to dhc-make-with-libs but uses the C++ compiler instead of the C compiler to generate the object file. This makes no difference except that one can use C++ constructs when inlining C code with cinline . This is particularly handy when interfacing external C++ libraries.

This is the elementary function called by dhc-make , dhc-make-with-libs and dhc-make-with-c++ .

Argument fspeclist is a list containing the fspec1 ... fspecN arguments of function dhc-make . Argument fname is similar to the fname argument of functions dhc-make or dhc-make-with-libs . Argument libs is similar to the libs argument of function dhc-make-with-libs .

The following example, for instance, forces the compilation of functions foo and bar .

(let ((dhc-make-force t))
   (dhc-make () (foo bar)) )

 (let ((dhc-make-lushflags (concat dhc-make-lushflags " -w -DMMX")))
    (dhc-make () (foo bar)) )

The initial value is derived from the Makefile variable LUSHFLAGS . It might reference other predefined varibales prefixed with character $ . These variables are expanded using dhc-substitute-env .

• $LUSHFLAGS Compilation flags defined by dhc-make-lushflags or by the optional argument of dhc-make-o .
  
• $INCS Flags specifying all the include directories specified by variable c-include-path . With this option, the C compiler can locate all include files that can be located using function find-c-include .
  
• $SRC The pathname of the source file.
  
• $OBJ The pathname of the object file.
  

The following example, for instance, invokes the C++ compiler instead of the C compiler:

(let ((dhc-make-overrides 
         (cons (cons "CC" (dhc-substitute-env "$CXX")) dhc-make-overrides) ))
  (dhc-make () (foo bar)) )

This function is useful to avoid processing long files when we are only interested in precompiled functions. See "packages/lapack/lapack-s.lsh" for an example.

The design philosophy of the Lush compiler is somewhat unusual, and, admitedly, in stark violation of many commonly accepted principles of good programming language design. First, The Lush compiler is not designed to replace the interpreter, but to complement it. Lush applications are often a combination of compiled code for things where performance takes precedence over flexibility (e.g. "expensive" and heavily numerical functions), and interpreted code for things where flexibility takes precedence over performance (high-level logic, user interface, memory management).

Therefore, the Lush compiler is designed to generate very efficient code (practically as good as its C equivalent), but has limitations as to what it can and cannot compile. What that means for the programmer is that, although the compiled and interpreted Lush dialects use the same syntax, they are really two different languages. From now on, the compiled Lush dialect will be called CLush, while the interpreted Lush dialect will be called simply Lush.

This deliberate difference between the two dialects allows all kinds of really juicy hacks (with no performance penalty), that would be difficult to achieve if the two dialects were identical by design. Within a single program you can simultaneously have in-line C code in the compiled part, run-time construction of code in the interpreted part (including run-time self modifying expressions like DMD splicing macros) with no hidden performance penalty.

The main differences between Lush and CLush are as follows:

• CLush is strongly typed (the type of all the variables must be declared, and is immutable), while Lush is not typed (e.g. a single variable can be assigned values of different types at different times in the same program).
  
• CLush uses lexical scoping, while Lush uses dynamic variable binding (if you don't know the difference, don't worry).
  
• The CLush compiler refuses to compile code that dynamically allocates objects that would require garbage collection. This allows the compiler to allocate all temporaries, local variables, and return values on the stack. While this provides clear performance advantages, it also puts strong limitations on what can and cannot be compiled (in particular most list manipulation functions are not usable in CLush)
  
• Dynamic allocation can be done is CLush, but the memory management must be done "by hand", and allocated objects must be deallocated explicitely, as with C. A pool class is provided in libstd/dyamic.lsh to facilitate dynamic allocation/deallocation of objects in compiled code. A good example of its use is the graph library in lsh/libgraph .
  

• 
  

There is a simple test to determine if a particular function is compilable or not: (compilablep func ). This may returns 'yes , 'maybe or () . The value 'maybe is returned when the argument is a macro whose compilability cannot be determined without expansion.

Basically the kinds of things that can be compiled by the CLush compiler are the kinds of things that you expect to be compilable in a "conventional" language like C. All the really "Lispish" stuff like eval , lambda , etc is not compilable (those things don't exist in C).

There are three types of things that cannot be compiled:

• Things that construct functions and code at run time (e.g. lambda , mlambda ), or things that evaluate dynamically generated code (e.g. eval ).
  
• Things that allocate dynamic structures that would require garbage collection
  
• Things that call non-compilable functions.
  

  ((-gptr- "int *") integer-pointer)
  ((-gptr- "FILE *") file-pointer)
  (cpheader "<my_header_file.h>")
  ((-gptr- "MyObjectType *") myobject-pointer)

Argument spec may also be a list containing a Lisp class name. This will define the type of the variable as a pointer to the C version of that class. Example:

  ((-gptr- (myclass)) myclass-pointer)

The C type of myclass-pointer will be "struct CClass_myclass *" .

• The types of all the slots of a compiled class must be declared.
  
• It is preferrable to include the constructor in the list of methods being compiled.
  
• The constructor of a compiled class must set initialize all the slots with an object of the type declared in the class definition.
  
• The parent class of a compiled class must be compiled prior to the class
  
• Methods defined in a subclass that overload a method of the parent class must have the same prototype as the method they overload (i.e. the same arguments with the same type, the same return type, and the same temporary variables).
  

Here is an example of compiled class declaration syntax (taken from packages/gblearning/modules.lsh ):

(defclass mle-cost sn-module
  ((-obj- (logadd-layer)) logadder)
  ((-obj- (idx1-ddstate)) logadded-dist)
  ((-idx1- (-int-)) label2classindex))

Compiling a class with dhc-make and its variants is done by passing a list whose first element is the class name and the remaining elements are the methods to be compiled. Here is an example:

  (dhc-make ()
    (sn-module)
    (mle-cost mle-cost fprop bprop bbprop))

In interpreted mode to-gptr simply returns a gptr containing the address the C object associated with the object passed as argument (which can be an index, storage, string, a compiled function, or any object).

This function may not be used to convert an integer (or other numerical types) to a gptr, but merely to obtain the address of an object.

In compiled mode, it is useful to know that a gptr does not carry enough information to allow the compiler to determine when the corresponding object must be released. The following function is a sure looser:

   (de never-do-this()
       (gptr (new myclass)) )

The compiler cannot understand that the object created by new is returned as a Gptr and may be used outside this function. The object is freed before returning, making the gptr useless. To prevent this, the object should be allocated in a pool using in-pool from lsh/libstd/dynamic.lsh .

• When argument arg is an object, this function checks that the object class is a subclass of class and returns the unmodified object.
  
• When this function is called in interpreted mode with a gptr argument, the lisp-to-c interface retrieves and returns the lisp object associated with the C object pointed to by the GPTR. It also checks that the object class is indeed a subclass of class .
  
• When this function is called in compiled mode with a GPTR argument, the compiler converts the GPTR into a pointer to an object of the specified class. Class membership is checked at run time.
  

Any C code can be written within a hash-brace construct. Lisp expressions and variables can be evaluated and refered to in the Lisp code by preceding them with a dollar sign.

• $legal-c-identifier refers to the lisp variable of the same name
  
• ${legal-lisp-variable} refers to the lisp variable (the braces are required if the lisp variable name is not a legal C name, e.g. if it contains dashes).
  
• $(lisp expression) refers to the result returned be the lisp expression.
  

Examples:

 [under construction]
 [insert juicy hash-brace examples here]

Example:

 (de add-to-integer( intg arg )
   ((-int-) intg arg)
   #{ $intg += $arg; #}
   intg )

expands to:

 (de add-to-integer (intg arg)
   ((-int-) intg arg)
   (cinline " %s += %s; " intg arg)
   intg )

See: (cinline format [ arg1 [ arg2 ... [ argn ]]])

  (cpheader "#include <stdio.h>")

The lines are inserted at the beginning of the C file, before all the Lush header files.

  ;; set element 2 of v to x.
  (de choucroute (v x) 
    ((-idx1- (-int-)) v) 
    ((-double-) x)
    (cinline "(IDX_PTR(%s,int))[2] = (int)(%s);" v x) ())

    extern_c char
    C_choucroute (struct idx *L1_v, real L1_x)
    {
      TRACE_PUSH ("C_choucroute");
      {
        (IDX_PTR (L1_v, int))[2] = (int) (L1_x);
        TRACE_POP ("C_choucroute");
        return 0;
      }
    }

This variable is initialized by the Lush startup code and contains a list of directories containing potential include files. Include files along this path can be searched using function find-c-include .

Furthermore, the compilation command, defined by variable dhc-make-command , typically uses the macro "$INCS" to specify all these directories to the compiler. Appending directories to c-include-path is then the easiest way to handle additional directories for include files.

This variable is initialized by the Lush startup code and contains a list of directories containing potential shared libraries. Function find-shared-library should be used to locate libraries located along this path, or in standard system locations.

This variable is initialized by the Lush startup code and contains a list of directories containing potential shared libraries. Function find-static-library should be used to locate libraries located along this path.

? (find-c-include "stdio.h")
= "/usr/include/stdio.h"

  (libload "libc/constants")
  (defconstant qqq (sprintf "#include \"%s\"" (find-c-include "stdio.h")))
  = "qqq"
  ? (de asd () (cpheader @qqq) #{ printf("asd\n") #} ())
  = asd
  ? (dhc-make "junk" asd)

 (de blah (x) ((-double-) x) (to-double #{ cblah($x) #}))

 (dhc-make-with-libs ()
   '("libasd.so")
   #{ #include <asd.h> #}
   blah)

Let's say that libasd.so and asd.h were placed in the same directory as our Lush file, we can do the following:

 (de blah (x) ((-double-) x) (to-double #{ cblah($x) #}))

 (libload "libc/make")
 (let* ((current-dir (dirname file-being-loaded))
        (dhc-make-lushflags (concat dhc-make-lushflags (sprintf " -I%s" current-dir))))

   (dhc-make-with-libs ()
     (list (concat current-dir "/libpng.so"))
     #{ #include "asd.h" #}
   blah))

Here is a simple example of a Lush file that dynamically loads asd.o and defines a function blah that calls the C function cblah defined in asd.o :

 (de blah (x) ((-double-) x) (to-double #{ cblah($x) #}))
 
 (mod-load "/wherever/asd.o")
 (dhc-make ()
   #{ double cblah(double); #}
   blah)

Rather than explicitely defining the C prototype of cblah in the dhc-make call, one may prefer to include the header file asd.h that corresponds to asd.o . This can be done easily:

 (de blah (x) ((-double-) x) (to-double #{ cblah($x) #}))
 
 (mod-load "/wherever/asd.o")
 (dhc-make ()
   #{ #include "/wherever/asd.h" #}
   blah)

Naturally, rather than refering to absolute paths, we may prefer to put asd.o and asd.h in the same directory as our Lush file. In that case, we need to tell Lush and the C compiler where to find everything:

 (de blah (x) ((-double-) x) (to-double #{ cblah($x) #}))
 
 (let* ((current-dir (dirname file-being-loaded))
        (dhc-make-lushflags (concat dhc-make-lushflags (sprintf " -I%s" current-dir))))
   (mod-load (concat current-dir "/asd.o"))
   (dhc-make ()
     #{ #include "asd.h" #}
   blah))

Naturally, all of this assumes that asd.o has previously been compiled. Lush conveniently provides a "make"-like utility to compile C files and specify dependencies called LushMake (see the corresponding documentation. Using LushMake as shown below will automatically generate asd.o when the Lush file is loaded, whenever asd.c or asd.h have been modified:

 (de blah (x) ((-double-) x) (to-double #{ cblah($x) #}))

 (libload "libc/make")
 (let* ((current-dir (dirname file-being-loaded))
        (dhc-make-lushflags (concat dhc-make-lushflags (sprintf " -I%s" current-dir)))
        (lm (new LushMake current-dir)))
   ;; compile asd.c to asd.o and mod-load asd.o
   (==> lm setflags (sprintf "-I%s" current-dir))
   (==> lm rule "asd.o"  '("asd.c" "asd.h"))
   (==> lm load)

   (dhc-make ()
     #{ #include "asd.h" #}
   blah))

Perhaps the functions defined in asd.o call functions from a library that must be linked-in, say libpng.so . These libraries can easily be added into using dhc-make-with-libs instead of dhc-make :

 (de blah (x) ((-double-) x) (to-double #{ cblah($x) #}))

 (libload "libc/make")
 (let* ((current-dir (dirname file-being-loaded))
        (dhc-make-lushflags (concat dhc-make-lushflags (sprintf " -I%s" current-dir)))
        (lm (new LushMake current-dir)))
   ;; compile asd.c to asd.o and mod-load asd.o
   (==> lm setflags (sprintf "-I%s" current-dir))
   (==> lm rule "asd.o"  '("asd.c" "asd.h"))
   (==> lm load)

   (dhc-make-with-libs ()
     '("libpng.so")
     #{ #include "asd.h" #}
   blah))

If our code in asd.c is written in C++ instead of C, we must replace the call to dhc-make-with-libs with dhc-make-with-c++ .

Basically, this works like calling C code. FORTRAN libraries can be loaded

Similar to the SN3 function but takes a class as argument instead of now obsolete cclass objects.

Similar to the SN3 function but takes a class as argument instead of now obsolete cclass objects.

This function displays the internal data structure of the interface between Lisp and Compiled code. The LISP-C interface maitains a sorted avl tree of all objects currently allocated. This table relates the address of the compiled structure, the type of the structure, the way the object was created and possibly the Lisp representation of this object.

• Calling this function without arguments returns the number of entries in the table.
  
• Calling this function with arg equal to 0 or () prints all the table. You better check the size of the table before doing this.
  
• Calling this function with arg equal to 1 prints all the table entries that needs to be synchronized before calling a compiled function.
  
• Calling this function with arg equal to 2 prints all the table entries that represent temporary objects.
  
• Calling this function with arg equal to an object, a storage, an index, or a strng prints the table entry (if any) associated with this object.
  

You can redirect the output of this command using function writing .

Example:

? (lisp-c-map)
= 12
? (lisp-c-map ())
  =                L 0x1e29b0 str              "A string for class c1"
 <                 L 0x2f10a0 idx              ::INDEX1:<3>
<                  L 0x2f12e0 obj:c2           ::c2:2f1330
 =                 L 0x2f1370 idx              ::INDEX1:<3>
=                   L 0x2f1c98 srg              ::FSTORAGE:static@2fe820
  =                L 0x2f1e18 str              "A second string for class c1"
 =                 L 0x2ff050 str              "A third string for class c1"
  =                L 0x2ff068 obj:c1           ::c1:2f1b08
=                  L 0x2ff080 idx              ::INDEX1:<3>
  =                L 0x2ff0a8 srg              ::FSTORAGE:static@2fe82c
 =                 L 0x31cb20 obj:c1           ::c1:1e2980
  =                L 0x31cb38 srg              ::FSTORAGE:ram@1e2968:<3>
= 12
The first character describes the position of the entry in the balanced
tree.  The second character tells wether the object was created by LISP (L)
or by compiled code (C) (e.g. Pool). Each entry displays then the address
of the compiled object and the lisp representation of the object. An object
created by C code may have no lisp representation until it is referenced by
a represented object or until the proper GPTR is casted by function <obj>.