Compiling Common Lisp to an Executable Using SBCL

Developing Common Lisp typically takes place in an interactive environment: SLIME/Sly with Emacs, Vlime/Slimv with (neo)vim, Alive with VS Code, or one of the many other options available. However, when it comes time to deploy your application, you typically do not want to deploy a program running in an interactive Lisp debugger. There are tools such as Buildapp that are designed to do this in a portable (across Common Lisp implementations) way, but today we're going to peel back the curtains and take a look at how to do this with SBCL directly. Then, for us Mac developers out there, we're going to look at how to cross compile to Linux using containerization with the new Apple Container CLI tool.

Let's say we have some simple code in `main.lisp` like this:

(defpackage :my-program
  (:use :cl))

(defun refresh-data ()
  "Downloads data from the internet, and caches it on this computer."
  (print "Getting data!"))

(defun display-data ()
  "Loads the cached data and displays it to the user."
  (print "Displaying data!"))

We would like to turn this into a CLI that accepts two commands: `refresh` and `display`. The SBCL function for saving a live image to disk is `sb-ext:save-lisp-and-die`. For building an exe, we can use it like so:

(sb-ext:save-lisp-and-die
 "program-name"
 :toplevel 'main                                ;; [1]
 :executable t                                  ;; [2]
 :save-runtime-options :accept-runtime-options  ;; [3]
 :compression t)                                ;; [4]

To go one keyword at a time:

1. `:toplevel` defines a function that will be the entrypoint to our application.
2. `:executable` tells SBCL to dump the image in an executable format. Specifying `nil` here would be useful for just saving your existing image to disk if you've carefully set up some state that you'd like to come back to later.
3. `:save-runtime-options` tells SBCL how to handle arguments passed to this executable. Your options are `t` (inherit all SBCL stack and heap settings from the currently running image), `nil` (pass any arguments to the executable directly to SBCL to control the runtime) or `:accept-runtime-options`. This third option will intercept `–dynamic-space-size` and `–control-stack-size` flags passed to the executable to be used by SBCL, but will otherwise pass the arguments through.
4. SBCL is built with executable compression. You may want this - you're trading a few milliseconds of startup time for a lot of disk space (~75MB -> ~17MB on my machine). I'll be leaving it out for the rest of the tutorial for brevity's sake, but it's very useful to know.

Now that we know how to dump an image to an executable, let's write that main function and give it a try.

(defpackage :my-program
  (:use :cl))

(defun refresh-data ()
  "Downloads data from the internet, and caches it on this computer."
  (print "Getting data!"))

(defun display-data ()
  "Loads the cached data and displays it to the user."
  (print "Displaying data!"))

;; NEW
(defun main ()
  (print "Running the program."))

Let's create the exe with the following commands:

1. `sbcl`
2. `(load "./main.lisp")`
3. `(in-package :my-program)`
4. `(sb-ext:save-lisp-and-die "program-name" :toplevel 'main :executable t :save-runtime-options :accept-runtime-options)`

After that completes, you should have a runnable executable - "my-program".

Now, let's handle command line arguments:

(defun main ()
  (let ((args (cdr sb-ext:*posix-argv*)))
    (when (member "refresh" args :test #'string=)
      (refresh-data))
    (when (member "display" args :test #'string=)
      (display-data))))

Great! One last thing remains to make this feel like a "normal" CLI application - error handling. If an error happens in the existing application, the CLI user will be dropped into an SBCL debugger. This is pretty neat, but probably not what we want if we were to ship/deploy this application. SBCL offers a function `sb-ext:disable-debugger` for this exact purpose.

(defun main ()
  (sb-ext:disable-debugger)
  (let ((args (cdr sb-ext:*posix-argv*)))
    (when (member "refresh" args :test #'string=)
      (refresh-data))
    (when (member "display" args :test #'string=)
      (display-data))))

Note that the error messages produced by the application will be kind of ugly. One option is to wrap the bulk of the application in a handler-case:

(defun main ()
  (sb-ext:disable-debugger)
  (handler-case
      (let (()))      ;; ... application goes here
    (error (e)
      (format t "ERROR: ~a~%" e)
      (sb-ext:exit :code 1))))

Here's what the final program would look like:

(defpackage :my-program
  (:use :cl))

(defun refresh-data ()
  "Downloads data from the internet, and caches it on this computer."
  (print "Getting data!"))

(defun display-data ()
  "Loads the cached data and displays it to the user."
  (print "Displaying data!"))

(defun main ()
  (sb-ext:disable-debugger)
  (handler-case
      (let ((args (cdr sb-ext:*posix-argv*)))
        (when (member "refresh" args :test #'string=)
          (refresh-data))
        (when (member "display" args :test #'string=)
          (display-data)))
    (error (e)
      (format t "ERROR: ~a~%" e)
      (sb-ext:exit :code 1))))

Let's say we have a more sophistated example with a full blown ASDF system.

(defsystem #:my-system
  :components ((:file "main")))

As long as this system can be bootstrapped by running `(asdf:load-system :my-system)`, we could build an exe from this system with the following command:

sbcl --eval '(asdf:load-system :my-system)' --eval "(sb-ext:save-lisp-and-die
   \"program-name\"
   :toplevel 'main
   :executable t
   :save-runtime-options :accept-runtime-options)"

If you're on an M-series Macbook and want to build an amd64 executable for linux, you can use Apple's container CLI to build the exe via containers. Here's an example dockerfile if you're building for an ubuntu based system:

FROM docker.io/ubuntu:latest
WORKDIR /build
RUN apt update && apt install sbcl -y
CMD sbcl --eval '(asdf:load-system :my-system)' --eval "(sb-ext:save-lisp-and-die \"program-name\" :toplevel 'main :executable t :save-runtime-options :accept-runtime-options)"

Then build the image with:

container build -a amd64 --tag lisp-builder --file ./Dockerfile .

Now we have an amd64 image ready to go. Next we just need to mount our source directory to the image and build an executable.

container run -a amd64 --volume <path to your directory>:/build lisp-builder

And voila! You've just cross compiled your SBCL executable to Linux.

• If you're running into trouble with your image being unable to find your project's dependencies, you may need to mount them as well (e.g. `–volume home/user/quicklisp/local-projects:/build/local-projects`). Note that asdf itself is trivially packaged into a single file (just run make). Then you can configure your asdf source registry to look in that directory. Your final invocation might look something like:

sbcl --eval \
"(asdf:initialize-source-registry '(:source-registry (:tree \"/build/local-projects\") :inherit-configuration))" \
       --eval '(asdf:load-system :my-system)' \
       --eval "(sb-ext:save-lisp-and-die \"program-name\" :toplevel 'main :executable t :save-runtime-options :accept-runtime-options)"

See the ASDF manual on the Configuration DSL for more details.

• On portability - this code was all written to be SBCL specific to remove as many layers of abstraction as possible. Here's a more portable version of the example program written above.

(defpackage :my-program
  (:use :cl))

(defun refresh-data ()
  "Downloads data from the internet, and caches it on this computer."
  (print "Getting data!"))

(defun display-data ()
  "Loads the cached data and displays it to the user."
  (print "Displaying data!"))

(defun disable-debugger ()
  (setf *debugger-hook*
        (lambda (condition hook)
          (declare (ignore hook))
          (format t "Unhandled error: ~a~%" condition)
          (uiop:quit 1))))

(defun main ()
  (disable-debugger)
  (let ((args (cdr (uiop:raw-command-line-arguments)))
    (when (member "refresh" args :test #'string=)
      (refresh-data))
    (when (member "display" args :test #'string=)
      (display-data))))))

Additionally, there's `uiop:dump-image` for creating an executable.

(setf uiop:*image-entry-point* 'main)
(uiop:dump-image "my-program" :executable t :compression t)