Dark Chestnut

Set up Verbose for multi-threaded standalone applications.

Sat, 20 Nov 2021 00:00:00 +0000

Although Verbose is one of few logging libraries that work with threaded applications (See Comparison of Common Lisp Logging Libraries), I had some trouble getting it to work in my application. I have a Hunchentoot web application which handles each request in a separate thread that is built as a standalone executable. Getting Verbose to work in Slime was trivial but once I built the standalone, it kept crashing.

The Verbose documentation provides all the information needed to make this setup work but not in a step-by-step fashion so this took me some time to figure out.

To work with threaded applications Verbose must run inside a thread of its own. It tries to make life easier for the majority case by starting its thread as soon as it is loaded. Creating a standalone application requires that the running lisp image contains only a single running thread. The Verbose background thread prevents the binary from being built. This can be remedied by preventing Verbose from immediately starting its background thread and then manually start it inside the application.

When Verbose is loaded inside Slime it prints to the REPL’s *standard-output* without fuss but when I loaded it inside my standalone binary it caused the application to crash. I did not investigate the *standard-output* connection logic but I discovered that you must tell Verbose explicitly about the current *standard-output* in a binary otherwise it won’t work.

Steps:

(pushnew :verbose-no-init *features*)

This feature must be set before the Verbose system is loaded. It prevents Verbose from starting its main background thread, which it does by default immediately when it is loaded.

I added this form in the .asd file immediately before my application system definition. While executing code inside the .asd file is considered bad style it provided the cleanest way for me to do this otherwise I would have to do it in multiple places to cover all the use cases for development flows and building the production binary. There may be a better way to set *features* before a system is loaded but I have not yet discovered it.
(v:output-here *standard-output*)

This form makes Verbose use the *standard-output* as it currently exists. Leaving out this line was the cause of my application crashes. I am not sure what the cause is but I suspect Verbose tries to use Slime’s version of *standard-output* if you don’t tell it otherwise, even when it is not running in Slime.

This must be done before starting the Verbose background thread.
(v:start v:*global-controller*)

Start the Verbose background thread.
(v:info :main "Hello world!")

Start logging.

I use systemd to run my applications. Systemd recommends that applications run in the foreground and print logs to the standard output. The application output is captured and logged in whichever way systemd is configured. On default installations this is usually in /var/log/syslog in the standard logging format which prepends the timestamp and some other information. Verbose also by default prints the timestamp in the logged message, which just adds noise and makes syslog difficult to read.

Verbose’s logging format can be configured to be any custom format by subclassing its message class and providing the proper formatting method. This must be done before any other Verbose configuration.

Combining all the code looks like below.

In app.asd:

(pushnew :verbose-no-init *features*)

(defsystem #:app
  ...)

In app.lisp:

(defclass log-message (v:message) ())

(defmethod v:format-message ((stream stream) (message log-message))
  (format stream "[~5,a] ~{<~a>~} ~a"
          (v:level message)
          (v:categories message)
          (v:format-message NIL (v:content message))))

(defun run ()
  (setf v:*default-message-class* 'log-message)
  (v:output-here *standard-output*)
  (v:start v:*global-controller*)
  (v:info :main "Hello world!")
  
  ...)

User feedback during long running external processes.

Sun, 02 May 2021 00:00:00 +0000

Sometimes it may be necessary to execute an external command that takes a long time to complete, long enough that the user needs visual feedback while it is running to show that the process is still alive.

UIOP provides fantastic tools for running external commands in a portable manner but it was not obvious to me how to show the external command’s output to the user while it was still busy. I also wanted to execute the external command in a synchronous fashion, i.e. my lisp application must wait for the external command to finish before continuing. The need for synchronicity sent me down the wrong path of using the synchronous uiop:run-program. It only returns when the external command has finished, which means you can only process the command output once it is completed.

I eventually realised I should use uiop:launch-program, the asynchronous version, and I came up with the following solution. In the example below the (ping) function pings a website and prints the results as they become available. Afterwards it returns the exit code of the ping command.

(defun ping ()
  (let (proc out exit-code)
    (unwind-protect
         (progn
           (setf proc (uiop:launch-program
                       (list "ping" "-c" "5" "www.darkchestnut.com")
                       :ignore-error-status t
                       :output :stream))
           (setf out (uiop:process-info-output proc))
           (iter
             (while (uiop:process-alive-p proc))
             (iter
               (while (listen out))
               (write-char (read-char out) *STANDARD-OUTPUT*))
             ;; ... Maybe do something here
             (sleep 0.5))
           (uiop:copy-stream-to-stream out *STANDARD-OUTPUT* 
                                       :linewise t))
      (setf exit-code (uiop:wait-process proc))
      (uiop:close-streams proc))
    exit-code))

In the first example the command’s output is shown to the user but it is not processed in any other way. If you need to do some extra processing on it after completion then the next example should provide a good starting point.

(defun ping-processing ()
  (let (proc out exit-code output)
    (with-open-stream (output-stream (make-string-output-stream))
      (with-open-stream (broadcast-stream 
                         (make-broadcast-stream *STANDARD-OUTPUT* 
                                                output-stream))
        (unwind-protect
             (progn
               (setf proc (uiop:launch-program
                           (list "ping" "-c" "5" 
                                 "www.darkchestnut.com")
                           :ignore-error-status t
                           :output :stream))
               (setf out (uiop:process-info-output proc))
               (iter
                 (while (uiop:process-alive-p proc))
                 (iter
                   (while (listen out))
                   (write-char (read-char out) broadcast-stream))
                 (sleep 0.5))
               (uiop:copy-stream-to-stream out broadcast-stream 
                                           :linewise t)))
          (setf exit-code (uiop:wait-process proc))
          (uiop:close-streams proc)
        (setf output (get-output-stream-string output-stream))
        ;; ... process output here
        exit-code))))

Process sub-command style command line options with Adopt.

Mon, 19 Apr 2021 00:00:00 +0000

How to process sub-command style command line arguments is a question that arises more and more. Many of the basic option handling libraries can not handle this at all, or they make it very difficult to do so.

One of the newer libraries in the option processing field is Adopt by Steve Losh. It was not designed to handle sub-commands but it is in fact very capable to do this without having to jump through too many hoops.

In a Reddit thread someone asked if Adopt can handle sub-command processing and Steve answered with the following example:

(eval-when (:compile-toplevel :load-toplevel :execute)
  (ql:quickload '(:adopt) :silent t))

(defpackage :subex
  (:use :cl)
  (:export :toplevel *ui*))

(in-package :subex)

;;;; Global Options and UI ----------------------------------------------------
(defparameter *o/help*
  (adopt:make-option 'help :long "help" :help "display help and exit" :reduce (constantly t)))

(defparameter *o/version*
  (adopt:make-option 'version :long "version" :help "display version and exit" :reduce (constantly t)))

(defparameter *ui/main*
  (adopt:make-interface
    :name "subex"
    :usage "[subcommand] [options]"
    :help "subcommand example program"
    :summary "an example program that uses subcommands"
    :contents (list *o/help* *o/version*)))

(defparameter *ui* *ui/main*)


;;;; Subcommand Foo -----------------------------------------------------------
(defparameter *o/foo/a*
  (adopt:make-option 'a :result-key 'mode :short #\a :help "run foo in mode A" :reduce (constantly :a)))

(defparameter *o/foo/b*
  (adopt:make-option 'b :result-key 'mode :short #\b :help "run foo in mode B" :reduce (constantly :b)))

(defparameter *ui/foo*
  (adopt:make-interface
    :name "subex foo"
    :usage "foo [-a|-b]"
    :summary "foo some things"
    :help "foo some things"
    :contents (list *o/foo/a* *o/foo/b*)))

(defun run/foo (mode)
  (format t "Running foo in ~A mode.~%" mode))


;;;; Subcommand Bar -----------------------------------------------------------
(defparameter *o/bar/meow*
  (adopt:make-option 'meow :long "meow" :help "meow loudly after each step" :reduce (constantly t)))

(defparameter *ui/bar*
  (adopt:make-interface
    :name "subex bar"
    :usage "bar [--meow] FILE..."
    :summary "bar some files"
    :help "bar some files"
    :contents (list *o/bar/meow*)))

(defun run/bar (paths meow?)
  (dolist (p paths)
    (format t "Bar-ing ~A.~%" p)
    (when meow?
      (write-line "meow."))))


;;;; Toplevel -----------------------------------------------------------------
(defun toplevel/foo (args)
  (multiple-value-bind (arguments options) (adopt:parse-options-or-exit *ui/foo* args)
    (unless (null arguments)
      (error "Foo does not take arguments, got ~S" arguments))
    (run/foo (gethash 'mode options))))

(defun toplevel/bar (args)
  (multiple-value-bind (arguments options) (adopt:parse-options-or-exit *ui/bar* args)
    (when (null arguments)
      (error "Bar requires arguments, got none."))
    (run/bar arguments (gethash 'meow options))))

(defun lookup-subcommand (string)
  (cond
    ((null string) (values nil *ui/main*))
    ((string= string "foo") (values #'toplevel/foo *ui/foo*))
    ((string= string "bar") (values #'toplevel/bar *ui/bar*))
    (t (error "Unknown subcommand ~S" string))))

(defun toplevel ()
  (sb-ext:disable-debugger)
  (multiple-value-bind (arguments global-options)
      (handler-bind ((adopt:unrecognized-option 'adopt:treat-as-argument))
        (adopt:parse-options *ui/main*))
    (when (gethash 'version global-options)
      (write-line "1.0.0")
      (adopt:exit))
    (multiple-value-bind (subtoplevel ui) (lookup-subcommand (first arguments))
      (when (or (null subtoplevel)
                (gethash 'help global-options))
        (adopt:print-help-and-exit ui))
      (funcall subtoplevel (rest arguments)))))

A list of Common Lisp command line argument parsers.

Sun, 18 Apr 2021 00:00:00 +0000

I was searching for a command line option parser that can handle git-style sub-commands and found a whole bunch of libraries. It appears as if libraries on this topic proliferate more than usual.

I evaluated them only to the point where I could decide to skip it or give it a cursory test. The information I gathered is summarised below.

If you only need the usual flag and option processing, i.e. not sub-commands, then I would suggest unix-opts. It appears to be the accepted standard and is actively maintained. It is also suggested by both Awesome Common Lisp and the State of the Common Lisp Ecosystem Survey 2020.

If your needs are very complex or specific you can investigate clon, utility-arguments or ace.flag.

For basic flags and options with sub-commands, there are a few libraries that explicitly support sub-command processing but you should be able to make it work with many of the other options and a bit of additional code.

Name	Print help	Native sub-commands	Notes
ace.flag	?	?	Not in QL.
adopt	Yes	No	Can generate `man` files.
apply-argv	No	No	Does not handle `-xyz` as three flags.
cl-just-getopt-parser	No	No	Easy to use.
cl-cli	Yes	Yes
cl-argparse	Yes	Yes
cli-parser	No	No	Does not handle free arguments, not in QL.
clon	?	Yes	Very complex, most feature rich.
command-line-arguments	?	?	Not well documented.
getopt	No	No	Does not handle `-xyz` as three flags, not well documented.
parse-args	No	No	Not in QL
utility-arguments	?	?	Complex to set up
unix-options	Yes	No	Easy to use.
unix-opts	Yes	No	The standard recommendation.

Database drivers for PostgreSQL and SQLite.

Mon, 13 Jul 2020 00:00:00 +0000

Index of picking libraries blog series

To interact with a database system from your code requires some database driver library. For most database systems there are multiple driver libraries, most of which are stable and work well.

There are also options of multi-system and single-system drivers. The multi-system drivers can interface with multiple database systems, usually MySQL, PostgreSQL and SQLite, while single-system drivers only work for a specific database system.

The two main reasons I see for using a multi-system driver are:

You want to reduce the risk in the event you need to switch your database system mid-project. It is rare but possible.
You work on multiple projects which use different database systems. With a multi-system driver you only need to learn one library which works the same for all databases.

My response to those two reasons are:

I consider it extremely unlikely that I will need to switch database systems for the project.
I prefer to use libraries that do one thing well rather than multi-purpose ones. I would rather learn the focused driver for each different database system I use than fighting the unavoidable complexities which come with generalised tools that cater for databases I will never use.

In another post I mentioned that I use PostgreSQL for my application database. I also use SQLite for local configuration files. The drivers I considered are:

CL-sxl
Sxql
CL-dbi
CL-sql
Postmodern
CL-sqlite

Postmodern and CL-sqlite are the only single-system drivers in that list. Due to the two reasons I mentioned above, I use Postmodern and CL-sqlite.

Index of picking libraries blog series

Database migration libraries for PostgreSQL.

Mon, 27 Apr 2020 00:00:00 +0000

Index of picking libraries blog series

It may be tempting at the start of a new project to create the first database tables manually, or write SQL scripts that you run manually, especially when you first have to spend a significant amount of time on sifting through all the migration libraries and then some more to get it working properly.

Going through this process did slow me down at the start of the project but I was determined to use a migration tool because hunting inexplicable bugs that only happen in production just to find out there is a definition mismatch between the production and development databases is not fun. Using such a tool also motivates you to write both the setup and teardown steps for each table while the current design is still fresh in your mind.

At first I considered a standalone migration tool because I expect them to be very good at that single task. However, learning the idiosyncrasies of a new tool and trying to make it fit seamlessly into my development workflow seemed like more trouble than it is worth.

I decided to stick with a Common Lisp library and found the following seven that work with PostgreSQL and/or Postmodern:

Postmodern-passenger-pigeon
database-migrations
cl-migrations
Mito
Crane
Orizuru-orm
cl-mgr

I quickly discounted Crane and Mito because they are ORM (Object Relational Mapper) libraries which are way more complex than a dedicated migration library. Development on Crane have stalled some time ago and I don’t feel it is mature enough for frictionless use yet. Mito declares itself as being in Alpha state; also not mature enough yet.

I only stumbled onto cl-mgr and Orizuru-orm long after making my decision so I did not investigate them seriously. Orizuru-orm is in any case an ORM which I would have discounted because it is too complex for my needs. CL-mgr looks simple, which is a good thing. It is based on cl-dbi which makes it a good candidate if you foresee switching databases but even if I discovered it sooner I would have discounted it for the same reason as CL-migrations.

CL-migrations looks very promising. It is a simple library focusing only on migrations. It uses clsql to interface with the database which bothered me because I already committed to using Postmodern and I try to avoid adding a lot of unused code to my projects. The positive side is that it interfaces to many different databases so it is a good candidate if you are not committed to using Postmodern. It is also a stable code base with no outstanding bug reports.

The two projects I focused on was Postmodern-passenger-pigeon and Database-migrations because they both use Postmodern for a database interface.

Postmodern-passenger-pigeon was in active development at the time and it seemed safer to use than Database-migrations because it can do dry runs, which is a very nice feature when you are upgrading your production database and face the possibility of losing data when things go awry. Unfortunately I could not get it working within a reasonable amount of time.

I finally settled on Database-migrations. It is a small code base, focused on one task, it is mature and it uses Postmodern so it does not pull in a whole new database interface into my project. There are however some less positive issues.

The first issue is a hindrance during development. Every time the migrations ASDF system (or the file containing it, as ASDF prefers that all systems be defined in a single file) is recompiled it adds all the defined migrations to the migrations list. Though each one will only be applied once to the DB it is still bothersome. One can then clear the list with (setf database-migrations::*migrations* nil) but then only newly modified migration files will be added. The solution then is to touch the .asd file after clearing the migrations list.

The second negative point is quite dangerous. The downgrade function takes a target version as parameter, with a default target of 0. This means that if you execute downgrade without specifying a target version you delete your whole database.

I am currently using Database-migrations and it works well for me. If for some reason I need to switch I will use cl-migrations.

Using Database-migrations

To address the danger of unintentionally deleting my database I created a wrapper function that does both upgrade and downgrade, and it requires a target version number.

Another practical issue I discovered is that upgrades and downgrades happen in the same order as they are defined in the migration file. If you create two tables in a single file where table 2 depends on table 1 then you can not revert / downgrade because Database-migrations will attempt to delete table 1 before table 2. The solution here is to use the def-queries-migration macro (instead of def-query-migration) which defines multiple queries simultaneously . If you get overwhelmed by a single definition that defines multiple tables the other option is to stick with one migration definition per file.

Index of picking libraries blog series

Web development starter pack.

Wed, 15 Apr 2020 00:00:00 +0000

Index of picking libraries blog series

A big part of a web development framework’s value is that it pre-selects a set of common libraries. If you use the framework as-is you will have a reasonable setup to build on without having to put any thought into it. More comprehensive frameworks express opinions on many more issues in addition to libraries and they often make it difficult to stray from their prescribed path. Lightweight frameworks confine their opinions to fewer topics and usually make it easier to follow your own path.

In this blog series I describe a list of libraries I picked for doing my web development. The list of libraries can be treated as a featherweight framework, a good foundation to make a quick start but with no fences to keep you on any particular path. The only exceptions are when a library depends on another library, for example easy-routes only works with Hunchentoot. Replacing Hunchentoot will require a replacement for easy-routes.

Below is the list of libraries I use. The options I considered and my reasoning for picking these particular libraries are set out in the various posts in the series.

Database : PostgreSQL

DB access : Postmodern

ORM / DAO : Postmodern

DB migrations : Database-migrations

HTTP server : Hunchentoot

Request routing and middleware : Easy-routes

JSON : ST-JSON

Templating : Djula

Testing : 5am

Logging : Verbose

Session management : Override Hunchentoot’s sessions with a custom class to store session data in PostgreSQL.

Password hashing : cl-pass

Configuration files : Sqlite

Building binaries : Buildapp with CCL

CLI parameters : unix-opts

Index of picking libraries blog series

Picking the database for a web application.

Wed, 01 Apr 2020 00:00:00 +0000

Index of picking libraries blog series

Picking a database for your web application can be surprisingly time consuming. There are a variety of databases of various types available and Quicklisp has an interface library for many of them.

Unless you need a very specific capability from the database and you are acutely aware of this fact, chances are that a standard relational database will be more than sufficient to provide for your needs.

Once you have decided to use a SQL database the three main contenders are MySQL, PostgreSQL and SQLite. They are all open-source, mature, actively developed and well-supported by Common Lisp.

For most projects SQLite will work very well, even in production. It can handle database sizes and query volumes far exceeding what most web apps will ever need, it stores the whole database in a single file so the backup process is simply a normal file copy and there is no database server application to configure and maintain.

The down side of SQLite is that the database file must be stored on the same machine as where the application is running because it is not a client/server system. If you need to run the application and the database on separate machines then SQLite will not work. One such reason would be if you need to build a highly available service, i.e. your application must run on multiple machines to ensure it remains available when one server stops working.

If you decide that you need a client/server database it comes down to MySQL or PostgreSQL. This is mostly a choice of personal preference. Both have very good support in Common Lisp. There are areas in which one is better than the other but few applications will ever run into those cases.

I use PostgreSQL in my projects, both for development and production environments. The idea is sometimes put forward that one should use SQLite in development and PostgreSQL in production. This is a bad idea. Even though both implement the SQL standard there are still distinct differences in how they function and in their capabilities. By using different database systems you must cater for two database variants instead of one. In addition, you will always go to production with untested code that can only be tested in production.

Index of picking libraries blog series

Testing command line interfaces

Tue, 07 Jan 2020 00:00:00 +0000

When you make a standalone application intended as a command line tool you may want integration tests to exercise the application through the normal command line user interface.

There are tools that can do this testing on OS executable applications but sometimes it is useful to run these tests as part of the test suite without having to first build a standalone binary.

The goal then is to test a function that only interacts via standard input and output. This code demonstrates how it can be done:

(defun application ()
  (let* (name greet)
    ;; Prompt 1
    (format *query-io* "Name: ")
    (setf name (read-line *query-io*))
    ;; Prompt 2
    (format *query-io* "Greeting: ")
    (setf greet (read-line *query-io*))
    ;; Result
    (format t "~&~A ~A!~%" greet name)))

(def-test greet ()
  (let* (;*STANDARD-INPUT* must contain all the responses at once.
         (*STANDARD-INPUT*
          (make-string-input-stream (format nil "John~%Hi~%")))
         (*STANDARD-OUTPUT* (make-string-output-stream))
         (*QUERY-IO*
          (make-two-way-stream *STANDARD-INPUT* *STANDARD-OUTPUT*))
         result)
    (application)
    (setf result (get-output-stream-string *STANDARD-OUTPUT*))
    ;; RESULT =>
    ;; "Name: Greeting: 
    ;; Hi John!
    ;; "
    (is (string-equal
         "Hi John!"
         (nth 1 (split-sequence:split-sequence
                 #\newline result))))))

The example’s method only works for functions that have a fixed request/response sequence, in other words, the test does not need to change its response dynamically, all the responses can be prepared before the function is executed.

The reason for this limitation is that make-string-input-stream creates streams with fixed content. Even if the original string changes, the stream content does not. Without the ability to change the stream content after creation there is no way to implement dynamic responses to application prompts.

While it does not seem possible to create interactive output-to-input streams with Common Lisp primitives, UIOP:RUN-PROGRAM can provide such output-to-input streams to external processes. This at least indicates that it can be done for some particular case. On CCL this is accomplished with file descriptor streams. I have not investigated the detail of the implementation nor have I looked how it is done for other implementations, so it may ultimately be a dead end.

It appears as if there is no easy and uncomplicated solution for setting up a *STANDARD-OUTPUT* to *STANDARD-INPUT* pipe that will enable test functions to apply logic in-between responses. If you need such a solution some avenues to investigate are:

Test the application as a standalone external process with UIOP:RUN-PROGRAM. This definitely works but it is exactly what we did not want to do.
Look for a library that implements some kind of pipe function. cl-plumbing may be an option.
Try to implement piping with files or sockets.

Parameterised decorators for easy-routes

Mon, 16 Dec 2019 00:00:00 +0000

The easy-routes library makes it easy and fast to define Rails-like HTTP routes for Hunchentoot. It has the concept of decorators which are functions that form part of the request handling sequence but not part of the main request handler. Decorator functions are usually general functions which are applicable to many request handlers, for example, to set up a database connection before handling a request.

Parameterless decorators

Most decorators are simple functions which do not take any parameters from the request handler. The DB connection decorator can be defined as below:

(defun @db (next)
  (postmodern:with-connection *db-spec*
    (funcall next)))

All decorators take the next parameter which is the function that should be called after this decorator has completed its task. The parameter is supplied by easy-routes.

A route that uses a decorator is defined like this:

(defroute api-handler ("/api"
                       :method :get
                       :decorators (@db @auth))
    ...)

The :decorators keyword takes a list of decorator names in the order they need to be executed. In the example above the @db function will run, followed by the @auth function and then api-handler. Since @auth runs after @db it can assume an open database connection. Note that the next parameter is not mentioned anywhere.

Parameterised decorators

For more complex cases where the decorator needs input from the route handler, it can be defined with extra parameters like this:

(defun @check (predicate http-error next)
  (if (funcall predicate)
      (funcall next)
      (http-error http-error)))

The next parameter must be last in the parameter list.

With a predicate function defined like below,

(defun test-p ()
  ...)

a route handler can be defined like this:

(defroute api-handler
    ("/api"
     :method :get
     :decorators (@db @auth 
                  (@check test-p hunchentoot:+http-forbidden+))
    )
    ...)

Decorators with additional parameters are passed as a list with the decorator name as the first list item followed by the parameters, excluding next. The list is not quoted.