I wrote on this topic before but the landscape has changed a lot since then.

Skip to the new Qlot workflow.

When you work on projects that become even slightly complex it is a matter of time until you run into problems where the specific version of a particular library becomes important. This happens in most, if not all, programming languages.

In the Common Lisp environment Quicklisp has become the de facto standard for loading libraries, including fetching and loading their dependencies. Quicklisp distributes libraries in “distributions” which are point-in-time snapshots of all the known and working libraries at the time of distribution creation.

An advantage of this approach is that you are guaranteed that all libraries available in the distribution can be loaded with any of the others. Some disadvantages are that 1) if a library was included in an older distribution but no longer loads cleanly, it gets removed from the distribution, and 2) libraries are only added or updated when a new distribution is cut.

Though libraries can be put in ~/quicklisp/local-projects/ in order to supplement or override those in the distribution, Quicklisp does not provide any mechanism for pinning the state of ~/quicklisp/local-projects/.

Some Quicklisp attributes:

1. All libraries in a distribution can be loaded with any of the others. Those that no longer do are removed from the distribution.
2. Libraries are only added or updated at distribution cutting time.
3. You specify a single distribution version, not individual library versions. The distribution is used globally, across all projects and across all lisp implementations.
4. Libraries can be loaded from ~/quicklisp/local-projects/. Anything there will override the version in the distribution.
5. Nothing about the ~/quicklisp/local-projects/ content can be specified using Quicklisp. It needs to be managed outside of Quicklisp.
6. Libraries are loaded from the current Quicklisp distribution. There is no way to specify which distribution version a particular project must use.
7. Quicklisp can create a bundle of all the loaded systems that can be committed with the project code and used from there rather than the distribution, i.e. vendoring.
8. The source code for libraries included in a distribution are stored in a dedicated location for Quicklisp, they are not read from the original source repo.

Depending on your situation these attributes may be positive, negative or irrelevant.

There are projects like Ultralisp that have different philosophies regarding the distribution content but they still depend on Quicklisp for all other aspects. Thus they share most of the above attributes.

Since my previous post on this topic much has changed in the library version arena. There are now many projects that address different aspects of the above list; the topic of vendoring has gained momentum; and Qlot has changed a lot, to the extent that some code samples in older posts no longer work.

Vendoring is the idea that all the libraries your project depend on are actually part of your project and as such should be committed as part of the project code. Both Quicklisp and Qlot support this with QL:bundle and QLOT:bundle, and the Vend tool is entirely focused on vendoring.

The significant changes in Qlot broke my development workflow. Since I now had to spend time fixing this it was a good opportunity to evaluate some of the other library versioning tools. The issues that made me hesitant to adapt to the new Qlot without considering other options are:

• Roswell: Qlot pushes hard for using Roswell. That ads Roswell as another intermediary and dependency in an already complex process.
• The Qlot documentation is heavily biased towards using it as an external tool rather than through the REPL. Previously I used Qlot from the REPL and wanted to continue doing so if possible.
• Qlot is developed on SBCL which means that any issues in CCL take longer to be discovered and fixed. As I mainly use CCL it is something to keep in mind.
• The documentation proposes that lisp be started as a subprocess of Qlot, e.g. $ qlot exec ros emacs or $ qlot exec ccl. This is to ensure that the lisp is properly configured to use the project local Quicklisp. This is a brittle solution because the standard lisp REPL is then no longer sufficient. When you forget to start lisp this way Quicklisp will load the wrong libraries without any indication, potentially causing subtle bugs which are normally absent.
• Using QLOT:quickload rather than QL:quickload always bugged me for similar reasons. It is a non-standard way to do a standard thing. Forgetting to do it is easy and then you have the wrong libraries loaded.

I evaluated many of the other version management tools and came to the conclusion that Qlot is the closest to what I wanted and I set off trying to find a workflow that will adhere to my requirements listed here:

• Does not involve Roswell
• Works with CCL
• Works inside the REPL of a lisp loaded without any funny requirements
• Gives me 100% certainty that the correct versions of all libraries are loaded, without having to interrogate (asdf:system-relative-pathname) for each.

After some fiddling with Qlot I learned that:

1. Qlot installs a complete and independent Quicklisp inside your project directory. This Quicklisp has no dependence, relation or knowledge about the global Quicklisp.
2. $ qlot exec ccl mostly arranges things such that Quicklisp is loaded from the project local installation. If you can arrange for that to happen without using Qlot then you can use start your lisp normally.
3. When the project local Quicklisp is loaded it doesn’t know about the global or any other project local Quicklisps. This gives the 100% certainty of where libraries were loaded from.
4. When the project local Quicklisp is loaded you use it exactly like you would the global one. That is, QL:quickload and not QLOT:quickload, nor do you use any other Qlot wrappers.
5. Qlot does not need to be present in your lisp image at all.

6. Loading Qlot inside you current REPL doesn’t work well because:

   1. The Qlot REPL API is still in development.
   2. Doing this loads Qlot from the global Quicklisp instead of the project local one. Distribution version mismatches between the two Quicklisps could trigger issues with Qlot itself. It also voids the certainty of library location.
7. Having Qlot as a standalone executable outside of your lisp image puts it on the same plane as other tools used for development such as make or git. It then doesn’t matter which implementation it prefers because it doesn’t affect your choices.
8. Qlot has gained the ability to bundle libraries in case you want to go the vendoring route.
9. Roswell is not needed for any of the above.

New workflow

Combining my requirements and my new understanding of Qlot, I modified my workflow for pinning library versions to be:

1. Qlot executable must be available in your path.
2. Specify the distribution and library versions in qlfile.
3. qlot install at the CLI.
4. Load lisp without executing init scripts, e.g. ccl -n.

5. Load Quicklisp from the project local installation.

   (load "PROJECT-PATH/.qlot/setup.lisp")

6. Use Quicklisp as before.

   (ql:quickload :alexandria)

If you would like to vendor your libraries then:

1. Specify the distribution and library versions in qlfile.
2. qlot install
3. qlot bundle
4. git commit
5. ccl
6. (load PROJECT-PATH/.bundle-libs/setup.lisp)

Qlot tasks

All Qlot related tasks such as initialising a project, installing libraries, upgrading libraries, etc must be performed in the CLI using the Qlot executable. These happen relatively infrequently and inside the REPL Qlot does not feature.