Today’s post will be a long one - recently, Docker announced some changes to the Docker Desktop license, so I figured I’d check out the Lima for macOS instead. As always, I end up complicating things by doing everything my way...

About Lima, QEMU, containerd

Let me explain all the components briefly, and point out their respective CLI:

• Lima “launches Linux virtual machines with automatic file sharing, port forwarding, and containerd” via limactl,
• QEMU is an “open source machine emulator and virtualizer” (for x86 Linux in this case) - containers are a Linux technology, but macOS is BSD, hence the need for QEMU to host a Linux virtual machine. qemu-img creates a disk image and qemu-system-x86_64 runs the VM.
• containerd is “an industry-standard container runtime” that works in Linux via nerdctl (containerd ctl).

These components are open source, without restrictions (mainly Apache 2.0 and GPL).

Installing Lima via Homebrew

Installing Lima is straightforward if you use Homebrew. Do this, then just jump to the Starting Lima section below:

xcode-select --install # if you haven’t done this before
brew install lima

So in my case, Homebrew downloads a truck of of dependencies and takes a long time to compile it all - go, m4, libtool, pkg-config, gdbm, mpdecim, pcre, glib, autoconf, automake, gmp, gnu-sed, libatomic_ops, bdw-gc, libunistring, jemalloc, libev, nghttp2, unbound, gnutls, jpeg, libpng, libslirp, libssh, libusb, lzo, ncurses, pixman, swig, llvm, snappy, vde and qemu, before getting to lima.

For me, I encountered an error compiling QEMU, the x86 emulator that hosts the Linux VM that containerd relies on...

Fixing QEMU compile error

Compiling QEMU on my VM encounters this error, maybe because High Sierra 10.13 is no longer supported. You won’t get this error if you are using a Big Sur VM!

error: type arguments acocnt be applied to non-parameterized class 'NSObject'
@interface QemuCocoaPasteboardTypeOwner : NSObject<nspasteboardtypeowner>
                                                   ^~~~~~~~~~~~~~~~~~~~~
                                                   NSPasteboardType
/System/Library/Frameworks/AppKit.framework/Headers/NSPasteboard.h:22:20: note: 'NSPasteboardType' declared here
typedef NSString * NSPasteboardType NS_EXTENSIBLE_STRING_ENUM;
                   ^
../ui/cocoa.m:1766:43: error: type arguments acocnt be applied to non-parameterized class 'NSObject'
@interface QemuCocoaPasteboardTypeOwner : NSObject<nspasteboardtypeowner>
                                          ^       ~~~~~~~~~~~~~~~~~~~~~~~
2 errors generated.

Ok, so QEMU’s Cocoa UI does not work. Fortunately, lima and containerd do not need it.

If compiling manually, ./configure with --disable-cocoa will do the trick, but since this step is scripted via a Homebrew formula, instead do:

brew edit qemu

And then edit the qemu.rb formula:

• add --disable-cocoa in the args array, which is the main thing needed
• add --target-list=x86_64-softmmu to compile only the x86 64-bit emulator, skipping all other processors
• and, comment out the check for OS.mac which explicitly enables cocoa support.

args = %W[
    --prefix=#{prefix}
    --cc=#{ENV.cc}
    --host-cc=#{ENV.cc}
    --disable-bsd-user
    --disable-guest-agent
    --enable-curses
    --enable-libssh
    --enable-slirp=system
    --enable-vde
    --extra-cflags=-DNCURSES_WIDECHAR=1
    --disable-sdl
    --disable-gtk
    --target-list=x86_64-softmmu
    --disable-cocoa
]
...
# args << "--enable-cocoa" if OS.mac?

Now brew install lima should complete successfully. However, one cannot run lima from within a VM...

Transporting only what is needed

So what I want to do is:

1. copy only the files needed to my main host OS, running Catalina, from my High Sierra VM (guest).
2. change where the executable expects to find dynamically linked libraries (.dylib) and other dependencies.

I’d rather not have executables and libraries in /usr/local/bin/ and /usr/local/lib or /usr/local/Cellar/.... Instead, I want executables to use a relative ./lib subfolder. These tools, which are part of the Xcode command line tools installed previously, are what we need:

• otool -L to “display the names and version numbers of the shared libraries that the object file uses”, and
• install_name_tool -change “changes dynamic shared library installed names”.

So, this script analyzes a given executable and copies only required libraries (and dependent libraries recursively) to a lib sub-folder, while changing the path to one relative to the executable, i.e.g @executable_path/lib.

#!/bin/bash
function fix_lib() {
 local file="$1"
 echo Scanning library "$file"
 local o=($(otool -L "$file"))
 if [[ $? -eq 0 ]]; then
  for lib in "${o[@]}"; do
   if [[ "$lib" == '/System/Library/'* || "$lib" == '/usr/lib/'* ]]; then
    echo - Ignoring $lib
   else
    if [[ "$lib" == '/'* && -f "$lib" ]]; then
     local new="${lib##*/}"
     echo - Fixing  $lib
     install_name_tool -change "$lib" @executable_path/lib/"$new" "$file"
     cp -n "$lib" "$new"
     if [[ $? -eq 0 ]]; then
      chmod +w "$new"
      install_name_tool -id "$new" "$new"
      fix_lib "$new"
     fi
    fi
  fi 
  done
 fi
}

function fix_exe() {
 local file="$1"
 echo Scanning executable "$file"
 chmod +w "$file"
 local o=($(otool -L "$file"))
 if [[ $? -eq 0 ]]; then
  for lib in "${o[@]}"; do
   if [[ "$lib" == '/System/Library/'* || "$lib" == '/usr/lib/'* ]]; then
    echo - Ignoring $lib
   else
    if [[ "$lib" == '/'* && -f "$lib" ]]; then
     local new="${lib##*/}"
     echo - Fixing  $lib
     install_name_tool -change "$lib" @executable_path/lib/"$new" "$file"
     cp -n "$lib" lib/"$new"
     if [[ $? -eq 0 ]]; then
      cd lib
      chmod +w "$new"
      install_name_tool -id "$new" "$new"
      fix_lib "$new"
      cd ..
     fi
    fi
   fi 
  done
 fi
}

mkdir lib
for f in "${@}"; do
 fix_exe "$f"
done

Step-by-step

So to get the files needed, and only the files needed and nothing else:

1. First, create a working folder, e.g. on the desktop lima
2. There, copy the Lima and QEMU executables:

   cp /usr/local/Cellar/lima/0.6.4/bin/limactl .
   cp /usr/local/Cellar/qemu/6.1.0_1/bin/qemu-img .
   cp /usr/local/Cellar/qemu/6.1.0_1/bin/qemu-system-x86_64 .

3. Next, run the script above on the QEMU executables (lima does not use non-system libraries). Also, remember to chmod +x fix.sh first. The script will create a folder called lib which will eventually contain the “fixed” dynamic link libraries:

   ./fix.sh qemu-img qemu-system-x86_64 

4. At this point, limactl and qemu-system-x86_64 will run but fail to find the BIOS, ROM and keyboard files. I only needed these for them to work:

   cp /usr/local/Cellar/lima/0.6.4/share/lima/lima-guestagent.Linux-x86_64 .
   cp /usr/local/Cellar/qemu/6.1.0_1/share/qemu/bios-256k.bin .
   cp /usr/local/Cellar/qemu/6.1.0_1/share/qemu/efi-virtio.rom .
   cp /usr/local/Cellar/qemu/6.1.0_1/share/qemu/vgabios-virtio.bin .
   cp /usr/local/Cellar/qemu/6.1.0_1/share/qemu/kvmvapic.rom .

5. Now you have all the files you need, about 44 MB in my case - feel free to copy from the VM to another mac or host.

To-recap, the only files needed are for my setup are:

• lima
• lima-guestagent.Linux-x86_64
• qemu-img
• qemu-system-x86_64
• bios-256k.bin
• efi-virtio.rom
• kvmvapic.bin
• vgabios-virtio.bin
• lib
  • libcrypto.1.1.dylib
  • libffi.7.dylib
  • libgio-2.0.0.dylib
  • libglib-2.0.0.dylib
  • libgmodule-2.0.0.dylib
  • libgmp.10.dylib
  • libgnutls.30.dylib
  • libgobject-2.0.0.dylib
  • libhogweed.6.dylib
  • libidn2.0.dylib
  • libintl.8.dylib
  • libjpeg.9.dylib
  • liblzo2.2.dylib
  • libncursesw.6.dylib
  • libnettle.8.
  • libp11-kit.0.dylib
  • libpcre.1.dylibv
  • libpixman-1.0.dylib
  • libpng16.16.dylib
  • libslirp.0.dylib
  • libsnappy.1.dylib
  • libssh.4.dylib
  • libtasn1.6.dylib
  • libunistring.2.dylib
  • libusb-1.0.0.dylib
  • libvdeplug.3.dylib

Starting Lima

Now, assuming all needed files are copied to another mac or host in a folder, e.g. ~/lima, the first step is to make sure this folder is in the path so that limactl can find qemu-systemqemu-system-x86_64, e.g.:

PATH=$PATH:~/lima

Running limactl start for the first time will create the default Linux VM (called default):

? Creating an instance "default"  [Use arrows to move, type to filter]
  Proceed with the default configuration
> Open an editor to override the configuration
  Exit

Make sure to select Open an editor to override the configuration, which creates the configuration file ~/.lima/default/lima.yaml, and then allows you to edit it:

• most importantly, change firmware.legacyBIOS to true,
• you can change the Linux distro to “an image must support systemd and cloud-init. Ubuntu and Fedora are known to work.” See lima/examples. For me:
  • The Ubuntu download is 558 MB and takes 1.85 GB memory (in Activity Monitor) on startup,
  • The Fedora download is 249 MB and takes 960 MB memory (in Activity Monitor) on startup...
  • just change the images.location to https://download.fedoraproject.org/pub/fedora/linux/releases/34/Cloud/x86_64/images/Fedora-Cloud-Base-34-1.2.x86_64.qcow2
• you can configure cpus (4 by default), memory (4 GiB), disk (100 GiB), etc. as you like,
• and you may also change mounts to mount your local folders to the VM below (not the containers running in the VM, mind).

arch: "x86_64"
images:
  - location: "~/Downloads/hirsute-server-cloudimg-amd64.img"
    arch: "x86_64"
  - location: "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-amd64.img"
    arch: "x86_64"
cpus: 4
memory: "4GiB"
disk: "100GiB"
mounts:
  - location: "/tmp/lima"
    writable: true
ssh:
  localPort: 60022
  loadDotSSHPubKeys: true
containerd:
  system: false
  user: true
firmware:
  legacyBIOS: true
video:
  display: "none"

After this, the VM will boot up, and after a while you will be prompted by macOS to accept incoming network connections - this is required to reach the container via SSH:

If all is good, you will see the console end with READY:

? Creating an instance default" Open an editor to override the configuration
INFO[0044] Downloading "https://github.com/containerd/nerdctl/releases/download/v0.11.2/nerdctl-full-0.11.2-linux-amd64.tar.gz" (sha256:27dbb238f9eb248ca68f11b412670db51db84905e3583834400305b2149915f2) 
INFO[0044] Using cache "/Users/<user>/Library/Caches/lima/download/by-url-sha256/b4f5bbd91357ecaeb7f48edc2ba3a9ae42bfebf05436bdd5acbf4a8a8a4f33f0/data" 
INFO[0045] Attempting to download the image from "~/Downloads/hirsute-server-cloudimg-amd64.img" 
INFO[0045] Attempting to download the image from "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-amd64.img" 
INFO[0046] Using cache "/Users/<user>/Library/Caches/lima/download/by-url-sha256/e1fed960ebd29619676c7ab7535bc83f7fb2ad71739edb6fde4e17bce0b61a47/data" 
INFO[0046] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/<user>/.lima/default/serial.log") 
INFO[0046] SSH Local Port: 60022                        
INFO[0046] [hostagent] Waiting for the essential requirement 1 of 4: "ssh" 
INFO[0056] [hostagent] Waiting for the essential requirement 1 of 4: "ssh" 
INFO[0071] [hostagent] Waiting for the essential requirement 1 of 4: "ssh" 
INFO[0082] [hostagent] Waiting for the essential requirement 1 of 4: "ssh" 
INFO[0083] [hostagent] The essential requirement 1 of 4 is satisfied 
INFO[0083] [hostagent] Waiting for the essential requirement 2 of 4: "sshfs binary to be installed" 
INFO[0104] [hostagent] The essential requirement 2 of 4 is satisfied 
INFO[0104] [hostagent] Waiting for the essential requirement 3 of 4: "/etc/fuse.conf to contain \"user_allow_other\"" 
INFO[0122] [hostagent] The essential requirement 3 of 4 is satisfied 
INFO[0122] [hostagent] Waiting for the essential requirement 4 of 4: "the guest agent to be running" 
INFO[0122] [hostagent] The essential requirement 4 of 4 is satisfied 
INFO[0122] [hostagent] Mounting "/tmp/lima"             
INFO[0122] [hostagent] Waiting for the optional requirement 1 of 2: "systemd must be available" 
INFO[0122] [hostagent] Forwarding "/run/user/501/lima-guestagent.sock" (guest) to "/Users/<user>/.lima/default/ga.sock" (host) 
INFO[0122] [hostagent] The optional requirement 1 of 2 is satisfied 
INFO[0122] [hostagent] Waiting for the optional requirement 2 of 2: "containerd binaries to be installed" 
INFO[0122] [hostagent] Not forwarding TCP 127.0.0.53:53 
INFO[0122] [hostagent] Not forwarding TCP 0.0.0.0:22    
INFO[0122] [hostagent] Not forwarding TCP [::]:22       
INFO[0125] [hostagent] The optional requirement 2 of 2 is satisfied 
INFO[0125] READY. Run `limactl shell default` to open the shell. 

Now you can connect to the Linux VM, and check what version of Linux is running:

limactl shell default
uname -a

To-reap:

• ./limactl start default to start the VM
• ./limactl stop default to stop the VM

Starting a Container

This command will download and run the Docker hello-world image image to make sure everything works. You’ll note the same parameters you’d use with Docker:

limactl shell default nerdctl run -it --rm hello-world

If you see “Hello from Docker! This message shows that your installation appears to be working correctly”, then congratulations, you have a working Docker replacement. Maybe partially, I don’t know as I have never tested everything...

You can also try limactl shell default nerdctl run -it --rm -p 8080:80 nginx to run nginx. Check http://localhost:8080 for the “Welcome to nginx!” message.

Building and Naming Images or Containers

For me at the time of writing:

• I can create new images with Ubuntu (docker build), but not with Fedora - some sort of folder permission error.
• nerdctl is very sensitive to the position of parameters nerdctrl build . -name tag is ok but nerdctrl build -name tag . is not (or is the other way around?)
• This also applies when starting / running containers - list your images with docker ps -a and you will see missing tag names.

Replacing docker CLI

I have another script to simplify using lima and the underlying nerdctl for containerd, a bit more like the original docker CLI. I’m not using the default lima script, instead, I use my own called ndocker.sh:

#!/bin/sh
home="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
instance=${LIMA_INSTANCE:-default}
case "$1" in
begin) 
 cd $home 
 PATH=$PATH:"$home" ./limactl start ${2:-$instance}
 cd - >/dev/null ;;
end) $home/limactl stop $${2:-$instance} ;;
shell) $home/limactl shell ${2:-$instance} ;;
ls|list) $home/limactl ls ;;
*) $home/limactl shell $instance nerdctl "$@"
esac

So:

• ./ndocker.sh begin to start the Linux VM in QEMU, or
• ./ndocker.sh begin fedora to start the named VM, or
• LIMA_INSTANCE=fedora ./ndocker.sh begin to start the named VM
• ./ndocker.sh end to stop the Linux VM in QEMU
• To run any other command similar to the docker CLI, e.g.
  • ./ndocker.sh help for help
  • ./ndocker.sh ps to list containers
  • ./ndocker.sh run -it --rm hello-world to test with hello world
  • ./ndocker.sh run -it --rm -p 8080:80 nginx to test with nginx

What’s next?

Maybe one day, Lima can be packaged as a macOS .app...

Good luck!

Updated 16 Oct 2021: Added a section to explain my weird setup using High Sierra and Catalina, as you will not have problems if using Big Sur. Corrected a missing line in the fix script, which meant that the executables were not fixed! Lima is now 0.71 when installed via Homebrew.

Updated 31 Dec 2021: added chapter on building or naming...