Resizing an snap LXD btrfs loopback image

Thursday, May 23rd, 2024

The problem

I’m using LXD (https://linuxcontainers.org/) to run some local containers for development on my desktop. This is mostly for security reasons, to prevent all kinds of supply chain attacks. You know, the whole well-known and near impossible “Don’t run untrusted code on your computer” thing.

After creating a fair amount of containers, I got a No space left on device error in the containers on startup.

I googled a bunch of suff, found a whole lot of posts on the Linuxcontainers.org forums, but there was a lot of misinformation and none of the solutions posted worked for me. So I decided to write up here what I did to fix this problem.

My setup

I’ve got a fairly simple setup on Ubuntu. Unfortunately, since I wanted to use LXD v4.0+, I had to use the Snap package for LXD.

I’m using a single storage volume with btrfs:

<host> $ lxc storage list
+---------+-------------+--------+--------------------------------------------+---------+
| NAME    | DESCRIPTION | DRIVER | SOURCE                                     | USED BY |
+---------+-------------+--------+--------------------------------------------+---------+
| default |             | btrfs  | /var/snap/lxd/common/lxd/disks/default.img | 24      |
+---------+-------------+--------+--------------------------------------------+---------+

The instructions here probably don’t work as-is with other setups so your millage may vary. I’ll try to explain each step so you can adapt the instructions as needed.

Resizing

The first step is to find the loopback device and image. In a container, execute:

<container> $ df /
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/loop25 92211432 28580028 62952180 32% /

So our loopback device is /dev/loop25. Yes, it’s kinda weird that the loopback device is the same on the host as in the container, but remember that these are not virtual machines, they’re containers, so the share a certain part of the kernel space with the host.

Next, stop all the containers using lxc stop. I had some problems stopping them, as they would just hang while shutting down. I assume this was caused to low diskspace because after the resize the problem disappeared. You can force-shutdown a container using:

<host> $ lxc stop <CONTAINER_NAME> --force

On the host system, outside the container, find out which image the loopback device is configured to use using losetup:

<host> $ losetup /dev/loop25
/dev/loop25: [64769]:8781881 (/var/snap/lxd/common/lxd/disks/default.img)

This gives you the path to the disk image on the host.

Resize that image (on the host) using the truncate and losetup command. Don’t worry about the truncate usage here, it only adds space to the end of the image, kind of like increasing a harddrive partition size. Here we add 20 Gb of space to the image and let the loopback know about the change too:

<host> $ truncate -s +20GB /var/snap/lxd/common/lxd/disks/default.img
<host> $ losetup -c /dev/loop25

Finally, we need to resize the filesystem stored in the image. I did this on the host, but I guess it would also work from within the container? At any rate, btrfs needs the filesystem to be mounted. First we need to install the btrfs-progs package on the host, which has the btrfs binary.

<host> $ apt install btrfs-progs

Then we mount the image via the loopback device we found earlier, and use btrfs to resize the filesystem on it.

<host> $ mount /dev/loop25 /mnt/loop25
<host> $ btrfs filesystem resize max /mnt/loop25
<host> $ umount /mnt/loop25

Both the image and filesystem should now be resized, and you can start your container(s) again.