Four Years On, Umoci Celebrates A Long-Awaited Release

After a nearly four-year hiatus, there is a new umoci release! umoci is the reference implementation of the Open Container Initiative (OCI) image specification, and provides users with the ability to create, manipulate, and otherwise interact with container images. It is designed to be as small and unopinionated as possible, so as to act as a foundation for larger systems to be built on top of.

I originally wrote umoci back in 2016 to allow SUSE to create container images natively with our existing OS image build system called Kiwi, which is in turn used by the Open Build Service (not to be confused with the other OBS) to automatically build and distribute all of openSUSE’s and SUSE’s operating system container images, most Base Container Images (BCI), as well as countless community-maintained openSUSE container images.

At the time there was no real tool to do this^†, and so I sat down and wrote one to generate OCI images without the need for a Docker daemon – something we couldn’t require for OBS builds (not to mention it made very little sense to require all of Docker just to generate a handful of files and tar archives). It would be fair to argue that as a result the Open Build Service was the first OCI-native container image building service, with umoci having been used in production way back in 2016. OBS does now support building images based on Dockerfiles, but a very large number of core images are still built using umoci.

I’m not sure that me from nearly a decade ago would’ve thought umoci would continue to be such a core part of our build infrastructure at SUSE, nor that it would end up becoming the reference implementation of the OCI image specification. It also bears mentioning that umoci has seen a fair amount of use outside of SUSE – the notably very-much-not-OCI-thank-you-very-much LXC and Incus container runtimes both use umoci to provide support for OCI images, and the fairly unique Stacker build system (used by Cisco for building their appliance images) uses umoci to power its builds. The main reason that I think umoci has found such varied usage is because it is quite unopinionated and acts like a trusty multi-tool you can adapt for any situation involving container images.

†: This is actually a little bit more nuanced. There was a tool called oci-image-tool that it would be unfair to leave unmentioned, but I will talk about that a little later.

How to use umoci?

We have a fairly detailed quick start guide, but I’ll quickly run you through an example of how you can use umoci today. First, you’ll need to have an OCI image stored in a local directory. There are quite a few tools that let you download images, but the most common one I use is skopeo:

$ skopeo copy docker://registry.opensuse.org/opensuse/tumbleweed:latest oci:tumbleweed:latest
Getting image source signatures
Copying blob c2ab4b3af30b done |
Copying config 3392320e34 done |
Writing manifest to image destination

And now the local directory tumbleweed contains an OCI image and all of its content-addressed blobs:

$ find ./tumbleweed -type f
./tumbleweed/blobs/sha256/c2ab4b3af30b317cfb2bfa660e90bad5e4bc8d588f34528cb4ee33d53d218a62
./tumbleweed/blobs/sha256/3392320e34c1656ae69f7b2868b5749ea9ba9350c6ca30029b9d4ce18cd0ca53
./tumbleweed/blobs/sha256/e9de6d8081d88ebd6a6cfe5cd711b55e6e95261ef4ce396fedbf2fd9d23ebf27
./tumbleweed/oci-layout
./tumbleweed/index.json

So now you can use umoci to unpack the OCI image into an OCI runtime specification bundle. The OCI runtime specification is the underlying configuration system used by OCI runtimes, which power many of the container tools you know and love (such as Docker and Podman). In this example we will use runc (the most widely used OCI container runtime) to run a container, create some changes, and then update the image with an new layer containing those changes.

$ sudo umoci unpack --image tumbleweed:latest bundle
$ sudo runc run -b bundle ctr
umoci-default:/ # echo "we are inside the container" > /foo
umoci-default:/ # exit
$ sudo umoci repack --image tumbleweed:latest bundle

And now the image has a new layer containing the changes you made, it’s as easy as that! The image can now be pushed to a registry using tools like skopeo, or re-used to create more container instances. Of course, umoci has full rootless containers support too and has plenty of other useful features, but this is the most basic workflow you can use to build umoci images.

Note that it wasn’t necessary for us to use containers to modify the image and create a new layer – you could just modify the root filesystem in bundle/rootfs directly just like any other directory. This flexibility is precisely what allows umoci to be used by tools like Kiwi, OBS, Stacker, and others that wish to create container images without being forced to use containers to build them.

What’s new in umoci 0.5?

Despite the long wait, for most users there are only a handful of notable features and improvements added in this release (the most notable is support for zstd-compressed images – an OCI image specification version 1.1 feature that is getting used more and more in the wild). There was also a fix for a notable performance regression in the last release (though it’s been so long that maybe it’s better to call it a “performance improvement” at this point). Unfortunately, full support for the latest OCI image specification has not yet been implemented, for reasons I will outline later.

The bulk of the changes in this release were related to code modernisation changes (Go 1.13 errors were still somewhat new back in 2021), as well as some very large overhauls and improvements to overlayfs support (something that is used by Stacker but is definitely not widely used). You can find a more detailed list of changes in our changelog. There are still a fair number of things that I would like to get done in umoci, but a 4-year gap between releases meant that a release was more than overdue.

So, why did it take so long? Well, there are quite a few reasons but the main technical one is that I had planned for image specification version 1.1 support to be included in this release but unfortunately there are a cacophony of issues that made this too big of a job to do with very little reward. We’ll get into the exact details in a minute but first, we need to talk a little bit about the historical relationship between oci-image-tool and umoci.

umoci and oci-image-tool

Before umoci, there was a tool called oci-image-tool which was a collection of small helpers to do some limited operations on OCI images.

Now, I am going to talk about the shortcomings of oci-image-tool in this section, but I don’t want this to be interpreted as an attack on the folks who worked on it (myself included). There were a variety of factors that resulted in oci-image-tool ending up in the situation it is today, and I would never stoop to think that my own projects don’t have their own substantial shortcomings.

At the time, the problem I was trying to solve was that we needed a simple tool that Kiwi could use to create our OS container images directly without needing to use Docker itself. The system we were using then was based around generating a tar archive and then submitting a pull request to the docker-library/official-images repo which is used for building all of the “official library” Docker images, which were based on Dockerfiles. Our Dockerfile was just a single ADD statement to the rootfs.tar.xz archive generated by Kiwi, stored in a GitHub repo.

Given that OBS was (and still is) a far superior build system for distributions (as it can rebuild images if dependencies have been updated and provide Bill-of-Materials-like information about what packages are included in images — features that, at time of writing, Docker still doesn’t really have), this kind of setup was quite unfortunate. We would ideally just push our images directly to the official registry. Or even better, we would just host our own registry that could be entirely managed by OBS – which is how the SUSE Trusted Container Image Registry and openSUSE Community Container Image Registry work today. And for that we needed a tool to generate Docker images that could easily fit into the existing build systems we had at SUSE.

As a developer heavily involved in the Open Container Initiative, I felt that building a tool that could operate on the at-the-time yet-to-be-standardised OCI image format would be a better target, especially since the OCI image format was designed so that it could (relatively) easily be converted to and from the Docker image format. So, if we built an OCI image we could make use of other community-built tools such as skopeo (after a bit of extra work to implement completely Docker-less conversions).

I was also a bit disappointed by the lack of an image specification equivalent to the OCI runtime specification’s runc – that is, a relatively unopinionated and flexible reference implementation of the specification. Now, runc’s relationship to OCI is a little bit of a historical oddity (the runtime specification was developed after runc was adopted by OCI because OCI was donated by Docker and was the initial project of the OCI) but I really wanted to be able to just run a few commands to download an image, unpack it, and spawn a container with runc (as in the example we went through earlier.)

The unfortunate problems with oci-image-tool

It was clear from the outset that oci-image-tool would not be enough for everything we needed (the design was incompatible with any kind of incremental build system where you progressively modify a rootfs and create new layers). But my first thought was “no matter, we could create a small wrapper around oci-image-tool‘s APIs”.

And that was the thinking I had when making the first incarnation of umoci — a fairly minimal set of wrappers around oci-image-tool. The only real added feature we would need was the usage of a library called go-mtree to generate an mtree(8)-style manifest we could use to generate diff layers relatively efficiently.

If oci-image-tool had been able to do what we needed, maybe those small extra features would’ve been folded into oci-image-tool or Kiwi and umoci would only have ended up being a small prototype that disappeared after a few months.

Unfortunately, it turns out that oci-image-tool was (in my view) basically only useful as a proof-of-concept. It was missing very core features we needed (namely, creating new image layers – something that might be slightly important for a container image builder) and had several very severe bugs in the few features that were implemented (in particular, it couldn’t extract any real image layers due to a series of critical bugs). In a word, the state of the project at the time made it unusable to use as-is for our needs (if you’re interested, there is a list of them in the project README).

I initially sent patches to try to improve the situation but it became more and more clear that there were deeper design issues, at which point I ended up reimplementing all of the parts of oci-image-tool we were using, with the intention of pushing the new API back into oci-image-tool. Sadly, it eventually seemed that there was a lack of interest in doing this (which I understand – after all, I was effectively suggesting that they replace most of the code with code that I wrote). I did eventually become a maintainer of oci-image-tool but development behind oci-image-tool basically stalled soon after umoci was ready for use, and the oci-image-tool project itself eventually ended up pushing people to use umoci for image unpacking and repacking.

After being used in production for a few years, I put forward a proposal that umoci be added to the OCI as a reference implementation, and the proposal was accepted in 2020. As part of the discussion of that proposal, the OCI Technical Oversight Board felt that oci-image-tool (which had not received any real development work for a few years at that point) should probably be archived in favour of umoci. But, there were a few things that have delayed this effort.

So, what up with image-spec 1.1?

So far I have neglected to mention that there is one feature oci-image-tool provides that umoci currently cannot do – validating that an image conforms to the specification.

For background, ever since I first started developing umoci, I always wanted to make sure we had strict validation of our generated images so that there would be no question that our project was producing valid images. Even today, most of the run time of our test suite is running image-spec validation against our images at every stage of our tests. And the only tool that currently exists for image-spec validation is oci-image-tool validate.

Unfortunately, those exact validation tools have not been actively developed for many years and cannot handle version 1.1 of the image specification. In fact, the validator will completely refuse to even try to validate an image if it has a different version to the one it supports. And, as discussed above, the rest of the features of image-tools have been deprecated by the existence of umoci. The codebase is so old that we have to hack around the repo in our CI in order to build it in a form that can actually validate images (it’s still using Godep, for the few old-timers who remember what that is…).

The talks about archiving oci-image-tool concluded that it would also be necessary to split out oci-image-tool validate and merge it into umoci, so that umoci could do validation of images. This would allow us to entirely deprecate oci-image-tool in favour of umoci, and the repo could then be archived.

However, in practice this will require rewriting most (if not all) of the validation code just for umoci’s test suite – and there is a decent argument to make that having the validation code live in the same repo as the implementation is not nearly as useful a check for bugs in our implementation. I would also want to be able to validate against multiple versions of the image-spec, which (as mentioned above) is something that oci-image-tool validate does not support.

What should we do?

I’m still of two minds about what I should do. I think it’s important to have proper validation code for specifications, but at this point it seems overwhelmingly obvious that umoci is the only project actually validating their image-spec images and I don’t really want to take on even more maintainership burden for code that it seems nobody except us uses. But I also don’t want to just remove the validation we do in our tests – what if we subtly break something but in a way that umoci cannot detect? And finally, not doing anything is what leads to 4-year gaps between releases.

A similar story is true for the oci-runtime-tools project we use for OCI runtime specification validation, and is the reason why umoci isn’t using version 1.2.1 of the OCI runtime specification. However, this is less critical for umoci because we generate a fairly standard runtime-spec configuration, and oci-runtime-tools does still get a modest amount of ongoing development.

I’m open to suggestions, but I am leaning towards just sitting down and rewriting the validation code so we can put this whole issue to bed. Of course, when I will have enough time for what (in my view) qualifies as pure busy work is a different question…

What’s next for umoci?

As I said, there are a lot of things I would like to do with umoci, and I have recently felt more invigorated about working on it than I have for some time, but it never feels like I have enough time so knowing what to prioritise would be very useful.

If you are an umoci user, feel free to let me know what features or improvements you would prefer I focus on. While umoci is not a particularly exciting project, I’ve always felt that it was well positioned to be a dependable workhorse (and the users I know of seem to agree with that sentiment).

This blog post was originally posted on my personal blog.