BuildStream in BuildStream

A brief introduction to BuildStream

BuildStream is a software integration tool to integrate software stacks, or to put it more clearly, BuildStream is to several projects what a Makefile is to a single project. I have been using BuildStream for around 10 months while contributing to freedesktop-sdk, but I've never tried using it to start a project from "scratch", and what better way to use BuildStream than to build BuildStream with BuildStream?

How a BuildStream project works

A BuildStream project consists, at its core, of a file for project-wide configuration - project.conf - and a directory for the elements you wish to build. These elements are defined in an easy-to-understand yaml format, which makes it much friendlier to look at than many similar tools. BuildStream runs all builds in a sandbox, so it's necessary to either use a base image or implement your stack from the base up. I opted to use a powerful feature of BuildStream called junctions to get most of the dependencies I needed from freedesktop-sdk.

Junctions

A junction is a way to point BuildStream at another BuildStream project and simply reuse the elements defined in that project. In order to use a junction you simply add a junction element with the upstream project as a source, and then get at dependencies from it by just adding an additional line when you define your element. For example many of the elements in the bst-in-bst project require python, which is in freedesktop-sdk. To specify this I simply specified

- filename: base/python3.bst
  junction: freedesktop-sdk.bst

in the .bst file for that element.

The benefits of such a feature are mainly that if there is a trusted project implementing an element already one needn't reproduce that work, and can simply use the exact element. This hugely reduces the workload required to both maintain a project and to set it up in the first place. The final build pipeline for bst-in-bst builds some 150 elements, but almost all of these are integrated already in freedesktop-sdk, and I didn't need to reinvent the wheel to get the packages I required.

Caching

Now some will be raising an eyebrow here at the 150 elements required to build BuildStream - won't that be an exceedingly long build? Thankfully BuildStream implements caching of builds out of the box. It didn't seem worth setting up a personal cache server for a small project like this, but - thanks to the cache freedesktop-sdk has - I didn't need to rebuild the whole stack of elements beneath those I added.

Plugins

BuildStream is a plugin based tool, which comes with many benefits and some drawbacks. The main drawback to this is that if a junctioned project uses an additional plugin, you will need to install that plugin too. Freedesktop-sdk uses several plugins from the not-really-stable bst-external repository, so if it is used in a junction this must be installed too.

However the plugins make BuildStream very extensible and versatile, allowing multiple different formats to be built from a single project. While for bst-in-bst this wasn't really possible by the nature of BuildStream as a CLI tool, the potential for this can be seen in Valentin David's blog post in which he packages firefox as both a flatpak and a snap in a single project. This could hugely reduce the maintenance cost of projects which ship their product in many formats.

BuildStream in BuildStream

To achieve my meta goal of building BuildStream in BuildStream I simply looked at the dependencies required to build BuildStream, and added these on top of a junction from freedesktop-sdk. In testament for the ease with which a project can be put together this took me, a relatively inexperienced software integrator, only around a day. Most of this time was build time spent trying to fix a few bugs in the final image produced, as well as getting back to the dissertation this project was procrastinating.

The produced BuildStream should be capable of building almost anything thrown at it - certainly all of the core plugins will function correctly. At present some of the plugins from bst-external will not work due to a lack of host tooling in the generated image, but adding these dependencies is on my todo list.

To test that the BuildStream built in BuildStream was fully functional I had only one choice. I exported the sysroot generated into a docker image, and used this to build BuildStream in a BuildStream built by BuildStream. After a few integration bugs this worked, allowing me to bst shell into the produced artifact inside the docker image, and run bst --version:

bash-4.4# bst shell buildstream.bst
[...]
[bf20321f@buildstream.bst:/]$ bst --version
1.2.5

See the project here

Running the build took about 40 minutes in a docker image on a refurbished Thinkpad x240, most of which was bottlenecked on Google's protobuf and grpcio.

NOTE: Unfortunately not all of the integration problems are solved as of yet, it seems that /tmp is not kept during the checkout of the artifact, so before using BuildStream in the docker image you need to manually create this directory.