A brief introduction to BuildStream
BuildStream is a software integration tool to integrate software stacks, or to put it more clearly, BuildStream is to several projects what a Makefile is to a single project. I have been using BuildStream for around 10 months while contributing to freedesktop-sdk, but I've never tried using it to start a project from "scratch", and what better way to use BuildStream than to build BuildStream with BuildStream?
How a BuildStream project works
A BuildStream project consists, at its core, of a file for project-wide
project.conf - and a directory for the elements you wish to
build. These elements are defined in an easy-to-understand yaml format, which
makes it much friendlier to look at than many similar tools. BuildStream runs
all builds in a sandbox, so it's necessary to either use a base image or
implement your stack from the base up. I opted to use a powerful feature of
BuildStream called junctions to get most of the dependencies I needed from
A junction is a way to point BuildStream at another BuildStream project and
simply reuse the elements defined in that project. In order to use a junction
you simply add a
junction element with the upstream project as a source, and
then get at dependencies from it by just adding an additional line when you
define your element. For example many of the elements in the bst-in-bst project
require python, which is in freedesktop-sdk. To specify this I simply specified
- filename: base/python3.bst junction: freedesktop-sdk.bst
in the .bst file for that element.
The benefits of such a feature are mainly that if there is a trusted project implementing an element already one needn't reproduce that work, and can simply use the exact element. This hugely reduces the workload required to both maintain a project and to set it up in the first place. The final build pipeline for bst-in-bst builds some 150 elements, but almost all of these are integrated already in freedesktop-sdk, and I didn't need to reinvent the wheel to get the packages I required.
Now some will be raising an eyebrow here at the 150 elements required to build BuildStream - won't that be an exceedingly long build? Thankfully BuildStream implements caching of builds out of the box. It didn't seem worth setting up a personal cache server for a small project like this, but - thanks to the cache freedesktop-sdk has - I didn't need to rebuild the whole stack of elements beneath those I added.
BuildStream is a plugin based tool, which comes with many benefits and some drawbacks. The main drawback to this is that if a junctioned project uses an additional plugin, you will need to install that plugin too. Freedesktop-sdk uses several plugins from the not-really-stable bst-external repository, so if it is used in a junction this must be installed too.
However the plugins make BuildStream very extensible and versatile, allowing multiple different formats to be built from a single project. While for bst-in-bst this wasn't really possible by the nature of BuildStream as a CLI tool, the potential for this can be seen in Valentin David's blog post in which he packages firefox as both a flatpak and a snap in a single project. This could hugely reduce the maintenance cost of projects which ship their product in many formats.
To achieve my meta goal of building BuildStream in BuildStream I simply looked at the dependencies required to build BuildStream, and added these on top of a junction from freedesktop-sdk. In testament for the ease with which a project can be put together this took me, a relatively inexperienced software integrator, only around a day. Most of this time was build time spent trying to fix a few bugs in the final image produced, as well as getting back to the dissertation this project was procrastinating.
The produced BuildStream should be capable of building almost anything thrown at it - certainly all of the core plugins will function correctly. At present some of the plugins from bst-external will not work due to a lack of host tooling in the generated image, but adding these dependencies is on my todo list.
To test that the BuildStream built in BuildStream was fully functional I had
only one choice. I exported the sysroot generated into a docker image, and used
this to build BuildStream in a BuildStream built by BuildStream. After a few
integration bugs this worked, allowing me to
bst shell into the produced
artifact inside the docker image, and run
bash-4.4# bst shell buildstream.bst [...] [email@example.com:/]$ bst --version 1.2.5
See the project here
Running the build took about 40 minutes in a docker image on a refurbished Thinkpad x240, most of which was bottlenecked on Google's protobuf and grpcio.
NOTE: Unfortunately not all of the integration problems are solved as of
yet, it seems that
/tmp is not kept during the checkout of the artifact, so
before using BuildStream in the docker image you need to manually create this