The KernelCI project brings automated build and boot testing to anyone wanting to check the quality and stability of Linux kernel releases. What started as a spare time project has gradually grown into a community-driven open source project that’s used by the likes of Google and the Automotive Grade Linux project to verify the quality of the kernels included in their products. Here at BayLibre, we’ve been contributing to the KernelCI project since its founding and we routinely use it in our customer work — it’s become one of our most indispensable tools.
But the success of the project wasn’t always so certain. When the KernelCI project initially started, it was just two kernel hackers sharing bash and expect scripts to automatically apply patches and build ARM kernels. The journey from spare time tool to Linux Foundation project has been one of struggles and uncertainty.
Let’s break the evolution of the KernelCI project down into three phases.
Phase 1: Spare time project
The arm-soc tree is a Linux kernel git repository that’s used to collect patches for the kernel’s ARM sub architectures, and every release tens of developers submit pull requests to the arm-soc tree. When it was initially created in 2012, the co-maintainers (Arnd Bergmann, Olof Johansson and Kevin Hilman) had to apply patches to the tree by hand and there was no way to easily test that the patches they were merging worked correctly for the all of the 120 kernel configs. The end result was that build errors were accidentally incorporated, especially for less popular boards where it was easier for problems to go unnoticed.
Independently, both Kevin and Olof came up with the same solution to stop this from happening: a set of scripts to automate the task of applying patches and make building kernels easier. Over time, the two co-maintainers merged their scripts together to create a single collections of maintainer tools.
This was still early days though, and this embryonic version of KernelCI was missing many features that are taken for granted in a modern CI tool. For one, when the tools detected that a patch caused the kernel to stop building, the maintainers would have to manually send emails to the submitter telling them about the build failure. It was a very time-consuming process.
The next step was to reduce the effort of mailing build reports by automatically notifying authors whenever they submitted git pull requests that didn’t build across all ARM defconfigs. Once building and error reporting were automated, the arm-soc maintainers set their sights on run time testing every kernel that they built. And that meant booting kernels on real hardware.
Over the years, Kevin and Olof had collected a small pile of different ARM boards and platforms, so they diligently wrote the code to automatically boot freshly built kernels on their boards. This code uploaded kernels to the test device and booted it to a shell inside of a ramdisk. With this new functionality supported, the KernelCI code grew a new testing dimension. Now, not only could Kevin and Olof perform basic checks to ensure kernels built , but they could also ensure that those kernels would pass the slightly more strict requirement of booting to a shell. This meant that the automated testing could not only detect when patches “broke the build” but also when patches “broke the boot” too.
Around 2013, other kernel subsystem maintainers started asking for access to the KernelCI build and boot reports. Linaro engineers Tyler Baker and Milo Casagrande registered the kernelci.org domain and began work on building a website using the existing maintainer tools as a backend. To give the wider audience the information they required, a new frontend was written by Milo that would allow users to view the build and boot reports.
It was at this point that the KernelCI project as we know it today was officially born.
Phase 2: Finding a community
Fast forward to early 2016 and Kevin was now at BayLibre and continuing his work on KernelCI. All of the founding developers had changed companies and were still involved, to varying degrees, with the project in their spare time. But without corporate backing a question naturally arose: is the KernelCI project sustainable?
Now the future of the entire project was in doubt and faced a kind of existential crisis. The developers knew they would be able to allocate dedicated time to work on the KernelCI project, but only if there was enough demand. The project needed to move beyond a spare time effort into one with a self-sustaining community.
KernelCI began to see adoption beyond the arm-soc tree as other kernel maintainers began making heavy use of the project. In particular, Greg Kroah-Hartman and Sasha Levin (the stable tree maintainers) were using KernelCI to ensure that every new release they produced was just as stable as the previous versions. In addition, BayLibre productized KernelCI.org for the Automotive Grade Linux project. This proved that the KernelCI.org project was a viable choice for projects focused on market verticals such as automotive and IVI.
This was a major turning point and KernelCI became a legitimate open source project. All of the code was moved to GitHub and mailing lists were created for developers and users to report bugs and discuss new features. With the backing of various companies, things began happening much more quickly.
The increased popularity of the project meant that the feature wishlist started growing faster than the small group of developers were able to finish tasks. It turned out that there was so much interest for the project that the developer were unable to keep up. So the next phase of the project was searching for a permanent home and a source of funding.
Phase 3: Going mainstream
Once the project started to be adopted as a foundational continuous integration tool for kernel developers and stable maintainers, the KernelCI developers began looking for a way to permanently fund the work and pay for the infrastructure to take the project to the next level.
A number of methods were considered, but ultimately the project decided on working with the Linux Foundation to achieve their ambitious goals, and there’ll be an official announcement of the project joining the Linux Foundation at the upcoming OSS Summit Europe 2019 conference.
One of the issues that the re-invigorated KernelCI project wants to tackle is the fragmentation that exists in the Linux kernel testing space. There are currently dozens of kernel testing projects that duplicate work to varying degrees, and consolidating them into a single project — or establishing common test and reporting formats — would go along way towards avoiding that duplicate work.
The next chapter for KernelCI
Even though the KernelCI project has been going for seven years now, we’ve got no plans to slow down. New features are proposed all the time and the list of tasks just keeps growing.
Perhaps the biggest request the project has received is to make it easy for developers and product owners using stable kernels to switch to newer versions. Today, this is a challenge because there’s no standard way to compare the quality of two stable kernel releases, so you never know if you’re going to run into bugs when moving to a newer release. That means products ship with known bugs because developers are discouraged from updating.
If you’d like to help shape the future of Linux Kernel quality and the Continuous Integration ecosystem join the LF KernelCI project today. To find out more about the project, head on over to kernelci.org to see the latest results or join in the discussions on the mailing list.