The history of the KernelCI project

The KernelCI project brings automated build and boot testing to anyone wanting to check the quality and stability of Linux kernel releases. What started as a spare time project has gradually grown into a community-driven open source project that’s used by the likes of Google and the Automotive Grade Linux project to verify the quality of the kernels included in their products. Here at BayLibre, we’ve been contributing to the KernelCI project since its founding and we routinely use it in our customer work — it’s become one of our most indispensable tools.

But the success of the project wasn’t always so certain. When the KernelCI project initially started, it was just two kernel hackers sharing bash and expect scripts to automatically apply patches and build ARM kernels. The journey from spare time tool to Linux Foundation project has been one of struggles and uncertainty.

Let’s break the evolution of the KernelCI project down into three phases.

Phase 1: Spare time project

The arm-soc tree is a Linux kernel git repository that’s used to collect patches for the kernel’s ARM sub architectures, and every release tens of developers submit pull requests to the arm-soc tree. When it was initially created in 2012, the co-maintainers (Arnd Bergmann, Olof Johansson and Kevin Hilman) had to apply patches to the tree by hand and there was no way to easily test that the patches they were merging worked correctly for the all of the 120 kernel configs. The end result was that build errors were accidentally incorporated, especially for less popular boards where it was easier for problems to go unnoticed.

Independently, both Kevin and Olof came up with the same solution to stop this from happening: a set of scripts to automate the task of applying patches and make building kernels easier. Over time, the two co-maintainers merged their scripts together to create a single collections of maintainer tools.

This was still early days though, and this embryonic version of KernelCI was missing many features that are taken for granted in a modern CI tool. For one, when the tools detected that a patch caused the kernel to stop building, the maintainers would have to manually send emails to the submitter telling them about the build failure. It was a very time-consuming process.

The next step was to reduce the effort of mailing build reports by automatically notifying authors whenever they submitted git pull requests that didn’t build across all ARM defconfigs. Once building and error reporting were automated, the arm-soc maintainers set their sights on run time testing every kernel that they built. And that meant booting kernels on real hardware.

Over the years, Kevin and Olof had collected a small pile of different ARM boards and platforms, so they diligently wrote the code to automatically boot freshly built kernels on their boards. This code uploaded kernels to the test device and booted it to a shell inside of a ramdisk. With this new functionality supported, the KernelCI code grew a new testing dimension. Now, not only could Kevin and Olof perform basic checks to ensure kernels built , but they could also ensure that those kernels would pass the slightly more strict requirement of booting to a shell. This meant that the automated testing could not only detect when patches “broke the build” but also when patches “broke the boot” too.

Around 2013, other kernel subsystem maintainers started asking for access to the KernelCI build and boot reports. Linaro engineers Tyler Baker and Milo Casagrande registered the kernelci.org domain and began work on building a website using the existing maintainer tools as a backend. To give the wider audience the information they required, a new frontend was written by Milo that would allow users to view the build and boot reports. 

It was at this point that the KernelCI project as we know it today was officially born.

Phase 2: Finding a community

Fast forward to early 2016 and Kevin was now at BayLibre and continuing his work on KernelCI. All of the founding developers had changed companies and were still involved, to varying degrees, with the project in their spare time. But without corporate backing a question naturally arose: is the KernelCI project sustainable?

Now the future of the entire project was in doubt and faced a kind of existential crisis. The developers knew they would be able to allocate dedicated time to work on the KernelCI project, but only if there was enough demand. The project needed to move beyond a spare time effort into one with a self-sustaining community.

KernelCI began to see adoption beyond the arm-soc tree as other kernel maintainers began making heavy use of the project. In particular, Greg Kroah-Hartman and Sasha Levin (the stable tree maintainers) were using KernelCI to ensure that every new release they produced was just as stable as the previous versions. In addition, BayLibre productized KernelCI.org for the Automotive Grade Linux project. This proved that the KernelCI.org project was a viable choice for projects focused on market verticals such as automotive and IVI.

This was a major turning point and KernelCI became a legitimate open source project. All of the code was moved to GitHub and mailing lists were created for developers and users to report bugs and discuss new features. With the backing of various companies, things began happening much more quickly.

The increased popularity of the project meant that the feature wishlist started growing faster than the small group of developers were able to finish tasks. It turned out that there was so much interest for the project that the developer were unable to keep up. So the next phase of the project was searching for a permanent home and a source of funding.

Phase 3: Going mainstream

Once the project started to be adopted as a foundational continuous integration tool for kernel developers and stable maintainers, the KernelCI developers began looking for a way to permanently fund the work and pay for the infrastructure to take the project to the next level.

A number of methods were considered, but ultimately the project decided on working with the Linux Foundation to achieve their ambitious goals, and there’ll be an official announcement of the project joining the Linux Foundation at the upcoming OSS Summit Europe 2019 conference.

One of the issues that the re-invigorated KernelCI project wants to tackle is the fragmentation that exists  in the Linux kernel testing space. There are currently dozens of kernel testing projects that duplicate work to varying degrees, and consolidating them into a single project — or establishing common test and reporting formats — would go along way towards avoiding that duplicate work.

The next chapter for KernelCI

Even though the KernelCI project has been going for seven years now, we’ve got no plans to slow down. New features are proposed all the time and the list of tasks just keeps growing.

Perhaps the biggest request the project has received is to make it easy for developers and product owners using stable kernels to switch to newer versions. Today, this is a challenge because there’s no standard way to compare the quality of two stable kernel releases, so you never know if you’re going to run into bugs when moving to a newer release. That means products ship with known bugs because developers are discouraged from updating.

If you’d like to help shape the future of Linux Kernel quality and the Continuous Integration ecosystem join the LF KernelCI project today. To find out more about the project, head on over to kernelci.org to see the latest results or join in the discussions on the mailing list.

KernelCI Joins Linux Foundation

We’re thrilled to announce that the KernelCI project has officially joined the Linux Foundation and that BayLibre is a founding member company along with Google, Collabora, Red Hat, Foundries.io and Microsoft. BayLibre’s Kevin Hilman was one of the original founders of kernelci.org in 2014, and since then Kevin and the BayLibre team have continued to contribute changes. This announcement is a milestone for the project, and the support from the Linux Foundation demonstrates how essential automated testing at scale is for the Linux kernel community.

KernelCI is a true battle-tested open-source project and is a vital tool for the upstream Linux kernel maintainers, allowing them to merge large numbers of changes during the merge window while still maintaining quality and stability. Thanks to the collection of board farms located in hardware labs around the world, new changes are boot and runtime tested across a wide range of devices and hardware, and results are checked via the KernelCI infrastructure before being applied to the official maintainer repositories.

We think this is the start of a really exciting time for KernelCI. With the help of the Linux Foundation, infrastructure costs will be covered and the overall operation of the service will be improved. Which means the project’s members will be able to focus on new ways to grow the testing service and make it even better.

We’re looking forward to the next stage of the KernelCI project, and continuing to help increase the long-term maintainability of the Linux kernel.

If you want to learn more about KernelCI, including how you can get involved, check out the kernelci.org website and the mailing list.

We’re a Top Contributor to Automotive Grade Linux

At the Automotive Grade Linux (AGL) All-Member Meeting in Dresden last month, a list of top contributors to the project was announced which included both BayLibre and our very own Kevin Hilman. Walt Miner presented slides that showed the top individual contributors along with the top contributions broken down by company. Kevin is a top 20 contributor to the project, clocking in 26 commits in the first three quarters of the year, and BayLibre is in ninth place (with 45 commits) on the companies list.

We’ve had the pleasure of working closely with the AGL project over the last couple of years. Primarily, we’ve enhanced the Kernel CI project for use with AGL, but our AGL contributions have covered everything from Kernel CI backend support to new tests.

Since the beginning of 2018 the team at BayLibre have contributed:

  • Yocto-ptest fixes and improvements
  • CAN bus tests fixes and improvements
  • Changes to add USB host controller emulation to the QEMU template
  • A new layer for energy aware (EAS) scheduling patches

Plus, we offered the use of our own version of the AGL backend (running from inside our lab) while the official AGL backend was under development and transitioning to kernelci-docker code.

It’s fantastic to see the recognition for our hard work and to be listed alongside such impressive engineers and companies. We’re looking forward to the continued success of the AGL project and the part we get to play in that.

kernelCI: Linux Kernel Testing at kernelci.org

The kernelci.org project aims to improve the quality of the mainline Linux kernel by improving testing and validation across the wide variety of diverse hardware platforms that run Linux.

There are so many different devices and platforms that run Linux, and Linux kernel development is moving so quickly that it is difficult to ensure that any given platform will remain working and stable with each Linux version.  As an example, the chart here shows the growth in the number of 32-bit ARM based devices supported by Linux, with the total number of unique devices as of v4.11 just shy of 1400!  That doesn’t even count the growing number of 64-bit ARM devices or any of the other architectures like x86 or MIPS.

With such an incredible range of supported hardware, how can the Linux kernel community continue to ensure that all of this hardware remains well supported and evolves with the rest of the Linux kernel?

The kernelci.org project set out to help solve that problem.

During the development cycle of the Linux kernel, whenever there are changes to the source-code repository, the kernel is built in a wide variety of configurations for several different architectures. Today, there are over 270 different build configurations across 4 architectures (x86, MIPS, ARM and ARM64.)

After a successful build, the kernel images are made available to the several distributed labs for testing. Due to the diversity of hardware that runs linux, no one lab is going to have all the hardware, so kernelci.org was designed for distributed testing. When builds are completed, each lab can download the images for the hardware available, and perform the testing. Currently there are 8 active labs contributing a total of more than 250 unique hardware platforms across 4 unique architectures.

BayLibre’s Kevin Hilman is a founding developer of the kernelci.org project, and today, BayLibre has the largest lab contributing results from over 80 unique boards across 25 unique SoC families and performing thousands of tests each day.

If you have hardware you’d like to see tested with the latest Linux kernel in the kernelci.org project, feel free to contact us.  We can help guide you through setting up your own lab, or you could just send us your hardware and we can add it to our lab.

Want to know even more?

For a more in-depth overview, Kevin gave an overview talk of the kernelci.org project at the 2016 edition of the Kernel Recipes conference in Paris.  Slides are available online and the full talk was recorded and available right here:

 

 

 

Nexbox A1 serial console

The Nexbox A1 which includes an 8-core Amlogic S912 processor, is now supported in v4.10 of the Linux kernel, thanks in part to the work of BayLibre.

If you’d like to help with kernel development on this platform, the first think you’ll need is access to the serial console.  The serial port is not brought out to a connector, but pads are easily accessible on the main board.

Once you open the case, you’ll pads for the UART signals between the heat sink and the edge of the board.

In the photo to the right, wires have been soldered to the pads:

  • Red: VCC (not used)
  • Orange: RX
  • Yellow: TX
  • Black: Ground

Hooking the newly soldered wires up to a USB serial cable such as this one, you’ll see the boot-loader and linux kernel messages on your as soon as you power on the board.  NOTE: signal levels are 3.3V, and no need to hook up the VCC line.

Now you’ll be ready to dive in and help with kernel development on Amlogic processors.   Enjoy!

Kernel Recipes 2016


BayLibre was proud to be a sponsor of this this years Kernel Recipes conference in Paris.  Kernel Recipes is a small, technical conference focused on various topics related to the Linux Kernel.

I was a speaker again this year, and  gave a talk about the kernelci.org project.  The talk was a brief overview of the project, its history and ways to contribute.   Slides available here and video recordings are also available.

One of the fun parts of Kernel Recipes was all the speakers and some lucky audience members got caricatures of them drawn by artist Frank Tizzoni.  Here is one of me and one of the crowd, with a bunch more available on Kernel Recipes Twitter feed.

 

BayLibre contributes to kernelci.org

The BayLibre Lab is online and test results are being contributed to the kernelci.org project. In addition to standard boot tests, BayLibre is
developing a power regression test suite using LAVA and the kernelci.org infrastructure. We can’t wait to push the state of the art forward
around automated testing and quality assurance for Linux.
Marc has led the effort to bring the BayLibre Lab online. He has been supported by Kevin, Tyler, Neil and Milo from Linaro who patiently answered all of our questions about setting up the LAVA environment and integrating it with the kernelci.org project. Thanks again everyone.

Our first results are available here : http://kernelci.org/boot/all/lab/lab-baylibre/