The BayLibre LAVA Box: An Embedded Linux CI Lab Inside a PC Case

At BayLibre, we have the pleasure of working with the world’s best OEMs and silicon vendors on a wide range of projects. We get to play with the latest technology to ensure that it’s supported in the upstream Linux kernel, which means we use a lot of development boards every day.

And as any embedded Linux developer will tell you, it can be a real pain to work with multiple boards while you’re writing and testing code. Connecting up new boards, power-cycling, plugging in peripherals. Doing things by hand consumes a lot of time. That’s time better spent fixing bugs and writing new features. Not to mention the office mess that occurs when you’ve got over 10 boards connected with serial, power, and USB peripherals.

To illustrate the point, here’s a picture of Kevin Hilman’s home lab. It shows how quickly things can get out of hand when you’re doing Continuous Integration (CI) and helping kernelci.org to perform 2,680 boots of the latest kernel changes across 271 unique boards every day.

This got us thinking: is there a way to contain the mess and simplify administration of our boards? Fitting everything inside a single case was a hard requirement since some of our staff work from home and don’t have the space to house industrial-grade racks. We also wanted to encourage new kernelci.org labs and have something that would provide a foundation for custom CI solutions for our partners.

For a BOM cost of under €400 (for brand-new components), we built an entire lab inside of a PC case that ticked all of these boxes and validated our idea. Kevin Hilman and Patrick Titiano unveiled the project at Embedded Linux Conference Europe 2017. We call it the “LAVA Box”.

 

LAVA Box Design

Because the LAVA box provides an entire CI solution, it contains both hardware and software parts. The software portion is made up of LAVA server (master) and dispatcher (slave). The server provides a web interface and handles things like job scheduling and priorities, and board description files which greatly simplifies administration.

The dispatcher manages connections between the boards and the real world. It provides services like DHCP, TFTP, NFS, etc, handles USB fastboot and mass storage, and power switching for all devices under test (DUT). Serial connections are handled by the dispatcher and udev rules are used to ensure that serial consoles have the same device path every time they come up. Once you’ve connected your devices and written the description files, the day-to-day tasks are completely automated.

Both the server and dispatcher run inside of individual containers which are managed with docker-compose. While it’s possible to run the two containers separately for a scaled-out lab, we host everything on one machine for the LAVA Box proof of concept.

On the hardware side, everything is neatly housed inside a PC case. Boards are mounted to the case’s drive bays which makes it simple to swap them out for maintenance or administration. A standard PC PSU is able to supply enough power rails (molex/ATX connections) for the host computer and 5 DUTs with connectivity (network, power control, USB, etc). It would be possible to power even more devices with a more powerful PSU and a bigger case (or if you’re willing to run cables outside of the case).

We found that the +5v and +12v rails from a standard ATX PSU worked well for powering our DUTs. Though the lack of configurable voltages is a minor limitation, we were able to live with it. A critical piece of functionality is power switching for DUTs and there are lots of inexpensive options available on the market, such as USB-controlled relays. We used the fully-open ACME Cape because, apart from controlling power, it also measures power consumption.

All the DUTs communicate on an 8-port switch and sit on an internal network that is isolated from the outside world. Even though the LAVA box master requires Internet access in our configuration (to pull down CI jobs from kernelci.org), it is possible to run in a LAN-only mode for local jobs.

Our prototype is just one possible way to configure the LAVA Box. Due to the modularity of the design, and the open documentation and software, you can build your own “lab in a box” using different hardware components and software provided it uses the kernelci API.

The Automotive Grade Linux project has taken advantage of this flexibility to build their own LAVA box. We improved the KernelCI API to level-up the quality of their releases and the new API allows AGL to run platform, AGL-specific, and automotive-specific tests everytime new code is committed. In a group with over 100 member companies, scaling engineering effort is super important and our automation and CI work has significantly improved the overall quality of the AGL CI loop.

 

The hard things about building labs

It took us a long time to build our LAVA box prototype. But we learned some valuable lessons along the way that will save you time and energy if you want to build your own.

1. You need to balance DUT power consumption

One of our wishlist items was more power rails. This doesn’t necessarily mean we want more power, simply that we want more rails to connect more DUTs. In the future, we’ll move to a more powerful ATX unit since they usually come with an increase in the number of connectors.

2. Cheap USB cables are flaky

There are lots of brands that produce low-cst USB serial cables. Few of them work for very long. We had many issues with connectors only working for a short period of time and requiring unplugging before they’d work again. We ended up settling on FTDI cables as we found them to be much more reliable than other brands.

3. There’s not a lot of space inside a PC case

Things are tightly packed inside the case which means that cooling may become an issue if your DUTs run hot. We specifically purchased a PC case with front and rear fans to keep the internal temperature down.

4. There’s no standard DC power jack

Every DUT has its own connector, so you need to be comfortable pulling out the soldering iron to connect the power. This obviously isn’t for everyone and right now we don’t have a good solution for this. Hopefully this problem will eventually be solved by increased adoption of USB Power Delivery via Type-C connectors, but for now, board rework is often required.

 

Future plans for the LAVA box

We poured all of our embedded systems experience into designing and building the LAVA box and validating our idea. Based on the feedback we’ve received from AGL, and from what we’ve seen in our own labs, it’s been a roaring success. The LAVA box has productized the CI workflow for embedded Linux.

We’re not done yet and we have some more tweaks in mind for the current design such as improving connectivity by adding Wi-Fi and Bluetooth. Ater that, we’re going to take the things we learned and address new scenarios like building a cost-effective solution for single-board labs and scaling up to professional-grade labs in racks. That way, labs of all sizes can benefit from the power of CI.

If you want to join in you can start by reading the documentation and contributing to the KernelCI project on GitHub. This entire project is open source and community contributions are greatly appreciated.

Or perhaps you want to talk to us about building your own lab in a box? Reach us at contact@baylibre.com to see how we can help!

Kernel CI: Linux Kernel Testing at kernelci.org

The kernelci.org project aims to improve the quality of the mainline Linux kernel by improving testing and validation across the wide variety of diverse hardware platforms that run Linux.

There are so many different devices and platforms that run Linux, and Linux kernel development is moving so quickly that it is difficult to ensure that any given platform will remain working and stable with each Linux version.  As an example, the chart here shows the growth in the number of 32-bit ARM based devices supported by Linux, with the total number of unique devices as of v4.11 just shy of 1400!  That doesn’t even count the growing number of 64-bit ARM devices or any of the other architectures like x86 or MIPS.

With such an incredible range of supported hardware, how can the Linux kernel community continue to ensure that all of this hardware remains well supported and evolves with the rest of the Linux kernel?

The kernelci.org project set out to help solve that problem.

During the development cycle of the Linux kernel, whenever there are changes to the source-code repository, the kernel is built in a wide variety of configurations for several different architectures. Today, there are over 270 different build configurations across 4 architectures (x86, MIPS, ARM and ARM64.)

After a successful build, the kernel images are made available to the several distributed labs for testing. Due to the diversity of hardware that runs linux, no one lab is going to have all the hardware, so kernelci.org was designed for distributed testing. When builds are completed, each lab can download the images for the hardware available, and perform the testing. Currently there are 8 active labs contributing a total of more than 250 unique hardware platforms across 4 unique architectures.

BayLibre’s Kevin Hilman is a founding developer of the kernelci.org project, and today, BayLibre has the largest lab contributing results from over 80 unique boards across 25 unique SoC families and performing thousands of tests each day.

If you have hardware you’d like to see tested with the latest Linux kernel in the kernelci.org project, feel free to contact us.  We can help guide you through setting up your own lab, or you could just send us your hardware and we can add it to our lab.

Want to know even more?

For a more in-depth overview, Kevin gave an overview talk of the kernelci.org project at the 2016 edition of the Kernel Recipes conference in Paris.  Slides are available online and the full talk was recorded and available right here:

 

 

 

Nexbox A1 serial console

The Nexbox A1 which includes an 8-core Amlogic S912 processor, is now supported in v4.10 of the Linux kernel, thanks in part to the work of BayLibre.

If you’d like to help with kernel development on this platform, the first think you’ll need is access to the serial console.  The serial port is not brought out to a connector, but pads are easily accessible on the main board.

Once you open the case, you’ll pads for the UART signals between the heat sink and the edge of the board.

In the photo to the right, wires have been soldered to the pads:

  • Red: VCC (not used)
  • Orange: RX
  • Yellow: TX
  • Black: Ground

Hooking the newly soldered wires up to a USB serial cable such as this one, you’ll see the boot-loader and linux kernel messages on your as soon as you power on the board.  NOTE: signal levels are 3.3V, and no need to hook up the VCC line.

Now you’ll be ready to dive in and help with kernel development on Amlogic processors.   Enjoy!