Announcing the ACME Firmware Beta 3 Release

On 8th June 2018, BayLibre released a new version of the ACME Firmware, Beta 3. This important release contains a fix for a critical issue that can cause incorrect reporting of current and voltage data.

We recommend that everyone upgrade to this latest release right away.

A high-level summary of the changes can be found in the changelog, but in the rest of this article we’ll look at them in more detail.


Fix for Incorrect Shunt Resistor Values

The Beta 3 release includes a fix for a major bug that can manifest as incorrect current and voltage data being reported at the application level.

The issue is that shunt resistor values are incorrectly copied from the probe EEPROM when creating entries in the Linux sysfs file system. Depending on the configuration of inserted probes, the value from the probe in the first slot may actually be copied to the sysfs entry for the second slot, the value from the probe in the second slot may be copied to the sysfs entry for the third slot, and so on. In other words, it’s possible for all sysfs entries to contain incorrect shunt resistor values.

The bug is caused by a mismatch between the way probes are numbered by ‘dut-dump-probe’ (starting at 1) and the way the OS enumerates devices in sysfs (starting at 0). Because of this off-by-one bug, it’s impossible to initialize the shunt resistor value for the first slot on the ACME cape, and the default value of 10mohm is always used. Note that users of JACK probes may not be affected by this issue since the shunt resistor value for this probe happens to be identical to the default value.


An Upgrade to the Latest Yocto Krogoth Release

We’ve also upgraded to the latest version of the community-maintained Yocto Krogoth release (2.1.3), which includes both functional and security fixes, bringing the latest software updates to ACME users.

You can find instructions for downloading the ACME Firmware Beta 3 release here.

The BayLibre LAVA Box: An Embedded Linux CI Lab Inside a PC Case

At BayLibre, we have the pleasure of working with the world’s best OEMs and silicon vendors on a wide range of projects. We get to play with the latest technology to ensure that it’s supported in the upstream Linux kernel, which means we use a lot of development boards every day.

And as any embedded Linux developer will tell you, it can be a real pain to work with multiple boards while you’re writing and testing code. Connecting up new boards, power-cycling, plugging in peripherals. Doing things by hand consumes a lot of time. That’s time better spent fixing bugs and writing new features. Not to mention the office mess that occurs when you’ve got over 10 boards connected with serial, power, and USB peripherals.

To illustrate the point, here’s a picture of Kevin Hilman’s home lab. It shows how quickly things can get out of hand when you’re doing Continuous Integration (CI) and helping to perform 2,680 boots of the latest kernel changes across 271 unique boards every day.

This got us thinking: is there a way to contain the mess and simplify administration of our boards? Fitting everything inside a single case was a hard requirement since some of our staff work from home and don’t have the space to house industrial-grade racks. We also wanted to encourage new labs and have something that would provide a foundation for custom CI solutions for our partners.

For a BOM cost of under €400 (for brand-new components), we built an entire lab inside of a PC case that ticked all of these boxes and validated our idea. Kevin Hilman and Patrick Titiano unveiled the project at Embedded Linux Conference Europe 2017. We call it the “LAVA Box”.


LAVA Box Design

Because the LAVA box provides an entire CI solution, it contains both hardware and software parts. The software portion is made up of LAVA server (master) and dispatcher (slave). The server provides a web interface and handles things like job scheduling and priorities, and board description files which greatly simplifies administration.

The dispatcher manages connections between the boards and the real world. It provides services like DHCP, TFTP, NFS, etc, handles USB fastboot and mass storage, and power switching for all devices under test (DUT). Serial connections are handled by the dispatcher and udev rules are used to ensure that serial consoles have the same device path every time they come up. Once you’ve connected your devices and written the description files, the day-to-day tasks are completely automated.

Both the server and dispatcher run inside of individual containers which are managed with docker-compose. While it’s possible to run the two containers separately for a scaled-out lab, we host everything on one machine for the LAVA Box proof of concept.

On the hardware side, everything is neatly housed inside a PC case. Boards are mounted to the case’s drive bays which makes it simple to swap them out for maintenance or administration. A standard PC PSU is able to supply enough power rails (molex/ATX connections) for the host computer and 5 DUTs with connectivity (network, power control, USB, etc). It would be possible to power even more devices with a more powerful PSU and a bigger case (or if you’re willing to run cables outside of the case).

We found that the +5v and +12v rails from a standard ATX PSU worked well for powering our DUTs. Though the lack of configurable voltages is a minor limitation, we were able to live with it. A critical piece of functionality is power switching for DUTs and there are lots of inexpensive options available on the market, such as USB-controlled relays. We used the fully-open ACME Cape because, apart from controlling power, it also measures power consumption.

All the DUTs communicate on an 8-port switch and sit on an internal network that is isolated from the outside world. Even though the LAVA box master requires Internet access in our configuration (to pull down CI jobs from, it is possible to run in a LAN-only mode for local jobs.

Our prototype is just one possible way to configure the LAVA Box. Due to the modularity of the design, and the open documentation and software, you can build your own “lab in a box” using different hardware components and software provided it uses the kernelci API.

The Automotive Grade Linux project has taken advantage of this flexibility to build their own LAVA box. We improved the KernelCI API to level-up the quality of their releases and the new API allows AGL to run platform, AGL-specific, and automotive-specific tests everytime new code is committed. In a group with over 100 member companies, scaling engineering effort is super important and our automation and CI work has significantly improved the overall quality of the AGL CI loop.


The hard things about building labs

It took us a long time to build our LAVA box prototype. But we learned some valuable lessons along the way that will save you time and energy if you want to build your own.

1. You need to balance DUT power consumption

One of our wishlist items was more power rails. This doesn’t necessarily mean we want more power, simply that we want more rails to connect more DUTs. In the future, we’ll move to a more powerful ATX unit since they usually come with an increase in the number of connectors.

2. Cheap USB cables are flaky

There are lots of brands that produce low-cst USB serial cables. Few of them work for very long. We had many issues with connectors only working for a short period of time and requiring unplugging before they’d work again. We ended up settling on FTDI cables as we found them to be much more reliable than other brands.

3. There’s not a lot of space inside a PC case

Things are tightly packed inside the case which means that cooling may become an issue if your DUTs run hot. We specifically purchased a PC case with front and rear fans to keep the internal temperature down.

4. There’s no standard DC power jack

Every DUT has its own connector, so you need to be comfortable pulling out the soldering iron to connect the power. This obviously isn’t for everyone and right now we don’t have a good solution for this. Hopefully this problem will eventually be solved by increased adoption of USB Power Delivery via Type-C connectors, but for now, board rework is often required.


Future plans for the LAVA box

We poured all of our embedded systems experience into designing and building the LAVA box and validating our idea. Based on the feedback we’ve received from AGL, and from what we’ve seen in our own labs, it’s been a roaring success. The LAVA box has productized the CI workflow for embedded Linux.

We’re not done yet and we have some more tweaks in mind for the current design such as improving connectivity by adding Wi-Fi and Bluetooth. Ater that, we’re going to take the things we learned and address new scenarios like building a cost-effective solution for single-board labs and scaling up to professional-grade labs in racks. That way, labs of all sizes can benefit from the power of CI.

If you want to join in you can start by reading the documentation and contributing to the KernelCI project on GitHub. This entire project is open source and community contributions are greatly appreciated.

Or perhaps you want to talk to us about building your own lab in a box? Reach us at to see how we can help!

ACME and pyacmegraph – Part 2 / 2

This is the second (and final) part of my series on ACME and pyacmegraph.

For the 1st post, see here: ACME and pyacmegraph – part 1 / 2

In this post I will detail pyacmegraph features and functioning.

Read more

ACME and pyacmegraph – Part 1 / 2

A couple of weeks ago, we made public the ‘pyacmegraph’ tool that might interest you if you use our ACME probes for power management study or debug.

This tool handles ACME probes data capture and display, with some fancy features added.
Simply put, it’s an easy to use tool that aims at getting the best of ACME for studying a device power consumption.

In this post I will introduce pyacmegraph and explain how ACME is used to measure power and send this information to pyacmegraph. In a subsequent post, I will detail pyacmegraph features and functionning.

Read more

BayLibre is attending Embedded Linux Conference at San Jose

BayLibre is attending ELC at San Jose this week.

Bartosz will have a talk: Sigrok: Adventures in Integrating a Power-Measurement Device.

Later we will show some ACME HW during Demo Showcase and Booth Crawl.

Last but not least, BayLibre is hiring, so if you want to meet us, we will be around.

See you there!

The BayLibre Team

BayLibre ACME supported by sigrok

Thanks to Bartosz and Uwe Hermann, libsigrok now supports the BayLibre ACME device.

The ACME (Another Cute Measurement Equipment) is a BeagleBone Black Cape with an I²C-attached Texas Instruments INA226 current/power monitor and an I²C-attached TI TMP435 temperature sensor.

The sensors are supported in mainline Linux. The drivers expose a standard interface via the Linux sysfs pseudo file system, which the libsigrok driver uses.

Bartosz will also present a Sigrok: Adventures in Integrating a Power-Measurement Device talk at the Embedded Linux Conference on March 24, 2015 (schedule) in San Jose, CA.