We’re a Top Contributor to Automotive Grade Linux

At the Automotive Grade Linux (AGL) All-Member Meeting in Dresden last month, a list of top contributors to the project was announced which included both BayLibre and our very own Kevin Hilman. Walt Miner presented slides that showed the top individual contributors along with the top contributions broken down by company. Kevin is a top 20 contributor to the project, clocking in 26 commits in the first three quarters of the year, and BayLibre is in ninth place (with 45 commits) on the companies list.

We’ve had the pleasure of working closely with the AGL project over the last couple of years. Primarily, we’ve enhanced the Kernel CI project for use with AGL, but our AGL contributions have covered everything from Kernel CI backend support to new tests.

Since the beginning of 2018 the team at BayLibre have contributed:

  • Yocto-ptest fixes and improvements
  • CAN bus tests fixes and improvements
  • Changes to add USB host controller emulation to the QEMU template
  • A new layer for energy aware (EAS) scheduling patches

Plus, we offered the use of our own version of the AGL backend (running from inside our lab) while the official AGL backend was under development and transitioning to kernelci-docker code.

It’s fantastic to see the recognition for our hard work and to be listed alongside such impressive engineers and companies. We’re looking forward to the continued success of the AGL project and the part we get to play in that.

Linux Kernel 4.19 released, our contributions

BayLibre has continued our contribution to the Linux community as seen with this new version of Linux Kernel 4.19, released on Sunday, 21th October 2018.  An excellent summary of this release can be found at KernelNewbies.

Here is a summary of our contributions, organized by SoC family and a summary graph of contributions by developer.

Amlogic SoC family:

  • Initial support of the audio hardware on the Amlogic AXG SoC : A113D
    • Audio clock controller
    • Audio reset controller
    • Pinctrl missing configurations
    • ALSA SoC card
    • TDM input/output
    • SPDIF output
    • HW FIFOs handling
    • es7134 codec support
    • tas517x support
    • DT audio and pinctrl nodes
  • Add S805X based P241 board
  • Finally enable graphics output on the FriendlyElec Nanopi-K2
  • Enable DMT (Display Monitor Timing) HDMI output to support generic, non-HDMI monitors (DVI, VGA via HDMI->VGA dongle or monitors with custom timings in EDID)

Ti DaVinci SoC Family:

  • switch davinci SoCs to use the ti-aemif driver
  • switch to the reset framework for the DSP remoteproc

Misc:

  • Lots of cleanup and some code removal from Corentin
  • Various Clock & ALSA SoC fixups to make AXG Audio work
  • Add support for the CEC functionality on the upcoming Asus ChromeBox via the Embedded Controller interface and the Linux CEC Framework maintained by Hans Verkuil

 

 

U-Boot v2018.07 released, our contributions

BayLibre has continued contribution to the open-source community as seen with this new version of U-Boot v2018.07, released on 27th July 2018.

This release permits booting over an USB Mass Storage (USB Stick or Hard Drive) with EFI on the Libre Computer AML-S905X-CC board !

With U-Boot 2018.07, you can download the nighly Debian-Installer or OpenSuse  TumbleTweed, copy U-boot to a blank SDCard, copy the installer to an USB, plug USB and SDCard, boot and install !

Here is a summary of our contributions:

Amlogic SoC family:

  • Add support for the USB PHY and Controller on the Meson GXL (S905X, S905D) SoC based on the Linux version done by Martin Blumenstingl
  • Add Analog-to-Digital (ADC) driver based on the Linux version done by Martin Blumenstingl
  • Add Amlogic Reset Controller
  • Add ADC cli command
  • Sync DT with Linux 4.17

Misc :

  • Add “bulk” commands for Reset and Clocks + tests
  • Add dwc3-of-simple USB DWC3 simple Glue driver based on the Linux version
  • Add support for any number of PHYs for the DWC3 controller

Linux Kernel 4.18 released, our contributions

BayLibre has continued our contribution to the Linux community as seen with this new version of Linux Kernel 4.18, released on Sunday, 12th August 2018.  An excellent summary of this release can be found at KernelNewbies.

Here is a summary of our contributions, organized by SoC family and a summary graph of contributions by developer.

Amlogic SoC family:

  • Multiple fixups/enhancements for the AXG A113D SoC (I2C, TDM)
  • Add write support for NVMEM
  • MMC/SDCard Fixups
  • Boot fixup with SCPI

Ti DaVinci SoC Family:

  • Fixup for NAND support
  • Fixups for RemoteProc support
  • Fixups for AEMIF support

Misc:

  • Lots of cleanup and some code removal

 

 

Linux Kernel 4.17 released, our contributions

BayLibre has continued our contribution to the Linux community as seen with this new version of Linux Kernel 4.17, released on June, 3rd 2018.  An excellent summary of this release can be found at KernelNewbies.

We’re happy to report BayLibre is once again featured in the LWN stats for this release. Corentin Labbe from our team is a Top 10 developer and BayLibre is a Top 20 employer, both by Lines Changed.

Here is a summary of our contributions, organized by SoC family and a summary graph of contributions by developer.

Amlogic SoC family:

  • clocks: major rework to switch to regmap; misc. cleanups
  • improve DT support for WeTek hub and play2
  • MMC: reduce max speed for Odroid-C2 boards after multiple problems reported
  • AXG family: enable hardware RNG
  • HDMI: Add support for DMT modes

Misc:

  • Lots of cleanup and dead code removal (see Lines removed by Corentin)

 

 

Linux Kernel 4.16 released, our contributions

Linux Kernel 4.16 released

BayLibre has continued our contribution to the Linux community as seen with this new version of Linux Kernel 4.16, released on April, 1st 2018.  An excellent summary of this release can be found at KernelNewbies.

Here is a summary of our contributions, organized by SoC family and a summary graph of contributions by developer.

Amlogic SoC family:

This release is important since it’s the first Linux release that can boot an Amlogic SoC with graphics with u-boot mainline, both unchanged !

  • Finally enable the Video Processing Unit power domain and all the missing bit :
    • drm/meson: Add missing VPU init
    • drm/meson: dw_hdmi: Add support for an optional external 5V regulator
    • ARM64: dts: odroid-c2: Add HDMI and CEC Nodes
    • ARM64: dts: meson-gx: grow reset controller memory zone
    • ARM64: dts: meson-gx: Add HDMI_5V regulator on selected boards
    • ARM64: dts: meson-gx: add VPU power domain
    • dt-bindings: display: amlogic, meson-dw-hdmi: Add optional HDMI 5V regulator
    • dt-bindings: display: amlogic, meson-vpu: Add optional power domain property
    • This means you can download Linux v4.16 and U-Boot v2018.01, build them, flash them and you will be able to boot to graphics with HDMI on most Amlogic S905, S905X and S912 supported boards !
  • drm/meson: fix vsync buffer update : fix a long-time issue causing glitches when rendering directly using GBM
  • Jerome improved the Meson S905X/S905D/S912 internal ethernet PHY support by :
    • net: phy: meson-gxl: add interrupt support
    • net: phy: meson-gxl: use genphy_config_init
    • net: phy: meson-gxl: add read and write helpers for banked registers
    • net: phy: meson-gxl: define control registers
    • net: phy: meson-gxl: check phy_write return value
    • ARM64: dts: meson-gxl: add internal ethernet PHY irq
  • Jerome added the “clock protection” feature to the clock framework

Misc:

  • Add support for the the Variscite DART-MX6 SoM and Carrier board with LVDS display
  • media: uvcvideo: Add a quirk for Generalplus Technology Inc. 808 Camera
  • ARM: davinci: fix the GPIO lookup for omapl138-hawk
  • i2c: davinci: fix the cpufreq transition
  • Corentin did some cleanup in the crypto directory and some overall remove of unused code/documentation

 

 

Zephyr 1.11.0 released, BayLibre contributions

A full changelog of this release are available on the project releases page.

For these last 3 Zephyr releases, BayLibre engineer Neil Armstrong worked to add a better support for the STM32F0 SoC family by adding SPI, I2C, and internal Flash support, following his work on the previous Zephyr release on the overall STM32 Microcontrollers.
A hard time was spent adding SPI Slave and I2C Slave support on Zephyr for the whole STM32 Microcontroller family, to align with other RTOS and Linux for the I2C Slave part.

Since Zephyr 1.8, supported in Zephyr :

Along the following STM32 Microcontrollers features :

And ongoing features still in discussions :

The last Pull Request is done in cooperation with Daniel Wagenknecht, and triggered a lot of technical discussions ! Thanks Daniel !

 

The BayLibre LAVA Box: An Embedded Linux CI Lab Inside a PC Case

At BayLibre, we have the pleasure of working with the world’s best OEMs and silicon vendors on a wide range of projects. We get to play with the latest technology to ensure that it’s supported in the upstream Linux kernel, which means we use a lot of development boards every day.

And as any embedded Linux developer will tell you, it can be a real pain to work with multiple boards while you’re writing and testing code. Connecting up new boards, power-cycling, plugging in peripherals. Doing things by hand consumes a lot of time. That’s time better spent fixing bugs and writing new features. Not to mention the office mess that occurs when you’ve got over 10 boards connected with serial, power, and USB peripherals.

To illustrate the point, here’s a picture of Kevin Hilman’s home lab. It shows how quickly things can get out of hand when you’re doing Continuous Integration (CI) and helping kernelci.org to perform 2,680 boots of the latest kernel changes across 271 unique boards every day.

This got us thinking: is there a way to contain the mess and simplify administration of our boards? Fitting everything inside a single case was a hard requirement since some of our staff work from home and don’t have the space to house industrial-grade racks. We also wanted to encourage new kernelci.org labs and have something that would provide a foundation for custom CI solutions for our partners.

For a BOM cost of under €400 (for brand-new components), we built an entire lab inside of a PC case that ticked all of these boxes and validated our idea. Kevin Hilman and Patrick Titiano unveiled the project at Embedded Linux Conference Europe 2017. We call it the “LAVA Box”.

 

LAVA Box Design

Because the LAVA box provides an entire CI solution, it contains both hardware and software parts. The software portion is made up of LAVA server (master) and dispatcher (slave). The server provides a web interface and handles things like job scheduling and priorities, and board description files which greatly simplifies administration.

The dispatcher manages connections between the boards and the real world. It provides services like DHCP, TFTP, NFS, etc, handles USB fastboot and mass storage, and power switching for all devices under test (DUT). Serial connections are handled by the dispatcher and udev rules are used to ensure that serial consoles have the same device path every time they come up. Once you’ve connected your devices and written the description files, the day-to-day tasks are completely automated.

Both the server and dispatcher run inside of individual containers which are managed with docker-compose. While it’s possible to run the two containers separately for a scaled-out lab, we host everything on one machine for the LAVA Box proof of concept.

On the hardware side, everything is neatly housed inside a PC case. Boards are mounted to the case’s drive bays which makes it simple to swap them out for maintenance or administration. A standard PC PSU is able to supply enough power rails (molex/ATX connections) for the host computer and 5 DUTs with connectivity (network, power control, USB, etc). It would be possible to power even more devices with a more powerful PSU and a bigger case (or if you’re willing to run cables outside of the case).

We found that the +5v and +12v rails from a standard ATX PSU worked well for powering our DUTs. Though the lack of configurable voltages is a minor limitation, we were able to live with it. A critical piece of functionality is power switching for DUTs and there are lots of inexpensive options available on the market, such as USB-controlled relays. We used the fully-open ACME Cape because, apart from controlling power, it also measures power consumption.

All the DUTs communicate on an 8-port switch and sit on an internal network that is isolated from the outside world. Even though the LAVA box master requires Internet access in our configuration (to pull down CI jobs from kernelci.org), it is possible to run in a LAN-only mode for local jobs.

Our prototype is just one possible way to configure the LAVA Box. Due to the modularity of the design, and the open documentation and software, you can build your own “lab in a box” using different hardware components and software provided it uses the kernelci API.

The Automotive Grade Linux project has taken advantage of this flexibility to build their own LAVA box. We improved the KernelCI API to level-up the quality of their releases and the new API allows AGL to run platform, AGL-specific, and automotive-specific tests everytime new code is committed. In a group with over 100 member companies, scaling engineering effort is super important and our automation and CI work has significantly improved the overall quality of the AGL CI loop.

 

The hard things about building labs

It took us a long time to build our LAVA box prototype. But we learned some valuable lessons along the way that will save you time and energy if you want to build your own.

1. You need to balance DUT power consumption

One of our wishlist items was more power rails. This doesn’t necessarily mean we want more power, simply that we want more rails to connect more DUTs. In the future, we’ll move to a more powerful ATX unit since they usually come with an increase in the number of connectors.

2. Cheap USB cables are flaky

There are lots of brands that produce low-cst USB serial cables. Few of them work for very long. We had many issues with connectors only working for a short period of time and requiring unplugging before they’d work again. We ended up settling on FTDI cables as we found them to be much more reliable than other brands.

3. There’s not a lot of space inside a PC case

Things are tightly packed inside the case which means that cooling may become an issue if your DUTs run hot. We specifically purchased a PC case with front and rear fans to keep the internal temperature down.

4. There’s no standard DC power jack

Every DUT has its own connector, so you need to be comfortable pulling out the soldering iron to connect the power. This obviously isn’t for everyone and right now we don’t have a good solution for this. Hopefully this problem will eventually be solved by increased adoption of USB Power Delivery via Type-C connectors, but for now, board rework is often required.

 

Future plans for the LAVA box

We poured all of our embedded systems experience into designing and building the LAVA box and validating our idea. Based on the feedback we’ve received from AGL, and from what we’ve seen in our own labs, it’s been a roaring success. The LAVA box has productized the CI workflow for embedded Linux.

We’re not done yet and we have some more tweaks in mind for the current design such as improving connectivity by adding Wi-Fi and Bluetooth. Ater that, we’re going to take the things we learned and address new scenarios like building a cost-effective solution for single-board labs and scaling up to professional-grade labs in racks. That way, labs of all sizes can benefit from the power of CI.

If you want to join in you can start by reading the documentation and contributing to the KernelCI project on GitHub. This entire project is open source and community contributions are greatly appreciated.

Or perhaps you want to talk to us about building your own lab in a box? Reach us at contact@baylibre.com to see how we can help!

Linux Kernel 4.15 released, our contributions

Linux Kernel 4.15 released

BayLibre has continued our contribution to the Linux community as seen with this new version of Linux Kernel 4.15, released on January, 28th 2018.  An excellent summary of this release can be found at KernelNewbies.

Here is a summary of our contributions, organized by SoC family and a summary graph of contributions by developer.

Amlogic SoC family:

  • Finally GPIO IRQ Support for Amlogic SoCs
  • Pinctrl cleanup for AXG future support
  • Meson GXL/GXM Internal PHY fixup for reliability
  • Support for new boards :
    • Vega S96
    • Khadas VIM2
  • Driver support for the Video Processing Unit Power Domain for mainline U-Boot support
  • Level Reset support for GX Family

Misc:

  • Support for the Winbond w25q16dw present on the Khadas VIM2
  • usb: musb: da8xx: Remove duplicated defines

 

 

How We Improved AmLogic Support in Mainline Linux

It was a match made in heaven: two years ago AmLogic and BayLibre joined forces to bring top-notch support for AmLogic SoCs into the mainline Linux kernel.

Since that time AmLogic has continued to become one of the most relevant and widely used silicon vendors in the world. Similarly, BayLibre has expanded the variety of SoCs, drivers and frameworks supported upstream all while supporting a growing number of OEMs basing their products off of recent kernel releases.

Along the way Baylibre has collaborated with a great community of product makers, kernel hackers and hobbyists to improve the AmLogic Meson SoC support in mainline Linux. We’ve added support for 64-bit SoCs, new drivers, and we’ve become maintainers for the AmLogic platform in Linux.

Our goal is straightforward: AmLogic system-on-chip processors should be so well-supported in the mainline Linux kernel that OEMs will go to market using the latest stable kernel release.

Last year at Embedded Linux Conference 2017 and this year at Linux Conf Australia, Neil Armstrong gave a well-received talk on our work. This post expands on Neil’s presentation and provides an update on some of the changes we’ve been involved with recently.

The Linux Meson project

The Linux Meson project is an effort to get support for the ARM-based SoCs produced by AmLogic into the upstream Linux kernel and GNU/Linux distributions. AmLogic chips are found in many Android-based multimedia devices such as set-top boxes, smart TVs, projectors, and tablets but are also starting to appear in products in the smart speaker market. Popular consumer products and community boards using the AmLogic Meson SoC include:

  • Amazon Fire TV (gen3)
  • Wetek Play/Hub
  • Nexbox A1/A95
  • Hardkernel ODROID-C2
  • Tronsmart Vega S96
  • R-Box Pro
  • HwaCom AmazeTV
  • Libre Tech CC (a.k.a “Le Potato”)
  • Khadas VIM and VIM2
  • NanoPi K2

AmLogic provides a kernel release and SDK that supports their hardware but it’s a heavily-modified 3.10-based or 3.14-based kernel for Android. Thankfully, most of the changes are confined to the `drivers/amlogic` directory and it has been possible to use them as a starting point to write the upstream support.

Carlo Caione added the initial support for the 32-bit Meson6 SoC in Linux 3.18, with Beniamino Galvani adding support for Ethernet and Andreas Färber contributing code for the GXBB platform. Since then, support has been added for Meson8, Meson8b, and the 64-bit ARM Meson GXBB, Meson GXL and GXM SoCs and a whole host of hardware. Codenames for the 64-bit SoCs are used in the kernel source and the following table shows how they map to their respective product names:

 

Codename Product name
GXBB S905
GXL S905X, S905D
GXM S912
AXG A111, A112, A113

 

Here’s How We Helped

Our efforts have mainly been focused on adding support for the 64-bit Meson SoCs. Kevin Hilman generalized the boot and serial console code for the 64-bit AmLogic family of SoCs so that more boards could be supported, starting with the Hardkernel ODROID-C2).

Michael Turquette reworked the Meson SoC clock driver to move to the `platform_driver` API and replaced the dynamic configuration code with statically initialized data. That was in preparation for adding clock driver support for the S905 SoC. We’ve also added S905 AO (Always-On) clock and reset controller drivers and support for PWM, RNG, Watchdog, IR, I2C, SPIFC devices.

Because some of AmLogic SoCs use a legacy version of the System Control and Power Interface (SCPI) protocol (pre-v1.0), we pushed changes to the upstream SCPI driver to support it. The SCPI driver is used for Dynamic Voltage and Frequency Scaling (DVFS) with the CPUFreq subsystem to dynamically alter the processor’s speed.

We’ve added a driver for the AmLogic SPI Communication Controller (SPICC) that’s capable of full-duplex transfers up to 30MHz. And in v4.10 Kevin added a brand new SD/eMMC driver that supports the host controller found on the AmLogic S905/GX* family of SoCs.

Neil wrote the AmLogic Meson DRM driver and has added HDMI support since his last update during his talk at Embedded Linux Conference 2017.

As the following graph shows, the number of Meson patches merged into the kernel has risen steadily over the last few kernel releases, and Baylibre is responsible for hundreds of them.

AmLogic kernel commits and BayLibre contributions
And because of our knowledge of the Meson code, we’ve had the honour of becoming (co-)maintainers for the majority of it.

  • Meson SoC code (Kevin Hilman)
  • DRM display driver (Neil Armstrong)
  • AO-CEC driver (Neil Armstrong)
  • Meson SoC clock framework (Neil Armstrong and Jerome Brunet)

As the number of supported boards has grown, it’s become increasingly important to maintain quality across all devices. For that, we’ve leaned heavily on kernelci.org. Baylibre is a founding member of kCI and you can expect more details in an upcoming blog post.

kernelci.org has been extremely useful for our AmLogic work. Part of our lab is made up of 8 AmLogic Meson boards which we use to check for regressions across 6 kernel trees. The kernelci.org infrastructure recently caught a kernel regression in the SCPI code which resulted in CPUFreq failures on boot. Thanks to kernelci.org and our collection of Meson boards, we were able to get the offending patch reverted before the final v4.15 release.

 

Links to Community Resources

AmLogic Meson support has come a long way in the last two years. That progress has been possible thanks to the supportive Linux Meson community and the kernel hackers (shout out to Martin Blumenstringl for his contributions as well as Carlo Caione and Andreas Färber for their early work and ongoing reviews) helping to improve the upstream AmLogic code. There’s lots more to come and we are currently working on support for the new AXG SoC (including audio) which is targeted at the smart speaker market.

If you want to know more about Linux Meson development you can subscribe to the linux-amlogic mailing list or join the #linux-amlogic IRC channel on irc.freenode.net. Also, the Linux Meson wiki is updated as support for new devices and SoCs lands in mainline Linux.

 

Post header image by Hardkernel – http://odroid.com/dokuwiki/doku.php?id=en:odroid-c2, CC BY-SA 3.0, Link