|
Deprecated
|
Papers /
MetabuildsForEmbeddedDevelopment
Recently there was a discussion on the Buildroot mailing list on how Buildroot should be extended to support new tasks. Buildroot is a general purpose metabuild system with origins, and continued emphasis, in the embedded development world. BeagleBox utilizes Buildroot as one component in its software build process. I felt the Buildroot discussion was an opportune moment to take a look at metabuild systems, why we use them, and what they need to do for us. Table of Contents
Prerequisite Terminology
Before diving in we should clarify some common terminology. First, the process of converting source code into an executable program is called compiling. When you compile software the files used to facilitate the process are known as the build system, more commonly referred to as the build. This typically includes configuration files of varying types and formats that are read by and direct programs that compile the software. A project is a collection of files managed by the build system that targeted at a specific purpose. Some projects are known as metabuilds. A metabuild is a project whose purpose is to download other software from the Internet, mix it with patches and locally stored source code, and compile it all into a useful set of programs and their associated data and configuration files. Since the software being downloaded already has its own build system, the metabuild is actually a build of builds. Thus the term metabuild. A software stack is a series of software projects, with higher level projects directly dependent on one or more projects below it in the stack. Android is a software stack that provides application interfaces for application developers. The embedded software stack is a collection of software that provides the infrastructure on which a stack like Android runs. The Android software stack applies to a different problem space than the embedded software stack. The Embedded Development Process
The modern computing world is expanding the reach of embedded systems to those who may not be familiar with what it takes to bring a piece of hardware up to a specific application level. If you're attempting to build a packaged collection of software specific to an application on a specific, non-generalized hardware platform you may need to become familiar with the entire embedded software stack. Most Linux users are familiar with a variety of Linux distributions. A distribution is a collection of software that encompasses two or perhaps three components of the software stack required for embedded development. The software components within embedded development include the following.
A typical Linux distribution includes the Linux Kernel and root file system. It typically also includes a boot loader though not necessarily the same type of bootloader you find in embedded systems. It may also include product specific applications though the most popular Linux distributions such as Fedora, Ubuntu, Debian or OpenSuSE are generalized software platforms that are not targeted at a specific end use. Cross Toolchain
The Cross Toolchain is the core component of an embedded development process. This includes a compiler (and associated utilities and libraries) that runs on your host computer but that compiles software to run on a different hardware platform. For embedded systems the toolchain is often configured to be optimized for a specific target hardware platform such as a specific implementation of a PowerPC, ARM or MIPS processor. In this discussion, host refers to the computer on which software is compiled and target is the hardware on which the build (re: compiled software) will be run. The toolchain is itself a collection of utilities, the most important of which are the GCC compiler, the bin utils and the C library. A variety of projects, including Crosstool-NG and Buildroot, exist for collecting these bits to create the toolchain. Precompiled (re: binary) toolchains are also made available from various free and commercial sources such as CodeSourcery.com. An embedded project that relies on off the shelf support for a target board will typically use a binary toolchain or compile a toolchain based on commonly known configurations. However, if you compile your own you may need to understand at least some elements of compiling a toolchain manually in order to properly configure a project such Crosstool-NG. Bootloader
The bootloader is a bit of software that the hardware can access at power up, load into memory and immediately run. Its purpose is to prepare the hardware for use by the operating system. The bootloader often loads bits of hardware with firmware stored in flash (NOR or NAND chips), browses various buses to find what hardware is on the board and then launches the operating system and passes this information to it. There are a number of open source bootloader projects including Das U-Boot, CFE and Redboot. The choice of which to use for your platform is often dependent on which project was chosen by the board development team to provide bootloader support. A typical embedded project that is relying on off-the-shelf support for the target board will typically only use a bootloader and not modify it greatly. However, patches from outside sources may be necessary to include in the build system in order to get the latest, fully functioning support for specific hardware features. Operating System
The Linux kernel is the bit of software that runs the product specific applications. It includes device drivers for the various bits of hardware on the board and provides APIs for using that hardware. Used in conjunction with libraries provided by the root file system it provides all the available resources an application will need to interact with the capabilities provided by a board. Root File System
The root file system is the initial collection of files and directories needed to start an application. The number of files and directories in the root file system varies dramatically depending on the intended purpose of the embedded system. It can include all the files you might find on a typical desktop Linux distribution or may only include a single binary. Much of the work in creating an embedded system lies in the selection of the files and configurations found in the root file system. Product specific applications
The root file system provides the base upon which product specific applications can be built. These applications form the utility of the embedded system. And they are typically created from scratch or at least extended from original applications by the embedded developer. The product specific application is where many embedded projects focus their design work and often encompass the features advertised for the project. Seldom do embedded projects make extensive claims about the other components in their embedded software stack. Metabuilds
The metabuild is the system used to put all these pieces together in a coherent collection that allows working on any of them independently or in any combination. The metabuild is the tool that makes management of the software stack possible. The Build Workflow
Creating the embedded software stack starts with generating a cross toolchain. This tool collection is mandatory to work with any other embedded software stack component. You won't get anywhere with your project without it. The toolchain is compiled and run on the host computer. It will compile most of the software stack on the host computer. In embedded development, most compilation work is done on the host computer, not on the target computer. In the past, cross toolchains have not been relocatable. That means you could not compile it and then move the output (the binaries, libraries, etc.) to another directory. This is no longer the case with GCC 4.x or later. That means it is possible to build the cross toolchain and place it into a package format such as RPM for use by multiple developers. Once a toolchain is created the next step depends on your hardware. Most boards will require a bootloader. In some cases the board will already have a bootloader from the board manufacturer. In other cases the board manufacturer will provide the code for the bootloader. Still other cases require you to use a freely available bootloader project. Some boards use more than one bootloader. These bootloaders are referred to as stage 1, stage 2, and so forth. On the BeagleBoard, for example, the X-Loader bootloader is a stage 1 bootloader and Das U-Boot is used as a stage 2 bootloader. If source is used for the bootloader, it will be compiled with the cross toolchain. The bootloader is packaged by its build process into a format that can be used on a board. The bootloader will typically bring up a board to a point where developers can interact with it through a serial console. But the bootloader is not a general purpose application environment. So the next step is to bring up the operating system. The operating system for this discussion is the Linux kernel. Other operating systems may be used, such as any of the BSD derivatives. Note, however, that environments such as Android are not operating systems. Android is a distribution, the heart of which is the Linux kernel. The Linux kernel is compiled with the cross toolchain. It may need to be packaged into a specific format to place it on the board, such as in NAND or NOR flash storage. Such packaging may required build steps not provided by the Linux kernel build. The operating system provides services but on its own doesn't do much. The root file system provides the applications and tools that utilize the operating system to perform useful tasks. A root file system is simply a directory structure that contains one program that the operating system can run. That one program then launches all the other programs that the embedded system will run. The root file system is created through the use of one or more metabuild systems. The metabuild will typically download and compile multiple programs, install them into a local directory structure, and then package that directory structure into a format that can be used by the embedded system. In some cases the packaging will be supported by the metabuild. In other cases the packaging may have to be handled by additional, external build processes. Finally, you're reached the point where application development can occur. If the root file system has been built to include build tools, IE a toolchain that runs on the target computer, separate from the toolchain you run on the host computer, then the application can be compiled on the target board. However, since the capability of building on the host platform is already available and building on the target platform is often extremely slow due to read/write performance of the available storage, it often makes the most sense to build applications on the host. Additionally, the root file system on the target may be limited in development tools and may not provide a user interface as easy to use as your desktop. Patch, Extend, Test, Rebuild
The build workflow describes the order components in the embedded software stack get compiled and packaged. But the embedded development process is iterative - developers will return to various components at different times to enhance a component. Often this is done because patches are made available that extend a feature set, such as new support in the compiler for specific hardware features or an updated device driver in the Linux kernel. So the build workflow also includes the typical software development processes of patch, extend, integrate, test and rebuild. Let's look at this extended workflow as it applies to a number of use cases for embedded development. Use Case Scenarios
These use cases describes unique but related instances of development of the embedded software stack. Use Case: Reference Distribution
Most boards have one or more software engineers who job it is to create a reference distribution of the embedded software stack specific to that board. The distribution is typically released as one or more images, with each image a packaged collection of compiled software. Release cycles vary and are often not scheduled. The purpose of the reference distribution is two fold. First, it provides an initial working environment in which end users can experiment with the hardware. The environment provides basic capabilities that showcase the board it runs on. Second, if the release is managed properly, it can provide a mechanism for reproducing the images so that end users can identify minimum build requirements for their own software stack. Users of a reference distribution do little to patch, extend or integrate new features into the distribution. This is due to the need to keep the reference implementation small in order to prevent its becoming bloated or application specific. However, users will heavily test the distribution on the hardware and are likely to want to rebuild the distribution in order to verify they can, at a minimum, reproduce a known working system. Use Case: Tiny Desktop
The reference distribution is limited in scope. It focuses on showcasing hardware features. The opposite end of the spectrum is the general purpose desktop distribution. This includes popular Linux versions such as Ubuntu, Debian, Fedora and OpenSuSE. These distributions contain everything a desktop user might need, including many things they never use. These distributions follow regular release cycles and are usually delivered in packaged images formats. The tiny desktop use case refers to the use of a desktop distribution on an embedded system that runs with very lower power requirements on small footprint boards, often with limited available storage. A desktop distribution is heavily based on a package management system that allows the installation and removal of 3rd party software, along with any software on which the requested software may be dependent. The desktop may also include development tools for creating new applications. Hardware support is broad. The desktop distribution is built without knowledge of the specific hardware on which it is bound for use. Users of the tiny desktop expect all the features of their typical desktop. They rely on updates from the distribution maintainer via updated managed packages installed over the network. Their embedded systems require large storage spaces, including large SD cards or USB attached external drives. Performance can be affected for applications that require heavy write access to storage devices. Tiny desktop users are not embedded developers. They focus on application development and rely on the desktop distribution maintainers for patch, extend and integrate operations of the embedded software stack. Tiny desktop users do test the distribution heavily but may not require rebuild capabilities for the distribution itself, relying instead on released binary images. In general, the tiny desktop user is application oriented and not oriented on the embedded system as an individual product. Use Case: Mobile Platform
The mobile platform is similar to the tiny desktop. It is often created by a group managing the distribution. It differs from the desktop distribution by providing APIs that limit applications to running within the bounds of the embedded hardware (the tiny desktop isn't built with knowledge of hardware limitations). Mobile platforms are typically delivered as reference implementations on which application development is done by end users. Android is an example of a the mobile platform scenario. Mobile platforms have focused hardware support because they are targeted at specific devices. For example, mobile platforms do not typically support all USB network devices but instead support specific, builtin WiFi hardware. Mobile platforms provide reference implementations that are ported to specific hardware devices. This means that users of a mobile platform will do varying amounts of patching and extending of the reference platform. Mobile platform users will also do extensive integration of third party software, testing of the reference platform and rebuilding of the entire software stack. Use Case: Sensor Arrays
A sensor array is simply a series of devices, each of which whose purpose is to identify changes to the local physical environment. A sensor array typically has a specific task to perform such as monitoring of visual and weather conditions, movement through infrared beams, radar and sonar, and so forth. These devices do not typically have display systems directly connected to them. In many cases they deliver data to a remote hub through a network connection. Users developing for sensor arrays often have very strict power and storage requirements. These devices typically boot from flash and only store operating parameters back to flash. They do not typically have removable storage media. They may operate on batteries or localized power supplies such as solar arrays. Sensor arrays have very customized distributions. Developers of these distributions target very specific devices and very specific applications. They require complete control over the embedded software stack and are heavily dependent on the application of patches to public releases of components in the stack. Developers for these systems are likely to extend and integrate custom software within any of these components. The sensor array developer creates a very customized build environment that may require rebuilding of any component at any time. Use Case: Entertainment Platform
An entertainment platform is similar to a mobile platform in that it is built with knowledge of the video and display capabilities on which it will run. Developers of this type of embedded system make use of their knowledge of a specific hardware platform to squeeze every ounce of performance out of the video and audio playback. However, the entertainment platform usually has a stricter application use. Entertainment platforms are heavily tied to the user experience so often have glitzy interfaces with targeted features. This includes hardware accelerated 3D displays for overlays and effects. However, the interfaces are not typically general purpose and easily extended. Entertainment platforms may support a variety of input devices from keyboards and mice to wireless RF remotes to bluetooth wands. Modern entertainment platforms often target input devices that are delivered with the rest of the hardware. Network support is usually wireless 802.11g or better with modern entertainment platforms, with support for wired interfaces provided as a backup. Embedded system development of the entertainment platform is as complex as that of sensor array development because of the need for very specific hardware support. Entertainment platform build systems encompass patching and extending all components in the embedded software stack. They also required extensive integration of custom applications and utilities and perform many test and rebuild iterations before delivery. Use Case: Real Time Platform
A real time platform has very distinct timing requirements of its applications and hardware. An example is radar processing. An embedded system with real time requirements based on the Linux kernel may iterate over many kernel releases before settling on a suitable version. Once selected, that release will require extensive patching since the CONFIG_PREEMPT_RT patches are not always fully integrated into the drivers that may be required by the real time project. A real time platform usually has a very small root file system, often with only the bare minimum of utilities required for the run time environment. In the case of government projects, extra utilities may not be allowed or may required additional qualifications before being permitted in the final release. Real time projects often choose their hardware for specific features such as a digital signal processor (DSP). In many cases the real time project may be required to patch and extend existing, if any, support for the DSP and other specialized hardware. Real time projects are the most development intensive scenarios and are heavily dependent on patch, extend, integrate, test and rebuild requirements. Buildroot as Metabuild
Buildroot is a metabuild designed as a build wrapper over all of the embedded software stack components. It can build a toolchain or utilize an external (previously acquired or built) toolchain. It will build a variety of Das U-boot bootloader and Linux kernel releases. It can build the entire root file system and can be extended to include building custom applications. But while Buildroot is a Swiss Army Knife of build systems, it is not perfectly fit to every type of development practice. Lets look at where Buildroot is lacking within each of the previously described use cases. Note that this discussion focuses on limitations in Buildroot but should not be taken to mean Buildroot is unworthy of consideration in an embedded project. On the contrary, it is very well suited to embedded projects if used in a manner suited to the appropriate use case. Use Case: Reference Distribution
Reference distributions are easily rebuilt by end users if the build is based on Buildroot. The configuration file for Buildroot is based on the same configuration structure as the Linux kernel, allowing for easy modification through the use of a menu based selection system. End users can experiment with add on tools as long as Buildroot already supports those tools. For the developer of the reference distribution Buildroot suffers from the same problem as other scenarios, and that is the difficulty that exists in rebuilding only a part of the entire configuration. It is not easy to rebuild just the toolchain, for example, without rebuilding everything else. It is not very easy to experiment with multiple versions of the Linux kernel without changing the single configuration. Rebuilding individual components of the root file system is possible but a slightly awkward. Custom tools provided by the reference distribution to showcase hardware features must be added into Buildroot if they are to be built by Buildroot (and thus make it possible for end users to easily reproduce). Those customizations are typically stored in an SCM (source code management) system. To integrate them into Buildroot requires a build system on top of Buildroot, at least until the custom applications are merged upstream with the Buildroot source repository. As we shall see, this issue is common to any use case. Use Case: Tiny Desktop
The Tiny Desktop includes everything plus the kitchen sink in the root file system. It often has custom configurations of UI components unique to the distribution. This is often what sets these types of distributions apart from each other. Buildroot does not yet support the breadth of utilities found in these distributions, but there is no reason it couldn't support them eventually. In fact, it already supports an impressive array of utilities for use in the root file system. Buildroot does not perform the type of package management required by the tiny desktop scenario. While support exists for ipkg and RPM packaging formats (and possibly others), Buildroot does not create the bootstrap environment in which the packaging of these files can occur. The developer of the tiny desktop is left to create that on their own. Due to the extremely wide range of utilities it can become cumbersome to rebuild individual components of the root file system when those components are updated from their source repositories. It is also somewhat unclear ho, or at least difficult to rebuild individual packages within Buildroot and then rebuild the the resulting image file. Use Case: Mobile Platform
The application specific environment found in the mobile platform use case requires extensive customization by the embedded developer. As it turns out, this scenario can work well with Buildroot because the custom environment can exist as a stand alone package that can be integrated upstream into Buildroot as long as the customizations are open source. If they are not, then the customizations can be integrated with Buildroot by patching Buildroot locally. In either case, chances are good that patching of Buildroot will be required to get the desired results for the mobile platform. The patching of Buildroot implies a metabuild around Buildroot. Use Case: Sensor Arrays
The sensor array use case is especially well suited to Buildroot because of the need to limit the contents of the root file system. But requirements for rebuilding and customizing the bootloader, Linux kernel and even the cross toolchain make it necessary to either separate out these components and use Buildroot only for the root file system or use multiple versions of Buildroot, each one targeted at a specific embedded software stack component. Again, the problem is one of patching and rebuilding individual components. Use Case: Entertainment Platform
Buildroot's use in the entertainment platform is similar to its use with the mobile platform and sensor arrays. Customizations require patching of Buildroot during development. And hardware support often requires customization of all components of the embedded software stack. Use Case: Real Time Platform
The real time platform is least suited to use with Buildroot. In this case, however, the problem is less Buildroot's and more a problem with patch integration with the Linux kernel. Even building the kernel by itself for real time support will require a fair amount of custom work. Since patching and rebuilding individual components is an issue with Buildroot, this use case is better suited to a custom metabuild. Metabuilds of Metabuilds
In practice, the use of Buildroot has required that it be wrapped inside a custom project metabuild. Originally, to address the problems of rebuilding individual stack components at will, I used multiple Buildroot builds, each with a custom configuration targeted at a specific stack component. Later, when I found Crosstool-NG better supported specific projects for PowerPC and ARM I switched out Buildroot's build of the toolchain to Crosstool-NG. Recent versions of Buildroot better support Crosstool-NG so it's unclear if I'll switch my own metabuilds back to just Buildroot. More research is needed. Additionally, the need to patch various components or use custom releases (such as a vendor release of Das U-boot which has not been pushed upstream yet) required a custom metabuild that can be patched based on release names. BeagleBox takes this one step further and allows mixing not just mainline releases but releases from different source repositories. This allows testing code from a developer who has not yet pushed their changes upstream or not yet had them accepted. The top level metabuild must be able to perform actions on individual components. Assume the following component names.
The following targets are likely candidates for each component. Most of these are implemented for each component in BeagleBox. The -createpatch target is implemented in other builds I have at work but not yet in BeagleBox. BeagleBox does not yet support the apps component because the project has not reached that
Consistency is a key factor here. Each component is managed, via targets, the same way. The underlying build system hides the variations of the other metabuilds. What is most important is that each component can be worked on independent of the others. The -clean targets are used to remove build artifacts from retrieved/unpacked/patched source while the clobber does the same but also removes the saved archive file. The -saveconfig target copies the component configuration file to the top level metabuild source tree so it can be SCM managed. It may also replace certain configurations with template tags. When the build for the component is run the template tags are replaced with environment variables set in the top level metabuild by the -config target. Note that even with this structure, BeagleBox does not address the issue of rebuilding individual subcomponents of the underlying metabuilds. For example, it still can't rebuild just one external project supported by the Buildroot build. Summary
An embedded system metabuild must be able to detach build dependencies between software components so they can be rebuilt independently. It must be able to rebuild sub-components easily, primarily with respect to the root file system but also for the cross toolchain In some cases the metabuild must be able to easily repackage the root file system with updated skeleton files or be able to support package releases such as .debs and RPMs. But there is more here that we haven't touched on. Release management must be taken into account when forming the build system. A metabuild should be able to build a reference distribution by name. This is true even of a custom metabuild like BeagleBox where a reference distribution might be made for one hardware platform while another reference is made for a later revision of the hardware. Another area not covered is the need for local archives and for long term archiving of build systems. A local archive is where downloads of external metabuilds and the projects they download are stored. Buildroot and Crosstool-NG support this concept already and setting their configuration files to point to a common top level archive directory is a start. But then you have to keep track of which versions of software are used for a given release. Do you store those locally and permanently in a single archive file? If so, how would you use them later if you need to rebuild that released branch to work on bug in the field? The top level metabuild would need to be able to switch from grabbing remote archives to using a locally stored archive. Even with a tool as extensive and well designed as Buildroot it is nearly impossible to ignore the need for a metabuild of metabuilds. Without a project specific metabuild, it will be difficult to track custom changes to project files without an SCM that does not include the source for Buildroot, Crosstool-NG, Das U-Boot and the Linux kernel. An embedded project shouldn't be tracking that code - they have their own repositories and a custom build pulls their releases or code base at build time. The embedded project only needs to track its customizations. |