cdtools, because who has just one project?


When I was just getting started with embedded development I found many tutorials on how to perform cross compiles required setting up some shell functions and variables before working on builds.  This is a necessity for embedded work because the embedded build for the target platform won't use the same toolchain and build libraries as the host platform.  Unfortunately, the setup for the various publicly available build systems were all different.  angstrom/openembedded vs yocto vs buildroot vs Mentor Graphics vs whoever:   they all have their own setup.

Then I created my own build platform, pibox.  I did this mostly to teach myself the whole platform bring-up process.  One of the side effects of this (along with having to build custom OPKG packages for PiBox) was the need to bounce from one build to another quickly.  I wanted them all to use the same navigation commands and automatically point where ever they needed to with environment variables.  Thus was born cdtools.

Use case

Let's say I have a build system for the toolchain, kernel and root file system for my Raspbeery Pi (re: PiBox).  I do out-of-tree builds and packaging to avoid clutter when looking for updates with git status.  So while my source lives under …/raspberrypi/src my builds are under …/raspberrypi/bld and packages generated from the build are under …/raspberrypi/pkg.  So far so good.  That's not overly complex.

But now I need to bounce over to a kernel tree and make some updates and test them.  The updates I end up with will be integrated back into the PiBox build but before that happens I want to test them in a kernel-only tree (remember: PiBox is a build of multiple components – it is, in fact, a metabuild that downloads components, unpacks them, patches them, builds them and packages them).  So how do I bounce over to the kernel-only tree and run tests and then bounce back to the PiBox tree to integrate the changes?

Here's another scenario:  I have a slew of metabuilds that do nothing but download 3rd party utilities, configure them as needed for PiBox and them package them as OPKG packages.  These builds are then wrapped in another metabuild that builds all of those package-metabuilds in order to simplify creating a complete PiBox release!  This top level metabuild has dependencies on the package metabuilds so I may find myself performing the top level metabuild, finding a bug (due to an upstream change to the branch I'm using)  in a 3rd party utility, switching over to that package metabuild to fix, commit and push it, and then bouncing back to the top level metabuild to rebuild the packages again.

What I need are common navigation tools for all projects and a way to change that navigation based on which project I want work on right at this moment.  The way to do that is with shell variables.  But I need to do this without having all kinds of project-specific shell scripts that are all different.  There needs to be some consistency in how they are used.  I want to set the project name but use the same navigation to end up in source, build and package trees no matter which project I'm in.

cdtools

This shell script acts as a front end to a directory of configuration scripts, one for each project in which I do builds.  This script lives in ~/bin and sources any shell files found under ~/bin/env*.  In this way I separate builds for work from builds I do at home by placing the environment configurations in different directories, as in

~/bin/env.muse
~/bin/env.work

Whenever I add a new project (and this happens frequently for me) I just drop in a new shell file in on of the env subdirectories and then reload cdtools.

. ~/bin/cdtools

This makes it easy to make changes without having to logout and back in.  It also makes it easy to do within a screen session so I can test new or modified configurations under one screen window without affecting the other windows until I'm ready for the update.

shell functions

Each project has its own shell file which contains a project specific shell function that sets up environment variables and aliases.   The environment variables are always the same but point to different things.  Typically I use variables prefixed with GM_, though not always.

  • SRCTOP – top of all source trees, under which you find all project specific directories.
  • GM_HOME – top of the project directory, under which you find the project source, build and package trees, along with others as needed.
  • GM_WORK – a scratch pad area often common between projects.
  • GM_BUILD – the build directory for the project
  • GM_SRC – the source directory for the project
  • GM_PKG – the packaging directory for the project

Other variables can be set to update the executable PATH, library path, java path, or anything else that's needed.  As long as the same variables are used in each shell file (with unused variables ignored) then switching from one project to another will always properly setup the environment.

For the use case, the PiBox shell function is rpi and the linux kernel shell function is kernel.  Typing either configures my environment to access, build and navigate their respective trees.

aliases

Remembering the environment variables is hard.  So is typing them.  So the shell function maps them to navigation aliases.  All navigation aliases are prefixed with cd, as in

  • cdt – top of all source trees, under which you find project directories
  • cdh – top of the project directory
  • cdx – source tree under the project directory
  • cdb – build tree under the project directory
  • cdp – package tree under the project directory
  • cda – archive directory, where a local copy of downloaded source is kept, to avoid having to redownload unless absolutely necessary (this includes git or mercurial managed trees, as needed)
  • cde – Extras directory

A special alias, cd?, is setup to call a project shell function that lists the aliases, important environment variable configuration and help in cloning and accessing remote trees.  All projects use the same aliases.  Additional aliases can be setup as long as the context of the alias is the same for each project.

help functions

Until you've used cdtools for awhile it can be hard to remember the aliases.  It's even harder to remember the sometimes convoluted git or mercurial (or other source code manager system)  commands necessary to work with remote repositories.  A help function is included in each shell file.  This is always prefixed with “list”.  In our use case, the helper function is listrpi or listkernel.  But you don't have to remember this because the alias cd? always points to the currently active projects help function.

$ cd?
raspberrypi Alias settings:
-----------------------------------------------------------------------------
cdt cd SRCTOP (/home/mjhammel/src/ximba)
cdh cd GM_HOME (/home/mjhammel/src/ximba/raspberrypi)
cdw cd GM_WORK (/home/mjhammel/src/ximba/work)
cdx cd GM_SRC (/home/mjhammel/src/ximba/raspberrypi/src)
cdb cd GM_BUILD (/home/mjhammel/src/ximba/raspberrypi/bld)
cda cd GM_ARCHIVE (/home/mjhammel/src/ximba/raspberrypi/archive)
cdp cd GM_PKG (/home/mjhammel/src/ximba/raspberrypi/pkg)
cde cd GM_EXTRAS (/home/mjhammel/src/ximba/raspberrypi/extras)
To checkout tree:
cdt
mkdir raspberrypi
cdh
git clone ssh://git@gitlab.com:pibox/pibox.git src

cdlist

So what happens when you have umpteen projects and you can't remember which shell function sets up a particular project?  You can get a list of available functions and their descriptions using the cdlist function, which is included in the cdtools script.  The cdlist function browses all shell files and extracts the shell function name, a description and the name of the file its found in.  The description uses a tag in a shell comment, as in this example.

 # DESC: PiBox: Embedded environment for ARM-based system using buildroot

To search for this I can pass any string in the DESC field, such as PiBox:

 $ cdlist pibox
 ...
 pmsui PiBox Media Center UI (pms) [pmsui.sh]
 pmsuip PiBox Media Center UI (pms) [pmsuip.sh]
 psp PiBox psplash opkg build [psplash.sh]
 rpi PiBox: Embedded environment for ARM-based system using buildroot [raspberrypi]

All shell files with descriptions that include the string “pibox” (case insenstive) are listed.  Now I can see the shell function (rpi)  which is listed first and the file in which that function is defined (raspberrypi), which is listed at the end of the line inside brackets.

Example usage

First, load all cdtools scripts.

. ~/bin/cdtools

Now setup to work in the PiBox tree.  Then change to the source tree, see what's there and then switch to the packaging tree and do the same.

 rpi
 cdx
 ls -C
 Makefile  README  config.mk  configs  docs  pkg  scripts  src  util.mk
 cdp
 ls -C
 firmware opkg rootfs.ext3
 kernel.img.3.10.y opkg.10 rpiToolchain-0.9.0_Fedora19_201409081418-1.x86_64.rpm
 kernel.img.3.14.y opkg.9 rpiToolchain-0.9.0_Fedora19_201409162158-1.x86_64.rpm
 mkinstall.sh pibox-0.9.0_Fedora19_201409081418-src.tar.gz staging.tar.gz
 mksd.sh pibox-0.9.0_Fedora19_201409081418.tar.gz

I can see that I have done packaging after multiple builds here.  Wait, let's check what's in the top level metabuild tree.

 meta
 cdx
 ls -C
 MakeChangelog.sh  build.sh  docs  functions.sh  src

Okay, that looks fine.  Now back to the PiBox tree.

 rpi
 cdx

Easy peasy.

Multiple versions of a single repo

A neat trick here is that if you format the shell function correctly you can append a number.  That number will allow you to have multiple versions of the same repo, as in this example.

 rpi
 rpi 2
 rpi 3

Checking the alias of the middle one, we see this.

raspberrypi Alias settings:
-----------------------------------------------------------------------------
cdt cd SRCTOP (/home/mjhammel/src/ximba)
cdh cd GM_HOME (/home/mjhammel/src/ximba/raspberrypi2)
cdw cd GM_WORK (/home/mjhammel/src/ximba/work)
cdx cd GM_SRC (/home/mjhammel/src/ximba/raspberrypi2/src)
cdb cd GM_BUILD (/home/mjhammel/src/ximba/raspberrypi2/bld)
cda cd GM_ARCHIVE (/home/mjhammel/src/ximba/raspberrypi2/archive)
cdp cd GM_PKG (/home/mjhammel/src/ximba/raspberrypi2/pkg)
cde cd GM_EXTRAS (/home/mjhammel/src/ximba/raspberrypi2/extras)
To checkout tree:
cdt
mkdir raspberrypi2
cdh
git clone ssh://git@gitlab.com:pibox/pibox.git src

Note the “2” in the directory paths.  I can now have multiple copies of the same repo.  And this trick can go further by using additional arguments to the shell function to embed different-yet-related repositories using a single shell function.

 bui fakekey
 bui network-config 2

And so forth.

Extras

The cdtools script currently includes support for color coding of the cdlist using ASCII escape sequences.  This works in most terminal types.  More importantly, there is no reason you can't extend cdtools to support additional functionality that you want to embed in your own set of shell files.

Consistency of practice

What you should get out of this is the consistency of use when moving from one project to another.  How I build projects can vary greatly: PiBox uses GNU Make while one of its packages (launcher, for example) uses Autoconf.  This happens all the time because when doing a system build (a build of an OS plus utilities plus custom kernels plus custom apps, etc.) you may depend on many 3rd party source trees with a variety of build patterns.  But that doesn't matter.  What matters is can I build it quickly by bouncing to it and having pretty much everything I need in place when I do.

With cdtools, the answer to that question is a resounding YES.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.