Michael J. Hammel – The Graphics Muse

Nicholas Copernicus.  I mention this only because I often name computers after scientists, physicists and engineers.  Over the years I’ve had a number of systems named copernicus.

I mentioned in a previous post that my wife got me some more old books this past holiday.  Last year she got me Pat and the Iron Horse, the first book I remember reading as a kid.  This year, she found me some Big Little Books similar to the one that the school teacher has in her desk in A Christmas Story.  These books are probably not very interesting to anyone but us old farts, but they sure bring back memories.  I do remember having both of these as a kid.  I wonder if I still have them somewhere buried in a box at my parents house.

After buying a variety of Wifi dongles (to use separately on a Pi, BeagleBoard and APC at the same time), a collection of USB extenders, three different keyboards, an extra powered USB hub and two new Raspberry Pi’s (which showed up unexpectedly yesterday even though Element14 made me think they were back ordered and not shipping until the end of the month), I finally found the problem with my keyboard and mouse not working with either the Pi or my BeagleBoard.  Turns out it was rather simple.

Don’t plug in a 5V power supply to the OTG port when you also plug in a external powered USB hub to the USB port on the board.  Instead, plug the powered hub into the onboard hub an then connect one of the ports of the powered hub to the OTG port to provide power to the board.

I”m not sure what’s happening with power coming from two inputs though I suspect it’s a grounding issue.  But I verified both external hubs were working and I’m also able to use the wireless keyboard and mouse via the powered external hub on both the Pi and BeagleBoard.  I’m not sure why the wired keyboard and wireless mouse worked at all originally.  No matter.

Of course, now I have lots of extra USB cables and Wifi dongles.  But hey, live an learn, right?

If you’re playing with the idea of getting into embedded Linux, you might start with the most basic question:  what does embedded Linux mean?  In the not so distant past that was a little easier to define.  An embedded board was typically physically smaller or at least used a very custom layout, used a low power processor or microcontroller and had limited computing resources like memory or storage.  Custom layouts mean that it would cost oodles of money to build each board, so getting your hands on one was an expensive proposition left to companies with very specific intentions.  The radar boards we make where I work are like that.  Very expensive for each board.

That’s changed a lot recently.  Many ARM chips have plenty of horsepower and boards can have up to a GIG of memory or more using technologies like PoP (Package on Package) for mounting chips on top of each other.  You can even get quad-core ARM processors on tiny little boards with USB, wired network and HDMI output.  General purpose systems unlike a general purpose desktop, yet very similar too.  Cool stuff.

So how do you define embedded Linux?  Well, the trick these days is probably asking yourself if you have to cross compile software for your target platform.  If you can compile a Linux kernel on the board and it doesn’t take a full day to complete, then it’s not exactly embedded.  It’s essentially a desktop.  But then if you compile on the board and all it has is flash memory, you’re not doing yourself any favors and you should probably be cross compiling.

So what does it mean to cross compile for an embedded platform?  Embedded systems, like desktops or servers, need four things to run application software:  a compiler toolchain, a bootloader, a Linux kernel and a root file system.  The toolchain is a collection of software that includes a compiler and utilities for manipulating binaries.  A gcc compiler can be compiled to run on your desktop or server but that creates binaries that run on a different type of machine, which is referred to as the target system.  That’s a cross compiler.  The toolchain adds libraries and tools needed along with the compiler to build programs to run on the target machine.

Once you have a toolchain you may need to use it to create a bootloader.  The bootloader prepares the hardware for the operating system, aka the Linux kernel.  There may be more than one bootloader for a system.  Intel systems have BIOS, or if you’re unlucky enough to have a recent Intel board, UEFI.  That’s a first stage bootloader.  GRUB would be a second stage bootloader.  They work together to get Linux booted.   On ARM systems such as most mobile phone and tablets the bootloader may be u-boot.  The Raspberry Pi has a proprietary bootloader that actually uses the GPU instead of the CPU.  Things are gettin’ weird out there.

After the bootloader runs, it eventually loads the Linux kernel.  The kernel, in turn, loads a root file system.  The root file system is the basic directory structure containing libraries, configuration files and data needed for basic operation.  You application sits in the root file system somewhere.  Where you put it and how the root file system gets it started is one of the places you’ll spend a lot of time in your embedded project.  The bootloader, kernel and root file system, along with your application, all need to be compiled using the cross toolchain.

Since most embedded boards have limited memory and limited storage, usually in the form of on-board NAND or NOR flash but more often these days in the form of SD flash cards, what you put into your root file system is important.  You typically don’t want extra crap in there, first because it takes up space but also because you don’t want users to muck up your very specific device purpose.  Which leads me to the big question.

What they heck are you doing putting Ubuntu on an embedded board for?  To make a tiny desktop?  That’s very creative of you.

There are embedded distributions such as OpenEmbedded and Angstrom that let you very finely tune your root file system.  Buildroot does the same thing but uses a different method to achieve a similar result.  Most embedded systems root file systems start with BusyBox, which is a lightweight set of GNU utilities originally intended for resource-limiited systems.  My own BeagleBox an PiBox are simply meta-builds, that is, build systems that wrap tools like Buildroot and the Linux kernel to create a custom embedded platform.  In other words, of the four software components I mentioned, three are pretty easy to build yourself.

That leaves the toolchain.  Getting the right toolchain is important in order to squeeze the best performance out of your embedded system.  Since gcc and friends are used for most systems, the trick is to find someone to tweak the gcc build to get the most out of it.  That’s what people like Mentor Graphics (aka CodeSourcery) do.  And Linaro, a group who adds features to GCC and friends that specifically help ARM systems.

It used to be hard to find a good toolchain.  And even harder to build your own.  Now there is Crosstool-NG, which makes building the toolchain yourself only slightly harder than understanding how to build a root file system with Buildroot or even to configure the Linux kernel.  And groups like Linaro are working with these open source projects (Crosstool-NG, Buildroot, etc.) to make it even easier.

So that’s what embedded Linux is all about.  Four software components, customized to a target platform intended for a specific purpose.  In the end, you end up with your own custom Linux distribution.  And isn’t that just the coolest thing you can think of?

I’ve finally got a build for PiBox that supports a proper XBMC build.  The latter took some finagling to force a cross compile that didn’t pick up any bits and pieces from the build system.  So it’s all checked in and the builds placed on the SD card and it’s ready to test.  Except for some reason none of my keyboards work with the Raspberry Pi anymore.

The original keyboard I was using was wired through a powered USB hub so I could also have a wireless mouse and a USB wifi dongle and so I could easily switch the powered hub between the Pi, my BeagleBox and an APC board.  The mouse still works (I haven’t checked the dongle) but the keyboard stopped working.  I plugged the keyboard into my desktop and it worked, but not quite completely.  I also tried it on my BeagleBox, where it didn’t work either.  I plugged the keyboard directly into the Pi and BeagleBox, skipping the powered hub, but no joy.

So I assumed the keyboard is broken.  I ordered a Logitech wireless keyboard.  When it arrived, I tried it on the Pi and BeagleBox.  No joy.  I plugged it into the desktop.  No joy.  Then I noticed it is one of the special Unified wireless devices from Logitech.  These require special software to enable the keyboard.  There is some open source software that can make this work, but I blew it off for now in favor of running to Target to get another wireless keyboard – one without special mongo features.

So the newer keyboard is a GE keyboard.  I plug it in to the Pi, BeagleBox and the desktop.  The first two – no joy.  The desktop worked fine with it.  So now I’m getting really freaked.  Did I munge the kernel config on both board builds?  I fiddled with the PiBox build but then decided to blow that off and go get another GE wired keyboard from Target.  Wired is what worked before.

And no joy with either board, though the wired keyboard does work with the desktop.  So now I’m thinking I somehow I fried the USB ports on both boards.  Weird, cuz the wireless mouse still seems to work.  I checked the USB ports using u-boot on the BeagleBoard and they didn’t seem to show the devices connected to the hub (not the OTG port).  You can’t do that kind of check on the PiBox because it doesn’t use a bootloader that’s of any use to integration testing.  At least not as far as I’ve seen yet.

So perhaps I shocked the ports.  I had a blanket in my office that was causing me to get a lot of static shock when I sat at my desktop.  I shocked my wired mouse on my desktop a few times, but I don’t remember shocking the Pi or BeagleBoard.  But maybe I did.

I’ve orderded two more Pi’s.  If these work with the keyboards, then I’ll order another BeagleBoard C4.  At the moment, I’m not convinced the ports are fried.  But I’m at a loss as to what else could be wrong with them.  And until I can fix the keyboard, I won’t be able to test the lastest PiBox with XBMC builds.  Bummer.

Once again, a new year means new toys.  This holiday season, my wife got me an Acer C7 Chromebook, a pair of PowerLine ethernet adapters and a new printer.  We also got new phones.  On the non-techy side, she got me some classic Big Little books. The techy stuff has its own set of issues.  The books, as usual, bring simple joy.  More on the books in a later post.

The Acer C7 is a dual Celeron processor netbook or laptop, depending on how you classify such things, which I’m using to write this blog entry.  It comes with Google ChromeOS.  Of course, the first thing I wanted to do is replace the OS with stock Linux.  Sadly, this version of the Chromebook uses an Intel-based board, which comes with the new and magical crap known as UEFI.  Don’t get me wrong – BIOS needed to be made modern.  But adding all that “secure” crap doesn’t help at all.  Not for the consumer.  All it helps is Microsoft and Intel.  And that’s a bad idea.  The ARM people seem to understand this and u-boot will thus rule the world.

In the mean time, you need to have a signed kernel to boot.  You get this on the Chromebook, which is running a Linux distro under the hood (it might even be Ubuntu for all I know).  You can’t just install a distro.  Instead, you have to resize the partitions on the disk and install your own root file system.  This is exactly what ChrUbuntu does.  You boot the box, wait for its display manager login page, switch VTs and login.  You then grab a script and run it.  It automatically resizes the partitions and downloads the rootfs for Ubuntu 12.04.  A few extra commands get the box to boot directly into Ubuntu.  From there you can upgrade to 12.10, install a usable desktop (XFce instead of GNOME 3) and get a development platform installed.  Sadly, there is no such script or methodology defined for a meaningful distro such as Fedora.  There may be some alternative distros available, such as Arch Linux and maybe Mint, but I’ve not investigated them because, well, neither of them is Fedora.

The problems start if you find the default kernel doesn’t have required components.  I wanted NFS but the kernel doesn’t include it.  That’s because you have to use the kernel that came with the Chromebook and the Chromebook is all about using your handy hardware to shove your data to the cloud (which seems silly if you’re a developer since you’ve probably already invested a ton in local storage).  That means Chromebook users aren’t expected to access local data stores.  Unfortunately, you can’t just compile a matching kernel. First, the kernel is 3.4.0, which you can’t even find on kernel.org (it’s in the ChromeOS git repo).  And even if you did, you can’t just add the loadable modules, at least not for NFS because the kernel modules require modifications to the kernel itself. 

My solution to this was to use sshfs instead.  Sshfs is a FUSE layer that allows “mounting” remote file systems to the local users directory structure using nothing more than SSH.  For the purposes of writing documents, this works fine so I can ignore NFS for now.  But sshfs is not robust enough for doing things like running a Buildroot build.  Extracting big tar files, for example, exposed some problems.  So as long as all I’m doing is editing a document on a remote file system, I’m okay.  Anything more and I’ll need NFS.

There is a fairly well described method of upgrading the kernel.  It’s not for the faint of heart, but anyone who can follow my technical posts should be able to handle it.  I got about half way through it without any issues before realizing I could use sshfs instead.  So I haven’t tried the complete process, but maybe by the time I need it the process will be further automated.

The Chromebook itself is rather nice.  The screen is clean and bright – very helpful for my aging eyeballs.  The keyboard is small but fits my fingers just right.  The keyboard layout is fine too – your average QWERTY layout without numkeys.  Sadly, the battery only lasts about 3.5 hours.  Pretty crappy compared to my Galaxy Tab S2 (last years birthday gift – momma knows I love my toys), which doesn’t need recharging for 3 or 4 days even if I use it to watch movies while I do my 45 minute runs at the local club. But wireless works out of the box.  It has bluetooth but I don’t have any bluetooth devices.  The touchpad doesn’t do two finger scrolling, though its supposed to (driver problem, I guess).  I use wireless USB mice on laptops anyway.  It’s got 3 USB ports, an SD card slot and wired ethernet.  All in all, it should work fine for getting me some quality writing time each night before I go to sleep.

There are a lot of things I don’t like about Ubuntu.  I think I’ll save that for a later post too.  But one thing:  what is it with Ubuntu distros being named Prefixbuntu?  Some unnatural level of narcissim there, I think. Just wait for the ARM desktop distros: ChARM, SwARM, AlARM, mARMalade, etc.  How clever all we software people are, eh?

My wife also got me a wrap-around pillow that I can sit up and lean back on in my bed, which means that with the new Chromebook I should be able to hack out a few more blog entries than last year.  Hopefully it will also mean I’ll finally get down to writing my first real work of fiction, something I’ve wanted to do since I was 10 years old.  I loaded Writer’s Cafe on the Chromebook.  We’ll see if that helps or is any more motivating than the notes on my home wiki.

As for the other techy stuff, the printer is wireless, includes a scanner (supported under SANE and GIMP) and works great.  I was able to install it in a fairly short time.  The Powerline ethernet adapaters were a failure.  While I did get some connections between the two adapters when plugged into an outlet, the connection was unstable and throughput was significantly worse than 802.11.g connections in the house.  The phones are also a bit of a bust.  I bought cheap Virgin Mobile phones with no contracts.  The phones need to be recharged daily even when they aren’t being used (no calls, no texting, no apps running), the back cover will pop off if you aren’t careful plugging in the power cord, the voice recording for my personal message is terrible, and I can’t find a whole lot of use for all the stupid Android apps.  It is, after all, a friggin phone!  We’re considering switching back to Sprint and getting their cheapest phones, which is what we had previously.  That phone (at least my previous bottom of the line Sprint phone) was going strong when last used and worked fine for what I needed it for – talking to people using our actual voices.  What a concept.

The outdoor holiday decorations came down today (to keep the HOA off our butts) so now all that’s left to do is spend a year enjoying the spoils of the past season.  Now I can spend more time finishing up PiBox and XBMC.  I actually have them compiled and installed to an SD card.  I’m just waiting for a replacement keyboard (the old one stopped working with the Pi and BeagleBoard) to see if I’ve finally conquered my own distro with XBMC.  Crossing my fingers…..

Tonight I pushed the build for PiBox, my version of the BeagleBox build for the Raspberry Pi.  The build isn’t complete, but it does generate a working cross toolchain, Linux kernel and root file system.  It utilizes binary firmware for the GPU that is used to boot the board (instead of u-boot).  The root file system boots to a login prompt but doesn’t boot into X yet.  Not sure if networking works yet since I have a wireless card on there – haven’t tried the builtin wired connection.

So that’s a start.  I’m moving toward getting a generic build system for apps that creates opkg packages for both BeagleBox and PiBox.  Then I’ll look into getting some tools on the Pi for doing some parallel programming.  Nothing would be cooler than to stack a bunch of Pi’s into my little PiRack.

It took some work, but I finally have SGX fully integrated into the build and have BeagleBox booting into an X session that can run opengl apps.  I ran the SGX demos and they all seem to run fine.  A couple showed frame rates betweeen 45 and 52 frames per second, and that should be fine for video playback with flash (at least I think it should be okay).

The biggest issue was that Buildroot has a 1.7.5 Xorg collection while SGX was built with 1.9.  I had to update the versions of a number of packages under Buildroot:

• xserver_xorg-server: 1.9.4 (xinput ABI = 11)
• xproto_dri2proto: 2.6
• xproto_fixesproto: 5.0
• xproto_xproto: 7.0.22
• xproto_xextproto: 7.2.1
• xlib_libXfont: 1.4.2
• xutil_util-macros: 1.6.0 (remove 1.3.0 patch)
• xdriver_xf86-input-evdev: disabled in buildroot config
• xdriver_xf86-input-mouse: 1.7.2
• xdriver_xf86-input-keyboard: 1.6.2

There's more work that could be done here, like removing SGX-specific directories under /opt that aren't really necessary  but I'm not sure how this might violate the licenses for the PVR stuff.  So for now it'll all just take up space.

Why all this work on SGX?  Because I need it for hardware accelerated Flash playback.  I haven't gotten around  to integrating the flash player yet (you need to request permission to get it), but this was a prerequisite for it. 

Now I need to complete the DSP integration. 

So I've integrated TI's SGX package into the BeagleBox build.  What this does is to run the SGX package's Makefile-based build which in turn installs a bunch of stuff from the archive into another directory.  This directory will then contain a bunch of files which need to be installed again, this time into the target root file system.

To do this post-installation step, the Imagination Technologies folks who provide the software for the PowerVR hardware, created an installation shell script.  The script offers options that allow it to run on the host or target platform.  But there is no way for it to work on the host platform for a cross-build system like BeagleBox because it assumes the host has the drivers you want on your target in its root file system.  A cross-build environment doesn't have this.  What's worse, the SGX build system (re: the build run before you run the install script) actually grabs *ALL* the drivers from your cross-built kernel tree and places it into its own tree as if the SGX build was your rootfs generator.  Wow.  Talk about clueless.  So the script has to be run on the target platform, the BeagleBoard.  And even then it installs everything under /usr/local/XSGX.  Yuck.  It's a mess.

The correct things to do for the SGX release would be:

• Provide a kernel drivers only package or build target that installs modules based on the standard modules build processes for modules built outside of the kernel tree.  Include a couple of make targets to generate stand alone modules.conf and modules.dep files and let integrators deal with integrating those with the target rootfs.
• Provide an X.org package or build target that does something similar for the X.org modules and config files.  These should be source code since they seem to provide very recent X.org drivers which are not necessarily compatible with build tools like Buildroot's X.org package (though I've yet to test that).
• Provide a libraries and boot-script package or build target that allows installation into a user-specified root.  Honestly, the selection of the destination directory is not dependent on the hardware platform.  The distribution creator is responsible for that choice, and its usually based on the LSB or something similar.  This package can generate a boot script (or as many as needed)  based on which version of ES is selected, but the paths inside that script should be the same no matter which version is chosen.  For platforms that need more than one version, create an include file that sets a variable that points to the default ES version.  The include file can be sourced from the init scripts.  Locations for all these files are standard.
• Provide documentation and binaries in another package that installs under <root>/usr/local,  <root>/usr/local/SGX, or <root>/opt/SGX where integrators can specify the location of <root>.

They could also make opkg's but then they would lock out distributions that don't use opkg.  BeagleBox will eventually support opkg, but doesn't yet.  It's better to provide the release files structured via the LSB definitions for where files go and let distribution creators determine how to package them.

I was going to rewrite the install script so it could be used in a cross-build environment, but the script has a builtin license clause that prevents modification and redistribution, effectively making their package useless in a clean distribution without a bunch of manual cleanup on the target on firstboot.  I don't know if any other distributions deal with this problem.  Robert Nelson provides the SGX stuff for Ubuntu using the install.sh script for on-target installation.  I don't know yet what, if anything, the Android and Angstrom distributions are doing with respect to this problem.

Now we see the benefit of open source licensing.  If this license was more open, I could fix this for them.  But it isn't so I don't see why I would waste my time with it.  I can hack around their poor implementation on my build system, even if it is a bit ugly. How much nicer it would have been to be able to fix the problem and submit a patch.

After removing DRM from being compiled into my 3.2.23 kernel, I had my first initial success with running SGX on my BeagleBoard C4 based on my BeagleBox build.  After an initial boot where I stopped my current Xorg session, I ran omap_demo to setup the environment.  Not sure what this does but I think it does a bunch of symlinking from /usr/local/XSGX to /usr/lib, and so forth.  Anyway, this tried to launch the SGX Xorg session, but it failed.  I rebooted and ran the rc.pvr script manually.  That launched the SGX Xorg on my DVI display but without any clients.

From the serial console I ran glxgears -display :0.  The gears ran on the DVI display at about 47 fps.  Not too good, but at least it ran.  Running it with -fullscreen dropped the FPS to about 5, which is terrible.

So it’s not perfect, but it’s running.  That means I have a working build that supports SGX, even if it doesn’t start at boot time.  Things are gettin’ better.