Debian/Ubuntu debootstrap images for VirtualBox



[Print This] By mjhammel ~ February 27th, 2015. Filed under: Fedora, Linux, Software Development, Ubuntu, virtualization.

I’ve been working on building custom Debian and Ubuntu distributions for use under VirtualBox.   One advantage that both have over Fedora is debootstrap.  This tool allows you to create a default rootfs from pre-compiled packages inside a directory.  You can then chroot into that directory to install the kernel image, extra packages and do any additional customizations.

Where this gets really cool is using debootstrap with a few additional tools on an image file.  The process is fairly simple.  First you create an image file using dd and add a partition table for two partitions, boot and rootfs, using parted. Loop mount (kpartx) and bind mount boot under rootfs.  Format the partitions (ext3.mkfs).  Now you’re ready to install the rootfs using debootstrap.  Once installed, chroot into that directory and run customization scripts.  Finally exit the chroot, unmount and install a bootloader such as grub.  That creates the raw disk image.  You can then use qemu-image to convert that to various VM image formats such as those for VirtualBox, Xen and KVM.

I handle this using a front end script that runs 7 steps, most of which are outside of the chroot but a few that are in it.  The seven steps are

  1. Create image file
  2. Create partition table
  3. Loop mount and bind mount
  4. Install rootfs with debootstrap
  5. Copy in chroot scripts and data files
  6. Chroot and run those script and unmount
  7. Convert to VM image

These seven are pretty common for either Debian or Ubuntu.  There are small variations, such as what you include wiht debootstrap and perhaps how you need to do your loop mounts.  The differences come from the chroot scripts and data files.  There is one script that sets up the chroot environment as necessary to perform additional package installations.  Setup can include network interfaces, setting the locale and installing prerequisites necessary to do other installations.

Within the chroot script is the installation of the Linux kernel image.  This gets installed under /boot, which was bind mounted outside of the chroot so we have separate boot and rootfs partitions when we boot the image.  What’s interesting is that most information you find online about this process assumes you’re building debian or ubuntu on debian or ubuntu.  But I’m not.  I’m on Fedora.

This means that when you get to installing grub the instructions you find don’t always match what you need to do.  Fedora 21 uses grub2.  Debian uses grub or grub2 and Ubuntu has grub.  Does it matter?  Turns out it doesn’t.  All that matters is that the grub from your distro is installed to /boot on the image and the MBR of the image file points to it.  That way qemu-image can create an appropriate image for your VM.

For bonus points the use of qemu-image allows you to convert this raw disk image into a variety of VM image formats.  So a single image build is quickly converted to the VM environment of choice.

In the end I found that testing the VM image under VirtualBox was pretty easy, including getting the guest utilities installed as part of a firstboot-process.  Those utils need to rebuild kernel modules and you can’t do that from within the chroot easily.  It’s easier to just create a firstboot script hat installs the utils and rebuilds the modules andthat gets run the first time the image boots in the VM and then removes itself aftward.

So now I can quickly bring up a a Debian VM.  The whole process takes about 15 minutes unattended.  The only problem is it’s not Fedora.  If I could do this under Fedora or CentOS, I’d be sooo happy.

Related posts

Raspberry Pi: Kodi vs PiBox use case comparison



[Print This] By mjhammel ~ February 6th, 2015. Filed under: Hardware, Linux, Movies, PiBox, Raspberry Pi, Software Development, XBMC.
PiBox is part of miot.

PiBox is a member of miot, which is a brand I’m using for a variety of solutions I’m working on for the Raspberry Pi. “miot” stands for “My Internet Of Things”.

I’ve been working on PiBox now for a couple of years.  The immediate goal is a consumer-oriented device for video playback while not connected to the Internet.  In other words, play movies while camping.  I have longer term goals for it, but that’s where the project is aimed currently.

At home I used to use XBMC (now known as Kodi) on a big-metal server sitting behind my big screen TV for media services.  It’s primary use was to play the 500+ movies I’ve ripped to ISO images.  That machine hasn’t run in quite some time as it got quite hot behind the TV and I think it basically died.  Instead, we have a Roku hooked up to that TV and we stream Hulu, Netflix and Amazon.   The problem with the Roku (and other streaming boxes and sticks) is that none seems to support playback of locally stored videos.  So I still have a use for XBMC (re: Kodi) type software.

I could use PiBox for this but because the omxplayer doesn’t support ISO images I would have to rip all my ISO’s into MP4s.  That’s doable, but my wife likes to have the DVD menu system and the ability to add and remove subtitles.  So MP4s by themselves are not sufficient for this use case, at least not without external subtitles ripped separately.

So I installed OpenELEC on an SD card and brought it up on a Raspberry Pi Model B+ with a wifi dongle.  Setup was simple enough and I was able to connect to my NFS export video collection quite easily.  They even do an NFS query something akin to an SMB query.  I didn’t know you could do that.  Anyway, it automatically found my exports and I was able to configure them without any fuss.  I then did a library update and had all the posters I needed, though some are incorrect.  Those can be fixed later if necessary.  The update found all the ISO images plus the mp4s, since both are on the NFS exports.

I then tried to play a movie ripped as an ISO.   The Pi was not overclocked and loading the initial videos leading to the menu took quite some time.  Eventually the loaded and played, albeit with lots of jumpy behaviour.  I didn’t have audio hooked up at the time.  I tried to jump the DVD menu but this never displayed correctly.  I was unable to play the movie.

Next I tried overclocking the Pi.  The lead-in videos didn’t play and I was only able to see the text for “Play Movie’ from the DVD menu.  I managed to start the movie and it played modestly well but still with too many freezes and jumps.  Due to the nature of the video playback I never tried testing the audio playback.

What this tells me is that the ISO use case is not well supported on the Pi.  At least not the Model B+.  The Model 2 might fair better.  I don’t have one yet and won’t be getting one for a while due to changing jobs.  It also tells me that the trailer use case – the use case for PiBox – would be much simpler to use if I just used PiBox and ripped everything to MP4s.  However, I will have to add configuration support for using separately ripped subtitle files.  The VideoFE app, which wraps omxplayer, doesn’t have this yet though omxplayer apparently supports it.

So my design of PiBox with the specific use case of MP4s seems to be a better solution than Kodi if all you want to do is playback your locally ripped videos on a Raspberry Pi.  That’s good to know.  I like Kodi and use it on my desktop.  But it’s not necessarily the right app for a device like the Pi given all use cases.  As a side note, I designed the VideoFE to use whatever playback tool you choose, so it could be used on my desktop as well.  I would just swap out omxplayer with xine or mplayer.  I haven’t tried that yet, but intend to before too long.

 

Related posts

Life update: new gig @ HGST



[Print This] By mjhammel ~ February 6th, 2015. Filed under: General, Personal.

Last November I left CEI where I had worked for 8 1/2 years.  I had plenty saved up to cover me to the next gig.  So out went the resumes into the void and we waited for the holidays to pass to get an idea of how things have changed in the techy job market.

Turns out they’ve changed a lot.  I got 8 hits by the end of first full week of January.  Four of those went the distance with 2 offers, 1 offer in progress and 1 that eventually bowed out. All of these were remote, either in Seattle or Portland or work-from-home.  A few others interviewed me but then disappeared.  Only Amazon interviewed me and then turned around and rejected me right away.  That’s the third time they’ve done that so I think I’m done interviewing with Amazon.   They don’t do any real embedded work anyway (their Lab126 subsidiary does, but not Amazon itself).

hgstWhile waiting for those offers to be finalized HGST swooped in at the last second.  They contacted me, set up an interview, brought me in and made an offer in less than a week.  HGST has a relatively new office here in Colorado Springs so I wouldn’t have to relo.   They also made a terrific offer, better than any of the others.  And the office is actually closer than CEI.  Considering my old commute was about 6 minutes, I found it surprising I’d get an offer from someone even closer to home.

We had been excited by the idea of moving to the Pacific Northwest.  We’ve been there and really loved it, despite what everyone says about it raining or being overcast all the time.  Unfortunately, the timing is wrong.  My house won’t sell for what I bought it for so I gotta wait that out for at least a couple more years.  Economic changes come slowly to Colorado Springs.  It takes a long time to drop down like everyone else, but then it takes a lot longer to rebound after everyone else has.   And if we wait a couple more years I may just stay here and buy a vacation spot somewhere else.  It’s hard to leave the mountains behind too.

So I’m happy to announce that I’ve accepted the HGST gig.  It’s a storage product group and I’ll be working with firmware – bootloaders, Linux distro, customization and product apps.  Cool beans, I say.  My luck never seems to run out, it seems.  My wife and I are planning to build out the basement (hello, media room), make major updates to the landscaping and settle in for awhile.  If we need to get away, we’ve still got the Pod.  And DisneyWorld.

Life is good.

Related posts

git changelog generation: one liner



[Print This] By mjhammel ~ October 27th, 2013. Filed under: General.

Needed some place to note this short one-liner for generating a changelog between version releases with git.

git log v0.5...v0.6 --pretty=format:'%s'

The output looks something like this.

RM #225: Fix xcc to build when gmp is already installed. This is a back-rev'd patch from Crosstool-NG that required no changes so xcc would be on Fedora 19.
RM #225: Disable XBMC for dev platform build except for tinyxml.
RM #225: crtmpserver depends on tinyxml from XBMC package added by PiBox to Buildroot.

This works as long as you've tagged your tree.  To tag the tree, use an annotated tag and then push to the origin to share it.

git tag -a v0.1 -m"Why did I do this?"
git push origin --tags

Tha'ts it.

Related posts

DisplayLink vs TFT resistive touch screen for PiBox



[Print This] By mjhammel ~ August 21st, 2013. Filed under: Hardware, Raspberry Pi.

Okay, so working alone you often find yourself falling behind the curve.  I just got my DisplayLink working the other night and today I found this little device – a 2.8" TFT resistive touch screen for the Pi

touchdisplay

This is a much better solution than the DisplayLink for the trailer.  But if I spend any more money getting this thing working I may have to sell one of the dogs.  Eeek.

I'm going to keep going with what I have and then return to the touch screen option a little later.  I need to get to a fully functioning prototype before I go to cost -reduction and feature-enhancement mode.

You can buy this display via eBay.uk.  It's about $32US, plus shipping.  There is a discussion topic about it on the Raspberry Pi forums for the original version, which has since been updated.

Related posts

PiBox: 0.5.0 released, many new features since



[Print This] By mjhammel ~ August 19th, 2013. Filed under: Hardware, Linux, omapfb, PiBox, Raspberry Pi, X11, XBMC.

A while back I made an official 0.5.0 release for PiBox which I announced on the Raspberry Pi Forums.  Since then I've made significant advances in preparing PiBox for use as a media server in my travel trailer.  Some of the features I've recently implemented include:

  • Support for running as a wireless access point
  • Support for full network configuration through a custom UI
  • Support for running dhcp server
  • Support for USB-attached DisplayLink Monitor
  • Enabled NFS support
  • Enabled webcam support

    Other features which are currently in development include

    • Web streaming of webcam
    • I2C monitor of sensors (temperature at a minimum)
    • Alsa sound support
    • Integration of XBMCBox package to play NFS mounted videos

    At the moment PiBox will depend on a 10 port USB powered hub with a 2.5A or higher rating.  The DisplayLink monitor will be used for system monitoring displays.  A miniature wireless keyboard/mouse device will initially be used for input but I'd like to switch that to either a touch screen DisplayLink monitor (preferred) or a handheld remote of some kind. 

    When I'm done I expect to be able to:

    • Stream webcam monitoring out the back window to a tablet's browser (so I can see behind the trailer from the tow vehicle using existing capabilities of tablets and phones)
    • Serve media files mounted on USB attached SD cards over NFS
    • Play videos wirelessly over NFS through XBMC using an HDMI attached micorDLP projector (that displays on the outside wall of the trailer but inside our R-Dome).

    Things are moving along pretty good now.  I still have some cleanup to do of the DisplayLink UI and will have to write special applications for monitoring and displaying output from sensors.  But I'm getting close to being able to stream the webcam and using XBMC on a second PiBox.  With luck I should have most of this working before Halloween.

    Related posts

    Evolution out, Thunderbird in (or how to migrate the quickly from Evolution to something else)



    [Print This] By mjhammel ~ August 19th, 2013. Filed under: Fedora, Linux.

    NOTE: This doesn't work!  It's missing all my recent saved mail.  Evolution saves the newer stuff in individual files with a ",S" suffix in a different directory.  Ugh.  I have Evolution.

    My Fedora 19 upgrade at work has gone badly.  I've reinstalled twice, back peddled to F18 (which didn't work at all) and tried switching to CentOS (which failed trying to read my Evolution folders).  Right now I'm on F19 and the XFce desktop is still fairly slow, though better than my first tries (multiple kernel and x.org updates since then).  But Evolution is completely unusable.  It's painfully slow to do anything, from switching accounts to just browsing a single accounts email.  I've tried a number of things I found online but none seemed to have any impact.  I've tried messing with hdparm in case that was a culprit.  Nothing helps.  Evolution in F19 is too slow to be usable.  So I've switched to Thunderbird.

    The trick to switching is understanding that Evolution keeps mail in mbox folders along with associated indexing files.  You can ignore the indexing files and just copy over the mboxes to Thunderbirds Local Folders directory manually.  When you start Thunderbird it will see the new mboxes and you're all set though you still have to set up your imap/pop3 accounts – I didn't try to do that outside of thunderbird.  I also didn't worry about contact lists or calendars because the former I can recreate as I go and the latter I don't really care about.

    To make it easier for anyone who hasn't tried this, here are the steps I took, including making a backup of the Evolution folders before doing any of this.  Much of the process for doing this was already online, except for the part of collecting just the mboxes from Evolution.  Let's look at that process first.

    1. mkdir ~/backups/evolution
    2. cd ~/.local/share/evolution/mail
    3. rsync -a local_mbox ~/backups/evolution
    4. cd ~/backups/evolution
    5. find local_mbox -type f -iname "*.cmeta" | egrep -v "Junk|Trash" | sed 's/.cmeta//' > filelist.txt
    6. mkdir mbox
    7. tar -T filelist.txt -cvf - | (cd mbox; tar xvf -)

    Now you have a collection of mboxes, in their original folder structure, that you can import into another mail reader.  To get them into either Thunderbird or SeaMonkey:

    1. In Thunderbird, right click on Local Folders, select Settings.
    2. Note Local directory path
    3. Open a file browser and navigate to that directory.
    4. Open second file browser and navigate to ~/backups/evolution/mbox/local_mbox
    5. Select all files/directories in second file browser.
    6. Choose Edit->Copy
    7. In first file browser choose Edit->Paste to copy all files and folders to Thunderbird/SeaMonkey

    All your old email is now available under Local Folders in Thunderbird or SeaMonkey.  To do the same with Claws-Mail:

    1. Create folders under mbox (which may or may not be available – I'm not sure if I created that accidentally or not)
    2. Import mbox into folders one at a time.

    It's harder with Claws-Mail because there is no default Local Folders under which you can just copy the mboxes.  At least I couldn't find one.

    When you're done, you'll want to delete the backup as it won't be required anymore.  But hang onto your original evolution folders (~/.local/share/evolution) in case you need to refer to it again later.

    Related posts

    F19 upgrade: is this really getting better?



    [Print This] By mjhammel ~ August 6th, 2013. Filed under: Fedora, GTK+, Linux, NVidia, PulseAudio, X11, XFce.

    I'm still a Fedora user, even after years of releases that seem to cause more headaches than they solve.  I understand it's a bleeding edge distro.  But at the current rate of change it's bound to bleed out by the end of the year.  Systemd is a pain for me as a developer and so far does not seem to improve any boot time issues.  We're not down to 1 second boots, though we do seem to be down to 2 second shutdowns.  Big deal.  I only reboot when I do a system upgrade.  And then a bunch afterwards to fix everything broken by the upgrade.

    Most of my systems (and I have many between work and home) are running F16 with a couple F14 and one F12.  Last week my desktop at work had a disk failure and I had to reinstall.  So I jumped to F19.  I expected bumps along the way, especially given all I'd heard about the modified installation process.  It turned out that the installation went fine.  It's the running system that's giving me problems.

    First, nouveau crashed repeatedly.  I put up with three of these before switching to nVidia's akmod.  Unfortunately, I put the wrong one on the first time.  After a few hours of cleaning up the mess I went back to nouveau, went through a few more crashes and then tried nVidia's drivers again.  This time I got it right.  System stable.  Just incredibly slow.

    The desktop experience is almost painful with this release.  I'm using XFce and Firefox.  Scrolling in the browser is jumpy.  Switching workspaces quickly causes window refreshes to be very slow, on the order of seconds (at times, up to 20 or 30 seconds).  Evolution is slow too.  Just clicking on one message and then the next takes a full second or longer to change the display.  Maybe this is a change to the way spamassassin is integrated.  It wasn't that slow in F16.

    These issues may not be related but the overall effect is the desktop experience is terrible right now.  I'm trying to work from a single workspace for most of my work but that's a change from my usual workflow.  And I'm not liking it.

    There are threads online that state that setting "slub_debug=-" will disable debugging in the kernel to improve performance.  However, my kernel is not supposed to be a debug kernel (though the kernel config shows lots of debugging enabled).  I disabled slub_debug and performance improved slightly.  Workspace switching may be a little better, but not much.  Evolution seems a little better, but that may be due to some config changes I made.  Firefox isn't much changed.  I'm muddling through. Shading on my windows is slow.  I can roll them up quickly.  But rolling them down gets a white window for a couple of seconds before it gets updated with correct content.  I'd blame nVidia's drivers, but nouveau crashed, so I'm kind of out of options there.  I've installed all the X.org updates, all the evolution updates, all the browser updates.  All the updates.  Reboot, reboot, reboot.

    Then there are all the application updates.  The first to annoy me (I'm old and easily annoyed by change on my desktop) was screen.  I use a hardstatus line that shows each screen window title centered at the bottom.  In the latest version of screen this no longer worked.  Instead, I get "user@hostname:<cwd>".  This overrides my window title setting (screen -t).  I verified it does not come from my BASH configuration.  It's a change to the way screen works.  The fix, after many hours of fiddling with changes to the hardstatus line and other configuration options, is to use "-T linux" to set the terminal type when I open my windows on startup.  So I went from this

    screen -t blah 1

    to this

    screen -T linux -t blah 1

    Now things work again.  And that's totally by accident.  There was nothing online I could find about this problem.  I just read the man page again and experimented.  Lucky me.

    There were additional issues with getting the volume control working again.  More twiddling with pulse-audio and plugins.  I'd write those down here but I've already forgotten what I did.  Trial and error.   Disabling the firewall is changed.  Google it.  Not hard, but it has something to do with systemd tools.  The GUI is gone.  Then there is the change to, I think, autotools.  A GTK+2 app built on F16 just fine.  On F19 I have to add -lphtread to the Makefile.am.  Both F16 and F19 libc require -lpthread.  In F16 autotools (autogen.sh/configure) picked it up automatically.  In F19 it doesn't.  Why?  Seems pretty important if libc requires it.  Who provides it to autogen?  Doesn't appear to be glib or gtk (based on pkconfig).  So something internal to autotools appears to have changed.  And broken.  I muddle on.

    I've already switched a number of systems that are pseudo-production oriented (media servers, dev servers) to CentOS because I need them stable for long periods.  I've wanted to stay with Fedora mostly because getting media playback was easier on it than on CentOS (and I have no interest in Ubuntu or variations thereof).  If I can find the tricks to getting media playback working as easily as Mauriat Miranda's Fedora instructions I may switch to CentOS on all my systems.  Interestingly, I still haven't tried the media playback stuff on F19 at work.  I just listen to GotRadio in a browser.  Maybe I'll switch to CentOS before I try them.  Till then, I muddle on.

    Then again, I have my own distro for my Raspberry Pi.  Maybe I should extend it for the desktop….

    Related posts

    PiBox: Another long break ends and new work begins



    [Print This] By mjhammel ~ July 7th, 2013. Filed under: General, Linux, PiBox, Raspberry Pi.

    First, let me say that the fire is long gone and we were never evacuated.  It was stressful waiting around for them to decide if we needed to go or not, but in the end the fire moved away from us and not closer to us.  Now we have everything unpacked, have made a list of what we'd grab next time (to make it easier) and even got rid of a bunch of junk we realized was just junk anyway while we packed to evacuate.  We're a bit better prepared for next time.  Hopefully, there won't be one.  But now back to business: PiBox.

    I've taken a long break from PiBox development to deal with some personal issues.  We bought a travel trailer, so it wasn't bad personal stuff – it was fun personal stuff.  And it leads me to some new ideas for PiBox.

    There are a couple of features I'm going to add to PiBox to support our entertainment and safety needs in our new trailer.  In short these are:
    # A wifi backup camera
    # Network export of SD card data (for movies over NFS)
    # Wireless audio
    # A pico DLP projector.
    # Make PiBox act as a wireless access point.

    The projector shouldn't require much extra.  Just plug into the HDMI and its ready to go.  But I'll need power to it and I'm not sure how I'll deal with that just yet.

    The wireless audio should be supported with a USB stick and external speakers. 

    The network support of SD card data over NFS means making sure the kernel and user space support is in place. 

    Also, since I'll need to use USB to add the SD cards (to give a variety of video options) I'll need an external USB hub with a bunch of ports.  It will need to be powered, and the Pi will be powered from it, including wireless keyboard/mouse.

    The wifi backup camera may prove challenging.  I've already experimented with a number of solutions and finally found a mix of ffmpeg, crtmpserver and Jw Player to provide the least amount of lag on Fedora.  The question is: can I cross compile crtmpserver and will JWPlayer work correctly if served from PiBox?  I think the answer to both is "yes", but I've yet to verify that.

    Making the Pi into a wireless access point/router should be fairly painless, especially since my default Wifi adapter already supports that feature.
     
    Another side project has been to build a portable power solution for the Raspberry Pi.  I've designed the board and have ordered and received the parts, but I've yet to put the thing together.  It should allow for battery power to run the Pi, which would be nice for use with the DPL projector.  However, I can also just use 110v power at the campsites since we don't plan (yet) on dry camping.

    In the end I hope to allow PiBox to provide the following:
    * Wireless backup camera serving via RTP to a tablet browser so I can see behind the trailer while we're driving.
    * A wireless media server providing AVI videos on SD cards.
    * A second Pi with the media server mounted to play AVI's through a DLP projector and wireless speakers while camping.
    * Self-powered media players using DLP projectors for the bunks in the trailer.

    Anyway, some fun stuff to work on.  Time to get back at it.
     

    Related posts

    Fire update – yeah, we’re close



    [Print This] By mjhammel ~ June 12th, 2013. Filed under: General.

    The Black Forest Fire is north and east of us.  We’re currently in voluntary evacuation mode, which is the second level of evacuation out of three.  Next mode is mandatory.  However, we’re on the far south side of the box that marks the voluntary evacuation.  And we’ve got a lot of grassland (not trees) between us and the fire, plus a major road on which its likely they could make a stand to prevent the fire jumping (like they did on Vindicator last year with the Waldo Canyon Fire).  There is little smoke here – winds are blowing north-northwest, which is away from us.  At least for now.

    So we’ve got both cars packed with valuables and we’re ready to bolt if we need to.  The dogs are clueless, thankfully.  We just don’t think its likely to head our way tonight.  Tomorrow could be different if the winds change.  They won’t have a handle on this thing for a couple of days.

    Weird thing is we just bought – yesterday – a Forest River R-Pod travel trailer.  So we have a “2nd home”, albeit reeeeally tiny.  Worse: my car can’t tow it yet.  I have an appointment to have the trailer hitch upgraded for next Wednesday.  If we’re still in voluntary evacuation tomorrow I may call to see if I can get that moved up.

    Life is weird.  Last year my sister was sitting on the edge of the Waldo Canyon Fire.  The fire fighters did a great job to help save her home.  This year, I’m hoping the same for our home.  But mother nature is fickle … and sometimes just plain mean.

    Related posts