Migrating from gitorious – it sucks.

[Print This] By mjhammel ~ April 1st, 2015. Filed under: General.

I can no longer push to gitorious so I tried to run the import of all my repos (and there are alot) into GitHub. This failed miserably. Only two of 34 repos were imported. Some of the repos actually got created on GitLab but there is nothing in them. So I tried to import one of them manually. It’s been running for about 20 minutes now and nothing is happening. No errors, no messages, no nothing. Just a spinning wheel.

I dug around and found the process for migrating a repo, be it to GitLab or GitHub or whatever.  It requires manual creation of each repository.  While gitorious allowed me to have different projects with different repos under them, GitHub does not.  At least I don’t see a way to do that.  GitLab does, but they call them groups.  The import appears to have created the groups but that’s all it did.

So I guess I can try doing this manually under each group, after manually creating each repo.  *sigh*  This is a strong argument for running my own git server and gitweb.

Related posts

apache 2.4/Fedora quirk in welcome.conf

[Print This] By mjhammel ~ March 10th, 2015. Filed under: Fedora, Linux, Software Development, Ubuntu.

I was trying to set up a directory index via a named vhost under Apache today.  Typically all you need to do for this scenario is something like this in an apache conf file.

<VirtualHost *:80>
ServerName files.gfxmuse.org
DocumentRoot /home/httpd/files
<directory "="" home="" <span="" class="hiddenSpellError" pre="" data-mce-bogus="1">httpd/files">
Options +Indexes

Then restart apache and make sure the server name resolves somehow (DNS or /etc/hosts) and you’re good to go.

But with Apache 2.4 something changed.  This doesn’t work.  You get the nice welcome page that says the web server is running but you don’t get the directory index.  I found a solution online that hid the solution behind a requirement to “like” the answer but it became very clear what the problem was: the welcome.conf configuration file overrides my directory index.

In welcome.conf you find this bit of code.

<LocationMatch "^/+$">
Options -Indexes
ErrorDocument 403 /.noindex.html

That Options -Index line is the problem.  The easiest solution, in this case where all I want is a web server that shows a directly listing, is to comment out that whole block of code.  Restarting the web server and accessing my file server works as expected now.

I had this problem on Fedora 21.  I’ve seen others have it with Ubuntu.  So the source of welcome.conf content may be the apache project itself.  In other words, this solution may be applicable to any distribution using Apache 2.4 or later.

Related posts

PiBox update: user and developer guides

[Print This] By mjhammel ~ March 2nd, 2015. Filed under: PiBox, Raspberry Pi.

I was job hunting from late November to late January.  Since few of the people needed to do interviews tend to be available during the 12th month of the year, I was able to devote a lot of time working on PiBox development.  However, once the new year rolled in I was bouncing around from office to office in my finest nerd sweaters and cleanest hiking shoes.   Fortunately, no one was interested in my fashion sense.  But it required me to do a bit of travelling and lots of phone chatting, which played havoc with the concentration required to do development work.  So little PiBox work got done.

Fortunately all that ended in mid February when HGST decided I was least offensive candidate that week.    But it takes awhile to get back into a project like PiBox.  So I thought I’d start slowly by finally getting around to writing some end user and developer documentation on the wiki.

I managed to complete a user guide to explain how to use the media systems.  Turns out it’s pretty easy, which is a good thing.  The developer document was a bit harder since I wasn’t sure if I wanted to explain how to use and modify the development platform’s build system or explain how to write apps for the media systems.  Turns out I did a little of both.  That’s a good thing too.  I think.

What’s still missing is the protocol document explaining how messages are passed around the media system components.  I’ve got some diagrams up and an idea of what I need to say.  I just need to get some time to finish it up.

And then I can get back to my issues list.  So much to do.  So little time.  Anyone wanna help?

Related posts

Debian/Ubuntu debootstrap images for VirtualBox

[Print This] By mjhammel ~ February 27th, 2015. Filed under: Fedora, Linux, Software Development, Ubuntu, virtualization.

I’ve been working on building custom Debian and Ubuntu distributions for use under VirtualBox.   One advantage that both have over Fedora is debootstrap.  This tool allows you to create a default rootfs from pre-compiled packages inside a directory.  You can then chroot into that directory to install the kernel image, extra packages and do any additional customizations.

Where this gets really cool is using debootstrap with a few additional tools on an image file.  The process is fairly simple.  First you create an image file using dd and add a partition table for two partitions, boot and rootfs, using parted. Loop mount (kpartx) and bind mount boot under rootfs.  Format the partitions (ext3.mkfs).  Now you’re ready to install the rootfs using debootstrap.  Once installed, chroot into that directory and run customization scripts.  Finally exit the chroot, unmount and install a bootloader such as grub.  That creates the raw disk image.  You can then use qemu-image to convert that to various VM image formats such as those for VirtualBox, Xen and KVM.

I handle this using a front end script that runs 7 steps, most of which are outside of the chroot but a few that are in it.  The seven steps are

  1. Create image file
  2. Create partition table
  3. Loop mount and bind mount
  4. Install rootfs with debootstrap
  5. Copy in chroot scripts and data files
  6. Chroot and run those script and unmount
  7. Convert to VM image

These seven are pretty common for either Debian or Ubuntu.  There are small variations, such as what you include wiht debootstrap and perhaps how you need to do your loop mounts.  The differences come from the chroot scripts and data files.  There is one script that sets up the chroot environment as necessary to perform additional package installations.  Setup can include network interfaces, setting the locale and installing prerequisites necessary to do other installations.

Within the chroot script is the installation of the Linux kernel image.  This gets installed under /boot, which was bind mounted outside of the chroot so we have separate boot and rootfs partitions when we boot the image.  What’s interesting is that most information you find online about this process assumes you’re building debian or ubuntu on debian or ubuntu.  But I’m not.  I’m on Fedora.

This means that when you get to installing grub the instructions you find don’t always match what you need to do.  Fedora 21 uses grub2.  Debian uses grub or grub2 and Ubuntu has grub.  Does it matter?  Turns out it doesn’t.  All that matters is that the grub from your distro is installed to /boot on the image and the MBR of the image file points to it.  That way qemu-image can create an appropriate image for your VM.

For bonus points the use of qemu-image allows you to convert this raw disk image into a variety of VM image formats.  So a single image build is quickly converted to the VM environment of choice.

In the end I found that testing the VM image under VirtualBox was pretty easy, including getting the guest utilities installed as part of a firstboot-process.  Those utils need to rebuild kernel modules and you can’t do that from within the chroot easily.  It’s easier to just create a firstboot script hat installs the utils and rebuilds the modules andthat gets run the first time the image boots in the VM and then removes itself aftward.

So now I can quickly bring up a a Debian VM.  The whole process takes about 15 minutes unattended.  The only problem is it’s not Fedora.  If I could do this under Fedora or CentOS, I’d be sooo happy.

Related posts

Raspberry Pi: Kodi vs PiBox use case comparison

[Print This] By mjhammel ~ February 6th, 2015. Filed under: Hardware, Linux, Movies, PiBox, Raspberry Pi, Software Development, XBMC.
PiBox is part of miot.

PiBox is a member of miot, which is a brand I’m using for a variety of solutions I’m working on for the Raspberry Pi. “miot” stands for “My Internet Of Things”.

I’ve been working on PiBox now for a couple of years.  The immediate goal is a consumer-oriented device for video playback while not connected to the Internet.  In other words, play movies while camping.  I have longer term goals for it, but that’s where the project is aimed currently.

At home I used to use XBMC (now known as Kodi) on a big-metal server sitting behind my big screen TV for media services.  It’s primary use was to play the 500+ movies I’ve ripped to ISO images.  That machine hasn’t run in quite some time as it got quite hot behind the TV and I think it basically died.  Instead, we have a Roku hooked up to that TV and we stream Hulu, Netflix and Amazon.   The problem with the Roku (and other streaming boxes and sticks) is that none seems to support playback of locally stored videos.  So I still have a use for XBMC (re: Kodi) type software.

I could use PiBox for this but because the omxplayer doesn’t support ISO images I would have to rip all my ISO’s into MP4s.  That’s doable, but my wife likes to have the DVD menu system and the ability to add and remove subtitles.  So MP4s by themselves are not sufficient for this use case, at least not without external subtitles ripped separately.

So I installed OpenELEC on an SD card and brought it up on a Raspberry Pi Model B+ with a wifi dongle.  Setup was simple enough and I was able to connect to my NFS export video collection quite easily.  They even do an NFS query something akin to an SMB query.  I didn’t know you could do that.  Anyway, it automatically found my exports and I was able to configure them without any fuss.  I then did a library update and had all the posters I needed, though some are incorrect.  Those can be fixed later if necessary.  The update found all the ISO images plus the mp4s, since both are on the NFS exports.

I then tried to play a movie ripped as an ISO.   The Pi was not overclocked and loading the initial videos leading to the menu took quite some time.  Eventually the loaded and played, albeit with lots of jumpy behaviour.  I didn’t have audio hooked up at the time.  I tried to jump the DVD menu but this never displayed correctly.  I was unable to play the movie.

Next I tried overclocking the Pi.  The lead-in videos didn’t play and I was only able to see the text for “Play Movie’ from the DVD menu.  I managed to start the movie and it played modestly well but still with too many freezes and jumps.  Due to the nature of the video playback I never tried testing the audio playback.

What this tells me is that the ISO use case is not well supported on the Pi.  At least not the Model B+.  The Model 2 might fair better.  I don’t have one yet and won’t be getting one for a while due to changing jobs.  It also tells me that the trailer use case – the use case for PiBox – would be much simpler to use if I just used PiBox and ripped everything to MP4s.  However, I will have to add configuration support for using separately ripped subtitle files.  The VideoFE app, which wraps omxplayer, doesn’t have this yet though omxplayer apparently supports it.

So my design of PiBox with the specific use case of MP4s seems to be a better solution than Kodi if all you want to do is playback your locally ripped videos on a Raspberry Pi.  That’s good to know.  I like Kodi and use it on my desktop.  But it’s not necessarily the right app for a device like the Pi given all use cases.  As a side note, I designed the VideoFE to use whatever playback tool you choose, so it could be used on my desktop as well.  I would just swap out omxplayer with xine or mplayer.  I haven’t tried that yet, but intend to before too long.


Related posts

Life update: new gig @ HGST

[Print This] By mjhammel ~ February 6th, 2015. Filed under: General, Personal.

Last November I left CEI where I had worked for 8 1/2 years.  I had plenty saved up to cover me to the next gig.  So out went the resumes into the void and we waited for the holidays to pass to get an idea of how things have changed in the techy job market.

Turns out they’ve changed a lot.  I got 8 hits by the end of first full week of January.  Four of those went the distance with 2 offers, 1 offer in progress and 1 that eventually bowed out. All of these were remote, either in Seattle or Portland or work-from-home.  A few others interviewed me but then disappeared.  Only Amazon interviewed me and then turned around and rejected me right away.  That’s the third time they’ve done that so I think I’m done interviewing with Amazon.   They don’t do any real embedded work anyway (their Lab126 subsidiary does, but not Amazon itself).

hgstWhile waiting for those offers to be finalized HGST swooped in at the last second.  They contacted me, set up an interview, brought me in and made an offer in less than a week.  HGST has a relatively new office here in Colorado Springs so I wouldn’t have to relo.   They also made a terrific offer, better than any of the others.  And the office is actually closer than CEI.  Considering my old commute was about 6 minutes, I found it surprising I’d get an offer from someone even closer to home.

We had been excited by the idea of moving to the Pacific Northwest.  We’ve been there and really loved it, despite what everyone says about it raining or being overcast all the time.  Unfortunately, the timing is wrong.  My house won’t sell for what I bought it for so I gotta wait that out for at least a couple more years.  Economic changes come slowly to Colorado Springs.  It takes a long time to drop down like everyone else, but then it takes a lot longer to rebound after everyone else has.   And if we wait a couple more years I may just stay here and buy a vacation spot somewhere else.  It’s hard to leave the mountains behind too.

So I’m happy to announce that I’ve accepted the HGST gig.  It’s a storage product group and I’ll be working with firmware – bootloaders, Linux distro, customization and product apps.  Cool beans, I say.  My luck never seems to run out, it seems.  My wife and I are planning to build out the basement (hello, media room), make major updates to the landscaping and settle in for awhile.  If we need to get away, we’ve still got the Pod.  And DisneyWorld.

Life is good.

Related posts

git changelog generation: one liner

[Print This] By mjhammel ~ October 27th, 2013. Filed under: General.

Needed some place to note this short one-liner for generating a changelog between version releases with git.

git log v0.5...v0.6 --pretty=format:'%s'

The output looks something like this.

RM #225: Fix xcc to build when gmp is already installed. This is a back-rev'd patch from Crosstool-NG that required no changes so xcc would be on Fedora 19.
RM #225: Disable XBMC for dev platform build except for tinyxml.
RM #225: crtmpserver depends on tinyxml from XBMC package added by PiBox to Buildroot.

This works as long as you've tagged your tree.  To tag the tree, use an annotated tag and then push to the origin to share it.

git tag -a v0.1 -m"Why did I do this?"
git push origin --tags

Tha'ts it.

Related posts

DisplayLink vs TFT resistive touch screen for PiBox

[Print This] By mjhammel ~ August 21st, 2013. Filed under: Hardware, Raspberry Pi.

Okay, so working alone you often find yourself falling behind the curve.  I just got my DisplayLink working the other night and today I found this little device – a 2.8" TFT resistive touch screen for the Pi


This is a much better solution than the DisplayLink for the trailer.  But if I spend any more money getting this thing working I may have to sell one of the dogs.  Eeek.

I'm going to keep going with what I have and then return to the touch screen option a little later.  I need to get to a fully functioning prototype before I go to cost -reduction and feature-enhancement mode.

You can buy this display via eBay.uk.  It's about $32US, plus shipping.  There is a discussion topic about it on the Raspberry Pi forums for the original version, which has since been updated.

Related posts

PiBox: 0.5.0 released, many new features since

[Print This] By mjhammel ~ August 19th, 2013. Filed under: Hardware, Linux, omapfb, PiBox, Raspberry Pi, X11, XBMC.

A while back I made an official 0.5.0 release for PiBox which I announced on the Raspberry Pi Forums.  Since then I've made significant advances in preparing PiBox for use as a media server in my travel trailer.  Some of the features I've recently implemented include:

  • Support for running as a wireless access point
  • Support for full network configuration through a custom UI
  • Support for running dhcp server
  • Support for USB-attached DisplayLink Monitor
  • Enabled NFS support
  • Enabled webcam support

    Other features which are currently in development include

    • Web streaming of webcam
    • I2C monitor of sensors (temperature at a minimum)
    • Alsa sound support
    • Integration of XBMCBox package to play NFS mounted videos

    At the moment PiBox will depend on a 10 port USB powered hub with a 2.5A or higher rating.  The DisplayLink monitor will be used for system monitoring displays.  A miniature wireless keyboard/mouse device will initially be used for input but I'd like to switch that to either a touch screen DisplayLink monitor (preferred) or a handheld remote of some kind. 

    When I'm done I expect to be able to:

    • Stream webcam monitoring out the back window to a tablet's browser (so I can see behind the trailer from the tow vehicle using existing capabilities of tablets and phones)
    • Serve media files mounted on USB attached SD cards over NFS
    • Play videos wirelessly over NFS through XBMC using an HDMI attached micorDLP projector (that displays on the outside wall of the trailer but inside our R-Dome).

    Things are moving along pretty good now.  I still have some cleanup to do of the DisplayLink UI and will have to write special applications for monitoring and displaying output from sensors.  But I'm getting close to being able to stream the webcam and using XBMC on a second PiBox.  With luck I should have most of this working before Halloween.

    Related posts

    Evolution out, Thunderbird in (or how to migrate the quickly from Evolution to something else)

    [Print This] By mjhammel ~ August 19th, 2013. Filed under: Fedora, Linux.

    NOTE: This doesn't work!  It's missing all my recent saved mail.  Evolution saves the newer stuff in individual files with a ",S" suffix in a different directory.  Ugh.  I have Evolution.

    My Fedora 19 upgrade at work has gone badly.  I've reinstalled twice, back peddled to F18 (which didn't work at all) and tried switching to CentOS (which failed trying to read my Evolution folders).  Right now I'm on F19 and the XFce desktop is still fairly slow, though better than my first tries (multiple kernel and x.org updates since then).  But Evolution is completely unusable.  It's painfully slow to do anything, from switching accounts to just browsing a single accounts email.  I've tried a number of things I found online but none seemed to have any impact.  I've tried messing with hdparm in case that was a culprit.  Nothing helps.  Evolution in F19 is too slow to be usable.  So I've switched to Thunderbird.

    The trick to switching is understanding that Evolution keeps mail in mbox folders along with associated indexing files.  You can ignore the indexing files and just copy over the mboxes to Thunderbirds Local Folders directory manually.  When you start Thunderbird it will see the new mboxes and you're all set though you still have to set up your imap/pop3 accounts – I didn't try to do that outside of thunderbird.  I also didn't worry about contact lists or calendars because the former I can recreate as I go and the latter I don't really care about.

    To make it easier for anyone who hasn't tried this, here are the steps I took, including making a backup of the Evolution folders before doing any of this.  Much of the process for doing this was already online, except for the part of collecting just the mboxes from Evolution.  Let's look at that process first.

    1. mkdir ~/backups/evolution
    2. cd ~/.local/share/evolution/mail
    3. rsync -a local_mbox ~/backups/evolution
    4. cd ~/backups/evolution
    5. find local_mbox -type f -iname "*.cmeta" | egrep -v "Junk|Trash" | sed 's/.cmeta//' > filelist.txt
    6. mkdir mbox
    7. tar -T filelist.txt -cvf - | (cd mbox; tar xvf -)

    Now you have a collection of mboxes, in their original folder structure, that you can import into another mail reader.  To get them into either Thunderbird or SeaMonkey:

    1. In Thunderbird, right click on Local Folders, select Settings.
    2. Note Local directory path
    3. Open a file browser and navigate to that directory.
    4. Open second file browser and navigate to ~/backups/evolution/mbox/local_mbox
    5. Select all files/directories in second file browser.
    6. Choose Edit->Copy
    7. In first file browser choose Edit->Paste to copy all files and folders to Thunderbird/SeaMonkey

    All your old email is now available under Local Folders in Thunderbird or SeaMonkey.  To do the same with Claws-Mail:

    1. Create folders under mbox (which may or may not be available – I'm not sure if I created that accidentally or not)
    2. Import mbox into folders one at a time.

    It's harder with Claws-Mail because there is no default Local Folders under which you can just copy the mboxes.  At least I couldn't find one.

    When you're done, you'll want to delete the backup as it won't be required anymore.  But hang onto your original evolution folders (~/.local/share/evolution) in case you need to refer to it again later.

    Related posts