[Print This] By mjhammel ~ January 9th, 2016. Filed under: General, Linux, Video, Movies, PiBox, Raspberry Pi.
PiBox was built as a proof of concept for a variety of purposes. One of those was to serve media in my trailer when we go camping. I use omxplayer to play videos that are provided over SMB between a server and a player system. That works okay but the front end is not particularly user friendly. At least it’s not easy to browse videos – they’re currently in one long list. It’s on my list to fix but who knows when I’ll get to tit.
The other day I was reading some tech headlines for Linux and ran across mention of running minidlna on a Raspberry Pi. I wasn’t familiar with minidlna, or ReadyMedia as it’s now known. A little digging uncovered that it’s an open source version of a NetGear product for streaming media: movies, music and pictures. It works by implementing DLNA compliant protocols, a specification designed just for media sharing. I decided I’d try it out at home first. To see how well it worked.
The project doesn’t have Fedora or CentOS packages but it does provide a statically linked binary. I downloaded that and found it only required install of two files: the binary daemon and a configuration file. The configuration file is short and well commented making it easy to setup. It helped that I had a large collection of music and videos ripped. So I installed the files and ran the daemon as my own user (not root).
Now I just needed some software player that could handle DLNA. Fortunatley, I have a house full of Rokus. There is a simple player app that supports DLNA. I clicked on it and there was my server. I drilled down a few folders and found my music and video. Just like that. In my many years of trying to stream media, that has got to be the simplest tool setup I’ve ever come across.
The bad news is that most of my 600+ movies are ripped as ISO images. That was done to get the DVD navigation, jumping to chapters, etc. Now I’ll have to re-rip them as MP4 or MKV since the Roku doesn’t play ISO images.
I also need to see how I can add movies posters. While MP3’s have album art, the MP4’s appear to just have filenames. Something tells me I can fix that. I just need to find a way to use poster art where it currently exists.
What’s interesting with this is the possibility of streaming from PiBox using ReadyMedia to a Roku in my trailer. The Roku has an HDMI output, which will support the pico projector we use to play movies on the side of the trailer. The PiBox provides the wifi access point, so the Roku should work fine, I think. I haven’t tried this, but it’s something now added to my todo list. Since omxplayer doesn’t do dlna (that I know of), this may be an alternative to using PiBox with omxplayer for the player.
[Print This] By mjhammel ~ December 27th, 2015. Filed under: Linux, Hardware, PiBox, Raspberry Pi, Arduino.
My first PCBs arrived in early December, but due to the holidays I haven’t done anything with them yet. I just wanted to show what they look like.
There are 5 small boards attached to my PCB:
- An FTDI Basic from Sparkfun for connecting a USB cable to my Linux box for programming the microcontroller
- An Arduino Mini-Pro board
- A RTC clock board
- A tiny color display board
- An ESP8266 board
There is also a small USB connector for power. I don’t have to use a real USB connector for this. I just needed the power and ground points from a USB part for Fritzing to not complain about unclosed loops in the PCB layout. I’ll wire this directly to a power converter that takes 110V input and drops it to 5V output. That way I can use the same power input used for the outlets in this project to power the PCB (and relays, for that matter).
Not all of these have to be connected. I could just use the pinouts for them and wire them manually. For example, I’ll probably just add L-shaped male headers to the board for the FTDI Basic. It doesn’t have to be installed for the board to work.
[Print This] By mjhammel ~ December 24th, 2015. Filed under: General, Hardware, Holiday lighting, PiBox, Raspberry Pi, Arduino.
What do you get when an old software geek tries to learn about electricity? The same thing you get when you teach a 5 year old not to put their hands on the red coils on the stove. Experience through interactivity. Today’s lesson: fat wires vs thin wires.
I’ve been using small gauge wire to connect relays to outlets in my power controller project. I did this because I thought I needed thick wire to handle potentially large loads through the outlets. I expected to have multiple lights and possibly multiple water pumps attached to each outlet. Fatter wires are used for larger loads (ever seen thin wires on power poles?). So I chose fat wires to connect the relays and outlets. Wire thickness is measured as a gauge. Fat wires have a smaller gauge, which means thin wires have a higher gauge. Why? Beats me. That’s just the way it is.
The smaller gauge wires don’t fit in the screw-down holes for the relays on the Sainsmart board. I had to trim off some of the wire to make them fit cleanly. But the relays only handle 10A max and the distance between the relays and the outlets is less than 6 inches. I’m running standard 110V from the wall through the relays to the outlets. So how thick do these wires need to be?
After looking at an online wire capacity chart I realized I didn’t need the smaller gauge. I could use the larger 22 gauge wire (remember that larger gauge means thinner wire) for the very short runs between the relays and outlets. This is because if I take an average load per outlet of 6 lights averaging about 75 watts per bulb (remember this is lighting for my aquaponics at the moment) I find I’m only using 450 watts over the 110V power input. Using an online calculator I can see I’m only pulling about 4.1 amps. That’s far below the max for the relays and well within the safe limits listed in the Capacity Chart for 22 gauge wire. And this gets better if the lights are switched from high wattage incandescent bulbs to LEDs and CFLs, which most of them are now anyway.
So now I’ll be able to use thinner wire inside the enclosure, leaving more room for the relays and my custom boards. But more importantly, I won’t have a bunch of frayed wires hanging out of the tie-downs to short out the whole thing. And remember: short-out is Latin for frying myself, because software guys are a danger themselves, their pets, their families and essentially entire neighborhoods when handling hardware.
And yet, I’m still doing it. Go figure.
[Print This] By mjhammel ~ November 20th, 2015. Filed under: Arduino.
So my little aeroponics project is growing beyond its real needs, but once you start assembling legos it’s hard to stop. I’ve added an ESP8266 for wireless connectivity and a really small OLED LCD display. The latter will let me monitor the system without either serial (FTDI Basic) or wireless connected.
Fritzing made it easy to layout the prototype on a breadboard, though after working through cleanup of the schematic and the PCB I found some problems. First, I intended to use a USB breakout board from SparkFun connected to some headers on the PCB. But if you try to wire this in Fritzing the circuits are not closed at the headers and Fritzing gets confused. Instead, I found a USB part (from SparkFun) in the Fritzing parts libraries that had through holes. Through holes are important because I need to place some traces on the PCB on the bottom side of the board. The USB part had connectors that were labeled Vcc and Ground which allowed Fritzing to close circuits and allowed me to create traces the way I needed them.
The next problem came after the PCB was fully routed. The size of the PCB makes it prohibitive to get made. OSH Park charges about $5 per square inch. With my PCB at 8.8 x 5.3 inches that cost for a small production run (3 boards) would have been around $200. Not horrible but really, this is such a simple project why waste the money?
So now the plan is this:
- Remove the FTDI Basic. The Pro Mini just needs headers to connect the FTDI Basic.
- The OLED LCD doesn’t need to be on the PCB. It can be connected with wires. This will allow the display to be freely placed in the most appropriate location without respect to the positioning of the PCB.
- With these changes I should be able to mount the ESP8266 on the back of the PCB and
- move the TinyRTC and USB power to a second PCB, mounted vertically with spacers. That will reduce the costs of the PCBs significantly. In fact, I should be able to do the whole thing with perfboard.
So my cost goes next to nothing. In the end I should have two perfboards mounted together with spacers with the LCD mounted on a third perfboard (or without possibly) on some user-facing wall of the enclosure.
I still haven’t gotten to the enclosure but that will come after I find out how big this thing will be. I still have to add in the 4 channel relay board too, plus the wires connecting all the boards and the power outlets together.
Just posted this, then went back to review the layout in Fritzing. I realized that the ruler part I used to measure the PCB had two sides: inches and cm. Guess what idiot was reading the cm side? Yeah. That’s the guy. So it looks like the whole design may fit completely on the perfboards. The PCB would only be 3.3″ x 2″. Much more affordable.
Just goes to show: there is always one more thing to review….
[Print This] By mjhammel ~ November 5th, 2015. Filed under: Hardware, Holiday lighting, Raspberry Pi, Arduino.
One of the directions my PiBox project is headed is as a central hub for sensor management. This implies the ability to communicate between the PiBox and the sensor node using BLE or Wifi. PiBox can do that already over Wifi (and supports basic BLE communication though lacks a meaningful UI or automated registration) so I only need to start work on sensor nodes.
Now it’s hard for me to toss together a temperature sensor node without having a purpose for knowing the temperature. So whatever node I created needed to have a purpose. Enter: my newly configured aeroponics systems. This system uses a 5 gallon bucket with a water pump inside to spray nutrients on the roots of the plants. Lamps are hung from a support structure (a wire frame shelving unit) . Lamps and pump are plugged into a simple manually configured timer. Nothing special there, except for a really leggy tomato plant that has yet to fruit. But that’s a different project.
So I thought to myself: what if I created a programmable timer for the lamps and pump? I could use an Arduino for the brains, plus added components to make it clever. That’s my sensors purpose: handle power on and off of the lamps and pump for the aeroponics systems. So was born apc: the Aeroponics Power Controller.
I’m sure the first comment will be: but that’s not a sensor. Yeah, it’s not. But the concepts are the same: embedded device controls some physical components based on input from a remote management interface. For apc the physical components are power outlets. APC’s initial implementation will use four controlled outlets, each capable of being programmed independently. It needs to be able to run scheduled events (power enabled and disabled on each outlet). There should be some way of knowing the status of the controller: running or not. And there should be some status indication of what’s happened, re: a display of some kind.
Based on the design I need the following components.
- Arduino Mini Pro, 5V 16MHz
- TinyRTC DS1307
- FTDI Basic breakout board
- USB breakout board
- Sainsmart 4 channel relay board
These are the main off-the-shelf boards I need. I also need a breadboard to hook them all up, an LED and associated resistor and a bunch of wires. I discovered, after finishing the version of the project, that soldering is limited to adding headers to the boards and a few wires to the headers.
Why the Mini Pro? Well, for one thing there were on sale at SparkFun back in March 2014 when I bought a bunch of similar boards without having a specific purpose for them. So it’s been sitting around for awhile waiting for an inspiratio on my part. But it also has a very small footprint without all the bells and whistles (re: bulky connectors) required for other beginner projects. And it has plenty of computing horsepower for this kind of project. It’s actually way overkill. But I already had the board, so what the heck.
The TinyRTC is a real-time clock with a battery. The battery isn’t really necessary but it’s nice to have if you lose power. Though without power the apc won’t work. Dealing with that problem is beyond the scope of this project.
The 4 channel relay is the board that converts the control signals from the Arduino into the enable/disable actions of the power outlets. The relays are optically isolated so when each is enabled they make a loud click sound. You know they’re working if the click is loud and distinct. If it’s a quiet click you probably don’t have enough voltage on the control lines of the 4 channel relay.
The FTDI Basic breakout board is what you use to connect your Linux box to the Mini Pro. It’s what you use to upload your programs to the Arduino.
The USB power from your computer is enough to handle programming the Arduino and running the program. But it’s not enough to change the state of the relays. So a USB breakout board is used to connect a wall-wart power supply with 5V output to the APC. This powers the Arduino and the relay board. Note that you can connect this power source AND the power from the FTDI Basic at the same time. It won’t hurt the board (as far as I can tell). That’s how you test your code.
Hook Up Guide
The nice thing about this project is how it can be done with a series of off-the-shelf breakout boards. The Mini Pro has plenty of digital lines to connect to more than one board. Strangely the pins I need from it for SDA/SCL to talk to the TinyRTC are not aligned to the breadboard. That’s a trade-off when using a smaller microcontroller board like this, but it’s a minor inconvenience. I just soldered two pins to the A4 and A5 holes in the Mini Pro, then used female connectors to wire them to the breadboard at the SDA/SCL lines on the RTC.
The 3.3V power from the Mini Pro is not strong enough to switch the relays (not shown in the Fritzing diagram) so a USB breakout is used to bring in power from a wall wart that provides 5V input. In fact, that’s why I added two additional outlets to the original set of four outlets. The two new outlets are not controlled and I can plug the wall wart into the USB breakout board. Easy-peasy. This is why IoT is so popular. It’s a bit like Lego building, but with wires and pins.
All the code for this project is available from the git repository at gitlab.com. The code utilizes two libraries for the RTC, the Wire library for serial communications and the Timer library for running scheduled events. The Timer library provides for events to be run on short intervals like 100ms. The RTC libraries allow checking the current date/time against the schedules to see if an event should be triggered. Events are simply when the outlets are enabled and disabled.
As a side note to programming: I’m not using the Arduino IDE. I think IDE’s are a crutch that prevents you from learning WHY things work. So I’m using Tim Marston’s arduino-mk to edit and build from the command line. I added support for opening serial monitor to it, which you can find in the source code under the docs directory. I also use screen to create a bunch of virtual terminals in a single window. This I do because I can then work from different systems (desktop or laptop in different rooms of the house, for example) without losing history or resetting my environment. Lastly, I use cdtools with my project because I have a zillion projects and cdtools functions make it easy to bounce around them. Config files for both screen and cdtools are in the docs directory.
Automated Registration and Web UI
The next phase of this project will add networking to the system. An ESP8266 will be added to provide wifi connectivity. The ESP8266 uses an AT command set to connect and send and receive data from remote servers. The plan is to have the Pro Mini perform an initial registration with a remote server running on PiBox. The registration is initiated with a multicast message from PiBox asking for new devices. The device receives the message and sends a whoami packet with it’s IP address and a registration token (something akin to a secure key, I think). Once registered the device can access the PiBox web server via a RESTful interface. The interface will provide scheduling data for the device and the device can report status back.
Of course this requires a web interface for setting schedules for all registered devices. Fortunately PiBox already has a web UI with a RESTful API. I just need to extend it to support IoT devices. Then I need to create a UI, potentially using a self-defining UI format so devices can create their own configuration options without having to place the knowledge in the UI. If done right this format won’t take up much space on the device because there will be little to configure in each IoT. That’s the point: IoT implies a device with a simple purpose that is remotely managed.
Beyond the ordinary
The ESP8266 will connect to the serial lines of the Pro Mini so that means the serial console won’t be available. There is a neat little trick around that: a color LCD display. These devices are cheap, around $10-$20. I’ve selected a four wire device that uses SDA/SCL lines for data plus a ground and Vcc. Luckily the TinyRTC board has extra SDA/SCL pins to piggy-back another device like the LCD. And Adafruit has libraries for these devices that make it very easy to use.
After all this is done there is only one thing left to do: extend scheduling for my holiday lights. After all, wasn’t that really where this was headed? I mean, do I really need brains in my outlet timer for a tomato plant? I guess that depends on if that thing ever starts to fruit….
[Print This] By mjhammel ~ July 27th, 2015. Filed under: Linux, Software Development, Hardware, Audio, Video, PiBox, Raspberry Pi.
One of the longterm goals for PiBox is to integrate it into the home. I want to see the PiBox Server provide centralized communication for an array of IoT devices. One way to do this is with Bluetooth Low-Energy, aka BLE, aka Bluetooth 4.0. This specification allows very low power sensors to report information to services that can do something useful with the information: close the shades when the temperature goes up, let the roomba loose when the dog’s sheddings are accumulating, turn off the stove when granny forgets to take her meds.
Before I can do this I needed to get bluetooth classic working on the server. This turned out to be fairly easy. The kernel supports it and there is the BlueZ userspace tools, plus the bluez-tools wrappers to make it even easier. After much procrastination I finally got around to testing these on a server with a 4.0-compatible USB dongle. After a little fiddling with the /etc/bluetooth/audio.conf and /etc/asound.conf files I was able to play a simple wav file to a bluetooth speaker. Easy-peazy.
The next logical step was to get omxplayer, the hardware-accelerated video player for the Raspberry Pi that I use to play videos full screen, to switch from the analog output to ALSA. Turns out this can’t be done, or at least it isn’t done yet. omxplayer doesn’t support ALSA. So you can’t play the audio stream from a video over bluetooth. So much for my drive-in movie with wireless speakers. I looked at what might be required for this (there were hints from the omxplayer developers) but it seems a bit over my head and would distract from the larger scale goals for the project. I’m a one man team. There’s only so much time in a year.
So I’ve started banging my head for alternatives. The obvious one is to plug an analog-to-bluetooth transmitter into the analog output port of the Pi. Too easy. Though I have one on order for the short term (hey, we still use it when we go camping). But that would also require a larger box to hide the transmitter. And how do I switch between bluetooth and analog output? I may not always want bluetooth. Nah, I’m just kidding. Everyone loves bluetooth.
Another alternative is to switch to a different video player, like mplayer. That would get me a software switchable option for audio output. I tried this on a Model B+ but the performance was terrible since the GPU isn’t used. First, to get a nearly smooth playback the image had to be halved while being decoded on the CPU. And there was no way to rescale it to the display size at display time. So this was functionally unusable.
The next alternative (this is computing and there is ALWAYS another alternative) is to switch boards. I looked at several competitors like BananaPi an Odroid. The former looks like a possibility. But then I took another look at the newest version of the Pi, the Raspberry Pi 2 Model B. It’s a quad-core with each core spec’d at 1GHz, slightly more than the original Pi’s 900MHz. Not to mention the possibility of overclocking and additional on-board memory. So mplayer, which does work with ALSA but which requires the CPU to do the decoding, might just work on this board.
So I placed an order for one Pi 2 which has a slightly different Broadcom chip. I’ve already rebuilt the toolchain for this new chip. That’s an adventure in itself, but fortunately doesn’t appear to require major changes to the toolchain. I’m unclear if I have the toolchain optimized. It doesn’t use the latest Linaro gcc releases so I’m doubting it. But if my toolchain shows any possibility of running mplayer then I can always upgrade my Crosstool-NG release to get access to the latest Linaro bits.
[Print This] By mjhammel ~ July 2nd, 2015. Filed under: General, PiBox.
This past weekend I took PiBox out in the trailer for a field trial. The box was mounted under a cabinet with power and HDMI wrapped around to a 7″ HDMI-input monitor. Power on worked fine and the system came right up. But not everything worked as expected. Here is the summary.
The first issue was with shared files over SMB. I first tried to watch a movie using my LG Volt phone, which runs Android 4.4.2. I use ES File Explorer on Android to get to the video files. The phone could access the files and start the playback but the network would drop out after a while. I then tried it with a Galaxy Tab 2 tablet running Android 4.2.2. This worked much better and I watched several movies without interruption. The PiBox Media Player also worked perfectly. So I think the problem was with the phone’s wifi. That’s interesting since I often use the phone at the gym when I run on the treadmill to watch Netflix without problem. Apparently it works better with 3G than it does with wifi.
One thing of interest here: the tablet notified me at one point of “unusual magnetic activity” in the area. I got this when I used an astronomy app (setting up to use my telescope – hey, it wasn’t all work on this field trial, we were camping). I have no idea how it sensed this but wonder, if it was real, if it had any impact on local wifi network performance.
The media server was cabled to a 7″ display in the trailer. This required an extra power cable. Then we also had a digital TV box that added additional power and HDMI cables. Finally I had a HDMI switch to go between the TV receiver and the Media Server. That amounts to a boat load of cables that didn’t exist before with the old TV/DVD player. Fortunately, the digital TV and HDMI switch will go away when I add a digital TV dongle to the Media Server. But I’ll still have a TV cable. Even so, I can see that there are still too many cables. I need to power a display and connect to it with fewer exposed cables. That would leave just the power and digital TV lines exposed, which is much better. I’m not sure how I’ll solve that problem yet.
The wifi setup works well when connecting to the local router. Using PiBox Media Server as a wireless access point had some minor problems. First there was a problem with the way I set ccmp vs tkip. There are two ways to configure the network: bui-network-config (the GTK+ app run from the launcher) and the web interface. I fixed the web interface to do the right thing, which is use tkip, in the web backend (piboxd). But bui-network-config uses its own code for the same thing.
I’ve fixed bui-network-config but the long term solution is to have bui-network-config use the same backend. The problem is that bui-network-config was designed as a standalone tool that can be used outside of PiBox. So the question is how to maintain that capability. I think the solution is going to be to have a shared library that both bui-network-config and piboxd can use for this purpose.
Multiple USB sticks
Something that kind of shocked me was a problem with multiple USB sticks. With one stick plugged in videos were available. With two or three they were not. This may have been a problem with the USB ports. I still need to check on that. Another problem may be with how VideoFE handles the databases created by VideoLib on the sticks. Or it might just be that the sticks were messed up. This simply needs more research.
Field trials are good to find out just how usable the device really is. In this case I found that the simplistic alphabetical list of videos was slow to move through using the keyboard’s arrow keys. I also found that, for some reason, the poster art was slow to load (or at least appeared that way). This poster issue is new, as I’d not seen this problem at home. Again, it may be due to networking issues at this site.
The important issue here is that I need a way to fine tune the list of videos. Pressing a few keys should be sufficient to reduce the displayed list. I would need a way to cancel that search (to get the full list back) and/or a way to timeout the current search. If it does timeout, does the display go back to the full list? How does the user know what the current search state is (as in timeout state, etc.)?
There’s a bit of work to do here.
[Print This] By mjhammel ~ June 11th, 2015. Filed under: PiBox.
It’s taken more than 2 years but I’ve finally finished the first versions of the PiBox Media Server and Media Player. The cases are stained wood boxes from a hobby store but they provide me with information on how to deal with layout in a 3D printed case. It also required me to look at acquiring parts such as specialized cables and connectors. Some of the small breakout boards I used can be combined into custom boards in future versions.
The goal of this project has always been to build a complete software system from scratch using off the shelf hardware as much as possible to create a self contained set of devices that networked with each other. The hardware is all off the shelf and the few breakout boards are open hardware designs. The software was built from the ground up starting with a custom cross toolchain up through the custom applications and UI. It’s even getting some Android apps to go with it (future development).
The only thing that didn’t come out as I’d hoped is the projector used in the Media Player. I was hoping to find a development board for this but all I can find is a “low-cost” pico projector. And integrating the TI DLP chip is a little beyond my scope at the moment (maybe with time that will change).
But it’s a working set of prototypes. I plan on deploying them this summer on our camping trips. I used them last year without cases. This past winter our trailer was broken into and the TV/DVD player was stolen. So now the Media Server, with a 7″ HDMI monitor, will replace it. The only thing missing from it is digital TV. I have a dongle but wasn’t able to get omxplayer working with it yet. That will come with more work this summer.
For now, it’s a start. If you read this and look over the photos then drop me a line and let me know your thoughts.
[Print This] By mjhammel ~ June 10th, 2015. Filed under: General.
I’ve been using Apache 2.2 for quite some time. Lately I’ve noticed CentOS and other distros using Apache 2.4. The new Apache breaks a bunch of stuff, as does newer php.
One thing that needed changing is a switch in directory listing tools. I have an archive on my web site that is running Scott Evan’s Indices software. It’s just a prettifier for file listings but it worked well and was easy to set up and configure for my personal taste. But this doesn’t work on Apache 2.4 for some reason. I haven’t figured out why. However I did find an alternative that works.
h5ai provides more functionality out of the box and is even easier to configure: copy a directory to your document root and add a new path entry to the default DirectoryIndex. I don’t have an example of this because so far I’m just using it behind a firewall at work. But it’s quite nice. Still need to look into customization options. But out of the box it’s quite useful.
[Print This] By mjhammel ~ April 6th, 2015. Filed under: General, Linux, Software Development, Raspberry Pi.
When I was just getting started with embedded development I found many tutorials on how to perform cross compiles required setting up some shell functions and variables before working on builds. This is a necessity for embedded work because the embedded build for the target platform won’t use the same toolchain and build libraries as the host platform. Unfortunately, the setup for the various publicly available build systems were all different. Angstrom/OpenEmbedded vs Yocto vs Buildroot vs Mentor Graphics vs whoever: they all have their own setup.
Then I created my own build platform, PiBox. I did this mostly to teach myself the whole platform bring-up process. One of the side effects of this (along with having to build custom OPKG packages for PiBox) was the need to bounce from one build to another quickly. I wanted them all to use the same navigation commands and automatically point where ever they needed to with environment variables. Thus was born cdtools.
Let’s say I have a build system for the toolchain, kernel and root file system for my Raspbeery Pi (re: PiBox). I do out-of-tree builds and packaging to avoid clutter when looking for updates with git status. So while my source lives under …/raspberrypi/src my builds are under …/raspberrypi/bld and packages generated from the build are under …/raspberrypi/pkg. So far so good. That’s not overly complex.
But now I need to bounce over to a kernel tree and make some updates and test them. The updates I end up with will be integrated back into the PiBox build but before that happens I want to test them in a kernel-only tree (remember: PiBox is a build of multiple components – it is, in fact, a metabuild that downloads components, unpacks them, patches them, builds them and packages them). So how do I bounce over to the kernel-only tree and run tests and then bounce back to the PiBox tree to integrate the changes?
Here’s another scenario: I have a slew of metabuilds that do nothing but download 3rd party utilities, configure them as needed for PiBox and them package them as OPKG packages. These builds are then wrapped in another metabuild that builds all of those package-metabuilds in order to simplify creating a complete PiBox release! This top level metabuild has dependencies on the package metabuilds so I may find myself performing the top level metabuild, finding a bug (due to an upstream change to the branch I’m using) in a 3rd party utility, switching over to that package metabuild to fix, commit and push it, and then bouncing back to the top level metabuild to rebuild the packages again.
What I need are common navigation tools for all projects and a way to change that navigation based on which project I want work on right at this moment. The way to do that is with shell variables. But I need to do this without having all kinds of project-specific shell scripts that are all different. There needs to be some consistency in how they are used. I want to set the project name but use the same navigation to end up in source, build and package trees no matter which project I’m in.
This shell script acts as a front end to a directory of configuration scripts, one for each project in which I do builds. This script lives in ~/bin and sources any shell files found under ~/bin/env*. In this way I separate builds for work from builds I do at home by placing the environment configurations in different directories, as in
Whenever I add a new project (and this happens frequently for me) I just drop in a new shell file in on of the env subdirectories and then reload cdtools.
This makes it easy to make changes without having to logout and back in. It also makes it easy to do within a screen session so I can test new or modified configurations under one screen window without affecting the other windows until I’m ready for the update.
Each project has its own shell file which contains a project specific shell function that sets up environment variables and aliases. The environment variables are always the same but point to different things. Typically I use variables prefixed with GM_, though not always.
- SRCTOP – top of all source trees, under which you find all project specific directories.
- GM_HOME – top of the project directory, under which you find the project source, build and package trees, along with others as needed.
- GM_WORK – a scratch pad area often common between projects.
- GM_BUILD – the build directory for the project
- GM_SRC – the source directory for the project
- GM_PKG – the packaging directory for the project
Other variables can be set to update the executable PATH, library path, Java path, or anything else that’s needed. As long as the same variables are used in each shell file (with unused variables ignored) then switching from one project to another will always properly setup the environment.
For the use case, the PiBox shell function is rpi and the Linux kernel shell function is kernel. Typing either configures my environment to access, build and navigate their respective trees.
Remembering the environment variables is hard. So is typing them. So the shell function maps them to navigation aliases. All navigation aliases are prefixed with cd, as in
- cdt – top of all source trees, under which you find project directories
- cdh – top of the project directory
- cdx – source tree under the project directory
- cdb – build tree under the project directory
- cdp – package tree under the project directory
- cda – archive directory, where a local copy of downloaded source is kept, to avoid having to redownload unless absolutely necessary (this includes git or mercurial managed trees, as needed)
- cde – Extras directory
A special alias, cd?, is setup to call a project shell function that lists the aliases, important environment variable configuration and help in cloning and accessing remote trees. All projects use the same aliases. Additional aliases can be setup as long as the context of the alias is the same for each project.
Until you’ve used cdtools for awhile it can be hard to remember the aliases. It’s even harder to remember the sometimes convoluted git or mercurial (or other source code manager system) commands necessary to work with remote repositories. A help function is included in each shell file. This is always prefixed with “list”. In our use case, the helper function is listrpi or listkernel. But you don’t have to remember this because the alias cd? always points to the currently active projects help function.
raspberrypi Alias settings: ----------------------------------------------------------------------------- cdt cd SRCTOP (/home/mjhammel/src/ximba) cdh cd GM_HOME (/home/mjhammel/src/ximba/raspberrypi) cdw cd GM_WORK (/home/mjhammel/src/ximba/work) cdx cd GM_SRC (/home/mjhammel/src/ximba/raspberrypi/src) cdb cd GM_BUILD (/home/mjhammel/src/ximba/raspberrypi/bld) cda cd GM_ARCHIVE (/home/mjhammel/src/ximba/raspberrypi/archive) cdp cd GM_PKG (/home/mjhammel/src/ximba/raspberrypi/pkg) cde cd GM_EXTRAS (/home/mjhammel/src/ximba/raspberrypi/extras)
To checkout tree: cdt mkdir raspberrypi cdh git clone ssh://firstname.lastname@example.org:pibox/pibox.git src
So what happens when you have umpteen projects and you can’t remember which shell function sets up a particular project? You can get a list of available functions and their descriptions using the cdlist function, which is included in the cdtools script. The cdlist function browses all shell files and extracts the shell function name, a description and the name of the file its found in. The description uses a tag in a shell comment, as in this example.
# DESC: PiBox: Embedded environment for ARM-based system using buildroot
To search for this I can pass any string in the DESC field, such as PiBox:
$ cdlist pibox ... pmsui PiBox Media Center UI (pms) [pmsui.sh] pmsuip PiBox Media Center UI (pms) [pmsuip.sh] psp PiBox psplash opkg build [psplash.sh] rpi PiBox: Embedded environment for ARM-based system using buildroot [raspberrypi]
All shell files with descriptions that include the string “pibox” (case insenstive) are listed. Now I can see the shell function (rpi) which is listed first and the file in which that function is defined (raspberrypi), which is listed at the end of the line inside brackets.
First, load all cdtools scripts.
Now setup to work in the PiBox tree. Then change to the source tree, see what’s there and then switch to the packaging tree and do the same.
rpi cdx ls -C Makefile README config.mk configs docs pkg scripts src util.mk cdp ls -C firmware opkg rootfs.ext3 kernel.img.3.10.y opkg.10 rpiToolchain-0.9.0_Fedora19_201409081418-1.x86_64.rpm kernel.img.3.14.y opkg.9 rpiToolchain-0.9.0_Fedora19_201409162158-1.x86_64.rpm mkinstall.sh pibox-0.9.0_Fedora19_201409081418-src.tar.gz staging.tar.gz mksd.sh pibox-0.9.0_Fedora19_201409081418.tar.gz
I can see that I have done packaging after multiple builds here. Wait, let’s check what’s in the top level metabuild tree.
meta cdx ls -C MakeChangelog.sh build.sh docs functions.sh src
Okay, that looks fine. Now back to the PiBox tree.
Multiple versions of a single repo
A neat trick here is that if you format the shell function correctly you can append a number. That number will allow you to have multiple versions of the same repo, as in this example.
rpi rpi 2 rpi 3
Checking the alias of the middle one, we see this.
raspberrypi Alias settings: ----------------------------------------------------------------------------- cdt cd SRCTOP (/home/mjhammel/src/ximba) cdh cd GM_HOME (/home/mjhammel/src/ximba/raspberrypi2) cdw cd GM_WORK (/home/mjhammel/src/ximba/work) cdx cd GM_SRC (/home/mjhammel/src/ximba/raspberrypi2/src) cdb cd GM_BUILD (/home/mjhammel/src/ximba/raspberrypi2/bld) cda cd GM_ARCHIVE (/home/mjhammel/src/ximba/raspberrypi2/archive) cdp cd GM_PKG (/home/mjhammel/src/ximba/raspberrypi2/pkg) cde cd GM_EXTRAS (/home/mjhammel/src/ximba/raspberrypi2/extras)
To checkout tree: cdt mkdir raspberrypi2 cdh git clone ssh://email@example.com:pibox/pibox.git src
Note the “2” in the directory paths. I can now have multiple copies of the same repo. And this trick can go further by using additional arguments to the shell function to embed different-yet-related repositories using a single shell function.
bui fakekey bui network-config 2
And so forth.
The cdtools script currently includes support for color coding of the cdlist using ASCII escape sequences. This works in most terminal types. More importantly, there is no reason you can’t extend cdtools to support additional functionality that you want to embed in your own set of shell files.
Consistency of practice
What you should get out of this is the consistency of use when moving from one project to another. How I build projects can vary greatly: PiBox uses GNU Make while one of its packages (launcher, for example) uses Autoconf. This happens all the time because when doing a system build (a build of an OS plus utilities plus custom kernels plus custom apps, etc.) you may depend on many 3rd party source trees with a variety of build patterns. But that doesn’t matter. What matters is can I build it quickly by bouncing to it and having pretty much everything I need in place when I do.
With cdtools, the answer to that question is a resounding YES.