The Plumbing and Heating site is here.


Yet Another 'Pooter Hacker

homo hackerensis

Not sure I can really justify the 'hacker' claim except to (hopefully) differentiate myself from from yer average nueless clewbie Windoze luser. I do use Linux (generally Debian with some forays into Ubuntu but M$-free since ... er <mumble>) and have written working programs (even if only "Hello, World\n") in a variety of languages including Perl, Bourne/Bash shell, C and C++ (and keep promising myself to upclue on Python ... when I get the necessary Tuits. Yeah, and learn emacs too :-))

What follows is some of my scripts and setups which I'm not too embarrased to share with the world, and hope may be of use and/or interest to others.

N.B. I'm running Debian-based systems: Debian stable (currently 'Lenny') on my desktop, and Testing ('Squeeze') on my laptop. YMMV porting any of this to other distros (mainly in terms of what applications your distro may come installed with, or make it easy to use).


File Listing and manipulation

The first script I install on any system is lsl:

ls -al --color "$@" | more

or sometimes

ls -al --color "$@" | less -eRX

This gives me colour-coded and paged directory listings. The less version allows me to page back through a listing (which is nice) but makes me explicitly exit with another RETURN when I reach the end of the listing.

I should probably learn to use Midnight Commander. I used to be fluent in Norton Commander on DOS systems but MC was too different to port my mind-and-finger-set to easily, and I never made it up the learning curve. I use KDE's konqueror for wysiwyg file manipulation, backed up by shell for stuff konq can't do - for example renaming files according to regular expressions using Michael Forman's ren-regexp Perl script. And for doing hard links, which KDE just doesn't seem to know about. :-(

Mail

Like many folks on home PCs I used to use Mozilla mail or Thunderbird (now called Icedove by Debian) to get mail from my ISP (NTL). Since I also have a domain (or three) registered to me which is hosted elsewhere my mail client got mail from that domain too. That included mail for my partner: I set Mozilla up to filter that into a mail folder which was shared with suitable file permissions. And once in a while I'd fire up mutt to read mail sent by the system to me (output of cron jobs etc). Then I got a laptop and wanted to read my mail when I was away from home. One way to do this is to ssh into my home PC from the laptop and run my mail client on the home PC with its display on the laptop: that works, but it runs like cold molasses. Also I wanted to try out sylpheed: but all my existing mail was stored in Mozilla's mail format which Sylpheed didn't understand. And I was running into a bug that made Mozilla keep crashing. So some hacking ensued and ...

Now I use getmail to get mail from my hosting providers (via pop) and put it into the Maildirs of local users. dovecot serves the mail via imap to our mail clients. And I configured exim (Debian's mail transfer agent) to deliver system mail to my Maildir instead of the usual mail spool at /var/mail/username.

Details of how I've set up my mail.


Audio

It used to be relatively straightforward to get BBC radio's streaming audio (both live and listen-again) as Real Audio, but since late July 2008 they changed the way they deliver it. Fortunately there's now get_iplayer which magically does everything for you.

Except allow you to call yourself a hacker while doing it :-(.

Details ...


Digital photography

I get the pictures from my digicams (a Canon compact, Pentax DSLR and the camera built into my phone) by putting the camera's memory card into a reader and using my getpix script: this locates and mounts the device the card reader shows up on (e.g. /dev/sdc1 or whatever), finds the picture files and moves (or optionally copies without moving) them to a directory with today's date in YYYYMMDD format as its name. (This directory is inside a directory named with the year as 4 digits, within a directory ~/pix/, so if today is Xmas day 2010 pictures will be placed in ~/pix/2010/20101225/.)

The script renames each image file, transposing UPPER CASE to lower and prefixing the camera-generated filename with a date-and-timestamp derived from the image creation time in YYYYMMDD_hhmmss format, so an image file that appears as, say, DSCN1234.JPG on the card may end up at ~/pix/2010/20101225/20101225_102234_dscn1234.jpg. Files not having exif creation-time data - for example video files - will not have the timestamp prefix, except in the case of Canon .avi movie files: these are accompanied by thumbnail JPEGs with the same serial-numbered filename but the extension .thm. When the script encounters one of these .thm files it looks for a corresponding .avi file in the destination directory and renames that with the timestamp derived from the exif data in the .thm. (This relies on the .avi file being copied from the card to the destination before the .thm is processed.)

Timestamping images is a recent change for my picture collection so I needed a way to convert my existing collection to timestamped format. This is relatively easily done with a one-liner:

exiftool '-FileName<CreateDate' -d %Y%m%d_%H%M%S_%%f.%%e <dir>

which operates on just one directory - or to recurse down a specified directory tree just say:

find <dir> -type d -exec exiftool '-FileName<CreateDate' -d %Y%m%d_%H%M%S_%%f.%%e {} \;

Converting the names of Canon .avi files is not so easy and I resorted to writing a litle script tstampavi.

The getpix script also auto-rotates images according to their exif orientation tag. For raw files it calls a companion script raw2jpg , which uses Dave Griffin's excellent dcraw converter for proprietary camera formats, with cjpeg (it could use ImageMagick's convert) to make a JPG copy of the raw file, and exiftool to copy the exif data from the raw file into the jpeg.

Once my digital pictures are uploaded to the PC it inevitably turns out that some are quite duff and I quickly delete the JPEGs of them. This leaves the raw .PEF files taking up space on my HDD, so to clean them up I run the script rmraworphans.

Occasionally the memory card gets mounted Read-only. Where this is not caused by the tiny write-protect switch on an SD card getting moved(!) it can be due to some sort of problem on the card's FAT filesystem. This typically results in kernel panic messages in /var/log/messages, and can be cured by running (as root)

fsck.vfat -a -v /dev/<memory-card-device>

e.g.

fsck.vfat -a -v /dev/memory_card_SD


Backup

There's backup, archiving and snapshots. I think of them thus:

Backup
A copy of all or part of a filesystem which can be used to replace/rebuild a broken/lost/stolen system by copying files back from external media - usually tape or discs
Archive
A set of copies of a particular file or group of files at particular points in their evolution, possibly at different versions.
Snapshot
An image of all or part of a filesystem at a particular point in time, which can be accessed like the filesystem it is derived from (though only for reading).

The advent of cheap internal and external hard drives (a 1 Terabyte external drive is currently under £100) coupled with the availability of hard links in unix-ey filesystems and rsync's ability to efficiently and easily synchronse files on different systems has made it easy to implement a snapshot-type backup/archiving system. A discussion of such systems is here.

More ... (TBD)


ssh

Any Real Hacker(tm)must have a variety of 'pooters networked together. My home network doesn't include my toaster, doorbell or kids which probably excludes me from the Real Hacker category but I do have enough systems that communicating between them is an issue. Leaving aside the regular nuts'n'bolts stuff like NFS, NIS, SMB/SAMBA, local web and mail hosting etc. one inevitably needs ssh (including scp and rsync) connectivity. And one does not want to by typing in passwords all the damn' time.


Contact me

at wibble@yaph.org.uk