Automated ssh reconnects and handling slow connections

Originally published at Dom's Blog. You can comment here or there.

When using a slow connection such as via a mobile phone, even a command line prompt via ssh can be slow to type on at times.

The ‘rlwrap’ command works around the fact that each character sometimes take a long time to be echoed back to your client, by allowing you to prepare the line locally before hitting return whilst still allowing you to pass control characters.

In addition, the ‘autossh’ command will keep trying to automatically reconnect a remote login session whilst a connection is down, and ‘GNU Screen’ for multiplexing windows and being able to leave commands running on a system even whilst disconnected or not viewing the output, and you have a pretty powerful toolset.

Here is an example of how I connect to a remote system with these commands:

rlwrap autossh -M 22007 -t user@example.com 'screen -dr'

Prompt with colour-changing smiley dependant upon exit code of command

Originally published at Dom's Blog. You can comment here or there.

The file ~/.bashrc can contain a definition of the variable PS1 which is used to determine the prompt used on a command line. Normally this would display the username, hostname, working directory and is often modified to include additional information such as the time of day the command completes, history number or exit code.

The exit code of a process is zero if the command has completed successfully, otherwise the number is determined by the application – these are normally documented in its manpage.

A function can be defined within the .bashrc file which prints different strings dependant upon the exit code of the previously executed application, and the prompt constructed to include this, but upon further testing an issue arises with the length of the line when the text being entered wraps, as the number of characters in the prompt is calculated incorrectly. The reason for this is that the escape codes used to change the colour within the prompt are assumed by bash to take up space, but in fact they do not. This can be worked-around by enclosing the variables expressing the colours in escaped square brackets, but the same is not true within the print command used within the exit-code dependant function – instead, the strings 01 and 02 are used to denote the non-printing colour changes:


Red="\33[0;31m"
Green="\33[0;32m"
Yellow="\33[0;33m"
Blue="\33[0;34m"
Magenta="\33[0;35m"
Cyan="\33[0;36m"
Normal="\33[m"

smiley() {
ret_val=$?
if [ "$ret_val" = "0" ]
then
printf "\001$Green02:)\001$Normal\002"
else
printf "\001$Red\002:(\001$Normal\002 ($ret_val)"
fi
}
PS1="\[$Magenta\]\u@\h\[$Normal\]:\[$Cyan\]\w \$(smiley) \[$Normal\]"

References:

1. http://superuser.com/questions/301353/escape-non-printing-characters-in-a-function-for-a-bash-prompt
2. http://www.gnu.org/software/bash/manual/bashref.html#index-prompting-257

Manual rename command on OS X using bash

Originally published at Dom's Blog. You can comment here or there.

OS X has no ‘rename’ command which can be used on some Linux distributions to apple a perl regular expression to modify the name of a file, so I cooked-up this line instead to change m4v filenames to mp4:

for i in *.m4v ; do mv "$i" "`echo $i | sed 's/m4v/mp4/'`" ; done

The three types of quotes are used to either protect strings which may contain spaces or, in the case of the backticks, execute commands and return the output. The single and double quotes are used separately since they are nested one within the other. Single quotes would not allow variables to be expanded within them if they were used on the outside.

Removing ASCII codes from filenames

Originally published at Dom's Blog. You can comment here or there.

Sometimes when files are downloaded by automated download scripts, or just saved directly from a website, the filenames contain ASCII characters encoded with the percentage symbol, e.g.:

This%20filename%20contains%20spaces.html

Whilst this is considered bad practice, it is sometimes unavoidable (for example when downloading content which is saved dependant on the title of a page, not the filename of the source), so I wrote this little script to rename each of the files in a directory:

#!/opt/local/bin/perl -w
# A script to replace the ASCII codes in filenames with their actual characters
# Note: no checking is done
if ($#ARGV < 0 ) {
print "usage: substitute_ascii.pl \n";
exit;
}

# Process each argument in turn, saving the original name
# then replacing each ASCII code with its actual character
foreach $value (@ARGV) {
$original = $value;
$value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
if ( rename("$original", "$value") ) { print "$original -\> $value\n"; } else { print "Rename operation failed\n"; }
}

Automating APOD

Originally published at Dom's Blog. You can comment here or there.

So, today’s APOD is particularly awesome and got me thinking about how to automate downloading the image to my desktop background once a day. I imagine the regexp here is a hideous hack, but it works (for today’s html); I shall have to look into this further tomorrow.

In summary:
1) If you havn’t already done so, check out the Astronomy Picture of the Day, it’s ace
2) for the technically minded, here’s what I just hacked together to save this to the same file (assuming it’s always a jpeg) each day:

imagepath=`wget -q -O - -- http://apod.nasa.gov/apod/  | grep "IMG SRC" | sed 's/\(\<IMG SRC="\)\(.*\)"/\2/'` ; imageurl=http://apod.nasa.gov/apod/${imagepath} ; wget -q -O apod.jpg -- $imageurl

USB attached disk hangs

Originally published at Dom's Blog. You can comment here or there.

I put a 1.5 TB SATA disk into an external USB enclosure to use my Eee as a media server, and it runs great with fuppes from my PS3, but when leaving the machine running for several hours, it would result in the disk not being accessible.

Apr 18 10:10:28 ubuntu-eee kernel: [183391.907522] usb 1-2: USB disconnect, address 6
Apr 18 10:10:28 ubuntu-eee kernel: [183391.926314] xfs_force_shutdown(sdd1,0x1) called from line 1031 of file /build/buildd/linux-2.6.35/fs/xfs/linux-2.6/xfs_buf.c.  Return address
= 0xf94d0858
Apr 18 10:11:01 ubuntu-eee kernel: [183425.224119] Filesystem "sdd1": xfs_log_force: error 5 returned.

I looked up the errors and was directed to kernel bug #32432 and Ubuntu bug #349767 and #701011.

The suggestion in one of the comments is to turn off power management by adding this line to rc.local; whilst I don’t see this as a permanent solution, it will suffice for the moment:

for i in /sys/bus/usb/devices/*/power/level; do echo on > $i; done

Thanks are due to everyone who has contributed to these bug reports, I’m happy to offer to help but I doubt there’s much I can do to fix the problem, unfortunately. I guess it’s up to the owner of the relevant part of the kernel.

Time Machine backup via NFS

Originally published at Dom's Blog. You can comment here or there.

There are many posts online regarding the fact that you can allow Mac OS X’s built-in backup software, Time Machine, to backup to an NFS volume, but what is not mentioned on many of these pages is the fact the format of the sparsebundle has changed, so an additional step is necessary in order to create the file manually which is to be used for the backup.

I connected my Mac to the same switch as the NFS server for the initial backup, in order for it to proceed much faster and potentially more reliably than if they were only connected through my wifi router.

You may also need to set your hostname in the system preferences ‘sharing’ dialogue to something without spaces nor quotes.

Set the option to allow backing up to unsupported filesystems, namely NFS in my case:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

Create the file on the desktop using hdiutil at first, my example creates a sparsebundle capable of holding up to 320 GB of data, although the file doesn’t take up that much space initially. The name should include the system’s hostname and MAC address.

hdiutil create -size 320g -type SPARSEBUNDLE -nospotlight -volname "Backup of Dominic-MBP" -fs "Case-sensitive Journaled HFS+" -verbose /Users/dominic/Desktop/Dominic-MBP_f81defec32e3.sparsebundle

Next, create an XML file which describes the package itself and replace “YOURUUIDHERE” with the system’s hardware UUID (from the system profiler), save the file as com.apple.TimeMachine.MachineID.plist file:

 

 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.backupd.HostUUID</key>
    <string>YOURUUIDHERE</string>
</dict>
</plist>

Now copy it into your disk image directory:

cp /Users/dominic/Desktop/com.apple.TimeMachine.MachineID.plist /Users/dominic/Desktop/Dominic-MBP_f81defec32e3.sparsebundle/

 

Make sure you can write to the NFS filesystem as root from the Mac (because Time Machine will run with sudo), and that you have your Mac plugged in and the option set not to sleep within a short space of time in case you walk away whilst it’s doing the first backup.

References: Macworld; rajeev.namelevelofindirection.com; Apple Clinic

on Mac hints and tips and mini Ubuntu servers

Originally published at Dom's Blog. You can comment here or there.

Here are some useful sites I found when looking up keyboard shortcuts for my Macbook:

I have considered triple-booting my Mac with Ubuntu and Windows, but I gather this is quite difficult; the alternative would be to Hackintosh my Eee with something like osx86 but this isn’t allowed by the OS X license.

Until I have my server back online or I’ve bought a sheevaplug on which to run it, I’m limited to having Ubuntu in a virtual machine, or using a colleague’s shell account, such is the cost of moving house at short notice to very small rented accommodation.