useful (shell) scripts by Karl Voit

Some useful (shell) scripts by Karl Voit

Update 2012-04-23: current scripts and tools are hosted on my github page. Following is considered out-of-date.

Here, you find some links to some scripts I wrote for myself and which can be useful for other people too. Some of them do not have any special homepage here because each of the scripts includes documentation. And yes, there is no special order here :-)

Most of the scripts here are pure shell scripts (bash) but more and more complex scripts will be written in Ruby (see below) which require the Ruby interpreter to be installed. The same holds for Python.

You might also wonder, why I am publishing so short and simple scripts here. The reason is, that those (indeed simple) scripts have functionality or an idea behind, that should be made public in order to simplify someones computer-life. Those scripts are marked with "Short but useful".

Some of the scripts presented on this site are still some kind of quickhack and not quite good documented and/or fool-prove written. Feel free to finish them and send me your version. I hope that everybody, who wants to use some of my scripts, understands the code behind and is able to modify them. I tried to write really error-prone scripts for the general purpose scripts such as YAMN or lgh.

Feel free to use and comment my scripts. In every file, there should be my email-address for this purpose. All scripts here are published under the GPL.

Why/How/When doing shell scripting?

Many people think, that shell scripting is obsolete because of more powerful languages like perl. For some things, they are right, for other things they are wrong, and for some other things, it both methods are OK. Real life seldom replies simple :-)

My historical background with shell scripts

My background with shell scripting starts in the early 90s, when I used DOS on my 386SX. There were those batch-files, that could do simple, stupid things sequentially without being supervised. I used to use this batch-files very often and I got better to take advantage of every feature possible (character injection into the keyboard buffer before staring programs and so on).

When I switched to use Linux as my main platform, I quickly realized, that bash-scripting (bash was the standard shell and so I used it right away) allows more things than DOS-batches.

Shell scripting to solve (almost) any problem

In combination with the GNU-tools like sed, awk, grep, and so forth, it opened a whole new universe to me. The GNU-tools are specialized for a wide range of things. Piped together, they seem to solve any problem one might imagine.

My shell scripting examples below shows, that for example the bash is a full blown programming language with lots of defined basic language features such as "if", "while", "for", (sub-)procedure calls, I/O-methods, and so on. And the "libraries" (as delivered with Java) are the GNU-tools, which offer special functionality that can be used to process data.

Alternatives and trade-offs

Shell scripting is not the only solution. Powerful tools like perl offer more language features and come with a large set of libraries. There are many other alternatives such as Ruby and Python as the main alternatives to pure shell scripting.

What the heck is Ruby? It is roughly is a mixture of perl (regular expressions) and smalltalk (OOP) which provides a very comfortable scripting language. A lot of nice language features (e.g. Ruby-style iterators) makes programming way more fun. A lot of sophisticated things give room to experimenting with (more or less) new programming concepts (AOP, closures, ...). Ruby is as good for simple things as for complex things. I simply like it very much and I tend to write more and more scripts using this powerful tool instead of simple bash scripts; you have been warned. *g*

When it comes to performance, higher level scripting languages (e.g. perl, Ruby) are often much better than shell scripts not only because of the internal "overhead" of the shell when calling "external" tools like grep.

Perl might be wide spread but I think that (ignoring non-UNIX operating systems) shell scripts are available on more systems. But you have to be aware that they are not that portable than perl-scripts because on many (commercial) UNIX systems (including OS X), they are using System-V compatible tools and not the GNU-tools. Then you might face problems because the GNU-tools offer many features and options, the System-V tools don't have. When you take a look at the man-pages, you will notice, that on System-V systems, they are really short compared to the man-pages of the GNU-tools.

When you write shell scripts, you should be aware, that you have to comment your code (as you surely do while programming in other programming languages too :-). In some cases such as heavy use of regular expressions (e.g. sed), it is fatal, when you don't have a very good documentation on what you're doing here. That's the same as writing perl programs. There are opinions, that perl is a write-only-language because of the many dirty things, you can do with perl. Others say, that perl is the only language, that looks the same before and after uudecoding :-)

Please don't misunderstand me here: I also used perl and it is a quite useful programming language. But you should always be aware of the problems of this language. For me personally, I think that shell-scripts are more readable than perl-scripts. But that differs from person to person and code from code. Ruby or Python is designed to be easily read but as I found out by myself, you can write bad or unreadable/unmaintainable code with *any* language, including Ruby and Python.

So learn the power of shell scripting and the power of more complex scripting languages such as Ruby or Python and choose for every job, what language is better. Don't ignore the other side :-)

Learning shell-scripting

There are lots of pretty good pages out there, that explain shell scripting on various levels of knowledge. I just list some of my bookmarked pages here, without a certain order. Take a look by yourself.

jpilot2google — converting contact information from JPilot-CSV to Google Contact CSV

jpilot2google is a Python-script to convert the CSV export format of J-Pilot to the import CSV format of Google Contact.

I am using the script for periodically exporting my contact information of my PalmOS T3 which I sync with J-Pilot to my GMail account. With this method, I get the latest contact informations on my Android cellphone that syncs with GMail.

I do not want to tell google all the details about my contact data I maintain in detail on my palm handheld. I just need phone numbers and email addresses on my Android cellphone. Therefore I implemented a lot of command line switches to omit a lot of information on demand in order to keep a certain level of privacy.

The script offers its syntax help and further information when executed like: ./jpilot2google --help

A nice (new) feature is the semi-automatically generation of the config file:

  1. get jpilot2google (if not already done)
  2. learn its features with ./jpilot2google --help
  3. find your favourite settings by using the command line switches
  4. make sure, that the directory $HOME/.config/ exists
  5. then use ./jpilot2google --contacts --overwrite-config-file YOUR_OTHER_OPTIONS to let jpilot2google generate your config file for you!
  6. be happy ;-)

Version 0.4 (2009-12-08) is a major extension to previous versions. Please upgrade and enjoy the new features like checking for double phone numbers or email addresses, saving your favourite settings in a config file, a third email and a fourth phone entry, intelligent category detection for phone numbers and email addresses, and even more. 

options:
  -h, --help            show this help message and exit
  --contacts            convert contact CSV format
  --calendar            convert calendar CSV format (FUTURE FEATURE ONLY!)
  --truncate-firstname  contacts: truncate the firstnames to first character
                        and a dot
  --truncate-lastname   contacts: truncate the lastnames to first character
                        and a dot
  --category-prefix     contacts: add a prefix to category fields
  --omit-phones         contacts: ignore all phone numbers
  --suppress-fax-warnings
                        contacts: do not show warning when fax numbers are not
                        processed
  --omit-emailaddresses 
                        contacts: ignore all email addresses
  --omit-if-no-email-and-phone
                        contacts: ignore all contacts that got no email and no
                        phone
  --omit-company-if-name-is-given
                        contacts: do not include company information if a
                        contact has a full name
  --omit-addresses      contacts: ignore all postal addresses
  --omit-organization   contacts: ignore the organization/company field
  --omit-title          contacts: ignore the title field
  --omit-notes          ignore the notes field
  --overwrite-config-file
                        overwrite the config file with a version that
                        includes all currently set options
  --version             display version and exit
  -v, --verbose         enable verbose mode
  -q, --quiet           do not output anything but just errors on console

Examples:

If you are interested in the changes between the different versions, please have a look at the first part of the script itself.

Note: I also have a working tool to push calendar events (which includes a certain keyword) from JPilot (directly through jppy interface) to Google Calendar. The script deletes expired events from Google calendar too. Drop me a line if you are interested in this tool.

go to top of page

bmconverter.rb — converting bookmarks of different browsers

bmconverter.rb is a Ruby-script to convert bookmarks of various formats. The initial version supports Opera Hotlist v2 and FireFox v0.8. You will have to provide Ruby in order to start this script (Debian: apt-get install ruby).

The script can be easily extended with other bookmark formats as long as you know some Ruby-basics. You only have to worry about parsing a new format and/or formatting the items (URLs, Folders, Seperators, Header). Everything else—state machine, bookmark stack, item storage, command line parameter parsing, error prone design (hopefully) and so on—should not be the problem any more.

I hope, this tool finds a lot of friends that extends it to a general bookmark converter, that supports all major types of bookmark formats. At least I couldn't find such a tool when I needed a (good!) conversion of my bookmarks (>3500) from Opera to FireFox. This is the reason, why I also added this tool to SourceForge.net.

go to top of page

YAMN — yet another maildir notifier

YAMN is a simple mail notifier for a Maildir-format which is used e.g. with the MTA qmail. It can play a soundfile and/or start a program to inform you of new mails in your inbox. This script got its own page where you can find more details.

go to top of page

cdcopy — simple cd-burning script

This script copies CDs (audio/data) onto CDR(W)s. This script got its own page where you can find more details.

go to top of page

ladiff.sh — file differences in LaTeX-documents

This script generates a LaTeX-document that points out modified words (using wdiff). This file is processed with pdflatex and displayed with acroread (or xpdf). So you can quickly see changes in two versions of the same document.

The markings for new or deleted words can be changed via a variable: you can choose using the package soul, \fbox{}, or changebar.

Sometimes the file can not be compiled by (pdf)latex without errors. In those cases, you have to modify it by yourself and look for:

  • wrong nestings (wrong "}"),
  • encapsulated "\item",
  • duplicated commands (i.e. two "\small{" but only one closing "}"),
  • change markings in the preamble (before "\begin{document}"),
  • or similar (depending on the chosen method to mark changes).
go to top of page

z2pal.rb — converting Sharp Zaurus OPIE-PIM data (todolist, datebook, addressbook) to the palcal-format

z2pal.rb is a Ruby-script that converts various informations from the PIM applications of the Sharp Zaurus (FAQ) OPIE environment to the palcal PIM-format.

go to top of page

do_rsync.sh — data synchronizing script

Do you have to keep your data accurate on a desktop and e.g. a notebook?

This script manages it all automatically via rsync using a very comfortably machanism: just add to a list in a textfile, which directories you want to have synchronized, mount the other computer (via Samba, nfs, ...) onto your machine and do_rsync.sh will do the rest for you ;-)

There's still an disadvantage: you have to define, which repository is the most current and deleted files aren't recognised and newly created on a sync.

If you have more than one machine to sync' with — no problem. Different configurations are welcome. This script got its own page where you can find more details.

NOTE: This script was a workaround, 'till I found the ultimative sync-tool for the desktop/notebook problem: Unison file synchronizer.

Unison is able to sync between different operating systems(!) and it even sync my PDA with my desktop! You can use Unison in the command line or with a gtk-GUI. But do_rsync.sh is still a quite fine script for some purposes.

go to top of page

lgh — little GnuPG helper: sign, accept and import keys

This script helps to manage the process of handling the network of GnuPG-signatures.

Possible command line switches are:

  • sign: a key is fetched, signed and sent so the owner.
  • accept: after verification of the emailaddress, the key is re-imported to the local keyring.
  • import: someone sent me my signed key, import/add it to my local keyring.
go to top of page

dircheck — check files and subdirectories per white- and blacklists

This script checks the structure of directories and files below the current directory and reports, if certain files and subdirectories exists (or not).

Of course, this directories and files can be defined in whitelists (they should exist) and blacklists (they should not exist). Names of directories in the starting directory aren't checked (in this version). Please note, that reported missing files from non-existent subdirectories aren't filtered out (yet).

go to top of page

mountusbstick — loops through all USB-sticks available and tries to mount them using several filesystems

This script checks all attached USB-sticks (using 'ide-scsi'-kernelmodule) and tries to mount the stick to a defined mountpoint using several filesystems such as msdos and vfat. Please modify the script to meet your requirements!

go to top of page

singleton — prevents multiple instances of programs

Short but useful: When you start programs through this wrapper-script, it ensures, that only one instance of this programs are running. I use it for my email-tool (mutt) and my newsreader (slrn) in order not to mess up my Maildir-directory or my newsrc. You might recognize this functionality from the design pattern which is also called 'singleton'.

go to top of page

any — print out any running job which corresponds to a word

Short but useful: I often need the information, if anything of XY is running. Therefore I am using a script, that finds all jobs that contains the keyword, which is added as command line option to 'any'.

go to top of page

date2name — adds the modification time to filename

I find it very practical to add a timestamp to a filename. So the files are listed always according to the timestamp instead of alphabetic order. This helps me following the development of a project in the course of time. date2name adds such a timestamp to files automatically.

2009-03-25: a complete rewrite of date2name using Python (instead of shell) offers way more possibilities like converting existing datestamps, modify directory names, more datestamp formats, dryrun-mode, ... date2name was tested on GNU/Linux and OS X 10.5.6 Leopard.

go to top of page

mkdirday — creates a directory whose name starts with the current day

Short but useful: For same reasons I mentioned above (date2name), I like to make directories, that contains the current day in their name. mkdirday does this thing.

go to top of page

underscore — adds and removes underscores to filenames

underscore is able to replace spaces in filenames with underscores (_) and vice versa. Very handy with mp3/ogg-files and so forth.

go to top of page

getfloppyimage — reads in floppy image(s)

If you ever did a backup of your old diskettes, you probably got as sick as I was while typing the same thing a couple of docent times. getfloppyimage does the stupid work while you can concentrate on doing more important stuff. The script reads in one diskette image or a row of images using the tool 'dd'.

go to top of page

abcdestatus — prints out unfinished 'abcde'-jobs and some other status informations

I once ripped all my audio-CDs and this was hard work because I have a couple of hundreds ;-)
This script helped me a lot while using the command-line ripper-tool 'abcde'. It prints out unfinished jobs (by scanning the temporary files), the free hard-disc space, the number of running encoders, and the number of rippers. In combination with my other 'abcde'-helperscript 'myabcde', it also prints out the status of the job queue.

go to top of page

myabcde — a callscript for the command-line rippertool 'abcde'

When I once ripped all of my audio-CDs, I needed a tool, that starts a new job, when the old one has finished, if enough hard-disc space is available. I added another starting method: with a special command-line argument, myabcde starts a new 'abcde'-job, when XXX MBs are free. I used 'abcde' in that way, that it deletes the temporary WAV-files after encoding and therefore, there was always more free space after a couple of minutes. With this script, it was more easy to realize unattended ripping. Please note, that I wrote another script, that simplifies job handling with myabcde.

go to top of page

Zget — downloads some HTML-pages for offline-usage on a PDA

This script is rather specific for my needs but you might still find it useful to adopt it to your requirements.
Zget downloads the heise newsticker (PDA-version) of the last 2, 4 or 7 days, the daily new nerd clippings (the most important comic strips), and the program for local cinema (here, local means Graz/Austria) for offline-usage on my Sharp Zaurus which has a web browser. Additionally, some linkpages are generated, in order to easily get to the downloaded content. As I already mentioned: exactly this version might not satisfy you but it may serve as a template for your own download-script.

go to top of page
Last modified: Mon May 24 13:45:53 CEST 2004