TAPE BACKUP SCRIPT FOR UNIX/LINUX

(Revised Apr 12, 2002)

Background and brief description

tpio is a unix shell script for backing up an arbitrary number of separate trees sequentially onto a single tape as a series of unix-standard "cpio" blocked archives. It allows specified tree-nodes to be excluded. It builds an index for each tree as it's processed, and archives that to tape, followed immediately by the actual file tree itself.

At the end of the backup run, these indexes (as many as there were trees recorded) are then concatenated into a single contents file and left in the archive directory as a convenient, searchable record of each tape's contents.

The script has an imposing number of switches, but as with the cpio and tar commands themselves, you'll usually only find yourself using 2 or 3 of them.


In reality, the process of saving any sort of archive onto tape and retrieving it is quite trivial under any Unix or Linux system. With Linux, for example, a tape device such as /dev/nst0 (= SCSI Tape 0 with no auto-rewind) can be written-to by just 'cat'-ing to it.

So one can in principle just create an archive file and then copy it to tape by nothing more complicated than:

% mt rewind # (Mag tape rewind)
% cat my_archive > /dev/nst0
Then eject the tape, label it, and store it away.

Later, if you need to do a recovery, you simply load the appropriate tape and go:

% mt rewind
% cat < /dev/nst0 > my_archive

and then retrieve files from the file my_archive at your leisure.

This all follows quite naturally from the general Unix/Linux concept that all "devices" (or device drivers) ought to act like files, thus allowing programs to write to them, read from them, rewind (or seek) on them, and so on.


So if it's all this easy, why bother with a script? Well, some people don't. But I must admit that my main reason for writing this "cpio shell" was my total irritation in having to waste hours physically searching tapes for files. So I decided to write a script which (apart from simplifying the args to cpio, dd, and mt - which I can never remember) would create an automatically numbered index file from every session. These index files would be kept in a convenient directory and can then be searched at any time to verify the existence or otherwise of file(s) on a tape before needing to physically find, load, and read the actual tape.

Incidentally, these index files are created and numbered by tpio beginning at 100. (The names 1 to 99 are reserved for various types of delta backups.) Their default location is tpio's archive directory, currently defined as /usr/local/adm/dumpdir/labels (but you can change this of course).

When tpio finishes a session, it rewinds the tape and reads the first archive back as a final check that all is (reasonably) well. I added this little extra after getting an "I/O error" one morning when trying to recover some files for a user. (New tapes should of course be perfect, but when they do fail - no amount of bad language and jumping up and down on the spot really helps the user.) Being able to read the first archive back (immediately after recording the last archive and doing a full rewind) does at least verify the process to some extent, and it's much better than no check at all.

Results of each backup run (good or bad) are mailed to $USER (normally root if you run things from cron). If $USER isn't defined, tpio defaults it to root anyway. You can reset this by editing the script, of course.

Files:

The script uses (and assumes the existence of) find, cpio, mt, sed, cat, echo, dd and Mail.

The use of dd was suggested by an ex-colleague, David Magnay. He noticed that our tape drives were spending a lot of time "searching around" during a recording run. We concluded that each time the system went to fetch more data from disc or off the network, the delay was just sufficient for the tape drive to run out of data, so it would have to stop.

It would then rewind a little way, pause briefly, move forward again (probably to locate the end of the last block that it just recorded), and finally pause again to await the next block of data. Unfortunately, this whole rewind, pause, forward search, and pause cycle seemed to be taking 4 or 5 seconds, and since it was only actually recording for 1 to 2 seconds after that, the nett result was that the tape drive was only spending about 10% of its time recording.

David's idea (way back in 1988) was to pipe through "dd" and set its buffer size to some high value, such as 1 or 2 Mb. At least that way the tape could then record in large slabs. And this has worked quite well. These days, of course - much higher values, such as 128 or 256 Mb would be feasible on most systems. In 1988, our largest Unix system only had 8Mb of RAM (and 2x86Mb = 172Mb total disc) for 7 or 8 users. The smaller one had 4Mb of RAM - and it still ran quite well using National Semiconductor's port of Bell Labs Unix!

Usage examples:

Just typing

tpio -+
gives a brief help menu and a couple of usage examples.

If you typed something like:

tpio -ox / /usr /usr/local
you would end up with a tape containing six cpio archives placed end to end - the root index, the root tree, the /usr index, the /usr tree, the /usr/local index, and the /usr/local tree. The -o flag simply says "out to tape". The -x flag causes the -xdev (-mount) flag to be passed into "find" to stop the directory recursion from crossing through any mount points.

The assumption in this example is that /usr/local is an additional, separate mount to /usr, which is fairly common. But if this is not the case, then of course you will only need /usr.)

A -f <filename> option may also be supplied. This specifies the name of an exceptions file which is assumed to contain a list of tree or file names that you specifically want to exclude. So (eg) if you wanted to skip /usr/local/src/XFree86 and /dev, create a file called (eg) "exceptions" and in this, put the regular-expression lines:

# Following path descriptors are sed RE's for exclusion of certain paths in
# tape backups via "tpio".  Must be absolute, ie: they must start from "/"
/^\/usr\/local\/src\/XFree86\//d
/^\/dev\//d

Note that these directory-exclusion lines are in the form of (sed) RE (Regular Expressions), since they're simply passed to a "sed" line inside tpio. The first and last / in each line are the sed delimiters. All actual filename slashes must then be escaped using backward (Microsoft) slashes. (The "^" (caret) simply anchors things to the start-of-line.)

To make use of this exceptions file, call tpio as:

tpio -ox -f exceptions / /usr /usr/local
and /usr/local/src/XFree86 and /dev will be skipped.


NOTE: Before you can use tpio, you may need to edit it to select the appropriate /dev/ entry for your SCSI tape drive. The if block near the top of the script will have a bit of a go at this for you, but you may still need to change it. The quickest way to test it is to simply type

tpio -r
and you should see something like
Rewinding tape (wait):
Tape rewound.
if you have the correct device and it has a tape in it.

Typing just "tpio" or "tpio -+" provides help (optimised for a 40 line screen).



RECOVERY

To recover (for example) /usr from the tape created by the above example, put the tape in the drive and type:

tpio -r
to rewind the tape back to the start, then:
tpio -s 2
to skip the first two archives (the root index and tree archives), then type:
tpio -i
to read in the /usr index (if you first want to check for the existence of certain files). If you don't need the index, then just type
tpio -s 3
instead. Either way, once the tape is at the end of the 3rd archive (the /usr index), just type:
tpio -i
and the /usr tree will be pulled off the tape into the current directory.

When I do this sort of thing, I typically go to an empty temp directory - that way, the tree is recovered into there instead. I can then recover selected files by copying back into /usr from there. (This has the added advantage of not causing any chaos if I've inadvertantly positioned myself at the start of the wrong archive on the tape!)



BUGS

The version presented here has just suffered a minor "hack" from its earlier Solaris/SunOS incarnation (where it was used with two Exabyte and one Archive Python drive) to now run with Linux and one 1994-vintage Connor Archive Python (4mm DAT) drive.

Linux quite happily recognised the Connor drive, but when trying to read old Sun tapes, I kept getting I/O errors until I made some quick modifications to the blocking factor as shown on Evgeny Rodichev's web page, currently at http://cddb.sai.msu.su/~er/zeus_st.html (*).

(*) Thanks to Greg Kaiser for mailing me this new URL for Evgeny's page

As I recently discovered, the wonderful Linux SCSI-Howto also explains this stuff, in particular re the need to have the BSD mt program. The GNU one doesn't support the setblk command.

You don't need to use the -4 (4mm DAT tape) flag now, BTW. I only added this when I was using Solaris with multiple drives to flag our DAT unit. This current hacked version now defaults to Linux/DAT.

The script really needs a bloody good tidy up to make it more elegant. (If you do have any problems running it, BTW - include the -D (debug) flag, and you'll at least be able to see what it's doing.)



VERSION INFO

I've now tidied up tpio somewhat and removed the need for the undocumented 'seed file' (100.dummy), which I must admit I'd quite forgotten about until I went to use the script on a new (Solaris 2.8) machine in April 2002!

The new version is 6.01, dated April 11, 2002.


  Back to (Bluehaze) software archive page

 Bluehaze home page