Insights from practitioners in Information Management

Issue 68 – Vital Records

Over the last couple of weeks I have been unfortunate enough to lose an external backup hard drive and then a few days ago the hard drive on my laptop decided to die. Thankfully I had managed to replace the external hard drive in the mean time, so data loss was minimal. But the loss in terms of time has been huge.

Since beginning this newsletter, 68 editions ago – we have looked at the issues surrounding disaster planning and prevention, media fragility and of course digital archiving more than once. We also asked what value do you place on your information?

And IEA is not the only organisation to look at these issues. Every day the various list servs around the world speak of the same kinds of problems people were facing many years ago. And the reason is simple – technology moves at a faster pace than the organisations record keeping systems and the organisations who run said programs can keep up with. Every time we think we have a definitive answer to the interesting question – we find out we don’t. In fact we’re just scratching the surface of the proverbial iceberg when it comes to record keeping, archiving and retrieval. Hopefully this edition of Overload will bring all the pieces together. Then again, by Monday we’ll probably need to download yet another new piece of software…but I could just be being cynical.

In this Issue we will be looking at:

  • Media Fragility – Backups, Updates and Changes in Technology
  • Migration
  • Backups and Archives

Media Fragility – Backups, Updates and Changes in Technology
We seem to have a very short memory when it comes to computer technology; we know that the operating system that we are running today will be replaced with a more modern equivalent in the not too distant future. We also know that it will be installed with little thought or regard as to how much data we will lose as a direct result of the upgrade. We’re assured everything will be fine, and we hope and pray it will be. But does anyone else remember trying to upgrade through the early versions of windows? Or am I really showing my age now. Do you remember not upgrading through each version – because you didn’t think you needed to, then finding out you couldn’t read the old documents in the newest version.

For example, the digital copy of my thesis is sitting on a 3″ – yes I did say 3″ – floppy. It was written using a proprietary piece of technology called the Amstrad. Whilst I am unable to read the electronic copy – unless I want to try and find a museum somewhere that may have a working version, I can still go back to the University and locate a copy on the shelves, or given that it is in the UK and I live in Australia these days, will have to rely on the paper version that is sitting in my personal “paper-based” archive…

And I know I am not the only one with these kinds of issues and dilemmas. Do we scan and store electronically every item we have in our archives, so we can search for and find those items we need? Do we scan everything and worry we don’t have the best “format” for long term electronic storage, and hope the migration strategies we have in place will cope? Or do we carry on and manage the paper based versions and hope we don’t have to find it and produce it for any kind of reason – let alone a legal challenge.

Karen Lillington, writer with the Irish Times asked “I am wondering what “more modern format” could possibly be able to promise us that memories would remain accessible not just for one lifetime, but for several?

And responded with the statement that – “there is no modern digital format that provides anything more than a short-term answer. Advancing technology and new developments in storage media mean ever shorter periods during which a given format reigns.”

As you and I know, technological obsolescence is still a major problem for all organisations. Despite many projects, thousands of dollars and hundreds of person hours, we are still wading through the problem of trying to keep pace with the changes in technology, and access to our material. Lets put this back to the personal level. As I mentioned in the introduction – I’ve lost an external hard drive that contained my entire backup. A few weeks later the hard drive on my lap top died. Thankfully they didn’t die at the same time, then I would have been devastated, because those two pieces of technology contain my entire writing archive. As a writer and author my writing is my life.
So what can we do to ensure that we can still read the data when we need to?

In personal terms, I am now doing what I used to do BEFORE the storage capabilities of the external hard drives became cost effective for people like me – I’m sending my personal archive to the Internet, interestingly enough this has been termed “cloud” computing. I prefer the “ether” because “either” you can get it back or you can’t – it just depends whether their storage capabilities is any better than anyone else’s. …. Anyway I have several internet email accounts. One of these has unlimited storage capabilities, so I am able to send most of my files to the ether. Some of the larger files of course I can’t do that with – well until we get unlimited file size attachments which won’t take long if enough people shout about it. But I have these items in hard copy and could reproduce them if I ever needed to.

Of course every business, company and organisation has a larger problem than any single person has. Multiply the number of people who work in your organisation and you can begin to see how big an internet based archive you would need. Which makes the decision some organisations are taking to use Internet E-mail rather than the traditional e-mail accounts an interesting one. Not least because, there is no guarantee in life that the servers they use to store your information will survive any better or longer than the servers you have within your own organisation. And they make no guarantees either.

One of the other issues they can’t guarantee is of course, privacy – these servers are just as vulnerable to attack as our own. They can be hacked, cracked and laid open just like any other server. Add an IT crowd who don’t have the same interests in keeping your personal /  business / mission critical information safe as you do, and you are risking a lot by using these outside organisations to store your information. But that doesn’t stop people doing so.

So why am I doing it? Well, my email archive is a backup of a backup. I have multiple storage locations in the hope the information will be secure. I can also hope, given the size of the organisations in question, will have enough money at their disposal to ensure the data will be migrated through the relevant upgrades with minimal data loss.
The truth is – there is still no single solution to the ubiquitous issue – of storing electronic information and maintaining access to it. All we can hope to do is manage the process as best we can and hope programs such as VERS and InterPARES will eventually find the definitive answer (of course they could also answer that they have…).


I’ve mentioned migration – migration is important.

If you keep your data “live” on the server you use for your day-to-day operations, you can ensure this material can be read – because the data will be upgraded as and when you need to change / update your software and hardware.

And then there is technology refreshing. As in you move data from one format to another. A good example is that of video tape. Few people I would suspect still have a tape player at home. Fewer still would have a Betamax recorder. Did you throw your tapes away at the same time as the player? Did you replace them with DVD’s? How about your personal camcorder tapes? Do you still own a camcorder? Have you converted these tapes into a new format so you can marvel at your children’s antics? Or more interestingly – your own !

One of the biggest problems to migration and technology refreshing is cost. Either you employ someone to copy each item – in real time, to another format – and check the results….so they get to watch it twice. Or you outsource the job to an organisation who does this kind of thing for a living. Either way, depending on the material you are trying to preserve, you may need to undertake this process more than once…if you want to be able to read / access the material at some point in the future. Which is all very well, it still does not complete the picture when it comes to what to do with electronic information – and how do you ensure you can find it, open it, and be able to read it / view it again.

Backups and Archives:

First and foremost – backups are not your archive. I know you know that, just thought I would say it “for the record”.

Backups are designed to ensure you can get back up and running after an incident. That incident could be the deletion of a file you shouldn’t have. But it can also include –

  • Servers melting, hard drives crashing, natural events such as storm damage, acts of terrorism and all out war.
  • Building failure – malfunctioning sprinklers, heating or air conditioning, leaks, faulty wiring
  • Industrial accidents – nuclear and/or chemical spills and
  • Criminal behaviour – hacking, theft, arson, espionage, vandalism, riots.

Ensuring your data is backed up can and does play an important role in disaster planning, and organisations should ensure that all key data is backed up to another media at least once a day, more if your organisation deems necessary. Being able to access those same backup tapes (or other media used for the purpose) can sometimes be the difference between survival or not for the company. It has been said that:

  • 70% of organisations that suffer paperwork and computer loss go under within 3 years (McDougall, 1989)
  • 43% of businesses never re-open and a further 29% go under within 12 months. (Datapro Research, 1990)
  • 50% never recover from a major incident. (Sarkus, 1992)
  • There is only a 10% survival rate after a major computer crash. (White, 1989)
  • 48% of organisations cannot tolerate more than 24 hours of downtime (KPMG, 2002)

It took me a week (given that I work full time and could only work on my “problem” during the evenings and on the weekend) to rebuild the databases, re-load the software. Well when I say “me” a very good friend did most of the hardware / software – I did the data re-build. A week. I am in the middle of writing a book – the information is on two USB drives and the new external hard drive, so I didn’t lose any of the information. What I did lose was time. A valuable commodity when you don’t have a lot of spare time to play with. In terms of mission critical information – I didn’t lose very much at all. I had kept every email (in the ether) I had received so I could rebuild my personal newsletter distribution list. And as I have mentioned I didn’t lose the book I am in the middle of writing. So I would consider myself very lucky. The gaps in my personal archive I could plug from the original sources. And now I understand what needs to be backed up I can and have done that.

How about your organisation? Do you know what data is classed as mission critical? Whilst some vital records may be similar to the organisation next door, only you can determine what you will need in order to get back up and running. Can you afford any “down time”? Can you recover all of your lost data, or only some of it?

If your system cannot be down for any period of time, then perhaps another option would be to house mirror information off site, and away from the seat of the main operations. However, with the large-scale disasters that have been happening of late – how far is far enough away?