futureArch, or the future of archives...

Digital Preservation: What I Wish I Knew Before I Started

2012-01-30T09:41:00.005Z

Tuesday 24th January, 2012

Last week I attended a student conference, hosted by the Digital Preservation Coalition, on what digital preservation professionals wished they had known before they started. The event covered a great deal of the challenges faced by those involved in digital preservation, and the skills required to deal with these challenges.

The similarities between traditional archiving and digital preservation were highlighted at the beginning of the afternoon, when Sarah Higgins translated terms from the OAIS model into more traditional ‘archive speak’. Dave Thompson also emphasized this connection, arguing that digital data “is just a new kind of paper”, and that trained archivists already have 85-90% of the skills needed for digital preservation.

Digital preservation was shown to be a human rather than a technical challenge. Adrian Brown argued that much of the preservation process (the "boring stuff") can be automated. Dave Thompson stated that many of the technical issues of digital preservation, such as migration, have been solved, and that the challenge we now face is to retain the context and significance of the data. The point made throughout the afternoon was that you don’t need to be a computer expert in order to carry out effective digital preservation.

The urgency of intervention was another key lesson for the afternoon. As William Kilbride put it; digital preservation won’t do itself, won’t go away, and we shouldn't wait for perfection before we begin to act. Access to data in the future is not guaranteed without input now, and digital data is particularly intolerant to gaps in preservation. Andrew Fetherstone added to this argument, noting that doing something is (usually) better than doing nothing, and that even if you are not in a position to carry out the whole preservation process, it is better to follow the guidelines as far as you can, rather than wait and create a backlog.

The scale of digital preservation was another point illustrated throughout the afternoon. William Kilbride suggested that the days of manual processing are over, due to the sheer amount of digital data being created (estimated to reach 35ZB by 2020!). He argued that the ability to process this data is more important to the future of digital preservation than the risks of obsolescence. The impossibility of preserving all of this data was illustrated by Helen Hockx-Yu, who offered the statistic the the UK Web Archive and National Archives Web Archive combined have archived less than 1% of UK websites. Adrian Brown also pointed out that as we move towards dynamic, individualised content on the web, we must decide exactly what the information is that we are trying to preserve. During the Q&A session, it was argued that the scale of digital data means that we have to accept that we can’t preserve everything, that not everything needs to be preserved, and that there will be data loss.

The importance of collaboration was another theme which was repeated by many speakers. Collaboration between institutions on a local, national and even international level was encouraged, as by sharing solutions to problems and implementing common standards we can make the task of digital preservation easier.

This is only a selection of the points covered in a very engaging afternoon of discussion. Overall, the event showed that, despite the scale of the task, digital preservation needn't be a frightening prospect, as archivists already have many of the necessary skills.

The DPC have uploaded the slides used during the event, and the event was also live-tweeted, using the hashtag #dpc_wiwik, if you are interested in finding out more.

What is ‘The Future of the Past of the Web’?

2011-10-18T13:40:00.001+01:00

‘The Future of the Past of the Web’,

Digital Preservation Coalition Workshop

British Library, 7 October 2011

Chrissie Webb and Liz McCarthy

In his keynote address to this event – organised by the Digital Preservation Coalition , the Joint Information Systems Committee and the British Library – Herbert van der Sompel described the purpose of web archiving as combating the internet’s ‘perpetual now’. Stressing the importance to researchers of establishing the ‘temporal context’ of publications and information, he explained how the framework of his Memento Project uses a ‘ timegate’ implemented via web plugins to show what a resource was like at a particular date in the past. There is a danger, however, that not enough is being archived to provide the temporal context; for instance, although DOIs provide stable documents, the resources they link to may disappear (‘link rot’).

The Memento Project Firefox plugin uses a sliding timeline (here, just below the Google search box) to let users choose an archived date

A session on using web archives picked up on the theme of web continuity in a presentation by The National Archives on the UK Government Web Archive, where a redirection solution using open source software helps tackle the problems that occur when content is moved or removed and broken links result. Current projects are looking at secure web archiving, capturing internal (e.g. intranet) sources, social media capture and a semantic search tool that helps to tag ‘unstructured’ material. In a presentation that reinforced the reason for the day’s ‘use and impact’ theme, Eric Meyer of the Oxford Internet Institute wondered whether web archives were in danger of becoming the ‘dusty archives’ of the future, contrasting their lack of use with the mass digitisation of older records to make them accessible. Is this due to a lack of engagement with researchers, their lack of confidence with the material or the lingering feeling that a URL is not a ‘real’ source? Archivists need to interrupt the momentum of ‘learned’ academic behaviour, engaging researchers with new online material and developing archival resources in ways that are relevant to real research – for instance, by helping set up mechanisms for researchers to trigger archiving activity around events or interests, or making more use of server logs to help them understand use of content and web traffic.

One of the themes of the second session on emerging trends was the shift from a ‘page by page’ approach to the concept of ‘data mining’ and large scale data analysis. Some of the work being done in this area is key to addressing the concerns of Eric Meyer’s presentation; it has meant working with researchers to determine what kinds and sources of data they could really use in their work. Representatives of the UK Web Archive and the Internet Archive described their innovations in this field, including visualisation and interactive tools. Archiving social networks was also a major theme, and Wim Peters outlined the challenges of the ARCOMEM project, a collaboration between Sheffield and Hanover Universities that is tackling the problems of archiving ‘community memory’ through the social web, confronting extremely diverse and volatile content of varying quality for which future demand is uncertain. Richard Davis of the University of London Computer Centre spoke about the BlogForever project, a multi-partner initiative to preserve blogs, while Mark Williamson of Hanzo Archives spoke about web archiving from a commercial perspective, noting that companies are very interested in preserving the research opportunities online information offers.

The final panel session raised the issue of the changing face of the internet, as blogs replace personal websites and social media rather than discrete pages are used to create records of events. The notion of ‘web pages’ may eventually disappear, and web archivists must be prepared to manage the dispersed data that will take (and is taking) their place. Other points discussed included the need for advocacy and better articulation of the demand for web archiving (proposed campaign: ‘Preserve!: Are you saving your digital stuff?’), duplication and deduplication of content, the use of automated selection for archiving and the question of standards.

Day of Digital Archives, 2011

2011-10-06T13:13:00.001+01:00

Today is officially 'Day of Digital Archives' 2011! Well, it's been quite a busy week on the digital archives front here at the Bodleian...

The week began with the arrival of our new digital archives graduate trainee, Rebecca Nielsen. During her year here with us, the majority of Rebecca's work will be on digital archives of one kind or another, she'll be archiving all sorts, from materials arriving on old floppies to web sites on the live web.

Another of my colleagues, Matthew Neely, has been spending quite a bit of time this week working on the archive of Oxford don, John Barton. The archive includes over 150 floppies and a hard disk as well as hard-copy papers and photographs.

Barton's digital material was captured in our processing lab back in the Spring of 2010, and now Matthew is busy using Forensic Toolkit software to appraise, arrange and describe the digital content alongside the papers. There are a few older word-processing formats in the collection, but all things that we can handle.

We've also been having conversations with quite a few archive depositors this week, about scoping collections and transfer mechanisms, among other things. There has been some planning work too, while we consider the requirements for processing the archive of Sir Walter Bodmer, which includes around 300 disks (3.5" and 5.25"). For more on the Bodmer archive see the Library's Special Collections blog, The Conveyor.

Today, I've spent a little time looking at our 'Publication Pathway' and thinking about where we need a few tweaks. This is the process and toolset that we are building to publish our digital archives to users (Pete called it CollectionBuilder, and you can have a look at a slightly out-of-date version of it here: http://sourceforge.net/projects/beamcollectionb/). We have a bit more work to do on this and our user interface, but quite a bit of material in the pipeline waiting to get out to our users.

To close out the week, two of our webarchiving pilot group are heading off to the DPC's The Future of the Past of The Web event tomorrow, to learn more about the state of the art in webarchiving.

Lastly, I can't resist returning to the start of the week. On Monday, we had a power cut and temporarily lost access to Bodleian Electronic Archives and Manuscripts (BEAM) services. An unsubtle reminder that digital archives require lots of things to remain accessible, power being one of them!

Another source for old software

2011-09-08T11:43:00.000+01:00

This one is called http://www.oldapps.com/. Looks like a useful range of apps.

Comparing software tools

2011-08-22T16:59:00.004+01:00

While looking at software relating to digital video earlier today I came across a handy website called alternativeTo. It's a useful means of comparing software applications and getting an idea of the tools that are out there to help perform a particular task. AlternativeTo gives a brief summary of each piece of software along with screenshots of the software in action. Another useful feature is that searches can be filtered by whether the tools are free or open source.

Mobile forensics

2011-08-10T17:06:00.000+01:00

Anyone who heard Brad Glisson's talk at the DPC event on digital forensics may be interested in the paper 'A comparison of forensic evidence recovery techniques for a windows mobile smart phone' published in Digital Investigation Volume 8, Issue 1, July 2011, Pages 23-36. Interesting to see what was recovered, and how the different tools behave.

Preserving born-digital video - what are good practices?

2011-07-27T13:40:00.000+01:00

http://www.flickr.com/photos/fensterbme/1990023423/

Interesting to see Killian Escobedo's post on digital video preservation over at the Smithsonian Archives' visual archives blog. Our trainee, Emma, is working on questions of these sort at the moment as we start to develop strategies for preserving the vast amount of born-digital video being deposited in our archive collections. While there's quite a lot of material out there on digitising analogue video, we've found a real shortage of guidance on the management of born-digital video collections. With that in mind I'd be really interested in hearing how other folks are dealing with this kind of material. Can you give us any pointers? At the moment we're particularly interested in learning more about existing practices, good tools, realistic workflows, and preservation-grade standards (for metadata and content - which ones and why?).

So, what kind of digital video do we have? It's a good question, and one I can't answer fully for the moment. What I can say is that our collections include digital video deposited on CDs, DVDs, Bluray discs, miniDV and mediumDV cassettes, and hard disks. Much of this material has yet to be captured from its original media so we don't have that inventory of codecs, wrapper formats, frame rates, metadata, etc. that Killian talks about. This kind of detailed survey work is a next step for us, but one that will have to wait until we have developed a workflow for initial capture (bit-level preservation comes first). I wonder if we'll see the same diversity of technical characteristics present in the Smithsonian's materials. It seems likely.

Hidden Pages

2011-05-11T08:30:00.003+01:00

Yesterday I foolishly uploaded a Pages document to my work machine (that isn't a Mac) before heading into the office. I needed the content because I was due to give it to Susan that morning. Luckily I stumbled upon this tip and thought I'd share it in case you ever find yourself faced with a Mac disk full of documents and no Mac to read them on...

I guess I shouldn't have been surprised to discover a Pages document is in fact a zip file (like Word docs) and if you unpack it not only do you find an XML representation of the document (which would let you get at the text - run it through tidy first though as there aren't any line breaks!) or, neater, in the QuickLook directory is a PDF (file reports PDF 1.3) of the document.

Day saved! Phew!

Media recognition - Floppy Disks part 3

2011-04-20T17:39:00.000+01:00

3 inch Disks (Mitsumi 'Quick Disk')

Type:	Magnetic storage media
Introduced:	?1985
Active:	Unlikely.
Cessation:	Used in the 1980s.
Capacity:	?128KB - 256KB
Compatibility:	Requires a 3” drive appropriate to the manufacturer's specifications.
Users:	Likely to have been individual users and small organisations. Used for word-processing, music and gaming.
File Systems:	Unknown. May vary according to use. The disks were manufactured by Mitsumi and offered as OEM to resellers and used in a range of contexts including Nintendo (Famicom), various MIDI keyboards/samplers (Roland) and the Smith Corona Personal Word Processor (PWP).
Common Manufacturers:	Disks: Mitsumi appear to have made the magnetic disk (the innards), while other manufacturers made the cases. This resulted in different case shapes and labelling. For example Sharp Corona labelled the disks as DataDisk 2.8". Drives: Mitsumi?

Recognition

The Smith Corona Personal Word Processor (PWP) variant of the disk is double sided with one side being labelled ‘A’ and the other ‘B’. Each side also had a dedicated write-protect hole, known as a 'breakout lug'.

2.8" Smith Corona 'Quick Disk'


3.5" floppy side-by-side with a 2.8" Smith Corona 'Quick Disk'


Nintendo Famicon disk Some rights reserved by bochalla

High Level Formatting

Unknown. Possibly varied according to use.

3 Inch Disk Drives
Varied according to disk. The Smith Corona word processing disks are most likely to turn up in an archival collection. These were used in a Smith Corona PWP and possible models nos. include: 3,5,6, 6BL, 7, X15,X25, 40, 50LT, 55D, 60, 65D, 75D, 80, 85DLT, 100, 100C, 220, 230, 250, 270LT, 300, 350, 355, 960, 990, 2000, 2100, 3000, 3100, 5000, 5100, 7000LT, DeVille 3, DeVille 300, Mark X, Mark XXX, Mark XL LT.

Lego mockup of a Nintendo Famicon drive

Some rights reserved by kelvin255

Useful links
http://www.cromwell-intl.com/technical/quickdisk-recovery.html
http://en.wikipedia.org/wiki/History_of_the_floppy_disk

Preserving Digital Sound and Vision: A Briefing 8th April 2011

2011-04-13T09:53:00.008+01:00

Last Friday I went along to the DPC briefing Preserving Digital Sound and Vision. I was particularly interested in the event because of digital video files currently held on DVD media at the Bodleian.

After arriving at the British Library and collecting my very funky Save the Bits DPC badge I sat down to listen to a packed programme of speakers. The morning talks gave an overview of issues associated with preserving audio-visual resources. We began with Nicky Whitsed from the Open University who spoke about the nature of the problem of preserving audio-visual content; a particularly pertinent issue for the OU who have 40 years of audio-visual teaching resources to deal with. Richard Ranft then gave a fascinating insight into the history and management of the British Library Sound Archive. He played a speech from Nelson Mandela’s 1964 trial to emphasise the value of audio preservation. Next Stephen Gray from JISC Digital Media spoke about how students are using audio-visual content in their research. He mentioned the difficulties researchers find when citing videos, especially those on YouTube that may disappear at any time! To round off the morning John Zubrycki from BBC R and D spoke about Challenges and Solutions in Broadcast Archives. One of the many interesting facts that he mentioned was that subtitle files originally produced by the BBC for broadcast have been used as a tool for search and retrieval of video content.

After enjoying lunch and the beautiful sunny weather on the British Library terrace we moved onto the afternoon programme based on specific projects and tools. Richard Wright of the BBC spoke about the Presto Centre and the tools it has developed to help with audio-visual preservation. He also spoke about the useful digital preservation tools available online via Presto Space. Sue Allcock and James Alexander then discussed the Outcomes and Lessons learnt from the Access to Video Assets Project at the Open University which makes past video content from the Open University’s courses available to OU staff through a Fedora repository. Like the BBC, discovering subtitle files has allowed the OU to index their audio-visual collections. Finally Simon Dixon from the Centre for Digital Music Queen Mary University spoke about emerging tools for digital sound.

A final wide ranging discussion about collaboration and next steps followed which included discussion about storage as well as ideas for a future event addressing the contexts of audio-visual resources. I left the event with my mind full of new information and lots of pointers for places to look to help me consider the next steps for our digital video collections… watch this space.

Sharp font writer files

2011-04-12T17:48:00.000+01:00

Not a format I'd come across before, but we now have files of this type in the collections. They were written on something like this. Luckily someone has written a migration tool, and it seems to work. See fwwputils.

Anyone know of other tools for this format?

Got any older?

2011-04-07T10:32:00.003+01:00

Interesting article about the "oldest working Seagate drive in the UK". When we talk about storage here, eventually someone says "storage is getting cheaper". If you ever needed concrete proof, this is it!

World backup day 2011

2011-03-31T19:59:00.000+01:00

Today is officially 'world backup day', apparently. Time to back up your files...

I'm sure you know why, and perhaps even how. There are a few tips here if you want them: http://www.worldbackupday.net/

Advisory Board Meeting, 18 March 2011

2011-03-24T10:04:00.004Z

Our second advisory board meeting took place on Friday. Thanks to everyone who came along and contributed to what was a very useful meeting. For those of you who weren't there here is a summary of the meeting and our discussions.

The first half of the afternoon took the form of an overview of different aspects of the project.

Overview of futureArch's progress

Susan Thomas gave us a reminder of the project's aims and objectives and the progress being made to meet them. After an overview of the percentage of digital accessions coming into the library since 2004 and the remaining storage space we currently have, we discussed the challenge of predicting the size of future digital accessions and collecting digital material. We also discussed what we think researcher demand for born-digital material is now and will be in the future.

Born Digital Archives Case Studies

Bill Stingone presented a useful case study about what the New York Public Library has learnt from the process of making born-digital materials from the Straphangers Campaign records available to researchers.

After this Dave Thompson spoke about some of the technicalities of making all content in the Wellcome Library (born-digital, analogue and digitised) available through the Wellcome Digital Library. Since the project is so wide reaching a number of questions followed about the practicalities involved.

Web archiving update

Next we returned to the futureArch project and Susan gave an overview of the scoping, research and decisions that have been made regarding the web archiving pilot since the last meeting. I then gave an insight into how the process of web archiving will be managed using a tracking database. Some very helpful discussions followed about the practicalities of obtaining permission for archiving websites and the legal risks involved.

After breaking for a well earned coffee we reconvened to look at systems.

Systems for Curators

Susan explained how the current data capture process works for digital collections at the Bodleian including an overview of the required metadata which we enter manually at the moment. Renhart moved on to talk about our intention to use a web-based capture workbench in the future and to give us a demo of the RAP workbench. Susan also showed us how FTK is used for appraisal, arrangement and description of collections and the directions we would like to take in the future.

Researcher Interface

To conclude the systems part of the afternoon, Pete spoke about how the BEAM researcher interface has developed since the last advisory board meeting, the experience of the first stage of testing the interface and the feedback gained so far. He then encouraged everyone to get up and have a go at using the interface for themselves and to comment on it.

Training the next generation of archivists?

With the end of the meeting fast approaching, Caroline Brown from the University of Dundee gave our final talk. She addressed the extent to which different archives courses in the UK cover digital curation and the challenges faced by course providers aiming to include this kind of content in their modules.

With the final talk over we moved onto some concluding discussions around the various skills that digital archivists need. Those of us who were able to stay continued our discussions over dinner.

TZX

2011-03-17T08:53:00.001Z

I know it is a bit sad, but I couldn't help but feel a little flutter of happiness when I read about TZX. Just wanted to share that.

what have the Romans ever Done For us?

2011-02-18T15:01:00.005Z

Today I presented an internal seminar on RDF to the Bodleian Library developers, the first in a series of (hopefully) regular R&D meetings. This one was to provide a practical introduction to RDF to give us a baseline to build from when we start building models of our content (at least one Library project requires we generate RDF). I called it "what have the Romans ever Done For us?":

The title is a line from the Monty Python film "The Life of Brian" and I chose it not just because I've been looking into Python (the language) but also because I can imagine a future where people ask "What has RDF ever done for us?" in a disgruntled way. In the film people suggest the Romans did quite a lot - bits of public infrastructure, like aqueducts and roads, alongside useful services like wine and medicine. I think RDF is a bit like that. Done well, it creates a solid infrastructure from which useful services will be built, but it is also likely to invisible, if not taken for granted, like sanitation and HTML. :-)

The seminar itself seemed to go well - though you'd have to ask the attendees rather than the presenter to get the real story! We started with some slides that outlines the basics of RDF, using Dean Allemang & Jim Hendler's nice method of distributing responsibility for tablular data and ending up at RDF (see pg. 32 onwards in the book Semantic Web for the Working Ontologist), and leapt straight in with LIBRIS (for example Neverwhere as RDF) as a case study. In the resultant discussion we looked at notation, RDFS, and linked data.

The final half of the seminar was a workshop in which we split into two groups: data providers and data consumers, and then considered what resources at the Bodleian might be suitable for publication as RDF (and linked data) and what services we might build using data from elsewhere.

The data providers discussed how there was probably quite a lot of resources in the Library that we could publish, or become the authority on - members of the University for example, or any of the many wonders we have in the collections. To make this manageable, it was felt it would be sensible to break this task up, probably by project. This would also allow for specific models to be identified and/or developed for each set of resources.

There was some concern among the providers regarding how to "sell" the benefits of RDF and linked data to management. The concerns paralleled what I imagine happened with the emergent Web. Is this kind of data publication giving away valuable information assets, for little or no return? At worst this leads people away from the Library to viewing and using information via aggregation services. Of course, there is a flip side to this argument. Serendipitous discovery of a Bodleian resource via a third-party is essentially free advertising and may drive users through our (virtual or real) door?

The consumers seemed realise early on that one of the big problems was the lack of usable data. Indeed, for the talk, I scoured datasets trying to find a decent match of data to augment Library catalogues and found it quite hard. That isn't to say there is not a lot of data available (though of course there could be more), it is just without an application for it, shoehorning data into a novel use remains a novelty item. However, one possible example was the Legislation API. The group suggested that reviews from other sites could be used to augment Library catalogue results. They also suggested that the people data the Library published could be very useful and Monica talked about a suggestion she had heard at Dev8D for a developer expertise database (data Web?).

All in all lots of very useful discussion and I hope everyone went away with a good idea of what RDF was and what it might do for us and what we might do with it. There (justifiably) remains some scepticism, mostly because without the Web, linked data is simply data and we've all got our own neat ways to handle data already, be it a SQL database, XML & Solr, or whatever. Without the Web of Data the question "What do we gain?" remains.

It is a bit chicken and egg and the answer will eventually become clear as more and more people create machine processable data on the Web.

For the next meeting we'll be modelling people. I'm going to bring the clay! :-)

Slides and worksheets are available with the source open office documents also published on this (non-RDF) page!

Migrating documents

2011-01-24T15:41:00.008Z

We have a collection that consists of several thousand documents in various archaic (well, 1980s/90s) word processor formats including Ami Professional and (its predecessor) Samna Word. Perhaps of interest to folks intent on discussing the implications of migration for authenticity of the items, some of those Ami Pro files contain the (automatically generated) line:

"File ... was converted from Samna Word on ..."

So which is the original now?

Migrating these file formats has not been straight forward. This is because it was proved remarkably tricky to ascertain a key piece of information - the file format of the original. This is not the fault of file format tools (I'm using FITS, which itself wraps the usual suspects JHOVE & DROID), but the broader problem that the files have multiple formats. Ami Pro files are correctly identified "text/plain". The command file reports them as "ASCII English text". Some (not all) have a file extension ".sam" which is usually Ami Word, but the ".sam" files are not all the same format.

Yet this small piece of metadata is essential because without it it is very difficult to identify the correct tool to perform the migration. For example, if I run my usual text to PDF tool - which is primed to leap into action on arrival of a "text/plain" document - the resultant PDF shows the internals of a Ami Pro file, not the neatly laid out document the creator saw. We have a further piece of information available too, and curiously it is the most useful. This is the "Category" from the FTK - which correctly sorts the Ami Pros from the Samna Words.

This leads to a complex migration machine that needs to be capable of collating file format information from disparate sources and making sense of the differences, all within the context of the collection itself. If I know that creator X used Ami Pro a lot, then I can guess that "text/plain" & ".sam" means an Ami Pro document, for example. This approach is not without problems however, not least of which is that it requires a lot of manual input into what should ultimately be an automated and unwatched process. (One day, when it works better, I'll try to share this code!)

Sometimes you get lucky, and the tool to do the migration offers an "auto" mode for input. For this collection I am using a trial copy of FileMerlin to perform the migration and evaluate it. It actually works better if you let it guess the input format rather than attempt to tell it. Other tools, such as JODConverter, like to know the input format and here you have a similar problem - you need to know what JODConverter is happy to accept rather than the real format - for example, send it a file with a content type of "application/rtf" and it responds with an internal server error. Send the same file with a content type of "application/msword" and the PDF is generated and returned to you.

Then there is a final problem - sometimes you have to make several steps to get the file into shape. For this collection, FileMerlin should be able to migrate Ami Pro and Samna Word into PDFs. In practice, it crashes on a very small sub-set of the documents. To overcome this, I migrate these same documents to "rich text format" (which FileMerlin seems OK with) and then to PDF with JODConverter - sending the aforementioned "application/msword" content type. I had a similar problem with WordPerfect files where using JOD directly changed the formatting of the original files. Using libwpd to create ODTs and then converting them to PDFs generated more accurate PDFs. (This is strange behaviour since OpenOffice itself uses libwpd!) Every time I hit a new (old) file format, the process of identifying it and generating a heuristic for handling it starts over.

I'm starting to think I need a neural network! That really would be putting the AI in OAIS!

Digital forensics and digital archives

2010-12-15T10:18:00.000Z

If you're interested in how digital forensics methodologies and tools could, and are, being applied to digital archives then you might like to take a look at a new report published by CLIR. See http://www.clir.org/pubs/reports/pub149/pub149.pdf. It also includes a sidebar outlining how forensics tools are incorporated into our workflow.

What I learned from the word clouds...

2010-11-26T12:49:00.007Z

Now, word clouds are probably bit out of fashion these days. Like a Google Map, they just seem shiny but most of the time quite useless. Still, that hasn't stopped us trying them out in the interface - because I'm curious to see what interesting (and simple to gather) metadata n-grams & their frequency can suggest.

Take for instance the text of "Folk-Lore and Legends of Scotland" [from Project Gutenberg] (I'm probably not allowed to publish stuff from a real collection here and choose this text because I'm pining for the mountains). It generates a "bi-gram"-based word cloud that looks like this:

Names (of both people and places) quickly become obvious to human readers, as do some subjects ("haunted ships" is my favourite). To make it more useful to machines, I'm pretty sure someone has already tried cross-referencing bi-grams with name authority files. I also imagine someone has used the bi-grams as facets. Theoretically a bi-gram like "Winston Churchill" may well turn up in manuscripts from multiple collections. (Any one know of any successes doing these things?).

Still, for now I'll probably just add the word clouds of the full-texts to the interface, including a "summary" of a shelfmark, and then see what happens!

I made the (very simple) Java code available on GitHub, but I take no credit for it! It is simply a Java reworking of Jim Bumgardner's word cloud article using Jonathan Feinberg's tokenizer (part of Wordle).

The as yet unpaved publication pathway...

2010-11-10T15:19:00.006Z

It has been a while since we had a whiteboard post, so I thought it was high time we had one! This delightful picture is the result of trying to explain the "Publication Pathway" - Susan's term for making our content available - to a new member of staff at the Library...

Nothing too startling here really - take some disparate sources of metadata, add a sprinkling of auto-gen'd metadata (using the marvelous FITS and the equally marvelous tools it wraps), migrate the arcane input formats to something useful, normalise and publish! (I'm thinking I might get "Normalise and Publish!" printed on a t-shirt! :-))

The blue box CollectionBuilder is what does most of the work - constructs an in memory tree of "components" from the EAD, tags the items onto the right shelfmarks, augments the items with additional metadata, and writes the whole lot out in a tidy directory structure that even includes a foxml file with DC, PREMIS and RDF data streams (the RDF is used to maintain the hierarchical relationships in the EAD). That all sounds a lot neater than it currently is, but, like all computer software, it is a work in progress that works, rather than a perfect end result! :-)

After that, we (will, it aint quite there yet) push the metadata parts into the Web interface and from there index it and present to our lovely readers!

Hooray!

The four boxes at the bottom are the "vhysical" layout - its a new word I made up to describe what is essentially a physical (machine) architecture, but is in fact a bunch of virtual machines...

For the really attentive among you, this shot is of the whiteboard in its new home on the 2nd floor of Osney One, where Renhart and I have moved following a fairly major building renovation. Clearly we were too naughty to remain with the archivists! ;-)

(ZX) spectrums & stained glass

2010-09-29T07:24:00.008+01:00

Only slightly off topic but I was pointed at http://www.zxspectrum.net/ this morning. I can't decide what is more impressive, the site itself, that Java can be used to emulate a ZX Spectrum or that there are games listed from 2010!

Slightly more on topic, I was curious to read this article about stained glass, or is it file formats. Skip over the rather flowery bit and start reading from:

"The archive problem is one of format" and concludes "Then in five hundred years time... pictures of the Auch windows will be stored and accessible in the cloud. But not one of them...will have the jaw-dropping impact of people seeing them for the very first time...and realising that humans could do wondrous things for themselves."

I read another article, this time from New View (not something I'd usually read, but an article about the "origins of computing" was recommended to me), which - in a different article - likened stained glass windows to computer screens - mostly because of the back lighting - and suggested, a little like Chris Mellor in the Register, that the cultural experience of seeing stained glass is a bit like experiencing computers for the first time.

(An other article berated digital technology, almost branding it evil, and certainly bad enough to make the author ill using it. I was reading this at DrupalCon where 90% of the people there were welded to their MacBooks and wondered how bad using a plough might make these people feel! ;-))

I guess many folks don't have the "big bang" experience of the digital world. It crept up slowly - a bit like watching the cathedral and its windows being built would probably reduce its wonder - going from a ZX81 to a Pet to a BBC B to an Amstrad 1640 to a... well, you get the picture...

But does that make it any less astonishing? It shouldn't. If anything, it is all about helping people do "wondrous things for themselves", just like zxspectrum.net...

Media players and the reader interface...

2010-09-17T16:02:00.014+01:00

This is quite a long post, so I'm going to put the final line at the top too in case you don't read that far... ;-)

Your thoughts on media players would be most welcome!

The trouble, um, I mean, beauty of digital collections is that they redefine what a "manuscript" is. This is nothing new. Once upon a time someone somewhere probably upset the apple cart when they arrived at the hallowed doors with a basket full of photographs. Now we have video, audio and images, all of which can be encoded in any number of "standard" ways. (Not to mention a zillion different binary formats for just about any purpose you can imagine from sheet music to the latest car designs, which may well require more than just document-like presentation too - 3D models for example). These new manuscripts bring challenges for preservation, of course, but they also present challenges for presentation.

To address this, I've been learning more about media players in browsers with a view to picking one for the reader interface. I'm no expert in this field, so here is my layman's consideration of what I've found out and if you want to read more then this is great!

The traditional method to render audio/video in browsers, which pre-dates their ability to handle video themselves, is to use a browser plug-in, either directly (for example VLC plugin) or (more commonly) to build on top of Flash (eg. Flowplayer) or Java (eg. Cortado). The exact mark-up required to use these players varies. Some will simply use the "embed" tag and others have JavaScript libraries to simplify their usage and allow for graceful degradation in the event that the browser does not have the correct plug-in and/or understand/run JavaScript. (This may be an issue when we deploy the interface into a reading room with machines we do not control the configuration of).

But the times, they are a-changin'. Just as old browsers knew what to do when presented with an "img" tag, most modern browsers are beginning to support HTML5's "video" and "audio" tags, allowing the browser itself to handle the playback rather than farming this out to a plug-in. (For more on HTML5 generally see this presentation - the video tag is mentioned at about 58 minutes in). As an added bonus of bringing video into the browser in this way is it has inspired folks to build media players that manipulate the Web page to add the correct mark-up, be it a video tag, an embed, or whatever to play the media. This is currently being used to generate some nice media players that'll use the browser, the Flash-plugin, or whatever is available (see OpenVideoPlayer and OSMPlayer).

So now we get to the crux of it. What should we do for the reader interface? Go old-school (and annoy Steve Jobs) and use a Flash-based player? Adopt the new ways of HTML5? Insist on an Open Source player? Buy something in?

To work out the answer I did a bit of investigating and have installed most of the players mentioned thus far in this post - Flowplayer, OSMPlayer, video-tag only, VLC and Cortado, as well as JWPlayer.

Flowplayer uses the Flash-plugin to play Flash video (and, with an additional plug-in, MP3 audio) - it does not support Ogg. It is very simple to use and very slick to look at. It is open source, released under GPL3 with an additional (and reasonable) "attribution clause" which basically means the Flowplayer logo must appear on the player unless you pay extra.

JWPlayer works much like Flowplayer (though there is also a beta HTML5 video player in the making) and seems pretty good. While the source code is available, it is not clear if this is an open source product or otherwise - the source files do not include a LICENSE.txt or any boilerplate. Probably I'm just missing something there though, and JWPlayer seems a good choice if you don't mind Flash.

OSMPlayer is also open source and has numerous options for installation including a Drupal module (untested), a PHP library and a "stand-alone" configuration. In theory it supports lots of different audio and video formats and uses several divs to create a nice browser based player. Unfortunately, following the guidelines for both PHP and stand-alone configurations, I could not get it to work on my test server.

Video-tag only works pretty well with Firefox 3.6 on Ubuntu 10.04 and is very easy to include in a Web-page. Unfortunately it isn't nearly as slick at playback as Flowplayer - there is a delay in starting the video and it is unclear what is going on.

The VLC plug-in is also open source and seems to work pretty well and should be able to handle many different formats, but it isn't nearly as refined as other players and the provided example code fails to stop the video or make it full-screen. The VLC desktop player is wonderful, but I'm not convinced by the Firefox plug-in.

Cortado is a Java-applet provided to play Ogg Theora among other things. Usage is very simple - you just add an applet tag to the page - but playback is jerky, slow and lacked sound. I do not know if my machine is to blame for this or if it is the player itself so will have to investigate further.

Were I sat on and forced to make a choice I think I'd struggle. Flowplayer is slick to use and easy to implement, but requires we convert everything to Flash video or MP3 (mind you, most media will arrive in suitable formats I imagine). JWPlayer is very similar in this regard. I'd like to adopt the video-tag as this supports a wide range of formats, including open ones, but currently the experience is not very smooth and refinements in this area provided by things like OSMPlayer are still in their early stages of development. JWPlayer's HTML5 offering is still beta for example.

I guess my feeling for now is to either go with Flowplayer (and swallow the conversions required - actually pretty easy with ffmpeg) or spend a bit of time with OpenVideoPlayer's HTML5 work and the video tag. At this stage I think we probably need both working in the interface and see where the better user experience is...

I should throw one more thing into the pot - the problem of formats. Video and audio files are complicated beasts consisting of containers and tracks and such - a bit like cassettes! The contents of these containers are encoded in a variety of ways, each requiring different software to decode and render their content. We have the same problem with documents and we solve that by converting all the text-based materials we get into PDFs (for presentation before anyone starts worrying about the preservation implications of PDF!) and use a PDF plug-in to display them.

Can we do the same with our audio/video material and if we can, what format (I'm using "format" as a general term to mean "container/encoding"!) do we use? (Victoria has already done some work along these lines, creating WAVs for storage and MP3s for presentation, from audio CDs). Is there any additional concerns given that most born-digital video/audio is likely to arrive at our doors in a compressed format? Should we uncompress it? Is such a thing even possible? Should we (and do we have the processing power to) convert all audio/video materials to open formats for both preservation and presentation purposes?

We're going to raise this final question at our next Library developer meeting and see what folks think. In theory we can delay the decision because most browsers and their plug-ins handle multiple formats, but perhaps we should have a standard delivery format much like we currently have PDF?

Oh dear. I started writing this post with the hope of finding all the answers! I have found out a lot about media players at least, which can only be a good thing, and I've also found out that that state of the art is not quite as far along as the proponents of HTML5 killing Flash would like us to believe - though there is good work going on here and this is the future. I'm also unclear just how much my experience of these things is hindered by using Ubuntu - I often wrestle with the playback of media files under Linux! :-)

Still, I think we're further along, nearer an answer and at least in a place to know where to start testing...

Your thoughts on media players would be most welcome! :-)

Wot I Lernd At DrupalCon

2010-09-03T17:26:00.006+01:00

I spent last week in the lovely city of Copenhagen immersed in all things Drupal. It was a great experience, not just because of the city (so many happy cyclists!), but because I'd not seen a large scale Open Source project up close before and it is a very different and very interesting world!

I'm going to pick out some of my highlights here as to cover it all would take days, but if you want to know more I'd encourage you to check out the conference Web site and the presentation videos on archive.org.

So, wot did I lernd?

Drupal Does RDF
OK, so I knew that already, but I didn't know that from Drupal 7 (release pending) RDF support will be part of the Drupal core, showing a fairly significant commitment in this area. Even better, there is an active Semantic Web Drupal group working on this stuff. While "linked data" remains something of an aside for us (99.9% of our materials will not make their way to the Web any time soon) the "x has relationship y with z" structure of RDF is still useful when building the BEAM interfaces - for example Item 10 is part of shelfmark MS Digital 01, etc. There is also no harm in trying to be future proof (assuming the Semantic Web is indeed the future of the Web! ;-)) for when the resources are released into the wild.

Projects like Islandora and discussions like this suggest growing utility in the use of Drupal as an aspect of an institutional repository, archives or even Library catalogues (this last one my (pxuxp) experiment with Drupal 6 and RDF).

Speaking of IRs...

Drupal Does Publishing
During his keynote, Dries Buytaert (the creator of Drupal) mentioned "distributions". Much like Linux distributions, these are custom builds of Drupal for a particular market or function. (It is testament the software's flexibility that this is possible!) Such distributions already exist and I attended a session on OpenPublish because I wondered what the interface would look like and also thought it might be handy if you wanted to build, for instance, an Open Access Journal over institutional repositories. Mix in the RDF mentioned above and you've a very attractive publishing platform indeed!

Another distro that might be of interest is OpenAtrium which bills itself as an Intranet in a Box.

Drupal Does Community
One of my motivations in attending the conference was to find out about Open Source development and communities. One of the talks was entitled "Come for the Software, Stay for the Community" and I think part of Drupal's success is its drive to create and maintain a sharing culture - the code is GPL'd for example. It was a curious thing to arrive into this community, an outsider, and feel completely on the edge of it all. That said, I met some wonderful people, spent a productive day finding my way around the code at the "sprint" and think that a little effort to contribute will go a long way. This is a good opportunity to engage with a real life Open Source community. All I need to do is work out what I have to offer!

Drupal Needs to Get Old School
There were three keynotes in total, and the middle one was by Rasmus Lerdorf of PHP fame, scaring the Web designers in the audience with a technical performance analysis of the core Drupal code. I scribbled down the names of various debugging tools, but what struck me the most was the almost bewildered look on Rasmus' face when considering that PHP had been used to build a full-scale Web platform. He even suggested at one point that parts of the Drupal core should be migrated to C rather than remain as "a PHP script". There is something very cool about C. I should dig my old books out! :-)

HTML5 is Here!
Jeremy Keith gave a wonderful keynote on HTML5, why it is like it is and what happened to xhtml 2.0. Parts were reminiscent of the RSS wars, but mostly I was impressed by the HTML 5 Design Principles which favour a working Web rather than a theoretically pure (XML-based) one. The talk is well worth a watch if you're interested in such things and I felt reassured and inspired by the practical and pragmatic approach outlined. I can't decide if I should start to implement HTML5 in our interface or not, but given that 5 is broadly compatible with the hotchpotch of HTMLs we all code in now, I suspect this migration will be gentle and as required rather than a brutal revolution.

Responsive Design
I often feel I'm a little slow at finding things out, but I don't think I was the only person in the audience to have never heard about responsive Web design, though when you know what it is, it seems the most obvious thing in the world! The problem with the Web has long been the variation in technology used to render the HTML. Different browsers react differently and things can look very different on different hardware - from large desktop monitors, through smaller screens to phones. Adherence to standards like HTML5 and CSS3 will go a long way to solving the browser problem, but what of screen size? One way would be to create a site for each screen size. Another way would be to make a single design that scales well, so things like images disappear on narrower screens, multiple columns become one, etc.

Though not without its problems, this is the essence of responsive design and CSS3 makes it all possible. Still not sure what I'm on about? dconstruct was given as a good example. Using a standards compliant browser (ie. not IE! (yet)) shrink the browser window so it is quite narrow. See what happens? This kind of design, along with the underlying technology and frameworks, will be very useful to our interface so I probably need to look more into it. Currently we're working with a screen size in mind (that of the reading room laptop) but being more flexible can only be a good thing!

There were so many more interesting things but I hope this has given you a flavour of what was a grand conference.

The Case for Digital Preservation

2010-09-01T16:09:00.004+01:00

Now, I'm pretty sure there is no need for me to make the case to the good readers of this blog, but if you're ever stuck for something to say about why your work is important - for example at parties - then the demise of the print edition of the OED seems a good candidate!

OK, so no one is about to ditch the Pocket version, or even the Shorter (I got one of those for a graduation present from my Grandma!), but even so...

The last print OED was published in 1989. I imagine, given the regular updates to the OED online, that there has been a substantial influx of words since 1989 and I guess (given how Chaucer looks now) English will undergo some significant changes in the future. Unless we (the DP community) decide to preserve the digital OED, we will condemn readers of 2489 to struggle on with an antique 1989 print copy and much will they wonder when they don't find things like "Internet"...

(Mind you, the electricity might have all run out by then so it wont really matter...)

On the flip side, and no doubt something someone at the party will point out, this is also a case for continuing to print the OED - at least a few copies, kept in safe places... ;-)

Homes for old software

2010-08-27T11:44:00.001+01:00

The more hybrid archives we work with, the more obvious it becomes that we need access to repositories of older software (or 'abandonware'). For older formats you often find that not only is the creating software obsolete, but any migration tool you can dig up is pretty out-of-date too. Recently I used oldversion.com to source older versions of CompuServe and Eudora to transform an old CompuServe account to mbox format with CS2Eudora. The oldversion site is really valuable and we could use more like it, and more in it. The trouble is, collecting and publishing proprietary 'abandonware' seems to be a bit of a grey area.

In 2003, the Internet Archive obtained some exemptions from the Digital Millennium Copyright Act (DCMA) that has allowed them to archive software, but this has to be done privately with the software being made available after copyright expiry. Not much help now, but promising for the long-term. The best thing that could happen (from an archivist's point of view) is that individuals and companies formally rescinded their interests in older software and put them in the public domain. Ideally they would put an expiry date into the initial licence before the software becomes abandonware.

I'm curious to hear about other good abandonware sites, especially ones that include 'productivity software' (our focus is here rather than gaming!). The Macintosh Garden is a good one, and Apple themselves also provide access to some older software, like ClarisWorks. What else is out there that we should know about?

Digup

2010-08-21T10:01:00.000+01:00

Might be of interest:

http://www.ubuntugeek.com/digup-a-console-tool-to-update-md5sum-or-shasum-digest-files.html

Balisage 2010 The Markup Conference

2010-08-17T13:50:00.004+01:00

Balisage 2010 The Markup Conference was
preceded by the International Symposium on XML for the Long Haul Issues in the Long-term Preservation of XML which opened with:

A brief history of markup of social science data: from punched cards to “the life cycle” approach covering the “25-year process of historical evolution leading to DDI, the Data Documentation Initiative, which unites several levels of metadata in one emerging standard.”

Sustainability of linguistic resources revisited looked at some of the difficulties facing language resources over the long-term.

Report from the field: PubMed Central, an XML-based archive of life science journal articles provided insight into the processes deployed to give public access to the full text of more than two million articles.

Portico: A case study in the use of XML for the long-term preservation of digital artifacts discussed some practices that can help assure the semantic stability of digital assets.

The Sustainability of the Scholarly Edition in a Digital World explored the need for “ tools to make XML encoding easier, to encourage collaboration, to exploit social media, and to separate transcriptions of texts from the editorial scholarship applied to
them”.

A formal approach to XML semantics: implications for archive standards examined whether “The application of Montague semantics to markup languages may make it possible to distinguish vocabularies that can last from those which will not last”.

Metadata for long term preservation of product data discussed the “valuable lessons to be learned from the library metadata and packaging standards and how they relate to product metadata”.

The day concluded with Beyond eighteen wheels: Considerations in archiving documents represented using the Extensible Markup Language (XML) which contemplated “strategies for extending the useful life of archived documents”.

Sessions in the main conference 2010 – covered topics such as :

gXML, a new approach to cultivating XML trees in Java which proposed “A single unified Java-based API, gXML, can provide a programming platform for all tree models for which a “bridge” has been developed. gXML exploits the Handle/Body design pattern and supports the XQuery Data Model (XDM)”.

Java integration of XQuery — an information unit oriented approach explored “a novel pattern of cooperation between XQuery and Java developer? A new API, XQJPLUS, makes it possible to let XQuery build “information units” collected into “information trays”.

XML pipeline processing in the browser discussed the benefits that providing XProc as a Javascript-based implementation would offer comprehensive client-side portability for XML pipelines specified in XProc.

Where XForms meets the glass: Bridging between data and interaction design explored using XForms which offers a model-view framework for XML whilst working within the conventions of existing Ajax frameworks such as Dojo as a way to bridge differing development approaches,data-centric versus starting from the user interface .

A packaging system for EXPath demonstrated how to adapt conventional ideas of packaging to work well in the EXPath environment. “EXPath provides a framework for collaborative community-based development of extensions to XPath and XPath-based technologies (including XSLT and Xquery)”.

A streaming XSLT processor Michael Kay (editor of the XSLT 2.1 specification) showed how he has been implementing streaming features in his Saxon XSLT processor;

Processing arbitrarily large XML using a persistent DOM covered moving the DOM out of memory and into persistent storage offering another processing option for large documents, by utilising, an efficient binary representation of the XML document that has been developed, with a supporting Java API.

Scripting documents with XQuery: virtual documents in TNTBase presented a virtual-document facility integrated into TNTBase, an XML database with support for versioning. The virtual documents can be edited, and changes to elements in the underlying XML repository are propagated automatically back to the database.

XQuery design patterns illustrated the benefits that might extend from the application of meta design patterns to Xquery.

Any old tapes - a true story - part 1

2010-08-09T13:49:00.004+01:00

My neighbour is a self-employed architect. He has worked digitally for at least ten years and now most of his work is done on either his old (but still perfectly serviceable) ThinkPad or a shiny new desktop PC. He works with a couple of different CAD packages along with some tax software and MS Office, all on Windows XP.

Recently, knowing what I do for a living, he asked if I could help with a problem he was having retrieving files from an external hard drive and, being easily persuaded by the promise of food and wine, I agreed to try to help (with all the usual caveats about probably not knowing anything about it all!).

We got the disk drive working quickly (this is often the way when solving other people's computer issues. Sit with them and they'll solve it themselves!) and so he asked me about his backups too - which should have been happening regularly to another external drive, but were not. I checked out the drive and found an old directory with a very uninformative name that contained some data files and a few manifests that didn't make much sense. I've forgotten the name already, but he told me this was the name of the backup software. Searching, this software was not on the PC. The new PC had been recently built on the basis of the old one by an outsourced IT support. They'd done a good job restoring the software, etc. but this one backup program (a commercial one) was missing.

The consequences where two-fold:

1) No backup was running
2) the data files (about 1.4GB worth) and manifest were, without the software, entirely unreadable.

My neighbour thought perhaps the backup software was about so he'd ask the IT support to install and configure it. I fired up MS Windows Backup (the first time I've ever used it - it seems OK) and ran a one off backup of his work, just to be on the safe side and suggested he ask his support about that (one thing you must never do is undo or override the work of the real support person!) too - it required a password to add it to Windows scheduler.

After it completed, he astutely asked where the files had gone, and so I showed him, on the external drive and was dismayed to find that Windows Backup had also dumped all the files into a 1.4GB (proprietary?) container. I wondered if we'd ever have to extract files from Windows Backup files and made a mental note to keep a copy of the software (bundled with XP) in the cupboard just in case! Worse, it was then impossible to reassure him that the files were there without a crash course in Windows Restore. Still, I remember MS Backup and Restore being a pain way back to MS-DOS! :-)

As we finished our wine and talked about these things, he seemed to suddenly remember my job and jumped up, rummaged in a cupboard. He pulled out an old tape cartridge:

Once his main backup media, but, like the files on the external drive, no longer usable. This time both the hardware and the software were long gone. He didn't seem worried - the files has probably been migrated off his old machine to the new one at some point - but still he wondered what was on it and said "I don't suppose it is readable now is it?". He hadn't meant it as a challenge, but I couldn't resist! I convinced him to let me take the tape with me and try to recover his data - all in the name of digital archaeology, of course!

My next post will be my first adventures in the land of the Travans...

Drupal

2010-07-16T11:27:00.003+01:00

I'm very excited! I just looked at a Web site that lists all the "high profile" folks using Drupal for the Web sites. I'm also nervous. The excitement and the nerves are linked. This is because I told Susan a few weeks ago that I would commit to Drupal as the front-end for our archival materials. So, there is a lot to live up to and I'm also stuck with a decision I made, so it is all my fault! No pressure! :-)

Why Drupal?

Well, when I was building the second incarnation of the archive interface (the first was a prototype put together by Susan), it started out as a bunch of Web pages and a Solr-based search engine. The back-end data was created using a combination of source data and metadata gleaned from the EAD catalogue, the output of FTK and a spreadsheet that was the result of some appraisal work by the archivist, all munged together by some Java code that did the transformations, created the thumbnails, etc.

As time moved on it became apparent that additional features would be nice to build into the interface. At least one of the Project Advisory Board members suggested it would be nice to see a more Web 2.0-like features and I've long thought that having reader-generated tags and (perhaps) comments attached to the manuscripts might be a nice idea. Other features also arose, and soon I realised that I'd have to either build database-driven site to make all this happen (which I suspect would've been rather ropey) or, far more sensibly, use one that already existed.

By wonderful coincidence (though the kind of thing that often happens) I saw some emails on the Fedora lists about Islandora. Secretly harbouring a desire to visit Prince Edward Island ;-), I took a closer look and it was there that I chanced on Drupal and it seemed to fit the bill quite nicely, offering comments, tagging, types of content, and user management. Further, it is extensible, has a bewildering, if full of promise API, and will hopefully mean I can build a "publication pathway" that interfaces with the preservation store (indirectly) and can be managed by the archivists in a nice Web-friendly way.

Does the excitement and the nerves start to make more sense now?

It is still early days, but I have re-factored the Java code to output content (fixing a major memory leak in the process!) suitable for import into Drupal and have developed a module that imports that content, including the structure of the collection as Collection - Shelfmarks - Items. It aint much to look at just yet, but it is getting there.

As I have further adventures in Drupal-land I'll keep you updated!

Have a lovely weekend!

Graduate Trainee Presentations - Archiving Digital Audio

2010-07-09T11:40:00.007+01:00

Since I am the graduate trainee for the futureArch project I also participated in Oxford's Graduate Library Trainee Scheme. During our year at Oxford all the trainees have to undertake a project. There aren't many restrictions on what this project can and can't be about; the only prerequisites are that it is additional to our day-to-day duties and is useful to each trainee's library. After talking it over with Susan I decided to research digital audio files and produce a guide to archiving digital audio.

On 7th July the trainees held a Project Showcase where each trainee gave a five minute presentation on their project. For anyone interested, most of the presentations (including mine) are now on slideshare. Five minutes is really not very long for a presentation and so I had to severely condense mine, although I've expanded my presentaion notes to include more detail - these are also on slideshare.

Our first local 'dead' hard disk acquisition

2010-07-02T17:03:00.000+01:00

We've imaged lots of removable media over the past year (~ 400, according to Victoria's stats), and I've also done a fair amount of forensic imaging of material on-site with donors (live acquisition) . One aspect of our 'forensic' armoury that has not been subject to so much testing is the imaging of whole hard disks at BEAM. So-called 'dead' acquisitions.

In the past few months two new accessions have presented us with an additional four hard disks. This is excellent news, as I have finally had the chance to use our forensic computer's Ultrabay (write-blocking device) to image a real 'collection hard disk'. Everything went smoothly. So far so good.

Vintage Computing Festival

2010-06-21T18:18:00.001+01:00

Yesterday I took a trip to the first official Vintage Computing Festival in Britain. I was a little surprised to hear that it was the first, but I imagine that there are plenty of 'unofficial' gatherings too. This event was held by the National Museum of Computing in Bletchley Park, which warrants a visit in its own right.

For the weekend's festival, Bletchley was transformed into vintage computing heaven: a couple of marquees and the ground floor of the house were packed with computers of all makes and models, each one up and running and ready for some hands-on time. The vast majority were being used for gaming - chuckie egg was all over the place - but I did spot the odd word-processing application here and there.

I thought I'd post some pictures from two exhibits that really caught my eye.

First was the BBC playing the 1980s BBC Domesday project from laserdisc. Look right and you'll see some video footage that we found having searched for 'falklands'. I've read quite a bit about the BBC Domesday laserdiscs over the years (after the CAMiLEON project they've become digital preservation folklore), but seeing the content at stake, and interacting with it on a contemporary platform is something quite special. I also suffer from BBC Micro nostalgia (though this is a Master).

This other I'm including partly for nostalgic reasons (I loved my spectrums, and so did my sister and my grandfather :-) ), and partly because it amused me. Twittering from a spectrum! Whatever next?!

Par2ty

2010-06-21T09:44:00.004+01:00

This is probably an old and battered hat for you good folks (seeing as the Web site's last "announcement" was in 2004!), but most days I still feel pretty new to this whole digital archiving business - not just with the "archive" bit, but also the "digital preservation", um, bit so it was news to me... ;-)

Perusing the latest Linux Format at the weekend, I chanced on an article by Ben Martin (I couldn't find a Web site for him...) about parchive and specifically par2cmdline.

Par-what? I hear you ask? (Or perhaps "oh yeah, that old thing" ;-))

Par2 files are what the article calls "error correcting files". A bit like checksums, only once created they can be used to repair the original file in the event of bit/byte level damage.

Curious.

So I duly installed par2 - did I mention how wonderful Linux (Ubuntu in this case) is? - the install was simple:

sudo apt-get install par2

Then tried it out on a 300MB Mac disk image - the new Doctor Who game from the BBC - and guess what? It works! Do some damage to the file with dd, run the verify again and it says "the file is damaged, but I can fix it" in a reassuring HAL-like way (that could be my imagination, it didn't really talk - and if it did, probably best not to trust it to fix the file right...)

The par2 files totalled around 9MB at "5% redundancy" - not quite sure what that means - which isn't much of an overhead for a some extra data security... I think, though I've not tried, that it is integrated into KDE4 too for a little bit of personal file protection.

The interesting thing about par2 is that it comes from an age when bandwidth was limited. If you downloaded a large file and it was corrupt, rather than have to download it again, you simply downloaded the (much smaller) par2 file that had the power to fix your download.

This got me thinking. Is there then any scope for archives to share par2 files with each other? (Do they already?) We cannot exchange confidential data but perhaps we could share the par2 files, a little like a pseudo-mini-LOCKSS?

All that said, I'm not quite sure we will use parchive here, though it'd be pretty easy to create the par2 files on ingest. In theory our use of ZFS, RAID, etc. should be covering this level of data security for us, but I guess it remains an interesting question - would anything be gained by keeping par2 data alongside our disk images? And, after Dundee, would smaller archives be able to get some of the protection offered by things like ZFS, but in a smaller, lighter way?

Oh, and Happy Summer Solstice!

OSS projects for accessing data held in .pst format

2010-06-10T08:34:00.000+01:00

Thanks to Neil Jefferies for a link to this article in The Register, which tells us that MS has begun two open source projects that will make it possible for developers to create tools to 'browse, read and extract emails, calendar, contacts and events information' which live in MS Outlook's .pst file format. These tools are the PST Data Structure View Tool and the PST File Format SDK, and both are to be Apache-licensed.

Developing & Implementing Tools for Managing Hybrid Archives

2010-06-02T11:10:00.003+01:00

As previously blogged, we were invited to talk at the University of Dundee's Centre for Archive and Information Studies seminar. I understand that the presentations along with a set of notes will be made available shortly, but in the mean time I thought I'd let you know my slides and notes are available on SlideShare and also my rather hastily thrown together home page! :-)

#4n6 event, and CLIR report on digital forensics as applied to cultural materials

2010-05-24T09:11:00.001+01:00

For a couple of days week before last I was at a meeting which went by the name of Computer Forensics and Born-Digital Content in Cultural Heritage Collections. The meeting was in support of a report bearing the same name (for now at least) which is currently being written by Matt Kirschenbaum, Richard Ovenden and Gabby Redwine. The final day of the workshop was dedicated to reviewing the first draft of the report, and the finalised version should be published by CLIR later this year.

We've been adapting digital forensics tools and techniques within BEAM (Bodleian Electronic Archives and Manuscripts) for a few years now, and this meeting was a useful event to talk about how we do this, and some of the issues (process, technical and ethical) it raises.

It was a good meeting, and I very much enjoyed hearing from other digital archivists and *real* forensics practitioners (they have rather different objectives to ours, but their tools are still useful!). Another highlight for me was Stephen Ennis' framing thoughts, presented in the first session. Ennis grounded the discussion, with three key - and very practical - points that should be important to any archivist:

1) What is the hard-cash value of born-digital archives?
Ennis contends that monetary value has been a preservation agent for literary manuscripts. If disks and digital data are of no value, their survival rate is likely to be poor. He cited the example of John Updike's archive (at Harvard), which contained software disks but no related data disks. It's worrying that dealers don't/won't appraise born-digital material, but this will surely change. Another issue is that we need dealers to be able to appraise digital archives without altering what they are appraising. Will they have to adopt digital forensic techniques too?

2) Are the steps that seem justified for celebrity authors justified for others?
This question is very important and equally applicable to 'papers', of course. In the digital domain, the obvious 'celebrity' example is the work Emory's MARBL have done to make one of Salman Rushdie's hard disks accessible to scholars through an emulator and a searchable database. We certainly won't be processing every digital archive submission at this level, and I suspect MARBL won't either. Where it's justified, I think it's a very good thing.

3) What is the researcher's object of study? Are we promoting new and different forms of enquiry?
This question, perhaps, gets closer to exploring our simultaneous excitement and concern when we consider the potential of combining scholarly enquiry and digital forensic tools in relation to born-digital archives. There's a good deal we need to learn about scholars' requirements and I'm looking forward to the day that we have more case studies so we can move this discussion beyond conjecture!

If you're interested in finding out more in advance of the report, you'll probably find that some of the slides will be published in due course at the event's website. You can also take a look at some photos and tweets.

I may extend this post with some of the more interesting tidbits if I find a moment.

Passwords you never created and never knew

2010-04-29T11:25:00.000+01:00

Every so often the more technically-savy in a family are called on to help set-up a new computer when an old one begins to fail. Experience tells you that there are a number of things that you'll need to do as part of this process, but there's generally one or more things you forget to check for and have to fix later. It's seldom a single-session process.

Last weekend the main problem was an unknown password for an email account. In a scenario which can't be that uncommon, an email account had been established by a friend and the password for it remembered by the email client but no human being. Luckily we were able to salvage the password using one of these tools and restore access to the email via a new client on the new computer.

It seems all to possible that we will encounter this scenario with a depositor at some stage, so it's handy to have an easy fix for it. On the other hand, it's a little worrying how easy a fix it is...

So long floppy, hello retro cool!

2010-04-28T08:51:00.004+01:00

If you've been following Victoria's rather brilliant posts about media, you'll be sad (or perhaps glad) to hear that the demise of the floppy draws ever closer now that Sony are discontinuing floppy disks. I suspect everyone has a story to tell that involves a floppy disk, the fear, the shear agony of that lost essay, the relief at the kindness of the geek who saved the file. These stories will become a thing of the past.

To balance this bad news, I also wanted to flag up the Vintage Computer Festival up the road at Bletchly Park. Lets hope they raise a glass to deprecated storage devices and their tales!

Do you know the way to Dundee?

2010-04-23T11:38:00.004+01:00

The Centre for Archive and Information Studies at the University of Dundee is putting on what looks to be a very interesting seminar entitled "Practical Approaches to Electronic Records: the Academy and Beyond". And I'm not just saying it'll be very interesting because we're talking at it either. Just take a look at the packed programme and I'm sure you'll agree.

I'll be covering the workflow we're adopting here at futureArch and hopefully demo part of it, as well as discussing our digital asset management system, the foundation for our archive and how those ideas may scale to smaller systems.

Hope to see you there and if not I'm sure we'll be reporting back right here so stay tuned!

(Also a bit (um, I mean big) thank you to Jennifer Johnstone for helping me find my way to Dundee! :-))

Using a D-Link DGE-530T Gigabit Network adapter in ESX 4.

2010-04-14T17:05:00.009+01:00

For our developers ESX testbed/playground I wanted to install two D-Link DGE-530T Gigabit PCI Desktop network adapters unfortunately they do not appear to be on the ESX supported list. These are the steps I took to get them to be recognised by ESX:

1. Acquire the skge.o driver which supports the Marvell Yukon 88E0001 chipset

The discussion Using a Marvell LAN card with ESXi 4 contains a link to a tarball sky2-and-skge-for-esxi4-0.02.tar.gz containing both the sky2 and skge driver

2. login to ESX 4.0 as root and copy the skge.o driver to /usr/lib/vmware/vmknod

2.1 download sky2-and-skge-for-esxi4-0.02.tar.gz

2.2 tar xvzf ../sky2-and-skge-for-esxi4-0.02.tar.gz

2.3 cp vmtest/usr/lib/vmware/vmkmod/skge.o /usr/lib/vmware/vmkmod

3. run 'lspci' and identify the NICs location (the xx:xx.x number in front of the description)

03:00.0 Ethernet controller: D-Link System Inc Unknown device 4b01 (rev 11)

4. run 'lspci -n' and determine the vendor and device IDs (for D-Link it should be 1186:xxxx)

lspci -n
00:00.0 0600: 8086:29b0 (rev 02)
(snipped)
03:00.0 0200: 1186:4b01 (rev 11)
03:02.0 0200: 8086:1026 (rev 04)

5. create the vmware pciid file '/etc/vmware/pciid/skge.xml' here's a listing of the mine

cat /etc/vmware/pciid/skge.xml


<?xml version='1.0' encoding='iso-8859-1'?>
<pcitable>
  <vendor id="1186">
    <short>D-Link System Inc</short>
    <name>D-Link System Inc</name>
    <device id="4b01">
      <vmware label="nic">
        <driver>skge</driver>
      </vmware>
      <name>DGE-530T Ethernet NIC</name>
      <table file="pcitable" module="ignore" />
      <table file="pcitable.Linux" module="skge">
        <desc>D-Link System|DGE-530T Ethernet NIC</desc>
      </table>
    </device>
  </vendor>
</pcitable>

6. create file /etc/vmware/init/manifests/vmware-skge.mf which contains a single line as shown
cat /etc/vmware/init/manifests/vmware-skge.mf

copy /usr/lib/vmware/vmkmod/skge.o

7. reboot the server and checking the /var/log/vmware/esxcfg-boot.log should confirm:

That the esxcfg boot process has loaded the skge.xml metafile , constructed the new vmware-devices.map file and included the skge.o driver in the initramfs image.

8. running 'lspci' after adding a second DGE-530T card now shows


03:00.0 Ethernet controller: D-Link System Inc DGE-530T Ethernet NIC (rev 11)
03:02.0 Ethernet controller: D-Link System Inc DGE-530T Ethernet NIC (rev 11)

Of course the normal caveats and disclaimers apply as in not supported by VMware etc.

Want to be our new graduate trainee?

2010-04-12T10:46:00.005+01:00

We are now advertising for our second graduate traineeship post within the project. This one-year post is intended to provide pre-course experience to a graduate prior to undergoing professional training on one of the recognised archive courses. Being based within futureArch, it particularly suits an applicant wishing to develop an understanding of how the shift to digital communications impacts the work of archivists.

The postholder will support the curatorial and technical work of the futureArch project, while sampling a variety of more traditional archival work, including providing services to researchers in the Special Collections Reading Room. The postholder will also participate in activities organised through the OWL Graduate Trainee Scheme.

Further details and application forms are available here. The closing date for applications is 10 May 2010 and we expect to interview on 1 June. For a flavour of some of the work Victoria has done during her time as a trainee take a look at some of her posts to this blog and to the Bodleian graduate trainee scheme blog.

Microsoft Works library

2010-03-31T14:07:00.000+01:00

Quick future aide-mémoire for accessing Works files. Available at the libwps site. Implemented in some openoffice variants.

Disk imaging for older floppies

2010-03-30T18:42:00.001+01:00

Thanks to Michael Olson for the link to Kryoflux , which is currently being developed by the Software Preservation Society (an organisation established to preserve disk-based computer games). Stanford are also using the Catweasel floppy disk controller; see Stanford's post on Catweasel and the Catweasel site itself. These could be handy to have around when we receive more in the way of unusual floppy formats.

Media Recognition Guide - Flash Media

2010-03-22T15:59:00.002Z

Flash memory is the alternative to byte-programmable memory, which is used by hard, floppy and Zip disks. It is much less expensive, meaning large capacity devices are economically viable and has faster access times and much better shock resistance and durability. Altogether this makes it particularly suitable for use as a portable storage device. Flash memory does have a finite number of write-erase cycles, but manufacturers can guarantee at least 100,000 cycles, which is a much larger number than with byte-programmable memory.

USB Flash Drive

Type:	Flash memory data storage device with USB interface
Introduced:	2000, though the company that invented the device is a legal issue.
Active:	Yes
Cessation:	-
Capacity:	First drive had a capacity of 8 MB but the latest versions can have capacities as large as 256 GB
Compatibility:	Widely supported by modern operating systems including Windows, Mac OS, Linux and Unix systems.
Users:	Broad. Has replaced 3.5” floppy disks as the preferred device for individuals and small organisations for personal data storage, transfer and backup.
File Systems:	FAT, NTFS, HFS+, ext2, ext3
Common manufacturers:	Many manufacturers and brands including Sandisk, Integral, HP, Kingston Technology and Sony

Recognition

USB flash drives can come in a range of shapes and sizes, but as a general rule they measure somewhere in the region of 70mm x 20mm x 10mm and all have a male USB connector at one end. Capacity also varies widely, though the majority of manufacturers specify this either by printing the information on the casing or etching it onto the connector.

Using the word ‘drive’ is misleading as nothing moves mechanically in a USB flash drive. However, they are read and written to by computers in the same way they read and write to disk drives, therefore they are referred to by operating systems as ‘drives’.

The only visible component is the male USB connector, often with a protective cap. Inside the plastic casing is a USB mass storage controller, a NAND flash memory chip and a crystal oscillator to control data output. Some drives also include jumpers and LEDS and a few also have a write-protect switch.

High Level Formatting

USB drives use many of the same file systems as hard disk drives, though it is rare to find a drive that contains a version that pre-dates its creation. Therefore, USB drives most likely contain FAT32, rather than FAT16 or FAT12. FAT32 is the file system most commonly found on USB drives due to its broad compatibility with all major operating systems. NTFS can be used but it is not as reliable when used on operating systems other than Windows. If a drives is intended for a specific operating system, you can expect to find either HFS+ (for Macs) or ext2 or ext3 (for Linux).

Formatting a disk is done in the same way as formatting a floppy disk. If being done on a Windows operating system for example the only difference is you will right click on the USB drive icon, rather than the floppy drive.

FireWire Flash Drive

Type:	Flash memory data storage device with firewire interface
Introduced:	2004
Active:	Yes
Cessation:	-
Capacity:	Either 4, 8 or 16 GB
Compatibility:	Compatible with any computer with a firewire connector
Users:	Limited. Never achieved the same popularity as USB flash drives. They come in smaller sizes and have slower memory
File Systems:	FAT, NTFS, HFS+, ext2, ext3
Common manufacturers:	Kanguru

Recognition

FireWire flash drives look similar and are similar in construction to USB drives, the one difference being that they use a FireWire connector, rather than a USB one. Due to this they have different data transfer rates and capacities than USB drives. Depending on which version of FireWire the drive has been manufactured with it has a transfer rate of either 49.13, 98.25 or 393 MB/s. With the exception of 40.13 MB/s, these rates exceed that of the latest USB version, however they have a much smaller capacity. Furthermore they are heavier and more expensive and fewer computers have the appropriate FireWire connectors compared to those with USB ports. Thus, FireWire flash drives have never dominated the market and are fairly rare.

High Level Formatting

FireWire drives only differ from USB drives in their type of connector, therefore they will contain the same file systems and can be formatted in the same way.

Media Recognition Guide - Iomega Zip Disks

2010-03-22T14:30:00.002Z

Type:	Removable disk storage
Introduced:	1994
Active:	Yes, but used by minority
Cessation:	-
Capacity:	Either 100, 250 or 750MB
Compatibility:	Zip drive needs to be of a matching or higher capacity than the Zip disk. Supports Windows OS, IBM OS/2, Mac OS 7.6 to 9.2, MAC OS X and some Linux OS.
Users:	Small businesses and personal users to backup files
File Systems:	NTFS, FAT, ext2, HFS/+, ADFS
Common manufacturers:	Iomega

Recognition

The Zip disk was introduced by Iomega in 1994 as a medium capacity removable storage device to rival 3.5” floppy disks. There are three versions with capacities of 100, 250 and 750 MB, which is considerably more than a 3.5” floppy disk, and Zip disks have a quicker data transfer rate: 1 MB/s compared with a HD 3.5” floppy disk’s rate of 15.6 KB/s. However, Zip disks never reached the same popularity as floppy disks and could not compete with other forms of removable storage, such as CDs which offer much larger capacities. Therefore sales declined and use is limited at the time of writing [2010], though zip drives and disks are still available for purchase from online retailers.

Zip disks are physically similar to floppy disks, except they are larger and not quite rectangular. Their dimensions are 97 x 98 x 6mm compared to 3.5” floppy disk dimensions of 90 x 94 x 3mm.

Zip drives are more compact than floppy drives. Dimensions vary, but are around 170 x 110 x 25mm, although drives with a SCSI interface are larger, measuring 193 x 139 x 44mm. Drives are external and attached to a computer via an interface, either PATA, SCSI, USB or FireWire. Very early drives had an IDE interface (the forerunner to PATA), but these are not common. Not every interface can be found on each type of Zip drive. Below is a table setting out which interface is compatible with each drive:

Drive	Interface
Drive	PATA	SCSI	USB	FireWire
Zip 100	Yes	Yes	Yes	No
Zip 250	Yes	Yes	Yes	Yes
Zip 750	Yes	No	Yes	Yes

High Level Formatting

Zip disks use the same file systems as floppy disks, with the most common being FAT for use with Windows, HFS or HFS+ for use with Mac OS and ext2 for use with Linux. Many disks come preformatted, but can still be reformatted by the user to suit their operating system.

Formatting with Windows

This is done in the same way as formatting floppy disks: with the disk inserted in the drive open ‘My Computer’, right click over the Zip disk drive icon and select ‘Format’. There are two options; either ‘Short Format’ or ‘Long Format’ and you can change the file system used with either option by selecting either Mac or PC. Click ‘Start’ and the disk will be formatted.

Formatting with Mac OS

Insert the disk into the zip drive. Open the IomegaWare folder then open the ‘Tools’ folder and double click on the Tools icon, which will open up the window. From here click on the appropriate icon for the disk you wish to format. There are two options, either ‘Short Erase’ or ‘Long Erase’. Long Erase should be used for disks containing errors. Select ‘Erase’ to begin formatting the disk. All content will be erased and the disk will be formatted to a file system appropriate for Mac OS (HFS/+).

Formatting with Linux

Formatting a zip disk using Linux is done in the same way as formatting floppy disks, the difference being you type man zip instead of man floppy. For more details see: http://linux.die.net/man/8/floppy

Media Recognition - Hard Disk Drive part 3

2010-03-15T14:24:00.003Z

Serial Attached SCSI (SAS) Interface

Type:	Magnetic storage media
Introduced:	2004
Active:	Yes [2010]
Cessation:	-
Capacity:	Varies, but majority do not exceed 300GB
Compatibility:	Compatible with all operating systems, though drives with a capacity of 137GB or more are only compatible with Windows 98 onwards and Mac OS 10.2 onwards. Not found on 8” or 5.25” drives.
Users:	Servers and high-end computers
File Systems:	FAT, NTFS, HFS/+, ext
Common manufacturers:	Western Digital, Seagate, Toshiba, Hitachi, Samsung

Recognition

SAS was born out of SCSI developments and entered the market in 2004. One feature making it preferable to SCSI is its higher transfer rate. Its fast speeds and high level performance make it suitable for high-end personal computer hard drives and servers. The first version was slower than the latest version of SCSI having a data transfer rate of 300 MB/s. However, in 2009 this rate increased to 600 MB/s and it is expected to reach 1200 MB/s by 2012. SAS uses point-to-point topology to connect the interface and can support multiple devices (up to 200), making it popular with servers. For the same reasons SAS hard disk drives are relatively expensive therefore they are not as common on standard personal computers as the more general purpose SATA interface.

The SAS connector is a 29-position connector. It is much smaller than its predecessor, SCSI, so as to be used on 2.5” drives. SAS connectors look similar to SATA connectors. The difference is that with the SATA interface the data and power connectors lie next to each other, but are separate, whereas with SAS the two form one connector, with a piece of plastic used to keep them distinct. This similarity is deliberate so that SAS connectors are compatible with SATA drives, but not the other way around.

External Hard Disk Drives

Early Apple Macintosh computers used external SCSI hard disk drives, despite internal hard disk drives being the standard for other PCs. More recently external hard drives are primarily used as additional storage devices.

Although the early Apple external drives were only compatible with Mac OS, later drives have been manufactured to support all modern operating systems. However, they cannot support any Windows OS preceding Windows 2000, Mac OS before version 8.5.1 or the Linux OS with a kernel earlier than version 2.4 unless updates are installed.

The hard disk in an external hard disk drive is no different to that in an internal drive, though an external drive is encased in plastic and the only visible part is the connector. This can either be a SCSI, eSATA, USB or FireWire connector.

FireWire (IEEE 1394): First released in 1995 this was originally developed as a replacement for the SCSI connector and many computers since 2003 have a built-in FireWire port, particularly Apple machines. FireWire has a higher transfer rate than USB and the latest version, FireWire 3200 has a rate of 393 MB/s, which also exceeds that of eSATA, although this rate varies with Windows OS. However it is more expensive than USB, hence it has never superseded USB’s popularity. It is compatible with Windows OS from Windows XP onwards, though issues with Vista have been raised. It is also compatible with Linux OS and Mac OS from version 8.6 onwards. The FireWire cables carry power and data on a single cable, therefore only one is needed for a device.

There have been several versions of FireWire each using different connectors. Here is a brief table setting this out:

Version	Cable Used	Date Introduced
FireWire 400 (IEEE 1394)	6-circuit	1995
FireWire 400 (IEEE 1394a)	4-circuit	2000
FireWire 800 (IEEE 1394b)	9-circuit	2002
FireWire S3200	9-circuit	2007

It is most common to find 6-cicuit connectors on desktop computers and 4-cicuit connectors on laptops. However, in 2000 amendments were made and the 4-cicuit connector was standardised resulting in more of these connectors being found on desktop computers.

FireWire 800 and 3200 are backwards compatible with these ports, but are manufactured with a 9-cicuit connector. Adaptors are available so that 9-circuit cables can be used with 4- and 6-circuit connectors on computers.

USB (Universal Serial Bus): USB was introduced in 1996 and has since become the dominant means to connect computer peripherals to the host controller. The original USB 1.0 has a transfer rate of 12 Mbits/s, which was increased to 60 MB/s (480 Mbits/s) by USB 2.0. This was released in 2000 and standardised in 2001. Like FireWire, USB connectors carry power as well as data, therefore do not require additional power cables.

There are several different USB cables available for different uses. The most common type found on computers is the A plug and port. A second type is similar in size and is usually found on extension cables. Mini plugs are also available for use with small devices such as cameras. The other sort is squarer and about half the width of the A plug. This is known as a B plug and is used on devices that use removable cables such as printers. Having two types of connector (A and B) prevents users accidentally creating an electrical loop.

eSATA: This is SATA’s own external connector introduced in 2004 with a transfer rate of 131 MB/s. Despite having a much larger data transfer rate, few computers have eSATA ports, favouring instead USB and FireWire.

Media Recognition - Hard Disk Drives part 2

2010-03-15T12:32:00.015Z

AT Attachment (ATA) Interface

Type:	Magnetic disk storage
Introduced:	1986
Active:	Only on existing computers
Cessation:	Almost completely superseded by SATA by 2007
Capacity:	750 GB
Compatibility:	Compatible with all operating systems, though drives with a capacity of 137GB or more are only compatible with Windows 98 onwards and Mac OS 10.2 onwards.
Users:	Desktop and laptop computers, particularly low-end ones. The primary interface used by IBM and adopted by Apple in 1994.
File Systems:	FAT, NTFS, HFS/+, CP/M, ext, MFS
Common manufacturers:	Present: Western Digital, Seagate, Toshiba, Hitachi, Samsung Past: IMB, Apple, Sony, Hewell-Packard, Maxtor

Recognition

ATA, designed in 1986 was an evolved form of Western Digital’s original Integrated Drive Electronics (IDE). It was used for desktop and laptop computers. After the development of SATA in 2003 ATA became retroactively known as Parallel ATA (PATA).

Here is an example of an ATA interface. It is identified as ATA because of the rectangular 40-pin connector socket used. The second image shows the male connector.

The first ATA hard disk drive was used in the Compaq Deskpro 386. It had a transfer rate of 16 MB/s and a capacity of 128 GB. Later versions had larger transfer rates, though none exceeded 80 MB/s until 2005. Before the advent of SATA, ATA hard disk drives were popular with IBM PCs due to their low cost. ATA drives are also found on post-1994 low-end Apple Microsystems, the first being the Apple Quadra 630. In 1997 Apple also adopted ATA for their high-end machines starting with the Apple Powerbook G3. The ATA connector has a relatively short cable so ATA hard disk drives are commonly inbuilt.

Serial ATA (SATA) Interface

Type:	Magnetic disk storage
Introduced:	2003
Active:	Yes [2010]
Cessation:	-
Capacity:	Up to 2TB [2010]
Compatibility:	Compatible with at least one version of all major operating systems: supports versions of Windows after XP. Also, drives with a capacity of 137GB or more are only compatible with Windows 98 onwards and Mac OS 10.2 onwards. Unlikely to be found on 8” or 5.25” drives.
Users:	Servers, desktop and laptop computers. Adopted by IBM and Apple to replace ATA.
File Systems:	FAT, NTFS, HFS/+, ext
Common manufacturers:	Present [2010]: Western Digital, Seagate, Toshiba, Hitachi, Samsung Past: Maxtor

Recognition

SATA was introduced in 2003 as a faster version of ATA and is used with desktops, laptops and servers. It was intended to replace ATA, and had almost completely done so by 2007. SATA uses AHCI (Advanced Host Controller Interface) and if a computer does not have it SATA runs in ATA emulation mode. This applies to Windows OS up to Windows XP, though Windows Vista and later versions support AHCI. Linux with a kernel version 2.16.19 onwards will support AHCI as will Solaris operating system.

The image below shows an example of a SATA interfaced hard disk. The manufacturer’s details are clearly displayed. This type of interface uses a different connector, which is more clearly visible in the second image.

The SATA interface uses a 7-pin connector and socket, as seen in the above image. SATA differs from its predecessors in that it has the option of two power sockets. There is the standard Molex power socket on the right, but SATA drives also have their own power socket. This is a 15-pin connector situated next to the data socket. The image below shows the reverse side of the hard disk drive. The data socket is on the left and the power connector is on the right.

The images below show the two male connectors, with the data connector first then the power connector.

Initially SATA had a transfer rate of 131 MB/s and the SATA revision 2.0 increased this to 300 MB/s. SATA is still currently being developed and the latest version released in 2009 has a transfer rate of 600 MB/s. External SATA hard disk drives often include their own eSATA connector. More information about this will be in the next post.

Small Computer Scientific Interface (SCSI) Interface

Type:	Magnetic storage media
Introduced:	1981 (under the name SASI)
Active:	Only on existing computers
Cessation:	Early twenty-first century
Capacity:	2 TB
Compatibility:	Compatible with all operating systems, though drives with a capacity of 137GB or more are only compatible with Windows 98 onwards and Mac OS 10.2 onwards.
Users:	Servers and primarily high-end computers. Used on all microcomputers until ATA established. Used by Apple for low-end machines until 1994 and high-end machines until 1999.
File Systems:	FAT, NTFS, HFS/+, CP/M, ext, MFS
Common manufacturers:	Present: Western Digital, Seagate, Toshiba, Hitachi, Samsung Past: IMB, Apple, Sony, Hewell-Packard, Maxtor

Recognition

SCSI was introduced in 1981 under the name SASI (Shugart Associates System Interface). It was used with various operating systems including Microsoft Windows, Mac OS, Unix and Linux. However, as ATA gained popularity SCSI became associated more with high-end machines and servers. Even then SCSI began to lose popularity in the 1990s and was abandoned by Apple Macintosh in 1999. In the last decade provision for SCSI on motherboards has been largely discontinued with manufacturers preferring SATA and SAS.

The SCSI interface uses either a 68 or 80-pin connector (68-pin is more common) and a Molex power connector. The images below show the female and male 68-pin socket and connector.

The first SCSI interface had a transfer rate of 3.5 MB/s which soon increased to 5 MB/S. The rate increased in 1994, 1997 and again in 2003 to reach 320 MB/s.