Sprucing up the TikaFileIdentifier

As it's International Archives Day tomorrow, I thought it would be nice to quickly share some news of a project we are working on, which should help us (and others!) to carry out digital preservation work a little bit more efficiently.

Following the SPRUCE mashup I attended in April, we are very pleased to be one of the organizations granted a SPRUCE Project funding award, which will allow us to 'spruce' up the TikaFileIdentifier tool. (Paul has written more about these funding awards on the OPF site.)

TikaFileIdentifier is the tool which was developed at the mashup to address a problem several of us were having extracting metadata from batches of files, in our case within ISO images. Due to the nature of the mashup event the tool is still a bit rough around the edges, and this funding will allow us to improve on it. We aim to create a user interface and a simpler install process, and carry out performance improvements. Plus, if resources allow, we hope to scope some further functionality improvements.

This is really great news, as with the improvements that this funding allows us to make, the TikaFileIdentifier will provide us with better metadata for our digital files more efficiently than our current system of manually checking each file in a disk image. Hopefully the simpler user interface and other improvements means that other repositories will want to make use of it as well; I certainly think it will be very useful!

Balisage: The Markup Conference 2009

Just back from the 2009 Balisage Symposium on Processing XML Efficiently and Balisage XML Conference : the presentations and updated papers will be published over the next few weeks. They were all very informative and to a high standard. Topics of particular interest included:

Performance of XML-based applications

One of the components of their XML-based publishing system discussed was the Schema, Addressing, and Storage System (SASS) - a data store that provides a unified view of metadata and content for publications they host which relies on the concept of storing metadata and related resources in a file-system based set of XML files along with the use of XQuery to feed these resources to an XSLT-based display layer.

XML in the browser

Explored how new XML vocabularies could be integrated into the browser. Thereby providing a new way forward for XML in the Browser.

Towards markup support for full GODDAGs and beyond: the EARMARK approach

Examined Overlapping markup and the utilisation of the standoff approach to markup to address the issues created by the purely hierarchical approach.

TNTBase: Versioned Storage for XML

Presented an open-source versioned XML database created by the integration of Berkeley DB XML into the Subversion Server.

Agile Business Objects Management Application for Electronic Records Archive Transfer Process

How XForms and Genericode are assisting the National Archives and Records Administration(NARA) in their goal to provide archivists with a modernised system with automatic workflow for the digital archive business process

“A practical introduction to EXPath: Collaboratively Defining Open Standards for Portable XPath Extensions.”

Covered the benefits to be derived from a Collaborative approach to the definition and implementation of standardised extensions in XPath which the core XML technologies such as Xquery and XSLT, Xproc, XForms would be able to utilise in a uniform way.

Automatic XML Namespaces

Explored the difficulties with Namespaces and proposed ways of addressing them.

CAIRO Content Model: A noob's overview (Part 2) Accessions

I-E-O-2. It is a curious one and no mistake. The idea of an accession wasn't something I'd come across before (though alien concepts are no surprise to me now having worked in libraries for some time) - and if you really want to confuse yourself about what it is, try this Wikipedia article on 'accession'.

All clear now? Didn't think so! (Though the bit about the museum is probably closest). For the purpose of ingest, I'm not too worried about legal ownership (this must be assumed) so my working definition - and I make no claims that it is a good or accurate - is:

"an accession is a unit of stuff that arrives at a given time, at a given archive, and needs to be added to that archive"

An accession then is defined by its origin, date of arrival and the collection to which it belongs.

Like IE01, IE02 uses the METS header element to record agent data - who did what and when to this record. This will usually "created by ingest process", "updated by CAIRO tool by user X", etc.

In IE02, the descriptive metadata is kept deliberately Spartan. This is because the concept of the accession remains largely transparent to researchers. While it is important to the archivist to know the source of the collection's parts, the researcher really just needs the collection and the items. So, dmdSec gives us space for an accession identifier (unitId) and title (unitTitle) (using controlled formats - which may or may not be specified yet - I'll find out as I move through the model!) and then a minimal EAD/DC/MODS description. Since we're an archive, I focussed on the EAD, but cross-walking to the others would be possible.

The dmdSec EAD description is quite minimal, listing just origination information (pricipal creator), a physical description (extent, in MBs), a description of the formats and a description of the software/hardware environment used. All these are given as free-text, manual fields, but I wonder how far we can get offering automated "tips" for these elements.

There is also room for a pointer to some rights statement (amdSec) (as with IE01) and also a pointer to a log file that records activity over this accession (fileSec). The format of that log file remains undefined, but it'll record things like "checked all files for viruses", "identified 15 obsolete files", "transformed obsolete formats to ODT", etc. We'll have to think about how we implement this linkage. The model says use a URN to point at the log file, but I wonder if we use a URI that points to a canned search of a generally logging service - something along the lines of beam.ouls.ox.ac.uk/audit?accessionID=12345. That way we can dynamically generate log reports for each accession. Needs thinking about anyways.

Finally there is, in the structMap, a map to the other "main divisions" of the accession and the model suggests that these are either folders or files (different types are given - like subject folder or email directory) and so it is apparent that the accession structMap could be used to reflect the entire accession structure or just the top level directories. Which is better is unclear, but I suspect we will be adding a manifest to the accession that does list all the files (so the structure can be browsed without getting near the real data objects and, if we do it that way, without having to parse the IE02 too much).

We could put a pointer to the manifest alongside the log file pointer in fileSec.

I guess that is probably no clearer than the aforementioned wikipedia page but at least I never used the word "Inaedificatio"!

CAIRO Content Model: A noob's overview (Part 1)

I have started to take a look at the CAIRO content model to see how this will be used for futureArch and, later, BEAM. Creation of the model predates me joining the team here at Oxford so it is possible I'll entirely misrepresent the work, so feel free to ask questions or shout! :-)

As there is quite a lot to the content model, I'll cover it in parts as I read them. This is to allow my brain to digest each bit rather than try to cram the entire thing in one go. Hopefully the same idea will help anyone trying to read about it here.

So without further ado, I present Intellectual Entity 01: Collection.

Affectionately known as "I-E-O-1" (the O is really a 0 but we don't say zero), this is described in the documentation as the "descriptive overview of the collection". The metadata here is designed to enable a curator to respond to researcher enquiries and freedom of information requests and also to provide the foundation for the rest of the collection's metadata (which is broken down - as we'll see in future posts - into accessions and items).

IE01 relates to both digital and physical (really need a better word - analogue?) components of the collection, but it is important to note that it does not replace the EAD record created by the archivist - though it links to it. This begs the question: are we currentlly able to link to our EAD records? (I'll need to find out the answer).

Like all of our entities, the object is specified using METS. From my perspective of creating digital object ingest tools, the fields I'm worrying about are (in no particular order):

• A list of agents, found in the metsHdr element, and recording who did what with this record.
• An embedded EAD and/or DC and/or MODS record used to describe the collection. Nb. this is NOT the archival EAD, but rather a subset of the given schema (EAD/DC/MODS) to record a minimal amount of metadata about this aspect of the entire collection - archive identifier, country code, dates, formats, scope and access.
• A link to a further entity (PR01) describing preservation rights of the collection.

There is also room to record a bit of structural information about the collection - the accessions that make it up for example - in the structmap.

That is a fairly sketchy outline of IE01 and there is a lot more to it than that of course, but that is my first impression. I hope it is useful to someone, and useful to me when I've recycled my paper notes! :-)