Thursday, 8 September 2011
Another source for old software
This one is called http://www.oldapps.com/. Looks like a useful range of apps.
Monday, 22 August 2011
Comparing software tools
While looking at software relating to digital video earlier today I came across a handy website called alternativeTo. It's a useful means of comparing software applications and getting an idea of the tools that are out there to help perform a particular task. AlternativeTo gives a brief summary of each piece of software along with screenshots of the software in action. Another useful feature is that searches can be filtered by whether the tools are free or open source.
Wednesday, 10 August 2011
Mobile forensics
Anyone who heard Brad Glisson's talk at the DPC event on digital forensics may be interested in the paper 'A comparison of forensic evidence recovery techniques for a windows mobile smart phone' published in Digital Investigation Volume 8, Issue 1, July 2011, Pages 23-36. Interesting to see what was recovered, and how the different tools behave.
Wednesday, 27 July 2011
Preserving born-digital video - what are good practices?
 |
| http://www.flickr.com/photos/fensterbme/1990023423/ |
So, what kind of digital video do we have? It's a good question, and one I can't answer fully for the moment. What I can say is that our collections include digital video deposited on CDs, DVDs, Bluray discs, miniDV and mediumDV cassettes, and hard disks. Much of this material has yet to be captured from its original media so we don't have that inventory of codecs, wrapper formats, frame rates, metadata, etc. that Killian talks about. This kind of detailed survey work is a next step for us, but one that will have to wait until we have developed a workflow for initial capture (bit-level preservation comes first). I wonder if we'll see the same diversity of technical characteristics present in the Smithsonian's materials. It seems likely.
Wednesday, 11 May 2011
Hidden Pages
Yesterday I foolishly uploaded a Pages document to my work machine (that isn't a Mac) before heading into the office. I needed the content because I was due to give it to Susan that morning. Luckily I stumbled upon this tip and thought I'd share it in case you ever find yourself faced with a Mac disk full of documents and no Mac to read them on...
I guess I shouldn't have been surprised to discover a Pages document is in fact a zip file (like Word docs) and if you unpack it not only do you find an XML representation of the document (which would let you get at the text - run it through tidy first though as there aren't any line breaks!) or, neater, in the QuickLook directory is a PDF (file reports PDF 1.3) of the document.
Day saved! Phew!
I guess I shouldn't have been surprised to discover a Pages document is in fact a zip file (like Word docs) and if you unpack it not only do you find an XML representation of the document (which would let you get at the text - run it through tidy first though as there aren't any line breaks!) or, neater, in the QuickLook directory is a PDF (file reports PDF 1.3) of the document.
Day saved! Phew!
Wednesday, 20 April 2011
Media recognition - Floppy Disks part 3
3 inch Disks (Mitsumi 'Quick Disk')
High Level Formatting
Type: | Magnetic storage media |
Introduced: | ?1985 |
Active: | Unlikely. |
Cessation: | Used in the 1980s. |
Capacity: | ?128KB - 256KB |
Compatibility: | Requires a 3” drive appropriate to the manufacturer's specifications. |
Users: | Likely to have been individual users and small organisations. Used for word-processing, music and gaming. |
File Systems: | Unknown. May vary according to use. The disks were manufactured by Mitsumi and offered as OEM to resellers and used in a range of contexts including Nintendo (Famicom), various MIDI keyboards/samplers (Roland) and the Smith Corona Personal Word Processor (PWP). |
Common Manufacturers: | Disks: Mitsumi appear to have made the magnetic disk (the innards), while other manufacturers made the cases. This resulted in different case shapes and labelling. For example Sharp Corona labelled the disks as DataDisk 2.8". Drives: Mitsumi? |
Recognition
The Smith Corona Personal Word Processor (PWP) variant of the disk is double sided with one side being labelled ‘A’ and the other ‘B’. Each side also had a dedicated write-protect hole, known as a 'breakout lug'.
 |
| 2.8" Smith Corona 'Quick Disk' |
 | ||
| 3.5" floppy side-by-side with a 2.8" Smith Corona 'Quick Disk' |
 | ||
| Nintendo Famicon disk Some rights reserved by bochalla |
High Level Formatting
Unknown. Possibly varied according to use.
3 Inch Disk Drives
Varied according to disk. The Smith Corona word processing disks are most likely to turn up in an archival collection. These were used in a Smith Corona PWP and possible models nos. include: 3,5,6, 6BL, 7, X15,X25, 40, 50LT, 55D, 60, 65D, 75D, 80, 85DLT, 100, 100C, 220, 230, 250, 270LT, 300, 350, 355, 960, 990, 2000, 2100, 3000, 3100, 5000, 5100, 7000LT, DeVille 3, DeVille 300, Mark X, Mark XXX, Mark XL LT.
Useful links
http://www.cromwell-intl.com/technical/quickdisk-recovery.html
http://en.wikipedia.org/wiki/History_of_the_floppy_disk
Varied according to disk. The Smith Corona word processing disks are most likely to turn up in an archival collection. These were used in a Smith Corona PWP and possible models nos. include: 3,5,6, 6BL, 7, X15,X25, 40, 50LT, 55D, 60, 65D, 75D, 80, 85DLT, 100, 100C, 220, 230, 250, 270LT, 300, 350, 355, 960, 990, 2000, 2100, 3000, 3100, 5000, 5100, 7000LT, DeVille 3, DeVille 300, Mark X, Mark XXX, Mark XL LT.
 | ||||
Lego mockup of a Nintendo Famicon drive
|
http://www.cromwell-intl.com/technical/quickdisk-recovery.html
http://en.wikipedia.org/wiki/History_of_the_floppy_disk
Wednesday, 13 April 2011
Preserving Digital Sound and Vision: A Briefing 8th April 2011
Last Friday I went along to the DPC briefing Preserving Digital Sound and Vision. I was particularly interested in the event because of digital video files currently held on DVD media at the Bodleian.
After arriving at the British Library and collecting my very funky Save the Bits DPC badge I sat down to listen to a packed programme of speakers. The morning talks gave an overview of issues associated with preserving audio-visual resources. We began with Nicky Whitsed from the Open University who spoke about the nature of the problem of preserving audio-visual content; a particularly pertinent issue for the OU who have 40 years of audio-visual teaching resources to deal with. Richard Ranft then gave a fascinating insight into the history and management of the British Library Sound Archive. He played a speech from Nelson Mandela’s 1964 trial to emphasise the value of audio preservation. Next Stephen Gray from JISC Digital Media spoke about how students are using audio-visual content in their research. He mentioned the difficulties researchers find when citing videos, especially those on YouTube that may disappear at any time! To round off the morning John Zubrycki from BBC R and D spoke about Challenges and Solutions in Broadcast Archives. One of the many interesting facts that he mentioned was that subtitle files originally produced by the BBC for broadcast have been used as a tool for search and retrieval of video content.
After enjoying lunch and the beautiful sunny weather on the British Library terrace we moved onto the afternoon programme based on specific projects and tools. Richard Wright of the BBC spoke about the Presto Centre and the tools it has developed to help with audio-visual preservation. He also spoke about the useful digital preservation tools available online via Presto Space. Sue Allcock and James Alexander then discussed the Outcomes and Lessons learnt from the Access to Video Assets Project at the Open University which makes past video content from the Open University’s courses available to OU staff through a Fedora repository. Like the BBC, discovering subtitle files has allowed the OU to index their audio-visual collections. Finally Simon Dixon from the Centre for Digital Music Queen Mary University spoke about emerging tools for digital sound.
A final wide ranging discussion about collaboration and next steps followed which included discussion about storage as well as ideas for a future event addressing the contexts of audio-visual resources. I left the event with my mind full of new information and lots of pointers for places to look to help me consider the next steps for our digital video collections… watch this space.
Tuesday, 12 April 2011
Sharp font writer files
Thursday, 7 April 2011
Got any older?
Interesting article about the "oldest working Seagate drive in the UK". When we talk about storage here, eventually someone says "storage is getting cheaper". If you ever needed concrete proof, this is it!
Thursday, 31 March 2011
World backup day 2011
Today is officially 'world backup day', apparently. Time to back up your files...
I'm sure you know why, and perhaps even how. There are a few tips here if you want them: http://www.worldbackupday.net/
I'm sure you know why, and perhaps even how. There are a few tips here if you want them: http://www.worldbackupday.net/
Thursday, 24 March 2011
Advisory Board Meeting, 18 March 2011
Our second advisory board meeting took place on Friday. Thanks to everyone who came along and contributed to what was a very useful meeting. For those of you who weren't there here is a summary of the meeting and our discussions.
The first half of the afternoon took the form of an overview of different aspects of the project.
Overview of futureArch's progress
Susan Thomas gave us a reminder of the project's aims and objectives and the progress being made to meet them. After an overview of the percentage of digital accessions coming into the library since 2004 and the remaining storage space we currently have, we discussed the challenge of predicting the size of future digital accessions and collecting digital material. We also discussed what we think researcher demand for born-digital material is now and will be in the future.
Born Digital Archives Case Studies
Bill Stingone presented a useful case study about what the New York Public Library has learnt from the process of making born-digital materials from the Straphangers Campaign records available to researchers.
After this Dave Thompson spoke about some of the technicalities of making all content in the Wellcome Library (born-digital, analogue and digitised) available through the Wellcome Digital Library. Since the project is so wide reaching a number of questions followed about the practicalities involved.
Web archiving update
Next we returned to the futureArch project and Susan gave an overview of the scoping, research and decisions that have been made regarding the web archiving pilot since the last meeting. I then gave an insight into how the process of web archiving will be managed using a tracking database. Some very helpful discussions followed about the practicalities of obtaining permission for archiving websites and the legal risks involved.
After breaking for a well earned coffee we reconvened to look at systems.
Systems for Curators
Susan explained how the current data capture process works for digital collections at the Bodleian including an overview of the required metadata which we enter manually at the moment. Renhart moved on to talk about our intention to use a web-based capture workbench in the future and to give us a demo of the RAP workbench. Susan also showed us how FTK is used for appraisal, arrangement and description of collections and the directions we would like to take in the future.
Researcher Interface
To conclude the systems part of the afternoon, Pete spoke about how the BEAM researcher interface has developed since the last advisory board meeting, the experience of the first stage of testing the interface and the feedback gained so far. He then encouraged everyone to get up and have a go at using the interface for themselves and to comment on it.
Training the next generation of archivists?
With the end of the meeting fast approaching, Caroline Brown from the University of Dundee gave our final talk. She addressed the extent to which different archives courses in the UK cover digital curation and the challenges faced by course providers aiming to include this kind of content in their modules.
With the final talk over we moved onto some concluding discussions around the various skills that digital archivists need. Those of us who were able to stay continued our discussions over dinner.
The first half of the afternoon took the form of an overview of different aspects of the project.
Overview of futureArch's progress
Susan Thomas gave us a reminder of the project's aims and objectives and the progress being made to meet them. After an overview of the percentage of digital accessions coming into the library since 2004 and the remaining storage space we currently have, we discussed the challenge of predicting the size of future digital accessions and collecting digital material. We also discussed what we think researcher demand for born-digital material is now and will be in the future.
Born Digital Archives Case Studies
Bill Stingone presented a useful case study about what the New York Public Library has learnt from the process of making born-digital materials from the Straphangers Campaign records available to researchers.
After this Dave Thompson spoke about some of the technicalities of making all content in the Wellcome Library (born-digital, analogue and digitised) available through the Wellcome Digital Library. Since the project is so wide reaching a number of questions followed about the practicalities involved.
Web archiving update
Next we returned to the futureArch project and Susan gave an overview of the scoping, research and decisions that have been made regarding the web archiving pilot since the last meeting. I then gave an insight into how the process of web archiving will be managed using a tracking database. Some very helpful discussions followed about the practicalities of obtaining permission for archiving websites and the legal risks involved.
After breaking for a well earned coffee we reconvened to look at systems.
Systems for Curators
Susan explained how the current data capture process works for digital collections at the Bodleian including an overview of the required metadata which we enter manually at the moment. Renhart moved on to talk about our intention to use a web-based capture workbench in the future and to give us a demo of the RAP workbench. Susan also showed us how FTK is used for appraisal, arrangement and description of collections and the directions we would like to take in the future.
Researcher Interface
To conclude the systems part of the afternoon, Pete spoke about how the BEAM researcher interface has developed since the last advisory board meeting, the experience of the first stage of testing the interface and the feedback gained so far. He then encouraged everyone to get up and have a go at using the interface for themselves and to comment on it.
Training the next generation of archivists?
With the end of the meeting fast approaching, Caroline Brown from the University of Dundee gave our final talk. She addressed the extent to which different archives courses in the UK cover digital curation and the challenges faced by course providers aiming to include this kind of content in their modules.
With the final talk over we moved onto some concluding discussions around the various skills that digital archivists need. Those of us who were able to stay continued our discussions over dinner.
Thursday, 17 March 2011
Friday, 18 February 2011
what have the Romans ever Done For us?
Today I presented an internal seminar on RDF to the Bodleian Library developers, the first in a series of (hopefully) regular R&D meetings. This one was to provide a practical introduction to RDF to give us a baseline to build from when we start building models of our content (at least one Library project requires we generate RDF). I called it "what have the Romans ever Done For us?":
The title is a line from the Monty Python film "The Life of Brian" and I chose it not just because I've been looking into Python (the language) but also because I can imagine a future where people ask "What has RDF ever done for us?" in a disgruntled way. In the film people suggest the Romans did quite a lot - bits of public infrastructure, like aqueducts and roads, alongside useful services like wine and medicine. I think RDF is a bit like that. Done well, it creates a solid infrastructure from which useful services will be built, but it is also likely to invisible, if not taken for granted, like sanitation and HTML. :-)
The seminar itself seemed to go well - though you'd have to ask the attendees rather than the presenter to get the real story! We started with some slides that outlines the basics of RDF, using Dean Allemang & Jim Hendler's nice method of distributing responsibility for tablular data and ending up at RDF (see pg. 32 onwards in the book Semantic Web for the Working Ontologist), and leapt straight in with LIBRIS (for example Neverwhere as RDF) as a case study. In the resultant discussion we looked at notation, RDFS, and linked data.
The final half of the seminar was a workshop in which we split into two groups: data providers and data consumers, and then considered what resources at the Bodleian might be suitable for publication as RDF (and linked data) and what services we might build using data from elsewhere.
The data providers discussed how there was probably quite a lot of resources in the Library that we could publish, or become the authority on - members of the University for example, or any of the many wonders we have in the collections. To make this manageable, it was felt it would be sensible to break this task up, probably by project. This would also allow for specific models to be identified and/or developed for each set of resources.
There was some concern among the providers regarding how to "sell" the benefits of RDF and linked data to management. The concerns paralleled what I imagine happened with the emergent Web. Is this kind of data publication giving away valuable information assets, for little or no return? At worst this leads people away from the Library to viewing and using information via aggregation services. Of course, there is a flip side to this argument. Serendipitous discovery of a Bodleian resource via a third-party is essentially free advertising and may drive users through our (virtual or real) door?
The consumers seemed realise early on that one of the big problems was the lack of usable data. Indeed, for the talk, I scoured datasets trying to find a decent match of data to augment Library catalogues and found it quite hard. That isn't to say there is not a lot of data available (though of course there could be more), it is just without an application for it, shoehorning data into a novel use remains a novelty item. However, one possible example was the Legislation API. The group suggested that reviews from other sites could be used to augment Library catalogue results. They also suggested that the people data the Library published could be very useful and Monica talked about a suggestion she had heard at Dev8D for a developer expertise database (data Web?).
All in all lots of very useful discussion and I hope everyone went away with a good idea of what RDF was and what it might do for us and what we might do with it. There (justifiably) remains some scepticism, mostly because without the Web, linked data is simply data and we've all got our own neat ways to handle data already, be it a SQL database, XML & Solr, or whatever. Without the Web of Data the question "What do we gain?" remains.
It is a bit chicken and egg and the answer will eventually become clear as more and more people create machine processable data on the Web.
For the next meeting we'll be modelling people. I'm going to bring the clay! :-)
Slides and worksheets are available with the source open office documents also published on this (non-RDF) page!
The title is a line from the Monty Python film "The Life of Brian" and I chose it not just because I've been looking into Python (the language) but also because I can imagine a future where people ask "What has RDF ever done for us?" in a disgruntled way. In the film people suggest the Romans did quite a lot - bits of public infrastructure, like aqueducts and roads, alongside useful services like wine and medicine. I think RDF is a bit like that. Done well, it creates a solid infrastructure from which useful services will be built, but it is also likely to invisible, if not taken for granted, like sanitation and HTML. :-)
The seminar itself seemed to go well - though you'd have to ask the attendees rather than the presenter to get the real story! We started with some slides that outlines the basics of RDF, using Dean Allemang & Jim Hendler's nice method of distributing responsibility for tablular data and ending up at RDF (see pg. 32 onwards in the book Semantic Web for the Working Ontologist), and leapt straight in with LIBRIS (for example Neverwhere as RDF) as a case study. In the resultant discussion we looked at notation, RDFS, and linked data.
The final half of the seminar was a workshop in which we split into two groups: data providers and data consumers, and then considered what resources at the Bodleian might be suitable for publication as RDF (and linked data) and what services we might build using data from elsewhere.
The data providers discussed how there was probably quite a lot of resources in the Library that we could publish, or become the authority on - members of the University for example, or any of the many wonders we have in the collections. To make this manageable, it was felt it would be sensible to break this task up, probably by project. This would also allow for specific models to be identified and/or developed for each set of resources.
There was some concern among the providers regarding how to "sell" the benefits of RDF and linked data to management. The concerns paralleled what I imagine happened with the emergent Web. Is this kind of data publication giving away valuable information assets, for little or no return? At worst this leads people away from the Library to viewing and using information via aggregation services. Of course, there is a flip side to this argument. Serendipitous discovery of a Bodleian resource via a third-party is essentially free advertising and may drive users through our (virtual or real) door?
The consumers seemed realise early on that one of the big problems was the lack of usable data. Indeed, for the talk, I scoured datasets trying to find a decent match of data to augment Library catalogues and found it quite hard. That isn't to say there is not a lot of data available (though of course there could be more), it is just without an application for it, shoehorning data into a novel use remains a novelty item. However, one possible example was the Legislation API. The group suggested that reviews from other sites could be used to augment Library catalogue results. They also suggested that the people data the Library published could be very useful and Monica talked about a suggestion she had heard at Dev8D for a developer expertise database (data Web?).
All in all lots of very useful discussion and I hope everyone went away with a good idea of what RDF was and what it might do for us and what we might do with it. There (justifiably) remains some scepticism, mostly because without the Web, linked data is simply data and we've all got our own neat ways to handle data already, be it a SQL database, XML & Solr, or whatever. Without the Web of Data the question "What do we gain?" remains.
It is a bit chicken and egg and the answer will eventually become clear as more and more people create machine processable data on the Web.
For the next meeting we'll be modelling people. I'm going to bring the clay! :-)
Slides and worksheets are available with the source open office documents also published on this (non-RDF) page!
Monday, 24 January 2011
Migrating documents
We have a collection that consists of several thousand documents in various archaic (well, 1980s/90s) word processor formats including Ami Professional and (its predecessor) Samna Word. Perhaps of interest to folks intent on discussing the implications of migration for authenticity of the items, some of those Ami Pro files contain the (automatically generated) line:
"File ... was converted from Samna Word on ..."
So which is the original now?
Migrating these file formats has not been straight forward. This is because it was proved remarkably tricky to ascertain a key piece of information - the file format of the original. This is not the fault of file format tools (I'm using FITS, which itself wraps the usual suspects JHOVE & DROID), but the broader problem that the files have multiple formats. Ami Pro files are correctly identified "text/plain". The command file reports them as "ASCII English text". Some (not all) have a file extension ".sam" which is usually Ami Word, but the ".sam" files are not all the same format.
Yet this small piece of metadata is essential because without it it is very difficult to identify the correct tool to perform the migration. For example, if I run my usual text to PDF tool - which is primed to leap into action on arrival of a "text/plain" document - the resultant PDF shows the internals of a Ami Pro file, not the neatly laid out document the creator saw. We have a further piece of information available too, and curiously it is the most useful. This is the "Category" from the FTK - which correctly sorts the Ami Pros from the Samna Words.
This leads to a complex migration machine that needs to be capable of collating file format information from disparate sources and making sense of the differences, all within the context of the collection itself. If I know that creator X used Ami Pro a lot, then I can guess that "text/plain" & ".sam" means an Ami Pro document, for example. This approach is not without problems however, not least of which is that it requires a lot of manual input into what should ultimately be an automated and unwatched process. (One day, when it works better, I'll try to share this code!)
Sometimes you get lucky, and the tool to do the migration offers an "auto" mode for input. For this collection I am using a trial copy of FileMerlin to perform the migration and evaluate it. It actually works better if you let it guess the input format rather than attempt to tell it. Other tools, such as JODConverter, like to know the input format and here you have a similar problem - you need to know what JODConverter is happy to accept rather than the real format - for example, send it a file with a content type of "application/rtf" and it responds with an internal server error. Send the same file with a content type of "application/msword" and the PDF is generated and returned to you.
Then there is a final problem - sometimes you have to make several steps to get the file into shape. For this collection, FileMerlin should be able to migrate Ami Pro and Samna Word into PDFs. In practice, it crashes on a very small sub-set of the documents. To overcome this, I migrate these same documents to "rich text format" (which FileMerlin seems OK with) and then to PDF with JODConverter - sending the aforementioned "application/msword" content type. I had a similar problem with WordPerfect files where using JOD directly changed the formatting of the original files. Using libwpd to create ODTs and then converting them to PDFs generated more accurate PDFs. (This is strange behaviour since OpenOffice itself uses libwpd!) Every time I hit a new (old) file format, the process of identifying it and generating a heuristic for handling it starts over.
I'm starting to think I need a neural network! That really would be putting the AI in OAIS!
"File ... was converted from Samna Word on ..."
So which is the original now?
Migrating these file formats has not been straight forward. This is because it was proved remarkably tricky to ascertain a key piece of information - the file format of the original. This is not the fault of file format tools (I'm using FITS, which itself wraps the usual suspects JHOVE & DROID), but the broader problem that the files have multiple formats. Ami Pro files are correctly identified "text/plain". The command file reports them as "ASCII English text". Some (not all) have a file extension ".sam" which is usually Ami Word, but the ".sam" files are not all the same format.
Yet this small piece of metadata is essential because without it it is very difficult to identify the correct tool to perform the migration. For example, if I run my usual text to PDF tool - which is primed to leap into action on arrival of a "text/plain" document - the resultant PDF shows the internals of a Ami Pro file, not the neatly laid out document the creator saw. We have a further piece of information available too, and curiously it is the most useful. This is the "Category" from the FTK - which correctly sorts the Ami Pros from the Samna Words.
This leads to a complex migration machine that needs to be capable of collating file format information from disparate sources and making sense of the differences, all within the context of the collection itself. If I know that creator X used Ami Pro a lot, then I can guess that "text/plain" & ".sam" means an Ami Pro document, for example. This approach is not without problems however, not least of which is that it requires a lot of manual input into what should ultimately be an automated and unwatched process. (One day, when it works better, I'll try to share this code!)
Sometimes you get lucky, and the tool to do the migration offers an "auto" mode for input. For this collection I am using a trial copy of FileMerlin to perform the migration and evaluate it. It actually works better if you let it guess the input format rather than attempt to tell it. Other tools, such as JODConverter, like to know the input format and here you have a similar problem - you need to know what JODConverter is happy to accept rather than the real format - for example, send it a file with a content type of "application/rtf" and it responds with an internal server error. Send the same file with a content type of "application/msword" and the PDF is generated and returned to you.
Then there is a final problem - sometimes you have to make several steps to get the file into shape. For this collection, FileMerlin should be able to migrate Ami Pro and Samna Word into PDFs. In practice, it crashes on a very small sub-set of the documents. To overcome this, I migrate these same documents to "rich text format" (which FileMerlin seems OK with) and then to PDF with JODConverter - sending the aforementioned "application/msword" content type. I had a similar problem with WordPerfect files where using JOD directly changed the formatting of the original files. Using libwpd to create ODTs and then converting them to PDFs generated more accurate PDFs. (This is strange behaviour since OpenOffice itself uses libwpd!) Every time I hit a new (old) file format, the process of identifying it and generating a heuristic for handling it starts over.
I'm starting to think I need a neural network! That really would be putting the AI in OAIS!
Wednesday, 15 December 2010
Digital forensics and digital archives
If you're interested in how digital forensics methodologies and tools could, and are, being applied to digital archives then you might like to take a look at a new report published by CLIR. See http://www.clir.org/pubs/reports/pub149/pub149.pdf. It also includes a sidebar outlining how forensics tools are incorporated into our workflow.
Friday, 26 November 2010
What I learned from the word clouds...
Now, word clouds are probably bit out of fashion these days. Like a Google Map, they just seem shiny but most of the time quite useless. Still, that hasn't stopped us trying them out in the interface - because I'm curious to see what interesting (and simple to gather) metadata n-grams & their frequency can suggest.
Take for instance the text of "Folk-Lore and Legends of Scotland" [from Project Gutenberg] (I'm probably not allowed to publish stuff from a real collection here and choose this text because I'm pining for the mountains). It generates a "bi-gram"-based word cloud that looks like this:
Names (of both people and places) quickly become obvious to human readers, as do some subjects ("haunted ships" is my favourite). To make it more useful to machines, I'm pretty sure someone has already tried cross-referencing bi-grams with name authority files. I also imagine someone has used the bi-grams as facets. Theoretically a bi-gram like "Winston Churchill" may well turn up in manuscripts from multiple collections. (Any one know of any successes doing these things?).
Still, for now I'll probably just add the word clouds of the full-texts to the interface, including a "summary" of a shelfmark, and then see what happens!
I made the (very simple) Java code available on GitHub, but I take no credit for it! It is simply a Java reworking of Jim Bumgardner's word cloud article using Jonathan Feinberg's tokenizer (part of Wordle).
Take for instance the text of "Folk-Lore and Legends of Scotland" [from Project Gutenberg] (I'm probably not allowed to publish stuff from a real collection here and choose this text because I'm pining for the mountains). It generates a "bi-gram"-based word cloud that looks like this:
Names (of both people and places) quickly become obvious to human readers, as do some subjects ("haunted ships" is my favourite). To make it more useful to machines, I'm pretty sure someone has already tried cross-referencing bi-grams with name authority files. I also imagine someone has used the bi-grams as facets. Theoretically a bi-gram like "Winston Churchill" may well turn up in manuscripts from multiple collections. (Any one know of any successes doing these things?).Still, for now I'll probably just add the word clouds of the full-texts to the interface, including a "summary" of a shelfmark, and then see what happens!
I made the (very simple) Java code available on GitHub, but I take no credit for it! It is simply a Java reworking of Jim Bumgardner's word cloud article using Jonathan Feinberg's tokenizer (part of Wordle).
Wednesday, 10 November 2010
The as yet unpaved publication pathway...
It has been a while since we had a whiteboard post, so I thought it was high time we had one! This delightful picture is the result of trying to explain the "Publication Pathway" - Susan's term for making our content available - to a new member of staff at the Library...
Nothing too startling here really - take some disparate sources of metadata, add a sprinkling of auto-gen'd metadata (using the marvelous FITS and the equally marvelous tools it wraps), migrate the arcane input formats to something useful, normalise and publish! (I'm thinking I might get "Normalise and Publish!" printed on a t-shirt! :-))
The blue box CollectionBuilder is what does most of the work - constructs an in memory tree of "components" from the EAD, tags the items onto the right shelfmarks, augments the items with additional metadata, and writes the whole lot out in a tidy directory structure that even includes a foxml file with DC, PREMIS and RDF data streams (the RDF is used to maintain the hierarchical relationships in the EAD). That all sounds a lot neater than it currently is, but, like all computer software, it is a work in progress that works, rather than a perfect end result! :-)
After that, we (will, it aint quite there yet) push the metadata parts into the Web interface and from there index it and present to our lovely readers!
Hooray!
The four boxes at the bottom are the "vhysical" layout - its a new word I made up to describe what is essentially a physical (machine) architecture, but is in fact a bunch of virtual machines...
For the really attentive among you, this shot is of the whiteboard in its new home on the 2nd floor of Osney One, where Renhart and I have moved following a fairly major building renovation. Clearly we were too naughty to remain with the archivists! ;-)
Wednesday, 29 September 2010
(ZX) spectrums & stained glass
Only slightly off topic but I was pointed at http://www.zxspectrum.net/ this morning. I can't decide what is more impressive, the site itself, that Java can be used to emulate a ZX Spectrum or that there are games listed from 2010!
Slightly more on topic, I was curious to read this article about stained glass, or is it file formats. Skip over the rather flowery bit and start reading from:
"The archive problem is one of format" and concludes "Then in five hundred years time... pictures of the Auch windows will be stored and accessible in the cloud. But not one of them...will have the jaw-dropping impact of people seeing them for the very first time...and realising that humans could do wondrous things for themselves."
I read another article, this time from New View (not something I'd usually read, but an article about the "origins of computing" was recommended to me), which - in a different article - likened stained glass windows to computer screens - mostly because of the back lighting - and suggested, a little like Chris Mellor in the Register, that the cultural experience of seeing stained glass is a bit like experiencing computers for the first time.
(An other article berated digital technology, almost branding it evil, and certainly bad enough to make the author ill using it. I was reading this at DrupalCon where 90% of the people there were welded to their MacBooks and wondered how bad using a plough might make these people feel! ;-))
I guess many folks don't have the "big bang" experience of the digital world. It crept up slowly - a bit like watching the cathedral and its windows being built would probably reduce its wonder - going from a ZX81 to a Pet to a BBC B to an Amstrad 1640 to a... well, you get the picture...
But does that make it any less astonishing? It shouldn't. If anything, it is all about helping people do "wondrous things for themselves", just like zxspectrum.net...
Slightly more on topic, I was curious to read this article about stained glass, or is it file formats. Skip over the rather flowery bit and start reading from:
"The archive problem is one of format" and concludes "Then in five hundred years time... pictures of the Auch windows will be stored and accessible in the cloud. But not one of them...will have the jaw-dropping impact of people seeing them for the very first time...and realising that humans could do wondrous things for themselves."
I read another article, this time from New View (not something I'd usually read, but an article about the "origins of computing" was recommended to me), which - in a different article - likened stained glass windows to computer screens - mostly because of the back lighting - and suggested, a little like Chris Mellor in the Register, that the cultural experience of seeing stained glass is a bit like experiencing computers for the first time.
(An other article berated digital technology, almost branding it evil, and certainly bad enough to make the author ill using it. I was reading this at DrupalCon where 90% of the people there were welded to their MacBooks and wondered how bad using a plough might make these people feel! ;-))
I guess many folks don't have the "big bang" experience of the digital world. It crept up slowly - a bit like watching the cathedral and its windows being built would probably reduce its wonder - going from a ZX81 to a Pet to a BBC B to an Amstrad 1640 to a... well, you get the picture...
But does that make it any less astonishing? It shouldn't. If anything, it is all about helping people do "wondrous things for themselves", just like zxspectrum.net...
Friday, 17 September 2010
Media players and the reader interface...
This is quite a long post, so I'm going to put the final line at the top too in case you don't read that far... ;-)
Your thoughts on media players would be most welcome!
The trouble, um, I mean, beauty of digital collections is that they redefine what a "manuscript" is. This is nothing new. Once upon a time someone somewhere probably upset the apple cart when they arrived at the hallowed doors with a basket full of photographs. Now we have video, audio and images, all of which can be encoded in any number of "standard" ways. (Not to mention a zillion different binary formats for just about any purpose you can imagine from sheet music to the latest car designs, which may well require more than just document-like presentation too - 3D models for example). These new manuscripts bring challenges for preservation, of course, but they also present challenges for presentation.
To address this, I've been learning more about media players in browsers with a view to picking one for the reader interface. I'm no expert in this field, so here is my layman's consideration of what I've found out and if you want to read more then this is great!
The traditional method to render audio/video in browsers, which pre-dates their ability to handle video themselves, is to use a browser plug-in, either directly (for example VLC plugin) or (more commonly) to build on top of Flash (eg. Flowplayer) or Java (eg. Cortado). The exact mark-up required to use these players varies. Some will simply use the "embed" tag and others have JavaScript libraries to simplify their usage and allow for graceful degradation in the event that the browser does not have the correct plug-in and/or understand/run JavaScript. (This may be an issue when we deploy the interface into a reading room with machines we do not control the configuration of).
But the times, they are a-changin'. Just as old browsers knew what to do when presented with an "img" tag, most modern browsers are beginning to support HTML5's "video" and "audio" tags, allowing the browser itself to handle the playback rather than farming this out to a plug-in. (For more on HTML5 generally see this presentation - the video tag is mentioned at about 58 minutes in). As an added bonus of bringing video into the browser in this way is it has inspired folks to build media players that manipulate the Web page to add the correct mark-up, be it a video tag, an embed, or whatever to play the media. This is currently being used to generate some nice media players that'll use the browser, the Flash-plugin, or whatever is available (see OpenVideoPlayer and OSMPlayer).
So now we get to the crux of it. What should we do for the reader interface? Go old-school (and annoy Steve Jobs) and use a Flash-based player? Adopt the new ways of HTML5? Insist on an Open Source player? Buy something in?
To work out the answer I did a bit of investigating and have installed most of the players mentioned thus far in this post - Flowplayer, OSMPlayer, video-tag only, VLC and Cortado, as well as JWPlayer.
Flowplayer uses the Flash-plugin to play Flash video (and, with an additional plug-in, MP3 audio) - it does not support Ogg. It is very simple to use and very slick to look at. It is open source, released under GPL3 with an additional (and reasonable) "attribution clause" which basically means the Flowplayer logo must appear on the player unless you pay extra.
JWPlayer works much like Flowplayer (though there is also a beta HTML5 video player in the making) and seems pretty good. While the source code is available, it is not clear if this is an open source product or otherwise - the source files do not include a LICENSE.txt or any boilerplate. Probably I'm just missing something there though, and JWPlayer seems a good choice if you don't mind Flash.
OSMPlayer is also open source and has numerous options for installation including a Drupal module (untested), a PHP library and a "stand-alone" configuration. In theory it supports lots of different audio and video formats and uses several divs to create a nice browser based player. Unfortunately, following the guidelines for both PHP and stand-alone configurations, I could not get it to work on my test server.
Video-tag only works pretty well with Firefox 3.6 on Ubuntu 10.04 and is very easy to include in a Web-page. Unfortunately it isn't nearly as slick at playback as Flowplayer - there is a delay in starting the video and it is unclear what is going on.
The VLC plug-in is also open source and seems to work pretty well and should be able to handle many different formats, but it isn't nearly as refined as other players and the provided example code fails to stop the video or make it full-screen. The VLC desktop player is wonderful, but I'm not convinced by the Firefox plug-in.
Cortado is a Java-applet provided to play Ogg Theora among other things. Usage is very simple - you just add an applet tag to the page - but playback is jerky, slow and lacked sound. I do not know if my machine is to blame for this or if it is the player itself so will have to investigate further.
Were I sat on and forced to make a choice I think I'd struggle. Flowplayer is slick to use and easy to implement, but requires we convert everything to Flash video or MP3 (mind you, most media will arrive in suitable formats I imagine). JWPlayer is very similar in this regard. I'd like to adopt the video-tag as this supports a wide range of formats, including open ones, but currently the experience is not very smooth and refinements in this area provided by things like OSMPlayer are still in their early stages of development. JWPlayer's HTML5 offering is still beta for example.
I guess my feeling for now is to either go with Flowplayer (and swallow the conversions required - actually pretty easy with ffmpeg) or spend a bit of time with OpenVideoPlayer's HTML5 work and the video tag. At this stage I think we probably need both working in the interface and see where the better user experience is...
I should throw one more thing into the pot - the problem of formats. Video and audio files are complicated beasts consisting of containers and tracks and such - a bit like cassettes! The contents of these containers are encoded in a variety of ways, each requiring different software to decode and render their content. We have the same problem with documents and we solve that by converting all the text-based materials we get into PDFs (for presentation before anyone starts worrying about the preservation implications of PDF!) and use a PDF plug-in to display them.
Can we do the same with our audio/video material and if we can, what format (I'm using "format" as a general term to mean "container/encoding"!) do we use? (Victoria has already done some work along these lines, creating WAVs for storage and MP3s for presentation, from audio CDs). Is there any additional concerns given that most born-digital video/audio is likely to arrive at our doors in a compressed format? Should we uncompress it? Is such a thing even possible? Should we (and do we have the processing power to) convert all audio/video materials to open formats for both preservation and presentation purposes?
We're going to raise this final question at our next Library developer meeting and see what folks think. In theory we can delay the decision because most browsers and their plug-ins handle multiple formats, but perhaps we should have a standard delivery format much like we currently have PDF?
Oh dear. I started writing this post with the hope of finding all the answers! I have found out a lot about media players at least, which can only be a good thing, and I've also found out that that state of the art is not quite as far along as the proponents of HTML5 killing Flash would like us to believe - though there is good work going on here and this is the future. I'm also unclear just how much my experience of these things is hindered by using Ubuntu - I often wrestle with the playback of media files under Linux! :-)
Still, I think we're further along, nearer an answer and at least in a place to know where to start testing...
Your thoughts on media players would be most welcome! :-)
Your thoughts on media players would be most welcome!
The trouble, um, I mean, beauty of digital collections is that they redefine what a "manuscript" is. This is nothing new. Once upon a time someone somewhere probably upset the apple cart when they arrived at the hallowed doors with a basket full of photographs. Now we have video, audio and images, all of which can be encoded in any number of "standard" ways. (Not to mention a zillion different binary formats for just about any purpose you can imagine from sheet music to the latest car designs, which may well require more than just document-like presentation too - 3D models for example). These new manuscripts bring challenges for preservation, of course, but they also present challenges for presentation.
To address this, I've been learning more about media players in browsers with a view to picking one for the reader interface. I'm no expert in this field, so here is my layman's consideration of what I've found out and if you want to read more then this is great!
The traditional method to render audio/video in browsers, which pre-dates their ability to handle video themselves, is to use a browser plug-in, either directly (for example VLC plugin) or (more commonly) to build on top of Flash (eg. Flowplayer) or Java (eg. Cortado). The exact mark-up required to use these players varies. Some will simply use the "embed" tag and others have JavaScript libraries to simplify their usage and allow for graceful degradation in the event that the browser does not have the correct plug-in and/or understand/run JavaScript. (This may be an issue when we deploy the interface into a reading room with machines we do not control the configuration of).
But the times, they are a-changin'. Just as old browsers knew what to do when presented with an "img" tag, most modern browsers are beginning to support HTML5's "video" and "audio" tags, allowing the browser itself to handle the playback rather than farming this out to a plug-in. (For more on HTML5 generally see this presentation - the video tag is mentioned at about 58 minutes in). As an added bonus of bringing video into the browser in this way is it has inspired folks to build media players that manipulate the Web page to add the correct mark-up, be it a video tag, an embed, or whatever to play the media. This is currently being used to generate some nice media players that'll use the browser, the Flash-plugin, or whatever is available (see OpenVideoPlayer and OSMPlayer).
So now we get to the crux of it. What should we do for the reader interface? Go old-school (and annoy Steve Jobs) and use a Flash-based player? Adopt the new ways of HTML5? Insist on an Open Source player? Buy something in?
To work out the answer I did a bit of investigating and have installed most of the players mentioned thus far in this post - Flowplayer, OSMPlayer, video-tag only, VLC and Cortado, as well as JWPlayer.
Flowplayer uses the Flash-plugin to play Flash video (and, with an additional plug-in, MP3 audio) - it does not support Ogg. It is very simple to use and very slick to look at. It is open source, released under GPL3 with an additional (and reasonable) "attribution clause" which basically means the Flowplayer logo must appear on the player unless you pay extra.
JWPlayer works much like Flowplayer (though there is also a beta HTML5 video player in the making) and seems pretty good. While the source code is available, it is not clear if this is an open source product or otherwise - the source files do not include a LICENSE.txt or any boilerplate. Probably I'm just missing something there though, and JWPlayer seems a good choice if you don't mind Flash.
OSMPlayer is also open source and has numerous options for installation including a Drupal module (untested), a PHP library and a "stand-alone" configuration. In theory it supports lots of different audio and video formats and uses several divs to create a nice browser based player. Unfortunately, following the guidelines for both PHP and stand-alone configurations, I could not get it to work on my test server.
Video-tag only works pretty well with Firefox 3.6 on Ubuntu 10.04 and is very easy to include in a Web-page. Unfortunately it isn't nearly as slick at playback as Flowplayer - there is a delay in starting the video and it is unclear what is going on.
The VLC plug-in is also open source and seems to work pretty well and should be able to handle many different formats, but it isn't nearly as refined as other players and the provided example code fails to stop the video or make it full-screen. The VLC desktop player is wonderful, but I'm not convinced by the Firefox plug-in.
Cortado is a Java-applet provided to play Ogg Theora among other things. Usage is very simple - you just add an applet tag to the page - but playback is jerky, slow and lacked sound. I do not know if my machine is to blame for this or if it is the player itself so will have to investigate further.
Were I sat on and forced to make a choice I think I'd struggle. Flowplayer is slick to use and easy to implement, but requires we convert everything to Flash video or MP3 (mind you, most media will arrive in suitable formats I imagine). JWPlayer is very similar in this regard. I'd like to adopt the video-tag as this supports a wide range of formats, including open ones, but currently the experience is not very smooth and refinements in this area provided by things like OSMPlayer are still in their early stages of development. JWPlayer's HTML5 offering is still beta for example.
I guess my feeling for now is to either go with Flowplayer (and swallow the conversions required - actually pretty easy with ffmpeg) or spend a bit of time with OpenVideoPlayer's HTML5 work and the video tag. At this stage I think we probably need both working in the interface and see where the better user experience is...
I should throw one more thing into the pot - the problem of formats. Video and audio files are complicated beasts consisting of containers and tracks and such - a bit like cassettes! The contents of these containers are encoded in a variety of ways, each requiring different software to decode and render their content. We have the same problem with documents and we solve that by converting all the text-based materials we get into PDFs (for presentation before anyone starts worrying about the preservation implications of PDF!) and use a PDF plug-in to display them.
Can we do the same with our audio/video material and if we can, what format (I'm using "format" as a general term to mean "container/encoding"!) do we use? (Victoria has already done some work along these lines, creating WAVs for storage and MP3s for presentation, from audio CDs). Is there any additional concerns given that most born-digital video/audio is likely to arrive at our doors in a compressed format? Should we uncompress it? Is such a thing even possible? Should we (and do we have the processing power to) convert all audio/video materials to open formats for both preservation and presentation purposes?
We're going to raise this final question at our next Library developer meeting and see what folks think. In theory we can delay the decision because most browsers and their plug-ins handle multiple formats, but perhaps we should have a standard delivery format much like we currently have PDF?
Oh dear. I started writing this post with the hope of finding all the answers! I have found out a lot about media players at least, which can only be a good thing, and I've also found out that that state of the art is not quite as far along as the proponents of HTML5 killing Flash would like us to believe - though there is good work going on here and this is the future. I'm also unclear just how much my experience of these things is hindered by using Ubuntu - I often wrestle with the playback of media files under Linux! :-)
Still, I think we're further along, nearer an answer and at least in a place to know where to start testing...
Your thoughts on media players would be most welcome! :-)
Friday, 3 September 2010
Wot I Lernd At DrupalCon
I spent last week in the lovely city of Copenhagen immersed in all things Drupal. It was a great experience, not just because of the city (so many happy cyclists!), but because I'd not seen a large scale Open Source project up close before and it is a very different and very interesting world!
I'm going to pick out some of my highlights here as to cover it all would take days, but if you want to know more I'd encourage you to check out the conference Web site and the presentation videos on archive.org.
So, wot did I lernd?
Drupal Does RDF
OK, so I knew that already, but I didn't know that from Drupal 7 (release pending) RDF support will be part of the Drupal core, showing a fairly significant commitment in this area. Even better, there is an active Semantic Web Drupal group working on this stuff. While "linked data" remains something of an aside for us (99.9% of our materials will not make their way to the Web any time soon) the "x has relationship y with z" structure of RDF is still useful when building the BEAM interfaces - for example Item 10 is part of shelfmark MS Digital 01, etc. There is also no harm in trying to be future proof (assuming the Semantic Web is indeed the future of the Web! ;-)) for when the resources are released into the wild.
Projects like Islandora and discussions like this suggest growing utility in the use of Drupal as an aspect of an institutional repository, archives or even Library catalogues (this last one my (pxuxp) experiment with Drupal 6 and RDF).
Speaking of IRs...
Drupal Does Publishing
During his keynote, Dries Buytaert (the creator of Drupal) mentioned "distributions". Much like Linux distributions, these are custom builds of Drupal for a particular market or function. (It is testament the software's flexibility that this is possible!) Such distributions already exist and I attended a session on OpenPublish because I wondered what the interface would look like and also thought it might be handy if you wanted to build, for instance, an Open Access Journal over institutional repositories. Mix in the RDF mentioned above and you've a very attractive publishing platform indeed!
Another distro that might be of interest is OpenAtrium which bills itself as an Intranet in a Box.
Drupal Does Community
One of my motivations in attending the conference was to find out about Open Source development and communities. One of the talks was entitled "Come for the Software, Stay for the Community" and I think part of Drupal's success is its drive to create and maintain a sharing culture - the code is GPL'd for example. It was a curious thing to arrive into this community, an outsider, and feel completely on the edge of it all. That said, I met some wonderful people, spent a productive day finding my way around the code at the "sprint" and think that a little effort to contribute will go a long way. This is a good opportunity to engage with a real life Open Source community. All I need to do is work out what I have to offer!
Drupal Needs to Get Old School
There were three keynotes in total, and the middle one was by Rasmus Lerdorf of PHP fame, scaring the Web designers in the audience with a technical performance analysis of the core Drupal code. I scribbled down the names of various debugging tools, but what struck me the most was the almost bewildered look on Rasmus' face when considering that PHP had been used to build a full-scale Web platform. He even suggested at one point that parts of the Drupal core should be migrated to C rather than remain as "a PHP script". There is something very cool about C. I should dig my old books out! :-)
HTML5 is Here!
Jeremy Keith gave a wonderful keynote on HTML5, why it is like it is and what happened to xhtml 2.0. Parts were reminiscent of the RSS wars, but mostly I was impressed by the HTML 5 Design Principles which favour a working Web rather than a theoretically pure (XML-based) one. The talk is well worth a watch if you're interested in such things and I felt reassured and inspired by the practical and pragmatic approach outlined. I can't decide if I should start to implement HTML5 in our interface or not, but given that 5 is broadly compatible with the hotchpotch of HTMLs we all code in now, I suspect this migration will be gentle and as required rather than a brutal revolution.
Responsive Design
I often feel I'm a little slow at finding things out, but I don't think I was the only person in the audience to have never heard about responsive Web design, though when you know what it is, it seems the most obvious thing in the world! The problem with the Web has long been the variation in technology used to render the HTML. Different browsers react differently and things can look very different on different hardware - from large desktop monitors, through smaller screens to phones. Adherence to standards like HTML5 and CSS3 will go a long way to solving the browser problem, but what of screen size? One way would be to create a site for each screen size. Another way would be to make a single design that scales well, so things like images disappear on narrower screens, multiple columns become one, etc.
Though not without its problems, this is the essence of responsive design and CSS3 makes it all possible. Still not sure what I'm on about? dconstruct was given as a good example. Using a standards compliant browser (ie. not IE! (yet)) shrink the browser window so it is quite narrow. See what happens? This kind of design, along with the underlying technology and frameworks, will be very useful to our interface so I probably need to look more into it. Currently we're working with a screen size in mind (that of the reading room laptop) but being more flexible can only be a good thing!
There were so many more interesting things but I hope this has given you a flavour of what was a grand conference.
I'm going to pick out some of my highlights here as to cover it all would take days, but if you want to know more I'd encourage you to check out the conference Web site and the presentation videos on archive.org.
So, wot did I lernd?
Drupal Does RDF
OK, so I knew that already, but I didn't know that from Drupal 7 (release pending) RDF support will be part of the Drupal core, showing a fairly significant commitment in this area. Even better, there is an active Semantic Web Drupal group working on this stuff. While "linked data" remains something of an aside for us (99.9% of our materials will not make their way to the Web any time soon) the "x has relationship y with z" structure of RDF is still useful when building the BEAM interfaces - for example Item 10 is part of shelfmark MS Digital 01, etc. There is also no harm in trying to be future proof (assuming the Semantic Web is indeed the future of the Web! ;-)) for when the resources are released into the wild.
Projects like Islandora and discussions like this suggest growing utility in the use of Drupal as an aspect of an institutional repository, archives or even Library catalogues (this last one my (pxuxp) experiment with Drupal 6 and RDF).
Speaking of IRs...
Drupal Does Publishing
During his keynote, Dries Buytaert (the creator of Drupal) mentioned "distributions". Much like Linux distributions, these are custom builds of Drupal for a particular market or function. (It is testament the software's flexibility that this is possible!) Such distributions already exist and I attended a session on OpenPublish because I wondered what the interface would look like and also thought it might be handy if you wanted to build, for instance, an Open Access Journal over institutional repositories. Mix in the RDF mentioned above and you've a very attractive publishing platform indeed!
Another distro that might be of interest is OpenAtrium which bills itself as an Intranet in a Box.
Drupal Does Community
One of my motivations in attending the conference was to find out about Open Source development and communities. One of the talks was entitled "Come for the Software, Stay for the Community" and I think part of Drupal's success is its drive to create and maintain a sharing culture - the code is GPL'd for example. It was a curious thing to arrive into this community, an outsider, and feel completely on the edge of it all. That said, I met some wonderful people, spent a productive day finding my way around the code at the "sprint" and think that a little effort to contribute will go a long way. This is a good opportunity to engage with a real life Open Source community. All I need to do is work out what I have to offer!
Drupal Needs to Get Old School
There were three keynotes in total, and the middle one was by Rasmus Lerdorf of PHP fame, scaring the Web designers in the audience with a technical performance analysis of the core Drupal code. I scribbled down the names of various debugging tools, but what struck me the most was the almost bewildered look on Rasmus' face when considering that PHP had been used to build a full-scale Web platform. He even suggested at one point that parts of the Drupal core should be migrated to C rather than remain as "a PHP script". There is something very cool about C. I should dig my old books out! :-)
HTML5 is Here!
Jeremy Keith gave a wonderful keynote on HTML5, why it is like it is and what happened to xhtml 2.0. Parts were reminiscent of the RSS wars, but mostly I was impressed by the HTML 5 Design Principles which favour a working Web rather than a theoretically pure (XML-based) one. The talk is well worth a watch if you're interested in such things and I felt reassured and inspired by the practical and pragmatic approach outlined. I can't decide if I should start to implement HTML5 in our interface or not, but given that 5 is broadly compatible with the hotchpotch of HTMLs we all code in now, I suspect this migration will be gentle and as required rather than a brutal revolution.
Responsive Design
I often feel I'm a little slow at finding things out, but I don't think I was the only person in the audience to have never heard about responsive Web design, though when you know what it is, it seems the most obvious thing in the world! The problem with the Web has long been the variation in technology used to render the HTML. Different browsers react differently and things can look very different on different hardware - from large desktop monitors, through smaller screens to phones. Adherence to standards like HTML5 and CSS3 will go a long way to solving the browser problem, but what of screen size? One way would be to create a site for each screen size. Another way would be to make a single design that scales well, so things like images disappear on narrower screens, multiple columns become one, etc.
Though not without its problems, this is the essence of responsive design and CSS3 makes it all possible. Still not sure what I'm on about? dconstruct was given as a good example. Using a standards compliant browser (ie. not IE! (yet)) shrink the browser window so it is quite narrow. See what happens? This kind of design, along with the underlying technology and frameworks, will be very useful to our interface so I probably need to look more into it. Currently we're working with a screen size in mind (that of the reading room laptop) but being more flexible can only be a good thing!
There were so many more interesting things but I hope this has given you a flavour of what was a grand conference.
Wednesday, 1 September 2010
The Case for Digital Preservation
Now, I'm pretty sure there is no need for me to make the case to the good readers of this blog, but if you're ever stuck for something to say about why your work is important - for example at parties - then the demise of the print edition of the OED seems a good candidate!
OK, so no one is about to ditch the Pocket version, or even the Shorter (I got one of those for a graduation present from my Grandma!), but even so...
The last print OED was published in 1989. I imagine, given the regular updates to the OED online, that there has been a substantial influx of words since 1989 and I guess (given how Chaucer looks now) English will undergo some significant changes in the future. Unless we (the DP community) decide to preserve the digital OED, we will condemn readers of 2489 to struggle on with an antique 1989 print copy and much will they wonder when they don't find things like "Internet"...
(Mind you, the electricity might have all run out by then so it wont really matter...)
On the flip side, and no doubt something someone at the party will point out, this is also a case for continuing to print the OED - at least a few copies, kept in safe places... ;-)
OK, so no one is about to ditch the Pocket version, or even the Shorter (I got one of those for a graduation present from my Grandma!), but even so...
The last print OED was published in 1989. I imagine, given the regular updates to the OED online, that there has been a substantial influx of words since 1989 and I guess (given how Chaucer looks now) English will undergo some significant changes in the future. Unless we (the DP community) decide to preserve the digital OED, we will condemn readers of 2489 to struggle on with an antique 1989 print copy and much will they wonder when they don't find things like "Internet"...
(Mind you, the electricity might have all run out by then so it wont really matter...)
On the flip side, and no doubt something someone at the party will point out, this is also a case for continuing to print the OED - at least a few copies, kept in safe places... ;-)
Friday, 27 August 2010
Homes for old software
The more hybrid archives we work with, the more obvious it becomes that we need access to repositories of older software (or 'abandonware'). For older formats you often find that not only is the creating software obsolete, but any migration tool you can dig up is pretty out-of-date too. Recently I used oldversion.com to source older versions of CompuServe and Eudora to transform an old CompuServe account to mbox format with CS2Eudora. The oldversion site is really valuable and we could use more like it, and more in it. The trouble is, collecting and publishing proprietary 'abandonware' seems to be a bit of a grey area.
In 2003, the Internet Archive obtained some exemptions from the Digital Millennium Copyright Act (DCMA) that has allowed them to archive software, but this has to be done privately with the software being made available after copyright expiry. Not much help now, but promising for the long-term. The best thing that could happen (from an archivist's point of view) is that individuals and companies formally rescinded their interests in older software and put them in the public domain. Ideally they would put an expiry date into the initial licence before the software becomes abandonware.
I'm curious to hear about other good abandonware sites, especially ones that include 'productivity software' (our focus is here rather than gaming!). The Macintosh Garden is a good one, and Apple themselves also provide access to some older software, like ClarisWorks. What else is out there that we should know about?
In 2003, the Internet Archive obtained some exemptions from the Digital Millennium Copyright Act (DCMA) that has allowed them to archive software, but this has to be done privately with the software being made available after copyright expiry. Not much help now, but promising for the long-term. The best thing that could happen (from an archivist's point of view) is that individuals and companies formally rescinded their interests in older software and put them in the public domain. Ideally they would put an expiry date into the initial licence before the software becomes abandonware.
I'm curious to hear about other good abandonware sites, especially ones that include 'productivity software' (our focus is here rather than gaming!). The Macintosh Garden is a good one, and Apple themselves also provide access to some older software, like ClarisWorks. What else is out there that we should know about?
Saturday, 21 August 2010
Tuesday, 17 August 2010
Balisage 2010 The Markup Conference
Balisage 2010 The Markup Conference was
preceded by the International Symposium on XML for the Long Haul Issues in the Long-term Preservation of XML which opened with:
A brief history of markup of social science data: from punched cards to “the life cycle” approach covering the “25-year process of historical evolution leading to DDI, the Data Documentation Initiative, which unites several levels of metadata in one emerging standard.”
Sustainability of linguistic resources revisited looked at some of the difficulties facing language resources over the long-term.
Report from the field: PubMed Central, an XML-based archive of life science journal articles provided insight into the processes deployed to give public access to the full text of more than two million articles.
Portico: A case study in the use of XML for the long-term preservation of digital artifacts discussed some practices that can help assure the semantic stability of digital assets.
The Sustainability of the Scholarly Edition in a Digital World explored the need for “ tools to make XML encoding easier, to encourage collaboration, to exploit social media, and to separate transcriptions of texts from the editorial scholarship applied to
them”.
A formal approach to XML semantics: implications for archive standards examined whether “The application of Montague semantics to markup languages may make it possible to distinguish vocabularies that can last from those which will not last”.
Metadata for long term preservation of product data discussed the “valuable lessons to be learned from the library metadata and packaging standards and how they relate to product metadata”.
The day concluded with Beyond eighteen wheels: Considerations in archiving documents represented using the Extensible Markup Language (XML) which contemplated “strategies for extending the useful life of archived documents”.
Sessions in the main conference 2010 – covered topics such as :
gXML, a new approach to cultivating XML trees in Java which proposed “A single unified Java-based API, gXML, can provide a programming platform for all tree models for which a “bridge” has been developed. gXML exploits the Handle/Body design pattern and supports the XQuery Data Model (XDM)”.
Java integration of XQuery — an information unit oriented approach explored “a novel pattern of cooperation between XQuery and Java developer? A new API, XQJPLUS, makes it possible to let XQuery build “information units” collected into “information trays”.
XML pipeline processing in the browser discussed the benefits that providing XProc as a Javascript-based implementation would offer comprehensive client-side portability for XML pipelines specified in XProc.
Where XForms meets the glass: Bridging between data and interaction design explored using XForms which offers a model-view framework for XML whilst working within the conventions of existing Ajax frameworks such as Dojo as a way to bridge differing development approaches,data-centric versus starting from the user interface .
A packaging system for EXPath demonstrated how to adapt conventional ideas of packaging to work well in the EXPath environment. “EXPath provides a framework for collaborative community-based development of extensions to XPath and XPath-based technologies (including XSLT and Xquery)”.
Processing arbitrarily large XML using a persistent DOM covered moving the DOM out of memory and into persistent storage offering another processing option for large documents, by utilising, an efficient binary representation of the XML document that has been developed, with a supporting Java API.
Scripting documents with XQuery: virtual documents in TNTBase presented a virtual-document facility integrated into TNTBase, an XML database with support for versioning. The virtual documents can be edited, and changes to elements in the underlying XML repository are propagated automatically back to the database.
XQuery design patterns illustrated the benefits that might extend from the application of meta design patterns to Xquery.
Monday, 9 August 2010
Any old tapes - a true story - part 1
My neighbour is a self-employed architect. He has worked digitally for at least ten years and now most of his work is done on either his old (but still perfectly serviceable) ThinkPad or a shiny new desktop PC. He works with a couple of different CAD packages along with some tax software and MS Office, all on Windows XP.
Recently, knowing what I do for a living, he asked if I could help with a problem he was having retrieving files from an external hard drive and, being easily persuaded by the promise of food and wine, I agreed to try to help (with all the usual caveats about probably not knowing anything about it all!).
We got the disk drive working quickly (this is often the way when solving other people's computer issues. Sit with them and they'll solve it themselves!) and so he asked me about his backups too - which should have been happening regularly to another external drive, but were not. I checked out the drive and found an old directory with a very uninformative name that contained some data files and a few manifests that didn't make much sense. I've forgotten the name already, but he told me this was the name of the backup software. Searching, this software was not on the PC. The new PC had been recently built on the basis of the old one by an outsourced IT support. They'd done a good job restoring the software, etc. but this one backup program (a commercial one) was missing.
The consequences where two-fold:
1) No backup was running
2) the data files (about 1.4GB worth) and manifest were, without the software, entirely unreadable.
My neighbour thought perhaps the backup software was about so he'd ask the IT support to install and configure it. I fired up MS Windows Backup (the first time I've ever used it - it seems OK) and ran a one off backup of his work, just to be on the safe side and suggested he ask his support about that (one thing you must never do is undo or override the work of the real support person!) too - it required a password to add it to Windows scheduler.
After it completed, he astutely asked where the files had gone, and so I showed him, on the external drive and was dismayed to find that Windows Backup had also dumped all the files into a 1.4GB (proprietary?) container. I wondered if we'd ever have to extract files from Windows Backup files and made a mental note to keep a copy of the software (bundled with XP) in the cupboard just in case! Worse, it was then impossible to reassure him that the files were there without a crash course in Windows Restore. Still, I remember MS Backup and Restore being a pain way back to MS-DOS! :-)
As we finished our wine and talked about these things, he seemed to suddenly remember my job and jumped up, rummaged in a cupboard. He pulled out an old tape cartridge:
Once his main backup media, but, like the files on the external drive, no longer usable. This time both the hardware and the software were long gone. He didn't seem worried - the files has probably been migrated off his old machine to the new one at some point - but still he wondered what was on it and said "I don't suppose it is readable now is it?". He hadn't meant it as a challenge, but I couldn't resist! I convinced him to let me take the tape with me and try to recover his data - all in the name of digital archaeology, of course!
My next post will be my first adventures in the land of the Travans...
Recently, knowing what I do for a living, he asked if I could help with a problem he was having retrieving files from an external hard drive and, being easily persuaded by the promise of food and wine, I agreed to try to help (with all the usual caveats about probably not knowing anything about it all!).
We got the disk drive working quickly (this is often the way when solving other people's computer issues. Sit with them and they'll solve it themselves!) and so he asked me about his backups too - which should have been happening regularly to another external drive, but were not. I checked out the drive and found an old directory with a very uninformative name that contained some data files and a few manifests that didn't make much sense. I've forgotten the name already, but he told me this was the name of the backup software. Searching, this software was not on the PC. The new PC had been recently built on the basis of the old one by an outsourced IT support. They'd done a good job restoring the software, etc. but this one backup program (a commercial one) was missing.
The consequences where two-fold:
1) No backup was running
2) the data files (about 1.4GB worth) and manifest were, without the software, entirely unreadable.
My neighbour thought perhaps the backup software was about so he'd ask the IT support to install and configure it. I fired up MS Windows Backup (the first time I've ever used it - it seems OK) and ran a one off backup of his work, just to be on the safe side and suggested he ask his support about that (one thing you must never do is undo or override the work of the real support person!) too - it required a password to add it to Windows scheduler.
After it completed, he astutely asked where the files had gone, and so I showed him, on the external drive and was dismayed to find that Windows Backup had also dumped all the files into a 1.4GB (proprietary?) container. I wondered if we'd ever have to extract files from Windows Backup files and made a mental note to keep a copy of the software (bundled with XP) in the cupboard just in case! Worse, it was then impossible to reassure him that the files were there without a crash course in Windows Restore. Still, I remember MS Backup and Restore being a pain way back to MS-DOS! :-)
As we finished our wine and talked about these things, he seemed to suddenly remember my job and jumped up, rummaged in a cupboard. He pulled out an old tape cartridge:
Once his main backup media, but, like the files on the external drive, no longer usable. This time both the hardware and the software were long gone. He didn't seem worried - the files has probably been migrated off his old machine to the new one at some point - but still he wondered what was on it and said "I don't suppose it is readable now is it?". He hadn't meant it as a challenge, but I couldn't resist! I convinced him to let me take the tape with me and try to recover his data - all in the name of digital archaeology, of course!
My next post will be my first adventures in the land of the Travans...
Friday, 16 July 2010
Drupal
I'm very excited! I just looked at a Web site that lists all the "high profile" folks using Drupal for the Web sites. I'm also nervous. The excitement and the nerves are linked. This is because I told Susan a few weeks ago that I would commit to Drupal as the front-end for our archival materials. So, there is a lot to live up to and I'm also stuck with a decision I made, so it is all my fault! No pressure! :-)
Why Drupal?
Well, when I was building the second incarnation of the archive interface (the first was a prototype put together by Susan), it started out as a bunch of Web pages and a Solr-based search engine. The back-end data was created using a combination of source data and metadata gleaned from the EAD catalogue, the output of FTK and a spreadsheet that was the result of some appraisal work by the archivist, all munged together by some Java code that did the transformations, created the thumbnails, etc.
As time moved on it became apparent that additional features would be nice to build into the interface. At least one of the Project Advisory Board members suggested it would be nice to see a more Web 2.0-like features and I've long thought that having reader-generated tags and (perhaps) comments attached to the manuscripts might be a nice idea. Other features also arose, and soon I realised that I'd have to either build database-driven site to make all this happen (which I suspect would've been rather ropey) or, far more sensibly, use one that already existed.
By wonderful coincidence (though the kind of thing that often happens) I saw some emails on the Fedora lists about Islandora. Secretly harbouring a desire to visit Prince Edward Island ;-), I took a closer look and it was there that I chanced on Drupal and it seemed to fit the bill quite nicely, offering comments, tagging, types of content, and user management. Further, it is extensible, has a bewildering, if full of promise API, and will hopefully mean I can build a "publication pathway" that interfaces with the preservation store (indirectly) and can be managed by the archivists in a nice Web-friendly way.
Does the excitement and the nerves start to make more sense now?
It is still early days, but I have re-factored the Java code to output content (fixing a major memory leak in the process!) suitable for import into Drupal and have developed a module that imports that content, including the structure of the collection as Collection - Shelfmarks - Items. It aint much to look at just yet, but it is getting there.
As I have further adventures in Drupal-land I'll keep you updated!
Have a lovely weekend!
Why Drupal?
Well, when I was building the second incarnation of the archive interface (the first was a prototype put together by Susan), it started out as a bunch of Web pages and a Solr-based search engine. The back-end data was created using a combination of source data and metadata gleaned from the EAD catalogue, the output of FTK and a spreadsheet that was the result of some appraisal work by the archivist, all munged together by some Java code that did the transformations, created the thumbnails, etc.
As time moved on it became apparent that additional features would be nice to build into the interface. At least one of the Project Advisory Board members suggested it would be nice to see a more Web 2.0-like features and I've long thought that having reader-generated tags and (perhaps) comments attached to the manuscripts might be a nice idea. Other features also arose, and soon I realised that I'd have to either build database-driven site to make all this happen (which I suspect would've been rather ropey) or, far more sensibly, use one that already existed.
By wonderful coincidence (though the kind of thing that often happens) I saw some emails on the Fedora lists about Islandora. Secretly harbouring a desire to visit Prince Edward Island ;-), I took a closer look and it was there that I chanced on Drupal and it seemed to fit the bill quite nicely, offering comments, tagging, types of content, and user management. Further, it is extensible, has a bewildering, if full of promise API, and will hopefully mean I can build a "publication pathway" that interfaces with the preservation store (indirectly) and can be managed by the archivists in a nice Web-friendly way.
Does the excitement and the nerves start to make more sense now?
It is still early days, but I have re-factored the Java code to output content (fixing a major memory leak in the process!) suitable for import into Drupal and have developed a module that imports that content, including the structure of the collection as Collection - Shelfmarks - Items. It aint much to look at just yet, but it is getting there.
As I have further adventures in Drupal-land I'll keep you updated!
Have a lovely weekend!
Subscribe to:
Posts (Atom)
Posts
