Behind the scenes: fully indexing the birth records with data manager Richard Jackson

I joined findmypast.co.uk at the start of the year and walked straight into my baptism of fire: reindexing the birth, marriage and death records.

As head of the data team, I am responsible for the quality of the data we receive from two separate transcription companies. I have to ensure that they meet their guarantees of quality so that everything falls into place in time for the launch.

Richard Jackson

We received the files in quarterly batches, usually with one file per quarter - for the births alone this amounted to more than 600 files and 113 million records. The challenges came in verifying that we were not missing any records and ensuring that all of the 800,000 images were in place and of high enough quality and that we could identify and standardise any fields that had been transcribed in error.

We shared knowledge with the transcription companies, provided them with lookup tables for valid districts, common first and last names and provided regular feedback so they could validate their transcriptions before delivering them to us. This ensured that we were as close as possible to our desired accuracy levels before we handled the data ourselves.

That said, there was still plenty of work to be done. By programmatically checking the files we received for gaps and inconsistencies, e.g., comparing the representation of first letters of surname across quarters, we were able to identify and plug plenty of gaps well before the project neared completion.

One of the most time consuming parts of the births project was cleaning up the registration districts from their incorrectly transcribed values into something that could be used in a search. Our list of invalid districts included over 60,000 incorrect values which all had to be standardised. My colleague Francesca Aiyeola and I spent many hours trying to work out if that ‘B’ should have been an ‘R’ or an ‘H’ and acquired a fine appreciation for the transcribers’ skills and patience in the process!

We hope that you enjoy the birth records and that you’re looking forward to the fully indexed marriages and deaths, coming soon.

5 Responses to “Behind the scenes: fully indexing the birth records with data manager Richard Jackson”

1. Caroline Gurney says:

   20 August 2010 at 9:24 am

   Very interesting. Thanks for sharing this. May I ask where the transcribers are based - UK or abroad? This has long been an issue for family historians because overseas transcribers do not have the inbuilt knowledge of British personal and place names which UK based transcribers have. No lists you supply, however comprehensive, can make up for that lack of native “feel” for what is right. As an example, I recently submitted a correction to FMP for a census entry where the street name had been transcribed as “Bessenbrook”. In fact, the census entry was “Bessboro” - short for Bessborough. As a native of these islands I know that “boro” is a common shortening of “borough” and that “Bessborough” is a fairly common street name, derived from an aristocratic title. I have never heard of “Bessenbrook” and Googling the name shows why - only two hits, both of which refer to the incorrect FMP census entry.

   Reply
   • jessmoore says:

     20 August 2010 at 10:35 am

     Hi Caroline,

     We’re glad you enjoyed the blog. We use professional transcribers from the UK and all over the world: China, the Philippines and Bangladesh, for example. Keep reporting any errors you find – we have a very high standard of accuracy but inevitably a small amount of errors slip through. Thanks!

     Reply
2. Denise Roberts says:

   19 January 2011 at 4:08 am

   Hi guys,

   I have recently taken out a 12-month subscription as I really enjoy using your website. I also love that I can submit corrections to the inevitable transcription errors in the census records, thus helping to make your data even more accurate. Are you planning to make this service available for the BMD indexes as well? (I’ve already found one little mistake that I would love to correct!)

   Reply
3. jessmoore says:

   19 January 2011 at 9:47 am

   Hi Denise,

   You can report transcription errors in our BMD records via the original image. The reason we designed it in this way is so you can report the error once you’ve confirmed it’s incorrect by viewing the original image.

   Thanks and we appreciate your diligence!

   Reply
4. Mary Brennan says:

   2 March 2011 at 11:19 pm

   Hi,

   It’s a shame that my sons birth record is still transcribed incorrectly, even though I’ve been repeatedly in touch with Claudia, Liam and Daniel of the FMP support team for FOUR MONTHS, who have continually assured me that it will be corrected. (And yes, I did report it several times via the original image, before contacting the support team.)

   I’m now at the point of being so disappointed, that I will not be renewing my subscription, and will go to Ancestry.com instead.

   Mary

   Reply