Letterlocking is a technique of folding and sealing letters so they cannot be read by prying eyes. I'm working with collaborators at MIT and the University of London to computationally analyze CT scans of sealed letters from 18th century Europe and virtually unfold them. The iron-gall ink used in these letters creates a nice contrast against the paper in the CT scanner, making it possible to read the ink off the page.

Recently, a similar technique has been used to unroll and read text on an ancient, damaged biblical scrolls found near the Dead Sea and papyrus scrolls found near Pompeii.

For the locked letters project, we are interested not only in reading the contents of the sealed letters, but also in analyzing and categorizing their folding patterns to better understand how this tradition evolved over time.

A fly-through of a sample CT dataset of locked letters:

Text is clearly visible, but even the relatively flat regions of the letters may be slightly curved in space, making reading them non-trivial.

Figure 1: Two example slices from the same dataset containing 11 letters.

Figure 1 shows slice #1114 (left) and slice #1598 (right) from another sample dataset containing eleven letters. The letters are separated from each other with foam and contained within a circular plastic container. For the most part, the letters are composed of many flat planes of paper, but at the folds the surfaces of the letters becomes highly curved. Pages are 2-3px wide and generally have some empty space between them, but in some regions of the data the gap between pages shrinks down to <1px (Figure 2).

Figure 2: A closeup of Figure 1 that shows some of the challenges in working with this data. Page separation visibility may be affected features of the letter such as ink (Letter 8) or dense seals (Letter 2) that scatter light into neighboring regions. Also in slice #1598, Letter 3 begins to show pages angled relative to the cross-sectional plane, resulting in pages appearing thicker in the slice.

Our current goal is to construct a mesh of points that define the folded surfaces of a letter with subpixel accuracy. One we have this mesh, we can unfold it and determine a mapping from the 3D CT data to a flat, unfolded state. From there we can reconstruct the letter in its unfolded state, read its contents, and characterize its folding pattern.

In order to construct this mesh, we first need to detect points that lie in the paper. This detection method should be unbiased to paper orientation so that is robust to differences in orientation of the letters in the CT scanner, folding patterns, and folded vs flat regions of the letter topology. This method should also take advantage of prior knowledge about the data set, namely, that the paper has a constant, known thickness.

What is the goal?

I would like to explore using constrained optimization techniques to identify individual layers of the letters in the CT dataset.

How will you evaluate progress?

I will compare these techniques against the mixed gaussian modeling and gradient-based image processing methods we've been using so far, by running them all on a ground truth data set, where the number of layers in the scan are known. Our current methods correctly identify the layers in a slice of the CT scan about 75% of the time - leaving more work to the linking algorithm and limiting the size of sections we are able to extract.

What is the prior art?

The papers I linked to above use local information in small regions of the CT scans to identify geometric markers and grow meshed geometry out from these starting points. This is the direction we have been exploring so far in our own dataset.

What techniques will you use?

By assuming an underlying sparse representation of the object used to generate the scan data (a folded 2D sheet), I want to use techniques from compressed sensing to find solutions that fit the data best. I think I will first start with some 1D and 2D section of my data.

What questions will you need to address?

What underlying model/representation do I use to represent the folded letter geometry?
How do I extend the compressed sensing article above to 2D and 3D?