Friday, January 09, 2009
TIFF's Catch 22
Section 22 of the TIFF 6.0 specification (PDF), on JPEG compression, has been a subject of ongoing controversy. The problems with it are discussed in Draft Tech Note 2 at remotesensing.org.
The major problem cited is that the tags defined in Section 22 require detailed understanding of the JPEG encoding. Even an application which simply modifies tags in a TIFF file, without any expectation of decoding JPEG codestreams, must parse the data which JPEG tags refer to in order to preserve it. For example, tag 521, JPEGACTables, points to a list of offsets to Huffman AC tables, whose format is given as follows:
16 BYTES of “BITS”, indicating the number of codes of lengths 1 to 16;
Up to 256 BYTES of “VALUES”, indicating the values associated with those codes, in order of length.
There is a similarly incomplete description of tag 520 (JPEGDCTables). This isn't sufficient information to determine the length of the table. The best a JPEG-unaware application can do, if it has to move the table, is to allocate 272 bytes for the copy of an AC table or 33 bytes for a DC table. If this were the only problem, it could be solved by assuming the maximum length for these blocks when copying them. However, various ambiguities in the section have been claimed, so that different implementations aren't compatible with each other.
What is clear, though, is that there's confusion in the TIFF world on how to handle JPEG compression. Tech Note 2 goes on to recommend an alternate specification. Adobe has adopted this alternative (or something very close to it) in its Photoshop TIFF Technical notes (PDF), but without rescinding Section 22. Undoubtedly there are still Section 22-based TIFF files in use and at least a few still being created. (Most TIFF files don't use JPEG compression at all, so this isn't a widespread problem.)
The basic problem is that Adobe hasn't revised the TIFF specification since 1992. Whatever problems there are with it look as if they'll remain the official standard forever. This is particularly a problem from the standpoint of digital preservation. "Real" TIFF files should follow the standard, not a technical note which (at least in name) is intended for a particular application. But this approach isn't realistic when so many files exist that depend on the Photoshop Notes. It has to be considered part of the specification.
JHOVE, by the way, deals with both sets of tags. It doesn't analyze the content of JPEG2000 blocks or modify files, so it doesn't run into the problems described.
Labels: TIFF
Tuesday, September 02, 2008
TIFF security
There's a furious discussion in progress on the TIFF mailing list on the security, or lack thereof, of libtiff. The general consensus is that libtiff isn't hardened against hostile files.
No major browser supports TIFF images, using libtiff or anything else, but creators of web applications that convert TIFF files for display should be wary of security issues. Code for displaying traditional Web formats such as GIF, PNG, and JPEG is relatively safe because it's stood the test of many attacks and most of its security-related bugs have been fixed by now. Code for displaying TIFF (libtiff gets singled out only because it's so widely used, not because it's less safe than alternatives) deals with a complex format, and there are many opportunities for false sizes and out-of-bounds pointers to lure an application into executing data.
The problem is that the most efficient code for computation-intensive tasks is written in C, which has often been called a high-level assembly language. It's efficient because it doesn't have any features to protect the code from running past the end of an array or using arbitrary pointers, so it's inherently hard to make C code safe.
One solution is to permit only trusted TIFF files. There's always a chance a hostile file can sneak in, though. None of the solutions are perfect. The same considerations apply to processing any other file format for the Web.
(This is, in part, a note to self for future projects.)
Labels: TIFF
Thursday, July 10, 2008
Update on BigTIFF
Aware Systems' BigTIFF page has just had a significant update. Previously it had strong warnings that the design was still in flux. It now states:
Whilst BigTIFF is not yet an actual official standard, there were quite a few people involved with developing the BigTIFF draft specification, including people from Adobe, and there have been no changes since that time. There are at least two independent implementations (our own AWare Systems AsTiff, and LibTiff since version 4.0). Furthermore, BigTIFF is a small and quite natural progression from TIFF. It seems highly unlikely that an organization with some apparent position of authority (i.e. a standards organization or Adobe) will produce a document on their own that contradicts this draft specification at this stage. We thus feel we can safely recommend adoption of BigTIFF now.
This seems to be more a matter of finally declaring it done (after nudging by me and several other people on the TIFF list) rather than any flurry of recent activity.
The formal description of BigTIFF -- to the extent we can call it "formal" -- is found here. Some issues, notably what MIME type to use, are still open.
There has been talk on the TIFF discussion list of sticking with "image/tiff". However, doing that would contradict IETF RFC 3302, which defines the type in terms of TIFF 6.0. The changes to BigTIFF from TIFF 6.0 are minor in a certain sense, but no TIFF reader following the TIFF 6.0 definition will accept a BigTIFF file. Registering "image/tiff-big" would require publication of a formal standard, and that isn't likely to happen. My own suggestion is informally adopting "image/x-tiff-big".
Anyway, congratulations to all those who have brought BigTIFF to this stage!
Related link: LibTIFF. Beware of imitations such as www dot libtiff dot org!
Thursday, April 10, 2008
The state of BigTIFF
A post by Bob Friesenhahn to the TIFF mailing list says: "The sponsored BigTIFF effort has come to completion. Libtiff4 is now in a more gradual 'maintenance' state which will hopefully lead to a stable release."
I hope this maintenance state isn't a euphemism for stagnation.
Related links:
Monday, July 23, 2007
New TIFF links site
Links to an assortment of information on TIFF can be found at a new site, tiffcentral.com.
Labels: TIFF
Thursday, July 05, 2007
Is BigTIFF TIFF?
An argumentenergetic discussion is developing on the TIFF mailing list on whether BigTIFF is a new version of TIFF or a new file format. There are good reasons for calling it a new format.
At the most obvious level, no existing application to read TIFF files can read any BigTIFF file. The TIFF specification says that bytes 2-3 contain "An arbitrary but carefully chosen number (42) that further identifies the file as a TIFF file." BigTIFF files contain the value 43 in this location. The TIFF spec says that bytes 4-7 contain "The offset (in bytes) of the first IFD." BigTIFF uses these locations for other purposes, and places the offset of the first IFD starting at byte 8. In addition, a tag structure is 12 bytes long in TIFF; it's 20 bytes in BigTIFF, to accommodate larger counts and offsets.
The BigTIFF proposal calls the value at offset 2 a "version number." But according to the TIFF specification, it isn't a version but a fixed identifier. Putting a different number there is a declaration that a file isn't TIFF.
File formats do change over time, and inevitably some files based on newer versions will not be readable by software based on older versions. But it's reasonable to expect that the newer files be recognizable as incompatible or broken instances of the format. When the format's signature information, other than the part which identifies the version, changes, it's no longer the same format.
BigTIFF does have a lot in common with TIFF, and it should be possible to modify a well-written TIFF reader to read BigTIFF without too much trouble. All tag values and definitions from TIFF are retained.
It's unfortunate that TIFF wasn't designed to include version information. The only way to tell a TIFF 6 file from a TIFF 4 file is by the features (tags and data types) it uses. This puts BigTIFF in a difficult situation. But from the standpoint of format identification, it makes more sense to call BigTIFF a new format derived from TIFF than to call it TIFF, and to give it its own MIME type and file extension(s).
The decision is actually in the hands of Adobe, which hasn't shown much interest in the TIFF standard in a long time. If Adobe does nothing, then the new format isn't TIFF, since it's fundamentally different from any Adobe-approved specification. Otherwise the decision is legally in Adobe's hands.
Monday, July 02, 2007
Bogus libtiff site
It's been pointed out on the TIFF mailing list that libtiff . org contains a long-outdated version of the libtiff library and is in untrustworthy hands. The location of the real libtiff library is at remotesensing.org. Frank Warmerdam writes:
Due to a slip up on our part the dns entry libtiff.org fell into the hands of DNS pirates and was sold to the current owner for several thousand euros to harvest it's positive google karma for a link farm.
It is out of our control. Really. Honestly. And I'd rather not angst over it any more.
About all that's to be done about it is to avoid linking to it, and to occasionally remind people that it's not a suitable place for getting libtiff code. Since it comes up first when you search in Google or Clusty for "libtiff", it is important to remember this.
Labels: TIFF
Monday, May 07, 2007
BIIIG TIFF!
Aperio Technologies has announced support for BigTIFF. It's not clear what this means, as BigTIFF is still in the proposal stage. The announcement says that "Aperio has donated these enhancements to the public domain, and is working with the TIFF standards body to incorporate them into a future standard release."
As a demonstration, Aperio has created what it claims is "the world's first terapixel image." Compressed, it's a mere 143 gigabytes, and the inspiring image which they chose for the occasion is a breast cancer slide replicated 225 times. It might take a while to download.
Labels: TIFF
Tuesday, January 02, 2007
BigTIFF
BigTIFF is a modified version of TIFF which has 64-bit offsets, allowing larger files. A project to implement it in LibTiff now has the sponsors needed to get the work done. Work in LibTiff 4.0, which will support BigTIFF, is planned to start in March.
The proposal -- and its creators stress that it's still a proposal, not a settled specification -- can be found here.
Labels: TIFF
Wednesday, December 06, 2006
Grayscale in TIFF
Grayscale. That's a nice black-and-white concept, isn't it? Maybe the truth is shades of gray; see the discussion on the TIFF list.
Tags: tiff
Labels: TIFF
Tuesday, August 29, 2006
Buggy TIFF writers
There is an interesting discussion on the TIFF developer mailing list about buggy tags. Part of the discussion on the IPTC tag, a private tag for IPTC (International Press Telecommunications Council) format metadata, is on the list archive, though the archive usually lags a few days behind. From the opening message:
For starters, the IPTC specification clearly shows how the data is organised in meaningful units of variable bitlength. The *single* correct way to encode that, is using a tag of datatype undefined, and passing the bytecount as tag count (i.e. length of datatype undefined is actually defined, it's the same length as byte).
However, some bozo somewhere in the early days of IPTC in TIFF decided to write the stuff as long datatype, with a count such that the tag data length is the length of the IPTC block padded with up to 3 zero bytes.
Adobe itself seems to be one of the violators. Good reading if you're implementing a TIFF reader or writer and need to get into the finer points, or just if you want to understand why some of us aren't convinced TIFF is a standard.
Labels: TIFF
Friday, June 23, 2006
When is black white?
Here's an interesting piece on the interpretation of the CCITT compression standard in TIFF, from the TIFF mailing list archive, by TIFF expert Joris Van Damme.
Labels: TIFF
