Category Archives: Formats

  • SDK’s and Libraries

    In our final post this week, let’s get an overview of off-the-shelf tools for developers to properly access formats and enable standard functionality.   

    Format documentation allows software developers to work effectively with a particular format. But there are other methods which are even more attractive for someone building a media app. Software Development Kits (SDK) and other software libraries are like mini-applications that can perform functions without requiring a developer to write the code from scratch. 

    For instance, camera manufacturers who use undocumented raw formats will typically publish an SDK that allows an application to properly read the raw files. Software applications and operating systems can incorporate the SDK into their application. This allows the camera manufacturer to have greater control over how the file is rendered. However, when software developers rely on manufacturer’s SDKs, they may also build in obsolescence since the SDK is not guaranteed to work with future operating systems.

    If you want to take a look at an SDK, you can download the Adobe DNG SDK here

    Software Libraries 

    Many software developers also depend on third party applications and software libraries to help decode media formats. In many cases, these apps and libraries are open source tools that are provided free of charge. This arrangement centralizes the development and allows it to be used for many different applications. If you work with media, then you probably use at least several of these all the time. They are embedded in many applications and services. 

    • EXIFtool is an open source command line application and set of libraries written by Phil Harvey. It can read and write metadata in many different file formats. It is used by many different applications to properly handle metadata. 
    • dcraw is an open source command line application that is written to open hundreds of different types of raw files. It is written by Dave Coffin and made available for free. The dcraw rendering engine is used in many free or inexpensive applications like RawTherapee and darktable. It is one reason that the many flavors of raw file continue to be so accessible. 
    • The XMPtoolkit is a software library provided by Adobe to properly handle reading and writing XMP metadata in images and other file types. 
    • FFmpeg is an open source suite of libraries and programs that can properly decode a vast number of video file formats. It is used by a large number of web services that support video and by some desktop applications. 
  • Format Structures

    In today’s post, we dive into the structure of image formats. We look at the purposes, specifications and standardization. 

    Formats are a way to stack up the bits in a file. These bits represent the components of a digital object. The format specifies a method to save all this data so it can be understood by various software programs. There are two traditional functions that the format can specify, and a third one I’m introducing here. 

    • Container formats – All the formats we will be discussing are, to some degree, container formats. This means that the format itself specifies a way to save multiple components into a single file. The exact way to encode the media object inside the container may be very tightly controlled or it may be only loosely specified. 
    • Encoding formats – Some media formats will also outline very specific ways to encode media. The JPEG format, for instance, provides a container for a few different components, but it primarily describes the options you must use to encode the media. To illustrate this (and make things a little more complicated),  JPEG encoding can also be used by other formats. 
    • Workflow formats – I’m coining this term to describe formats that go beyond what we typically understand as container or encoding formats. These formats are half media object and half project file. They can contain multiple images and other media. And they can contain program-specific instructions like a project file. They can also contain final output of the optimized image(s). In most cases, these additional capabilities are more specifically laid out than what you get with a general container format. 

    A bit of each

    Most formats for visual media specify both container and encoding methods, to varying degrees. 

    • Formats without container features cannot contain the components that are essential to modern media objects, except for very lightweight uses. For instance, GIF is used as a totally stripped down format for moving images. But it cannot contain metadata, so all provenance, rights and other other information cannot ride along with the file. 
    • Formats without any published encoding guidelines are typically too fragile and application-specific to be used outside of a single program. 

    As we examine specific formats, we will be able to see the balance between these two types of structure. 

    Format specification 

    Many formats will have some kind of written specification. This allows everyone to understand how the format is structured. By following this specification, files can be built that are compliant and therefore behave predictably. The specification tells software developers where and how the various components of a file are structured. 

    There is no universal way to write these specifications. Most specifications will have some type of narrative that helps you understand the purpose of the format and how to make use of it. Some specifications are extremely detailed, with many dozens of individual components that are all precisely spelled out. Some formats are quite loose, outlining the components but not providing a high level of detail. 

    Some formats do not have a published specification. Many camera raw formats, for instance, are not publicly documented. In order to work with these formats, software developers have to figure out the precise details of how each format is structured. Software developers will also use SDKs and other libraries to work with these files. As discussed below, undocumented formats present an increased danger of becoming unreadable in the future compared to ones that are well documented. 

    Links to some format specifications 

    If you’d like to geek out and actually read some format specifications, I’ve linked to some of them below. While they can be hard to understand, especially at first, some parts will be clear even to people without computer science degrees. (You might start with HEIF, since it’s less intimidating and more purpose-driven.)

    • The DNG specification is quite comprehensive and forward-looking. It is currently in its fifth version.
    • HEIF is much less mature, and the specification reflects that. But if you read the first part of the Technical Information page, you can see the objectives that the format is designed to address. It’s also useful to look at the examples on the github pages. 
    • The TIFF specification is a comprehensive and well-written document – for 1992, the last time it was officially updated. 
    • Video – If you really want to make your brain hurt, take a look at the MPEG-4 video spec. You can get an understanding of why video is so much harder than still images to standardize. 

    Components: Specified or freelance? 

    As we look at file formats, we will see there are a couple of ways to accommodate the need for complex image components. The TIFF format, for instance, has an extremely flexible structure allowing for the inclusion of nearly any kind of image component a software developer can dream up. 

    In later posts, we will see a different approach when we discuss DNG. Instead of an “anything goes” approach, DNG sets out some very specific requirements for image components and how to structure them within the file. (It also allows for flexibility, but lays out where and how to accomplish that). The more formalized approach was made necessary by the unruly proliferation of raw file variants and the prospect of many of them becoming unreadable. 

    I’m convinced that the more formalized approach to file formats as shown in DNG is the right way to go, at least for raw files. With more than 600 variants (and counting) available already, it’s essential to bring some order to the process. 

    In the next post, we will look at SDKs and libraries. These tools allow developers to properly access features supported by a format without having to write new code. 

  • Format Follows Function

    Now that we’ve defined the various characteristics of digital images, we can examine formats. We’ll outline the basic structures of file formats and their specifications. After that, we can look at specific file formats and see what kind of object each supports. 

    Purpose-driven

    Formats are developed to support a particular application, media type or process. To evaluate the suitability of a particular format, it’s useful to list the various classes of function that the format supports. Here are the most common options.

    Capture

    Some formats are specifically designed to facilitate image capture. The JPEG format was originally designed as a standardized format for digital cameras. Raw image formats are designed to serve the specific engineering needs of camera manufacturers. This may include special features and capabilities which are unique to a particular camera brand or model. . 

    Edit

    Optimizing images presents a series of challenges that are very different from the ones posed by capture. This can include the need to merge multiple images, to work on selected areas in an image, and/or replace pixels. Most importantly, formats for editing need to allow for reversible changes. TIFF and PSD formats have become the formats of choice for this kind of work because of their flexibility to include all kinds of components in the file. And DNG has become the most capable raw file editing format because of the components that it supports. HEIF is potentially ushering in a new era of edit formats. 

    Delivery

    Traditionally, file delivery puts a premium on predictable rendering through unambiguous color and tonal reproduction. This is usually coupled with either an emphasis on quality or on small file size. For traditional photos, a JPEG file is often the right choice for delivery if it is made with the proper level of compression.  In cases where a 16-bit file is required, a TIFF is a better delivery format. PNG is optimised for colors, hard edges and transparency, which makes it great for images with text or logos. For dimensional images or inherently connected images, you’ll need one of the more modern formats. 

    Archive

    When building an archive, it’s usually desirable to save the highest quality image in a standardized format. This can help ensure that the file renders properly and is accessible long into the future. TIFF is commonly understood to be the gold standard for conventional images, but a lossless JPEG or JPEG2000 can be just as good for 8-bit files. DNG offers archive functionality for raw images that is unmatched by any other format, including the ability to automatically check file integrity. And HEIF allows for storage of multimedia and dimensional objects that may be impossible with any other format. 

    Connectivity

    Some formats offer functions that enhance cloud connectivity in important ways. DNG supports editable proxy images that allow for distributed editing of raw images. And the HEIF format includes the ability to wrap application-specific proxies into a single file. 

    In the next post, we will examine what goes into the creation of a file format.