PDF Clown 0.1.0 — XRef streams, content rasterization and lots of good stuff

NOTE — On March 4, 2011 PDF Clown 0.1.0 has been released!

Hi there!

New features currently under development that will be available in the next (0.1.0) release:

1. Cross-reference streams and object streams
2. Version compatibility check
3. Content rasterization
4. Functions
5. Page data size (a.k.a. How to split a PDF document based on maximum file size)

It’s time to reveal you that I decided to consolidate the project’s identity (and simplify your typing life) changing its namespace prefix (it.stefanochizzolini.clown) in favor of the more succinct org.pdfclown: I know you were eager to strip that cluttering italian identifier! 😉

Last week I was informed that USGS adopted PDF Clown for relayering their topographic maps and attaching metadata to them. Although on a technical note it’s stated that its use will be only transitory, as they are converging toward a solution natively integrated with their main application suite (TerraGo), nonetheless its service in such a production environment seems to be an eloquent demonstration of its reliability. 8)

1. Cross-reference streams and object streams

After lots of requests, I’m currently busy on the development of cross-reference stream and object stream read/write functionalities [PDF:1.6:3.4.6-7]; in particular, stream reading has been partially based upon the code that Joshua Tauberer wrote some months ago while he was experimenting with PDF Clown on PDF files analysis for his US Congress activity tracker, GovTrack.

2. Version compatibility check

Working on cross-reference streams induced me to start supporting version-compatibility checking via annotations. This feature conveniently allows users to transparently control that the PDF files they are creating or modifying conform to a target PDF version (as specified in PDF file header) according to a configurable compatibility policy, defined through Document.Configuration.CompatibilityModeEnum — these are the alternative policies applicable:

• Passthrough: document’s conformance version is ignored; any feature is accepted without checking its compatibility.
• Loose: document’s conformance version is automatically updated to support actually used features.
• Strict: document’s conformance version is mandatory; any unsupported feature is forbidden and causes an exception to be thrown in case of attempted use.

Automatic compatibility checking is very handy as users can enforce generated PDF files’ conformance without manual intervention; for example, you don’t have to tweak your PDF file version to 1.5 if you plan to use the optional content functionality (OCG [PDF:1.6:4.10]), just sit back and see it be done! 🙂

3. Content rasterization

I’m quite impressed how naturally the existing model is integrating with PDF printing and image rasterization functionalities. Leveraging the existing model means that there’s a common infrastructure (see ContentScanner and ContentObject hierarchy) that serves disparate purposes (content creation, content analysis, content extraction, content rasterization, content editing, and so on), simplifying its understanding, use, maintenance and extension. I wanna stress that my goal is to come to an elegant viewer, NOT to a cumbersome retrofit component that’s added as an alien to fill the gap! 😉

Yes, I know these goodies had been outside my official plans for a long time, but during the last week of September, while crawling through the PDF Clown sources, I stumbled upon the above-mentioned ContentScanner and ContentObject hierarchy: I realized they were just ready for supporting content rendering, so I thought “What are we waiting for? Let’s do it!”… but don’t expect that 0.1 will deliver a full-fledged PDF viewer and printer — I’ll start prototyping the most basic graphics primitives such as space coordinates transformations, path drawing, color selection and so on. Advanced operations such as glyph outline drawing will necessarily appear afterwards. Anyway, I’m confident that at the end of the development process it will be possible to print and display PDF pages (and even independent parts of them such as external forms) along with their thumbnails.

The figure below compares an example of PDF Clown’s current rasterization capabilities (on the left, via Java 2D graphics) with its equivalent generated by Adobe Reader (on the right). As you can see, path drawing is highly conformant with the reference implementation, while no text rendering has been implemented yet.

Creating this figure was absolutely trivial — here it is the code sample used (line 34 executes the actual rendering of the first page of the document):

package org.pdfclown.samples;

import java.awt.Dimension;
import java.awt.image.BufferedImage;
import java.io.IOException;

import javax.imageio.ImageIO;

import org.pdfclown.documents.Document;
import org.pdfclown.files.File;
import org.pdfclown.tools.Renderer;

public class ContentRenderingSample
  extends Sample
{
  @Override
  public boolean run(
    )
  {
    String filePath = promptPdfFileChoice("Please select a PDF file");

    // 1. Open the PDF file!
    File file;
    try
    {file = new File(filePath);}
    catch(Exception e)
    {throw new RuntimeException(filePath + " file access error.",e);}

    // 2. Get the PDF document!
    Document document = file.getDocument();

    // 3. Rasterize the first page!
    Renderer renderer = new Renderer();
    BufferedImage image = renderer.render(document.getPages().get(0),new Dimension(1400,850));

    // 4. Save the rasterized image!
    try
    {ImageIO.write(image,"jpg",new java.io.File(getOutputPath() + java.io.File.separator + "ContentRenderingSample.jpg"));}
    catch(IOException e)
    {e.printStackTrace();}

    return true;
  }
}

As you can see in the following code chunk, Renderer.render(…) method takes care to prepare the target graphics context [line 31] delegating its rendering to the chosen content context [line 32] (that is, in this case, a Page object):

package org.pdfclown.tools;

import java.awt.Dimension;
import java.awt.image.BufferedImage;

import org.pdfclown.documents.contents.IContentContext;

/**
  Tool for rendering {@link IContentContext content contexts}.

  @author Stefano Chizzolini (http://www.stefanochizzolini.it)
  @version 0.1.0
  @since 0.1.0
*/
public class Renderer
{
    . . .

    /**
      Renders the specified content context into an image context.

      @param contentContext Source content context.
      @param size Image size expressed in device-space units (that is typically pixels).
      @return Image representing the rendered contents.
    */
    public BufferedImage render(
      IContentContext contentContext,
      Dimension size
      )
    {
      BufferedImage image = new BufferedImage(size.width,size.height,BufferedImage.TYPE_INT_BGR);
      contentContext.render(image.createGraphics(),size);

      return image;
    }
}

The Page object then delegates its own contents rendering to ContentScanner [line 36], which sequentially scans every graphics operation producing its corresponding raster representation:

package org.pdfclown.documents;

import java.awt.Graphics2D;
import java.awt.geom.Dimension2D;
import java.awt.print.Printable;

import org.pdfclown.PDF;
import org.pdfclown.VersionEnum;
import org.pdfclown.documents.contents.ContentScanner;
import org.pdfclown.documents.contents.IContentContext;
import org.pdfclown.objects.PdfDictionary;
import org.pdfclown.objects.PdfObjectWrapper;

/**
  [PDF:1.6:3.6.2] Document page.

  @author Stefano Chizzolini (http://www.stefanochizzolini.it)
  @since 0.0.0
  @version 0.1.0
*/
@PDF(VersionEnum.PDF10)
public class Page
  extends PdfObjectWrapper
  implements IContentContext,
    Printable
{
  . . .

  @Override
  public void render(
    Graphics2D context,
    Dimension2D size
    )
  {
    ContentScanner scanner = new ContentScanner(getContents());
    scanner.render(context,size);
  }

  . . .
}

4. Functions

Improving the color space definitions for content rasterization is forcing me to also manage functions [PDF:1.6:3.9] in all their flavors (Type 0 (Sampled), Type 2 (Exponential Interpolation), Type 3 (Stitching) and Type 4 (PostScript Calculator)).

5. Page data size (a.k.a. How to split a PDF document based on maximum file size)

org.pdfclown.tools.PageManager has been enhanced with the introduction of an elegant algorithm that accurately calculates the data size of PDF pages keeping shared resources (like fonts, images and so on) into consideration: this practically means that you can evaluate the incremental size of each page in a document, splitting the file when the collected pages reach the maximum file size you intended for your target split PDF files, without creating any cumbersome temporary file!

It’s really funny that proprietary products like PDFTron’s, which cost at least hundreds of bucks, suggest in their official knowledge base awkward trial-and-error strategies, iteratively creating horrible temporary files… evidently, only clowns can solve such a task better 😛 — Free Software rocks!

package org.pdfclown.samples;

import java.util.HashSet;
import java.util.Set;

import org.pdfclown.documents.Document;
import org.pdfclown.documents.Page;
import org.pdfclown.documents.Pages;
import org.pdfclown.files.File;
import org.pdfclown.objects.PdfReference;
import org.pdfclown.tools.PageManager;

public class SplitSample
  extends Sample
{
  private static final long MaxDataSize = 4 << 20; // 4 MBytes (you can obviously set it at your will).
  private PageManager manager;

  @Override
  public boolean run(
    )
  {
    // 1. Opening the PDF file...
    File file;
    {
      String filePath = promptPdfFileChoice("Please select a PDF file");
      try
      {file = new File(filePath);}
      catch(Exception e)
      {throw new RuntimeException(filePath + " file access error.",e);}
    }
    Document document = file.getDocument();
    Pages pages = document.getPages();

    // 2. Splitting the document...
    manager = new PageManager(document);
    int splitIndex = 0;
    long incrementalDataSize = 0;
    int beginPageIndex = 0;
    Set visitedReferences = new HashSet();
    for(Page page : pages)
    {
      long pageDifferentialDataSize = PageManager.getSize(page,visitedReferences); 
      incrementalDataSize += pageDifferentialDataSize;
      if(incrementalDataSize > MaxDataSize) // Data size limit reached.
      {
        int endPageIndex = page.getIndex();

        // Split the current document page range!
        splitDocument(++splitIndex,beginPageIndex,endPageIndex);

        beginPageIndex = endPageIndex;
        incrementalDataSize = PageManager.getSize(page,visitedReferences = new HashSet());
      }
    }
    // Split the last document page range!
    splitDocument(++splitIndex,beginPageIndex,pages.size());

    return true;
  }

  private void splitDocument(
    int splitIndex,
    int beginPageIndex,
    int endPageIndex
    )
  {
    // 1. Split the document!
    Document splitDocument = manager.extract(beginPageIndex,endPageIndex);

    // 2. Serialize the split file!
    serialize(splitDocument.getFile(),this.getClass().getSimpleName() + "." + (splitIndex),false);
  }
}