Sneak preview inside the 5D Mark IV Dual Pixel RAW files

                                                          -- a1ex, september 2016

The 5D Mark IV is now shipping, and sample images are already available on both camera review sites and blogs. In particular, this sample from was picked by quite a few others who wanted to look at dual pixel raw data:

In [1]:
wget -c
--2016-09-11 02:12:43--
Resolving (
Connecting to (||:80... connected.
HTTP request sent, awaiting response... 200 OK

    The file is already fully retrieved; nothing to do.


I've first noticed Laurent Clévy (you know him from releasing a command-line tool that could read the two image streams from a dual pixel CR2 (craw2tool.exe). Then, shortly after, Hirax attempted to create depth maps from some dual pixel CR2 images, using Laurent's tool to decode the files 1 2 3. These experiments actually motivated me to look inside the dual pixel raw. I've downloaded the above sample, and after a bit of fiddling with octave, I've got this animation, and also noticed the secondary image from the dual pixel raw contains one extra stop of highlights.

After browsing around, it turns out others were even faster. For example, Iliah Borg from RawDigger already wrote about dual pixel raw and found out the extra stop of highlights as well (before me). His findings confirmed my theories, although I did not agree 100% with what he wrote. Planet Mitch also did a nice summary of the initial findings around dual pixel. And - guess what - although most third party raw converters did not open the 5D4 CR2 files, dcraw did!

When reading all this, I thought "Oh boy, I'm way too late to the party", and went back to my usual stuff.

After doing some more experiments with this sample raw file, I realized there's still a lot to be discovered about the dual pixel to cover, which is why I've started writing this.

Enough chit-chat, let's look at the files.

Loading dual pixel images in octave

You will need Octave 4.0 with 16-bit image support. The default installation on most Linux distros only opens 8-bit images, so it won't work. If you are on Ubuntu, you may use ppa:aalpo/octave-q1; otherwise, you may have to compile it from source. I'm not sure what's the situation on other operating systems - you tell me.

In [2]:
# -E is undocumented; it outputs the raw image as with -D, also including optical black areas
dcraw -4 -E -s all -v _91A0045.CR2
Loading Canon EOS 5D Mark IV image from _91A0045.CR2 ...
Building histograms...
Writing data to _91A0045_0.pgm ...
Loading Canon EOS 5D Mark IV image from _91A0045.CR2 ...
Building histograms...
Writing data to _91A0045_1.pgm ...

As expected, we've got two PGM images from a dual pixel CR2. Let's open them in octave:

In [3]:
pkg load image
format compact
im0 = double(imread('_91A0045_0.pgm'));
im1 = double(imread('_91A0045_1.pgm'));
warning: your version of GraphicsMagick limits images to 16 bits per pixel
warning: called from
    imformats>default_formats at line 256 column 11
    imformats at line 79 column 3
    imageIO at line 106 column 11
    imread at line 106 column 30

Let's look at them. The images are huge, so I'll show a downsampled version. For simplicity, I'll show just each Bayer channel.

Why sqrt? The image looks too dark without it.

I know, purists will scream. It's just a quick preview, nothing more.

In [4]:
% subtract black level and apply a "gamma" curve
black = 2048;
im0_disp = sqrt(max(im0-black, 0));
im1_disp = sqrt(max(im1-black, 0));

% show each Bayer channel for each sub-image
im0_channels = [ im0_disp(1:32:end,1:32:end) im0_disp(1:32:end,2:32:end);
                 im0_disp(2:32:end,1:32:end) im0_disp(2:32:end,2:32:end); ];
im1_channels = [ im1_disp(1:32:end,1:32:end) im1_disp(1:32:end,2:32:end);
                 im1_disp(2:32:end,1:32:end) im1_disp(2:32:end,2:32:end); ];

% join both images in a single figure
imshow([im0_channels im1_channels], []);

You have already noticed:

  • the RGGB layout of the Bayer grid (same as with other Canon cameras)
  • the first subimage has clipped sky details in the green and blue channels
  • the second subimage looks darker and there are no clipped highlights

Let's try a color preview:

In [5]:
function imshow_bayer(im)
  % half-res debayer: turn each RGGB cell into a RGB pixel
  % not color-correct, but enough for a quick preview
  rgb(:,:,1) =  im(1:2:end,1:2:end);
  rgb(:,:,2) = (im(2:2:end,1:2:end) + im(2:2:end,1:2:end)) / 2;
  rgb(:,:,3) =  im(2:2:end,2:2:end);

  # black level, white balance and gamma correction (hardcoded)
  # octave also expects image data to be from 0 to 1 when using floating point (double)rgb = sqrt(max(rgb - 2048, 0));
  black = 2048;
  white = 16383;
  rgb = max(rgb-black, 0);
  rgb(:,:,1) *= 2.1;
  rgb(:,:,3) *= 1.4;
  rgb = rgb / (white-black);
  rgb = rgb.^(1/2.2);

  # show a downsampled version

% join both images in a single figure
imshow_bayer([im0 im1]);