Mysterious "extreme moiré" 3x3 readout on the EOS M

Bug reported here.

I would say even worse... the moiree is so extreme, that you can see it in liveview already.

Best guess: this must be 3x3 readout with line and column skipping!

Test images here. One is 3x3 "extreme moiré" readout, the other is 1:1 readout from x5 zoom. No camera motion between the two images.

Let's load them in Octave.

In [1]:
%%shell
wget -c -q https://www.dropbox.com/s/70ig9z535kixmk0/binningtest3.zip?dl=1 -O binningtest3.zip
unzip binningtest3.zip

Archive:  binningtest3.zip
  inflating: 98180001.DNG            
  inflating: 98180002.DNG
In [2]:
pkg load image

a = read_raw('98180001.DNG');
b = read_raw('98180002.DNG');

Some helper functions to display the images:

In [3]:
# raise the shadows
function y = gamma(x)
  y = sqrt(min(max(x - 2048, 0), 5000));
end

# show image
function show(x)
  imshow(gamma(x), [])
end

# show image, with zoom (integer factor, nearest neighbour interpolation)
function show_z(x, zoom)
  imshow(imresize(gamma(x), zoom, 'nearest'), [])
end

# show image difference, with zoom
function show_dz(a, b, zoom)
  imshow(imresize(a - b, 4, 'nearest'), [-100 100])
end

# blank space between 2 images
function y = blank(x)
  y = zeros(size(x,1), round(size(x,1)/20)) + inf;
end

# show 2 images side by side
function cmp(a, b, zoom, dif)
  if nargin > 3
    show_z([a, blank(a), b, blank(a), b - a + 2048 + (sqrt(5000)/2)^2], zoom)
#    imshow(imresize([gamma([a, blank(a), b]), blank(a), b - a], zoom, "nearest"), [])
  else
    show_z([a, blank(a), b], zoom)
  end
end

Let's extract a crop from each image (manual tweaking):

In [4]:
ac = a(401:600,751:950);
bc = b(283:882, 809:1408);

cmp(ac, imresize(bc,1/3), 2)

Hypothesis: from each 3x3 pixel block of the same Bayer color, only one pixel is captured. The other 8 are discarded.

In [5]:
for i = [1 3],
  for j = [1 3],
    figure, cmp(ac, bc(i:3:end, j:3:end), 2)
  end
end

Nope, the moiré is much worse than that!

Let's look at individual Bayer channels.

In [6]:
# trial and error - good match (red channel)
cmp(ac(1:2:end, 1:2:end), bc(1:6:end,3:6:end), 3, 1)
In [7]:
# trial and error - another good match (green1 channel)
cmp(ac(1:2:end, 2:2:end), circshift(bc(1:6:end,2:6:end), [0,-1]), 3, 1)
In [8]:
# some bad match (green1 channel)
cmp(ac(1:2:end, 2:2:end), bc(1:6:end,6:6:end), 3, 1)

That seems to give a good match. Maybe each Bayer channel has its own phase (i.e. position of the active pixel in the 3x3 grid) ?

Let's find out. When all else fails, brute force prevails.

In [9]:
# brute-force scanning to find the binning mode
# hypothesis: only one pixel from a 3x3 group is going to be used
# we want to find which pixel (of these 3x3), for each Bayer channel

# create a Bayer grid for showing the result
binmode(:,:,1) = zeros(size(bc));
binmode(:,:,2) = zeros(size(bc));
binmode(:,:,3) = zeros(size(bc));
binmode = uint8(binmode * 255);
colors = [1 2; 2 3];

# set all channels to "unused" (washed out)
for di = 1:2
  for dj = 1:2
    binmode(di:2:end, dj:2:end, colors(di,dj)) = 50;
  end
end

# for each Bayer channel
for di = 1:2
  for dj = 1:2
  
    # find the best match
    # we may also want to shift the image a bit to keep alignment
    # search space is small, so we can just brute-force things
    E = [];
    for dr = 1:6
      for dc = 1:6
        for ci = -2:2
          for cj = -2:2
            E(dr,dc,ci+3,cj+3) = norm((ac(di:2:end, dj:2:end) - circshift(bc(dr:6:end, dc:6:end), [ci, cj]))(11:end-10, 11:end-10)(:));
          end
        end
      end
    end

    # show the best match
    [m,ind] = min(E(:));
    [dr,dc,ci,cj]=ind2sub(size(E), ind); ci -= 3; cj -= 3;
    printf("channel (%d,%d) phase (%d,%d) offset (%d,%d)\n", di, dj, dr, dc, ci, cj)
    figure, cmp(ac(di:2:end, dj:2:end), circshift(bc(dr:6:end, dc:6:end), [ci, cj]), 3, 1)

    # mark it on the Bayer map
    binmode(dr:6:end, dc:6:end, colors(di,dj)) = 255;
  end
end

channel (1,1) phase (1,3) offset (0,0)
channel (1,2) phase (1,2) offset (0,-1)
channel (2,1) phase (4,3) offset (0,0)
channel (2,2) phase (4,2) offset (0,-1)

So far, so good; let's see the results.

In [10]:
# for some reason, imshow will display a dark image, figure out why
# workaround: imagemagick + markdown
imwrite(binmode(1:12, 1:12, :), 'binmode.png');
system("mogrify -filter Point -resize 3000% binmode.png");

binmode.png

Ta-da!

It is, indeed, 3x3 readout with line/column skipping. Only one pixel from each 3x3 group (of pixels with the same Bayer color) is captured. The other 8 are skipped.

However, the distance between the pixels actually read out is not uniform on the X direction. Rather, every two columns are adjacent.

Ideally, a 3x3 readout with line/column skipping would have been like this:

binmode-3x3-rcskip-ideal.png

With the "extreme moiré" readout, the same number of pixels is being captured. However, since we may have 4 columns skipped between two captured pixels, the Nyquist frequency is much lower.

Overall, aliasing in this "extreme moiré" mode is worse than in 720p, where a 5x3 readout (line skipping / column binning) is used. There, the camera reads out 3 pixels out of each 5x3 block of pixels having the same Bayer color.

With regular 1080p 3x3, most Canons use the following pixel binning method (source):

5d2-lv-binning-cell.png

Does that mean the pixel binning phase, at least horizontally, can be adjusted by software?

That could be pretty interesting.