5D Mark IV - Dual Pixel RAW from LiveView¶
-- a1ex, september-november 2018By default, the 5D Mark IV saves an unusual kind of Bayer data from LiveView - smaller size and with many artifacts. A closer look reveals what they actually are - Dual Pixel RAW frames from LiveView!
Let's decode them.
Background:
- 5D4 initial dual pixel notes (from CR2 files)
- 70D LiveView dual pixel notes (lv_daf_mode)
- LiveView raw types (lv_set_raw - we can extract Bayer data from various stages of Canon's image processing pipeline)
lv_raw_dump (after lv_save_raw) saves a file named 46201988.RAW.
In [1]:
%%shell
wget -c -q https://a1ex.magiclantern.fm/bleeding-edge/5D4/dual-pixel/kichetof/46201988.RAW
wget -c -q https://a1ex.magiclantern.fm/bleeding-edge/5D4/dual-pixel/kichetof/DEBUGMSG.LOG
In [2]:
%%shell
xxd -l 128 46201988.RAW; xxd -o 0x100000 -l 128 46201988.RAW
Smells like 16-bit data.
Let's fire up Octave (4.x with 16-bit image support) to have a closer look.
In [3]:
f = fopen("46201988.RAW");
x = fread(f, "uint16");
fclose(f);
size(x)
From the log file, we'll look for the output of this code snippet, to find out the image size:
#ifdef CONFIG_5D4 /* 1.0.4 */
DryosDebugMsg(0, 15, "Raw buffer size: %d x %d", MEM(0x133FC), MEM(0x13400));
DryosDebugMsg(0, 15, "FPS timer B: %d", MEM(0x12D10));
#endifIn [4]:
%%shell
cat DEBUGMSG.LOG | grep "Raw buffer size" -C 1
In [5]:
x = reshape(x, [1824 774])';
imshow(x, []);
In [6]:
# quick and dirty half-res Bayer preview
function show_raw(im, black, white)
r = im(1:2:end,1:2:end);
g1 = im(1:2:end,2:2:end);
g2 = im(2:2:end,1:2:end);
b = im(2:2:end,2:2:end);
g = (g1 + g2) / 2;
show_raw_rgb(r, g, b, black, white);
end
function show_raw_rgb(r, g, b, black, white)
r = r - black;
g = g - black;
b = b - black;
white = white - black;
m = log2(white);
# white balance and gamma (just good enough to see something, nothing more)
IM(:,:,1) = (log2(max(1,r*1.5)) / m).^3;
IM(:,:,2) = (log2(max(1,g)) / m).^3;
IM(:,:,3) = (log2(max(1,b*2)) / m).^3;
imshow(IM,[]);
end
prctile(x(:), [0 1 10 50 90 99 100])
show_raw(x, 4096, 5200)
Notice the artifacts in the blurred areas. These are not Bayer artifacts.
Let's decompose into Bayer channels.
In [7]:
ra = x(1:2:end, 1:4:end);
rb = x(1:2:end, 3:4:end);
imshow([ra rb], [])
Aspect ratio looks about right and artifacts disappeared!
Full color:
In [8]:
function [A, B] = dual_pixel_split(x)
ra = x(1:2:end, 1:4:end);
rb = x(1:2:end, 3:4:end);
g1a = x(1:2:end, 2:4:end);
g1b = x(1:2:end, 4:4:end);
g2a = x(2:2:end, 1:4:end);
g2b = x(2:2:end, 3:4:end);
ba = x(2:2:end, 2:4:end);
bb = x(2:2:end, 4:4:end);
A = zeros(size(ra) * 2);
A(1:2:end,1:2:end) = ra;
A(1:2:end,2:2:end) = g1a;
A(2:2:end,1:2:end) = g2a;
A(2:2:end,2:2:end) = ba;
B = zeros(size(ra) * 2);
B(1:2:end,1:2:end) = rb;
B(1:2:end,2:2:end) = g1b;
B(2:2:end,1:2:end) = g2b;
B(2:2:end,2:2:end) = bb;
end
[A, B] = dual_pixel_split(x);
In [9]:
show_raw([A B], 4096, 5200)
In [10]:
imwrite(uint16(A), "a.pgm");
imwrite(uint16(B), "b.pgm");
In [11]:
%%shell
# http://a1ex.magiclantern.fm/bleeding-edge/pgm2dng.c
pgm2dng a.pgm; exiftool a.DNG -BlackLevel=4096 -WhiteLevel=5200 -overwrite_original
pgm2dng b.pgm; exiftool b.DNG -BlackLevel=4096 -WhiteLevel=5200 -overwrite_original
In [12]:
%%shell
dcraw -r 1.35 1 2.5 1 a.DNG b.DNG
convert -loop 0 -delay 100 a.ppm b.ppm flip.gif
In [13]:
%%shell
identify flip.gif
Not that bad :)
Older sample¶
In [14]:
%%shell
wget -c -q http://www.redyeti.net/test/TST9_5D4/1080p_59.94/files.7z
7z -so e files.7z 46201988.RAW > 46201988_old.RAW
In [15]:
f = fopen("46201988_old.RAW");
x = fread(f, "uint16");
fclose(f);
x = reshape(x, [1824 774])';
[A, B] = dual_pixel_split(x);
imwrite(uint16(A), "a.pgm");
imwrite(uint16(B), "b.pgm");
prctile([A(:) B(:)], [0 1 10 50 90 99 100])
In [16]:
%%shell
pgm2dng a.pgm; exiftool a.DNG -BlackLevel=4096 -WhiteLevel=15000 -overwrite_original
pgm2dng b.pgm; exiftool b.DNG -BlackLevel=4096 -WhiteLevel=15000 -overwrite_original
dcraw -r 1.35 1 2.5 1 a.DNG b.DNG
convert -loop 0 -delay 100 a.ppm b.ppm flip2.gif
Not the prettiest subject, but... we've got some depth info!
4K30 sample¶
In [17]:
%%shell
wget -c -q http://www.redyeti.net/test/TST9_5D4/4k_29.97/files.7z -O files_4k.7z
7z -so e files_4k.7z 46201988.RAW > 46201988_4k.RAW
In [18]:
f = fopen("46201988_4k.RAW");
x = fread(f, "uint16");
fclose(f);
x = reshape(x, [2360 886])';
[A, B] = dual_pixel_split(x);
imwrite(uint16(A), "a.pgm");
imwrite(uint16(B), "b.pgm");
prctile([A(:) B(:)], [0 1 10 50 90 99 100])
In [19]:
%%shell
pgm2dng a.pgm; exiftool a.DNG -BlackLevel=4096 -WhiteLevel=18000 -overwrite_original
pgm2dng b.pgm; exiftool b.DNG -BlackLevel=4096 -WhiteLevel=18000 -overwrite_original
dcraw -r 1.35 1 2.5 1 a.DNG b.DNG
convert -loop 0 -delay 100 a.ppm b.ppm flip3.gif
Different image layout?
In [20]:
imshow(min(8000, [
[
x(1:4:end,1:4:end);
x(2:4:end,1:4:end);
x(3:4:end,1:4:end);
x(4:4:end,1:4:end)
], [
x(1:4:end,2:4:end);
x(2:4:end,2:4:end);
x(3:4:end,2:4:end);
x(4:4:end,2:4:end)
], [
x(1:4:end,3:4:end);
x(2:4:end,3:4:end);
x(3:4:end,3:4:end);
x(4:4:end,3:4:end)
], [
x(1:4:end,4:4:end);
x(2:4:end,4:4:end);
x(3:4:end,4:4:end);
x(4:4:end,4:4:end)
]]))
Some sort of dual RGGB GBRG?!
In [21]:
x = x(1:884, :); # 2 extra lines?!
# half-res in both directions, to keep it simpler
ra = (x(1:4:end, 1:4:end) + x(4:4:end, 1:4:end)) / 2;
g1a = (x(1:4:end, 2:4:end) + x(4:4:end, 2:4:end)) / 2;
g2a = (x(2:4:end, 1:4:end) + x(3:4:end, 1:4:end)) / 2;
ba = (x(2:4:end, 2:4:end) + x(3:4:end, 2:4:end)) / 2;
rb = (x(1:4:end, 3:4:end) + x(4:4:end, 3:4:end)) / 2;
g1b = (x(1:4:end, 4:4:end) + x(4:4:end, 4:4:end)) / 2;
g2b = (x(2:4:end, 3:4:end) + x(3:4:end, 3:4:end)) / 2;
bb = (x(2:4:end, 4:4:end) + x(3:4:end, 4:4:end)) / 2;
A = zeros(size(ra) .* [2 1]);
A(1:2:end,1:2:end) = ( ra(:,1:2:end) + ra(:,2:2:end)) / 2;
A(1:2:end,2:2:end) = (g1a(:,1:2:end) + g1a(:,2:2:end)) / 2;
A(2:2:end,1:2:end) = (g2a(:,1:2:end) + g2a(:,2:2:end)) / 2;
A(2:2:end,2:2:end) = ( ba(:,1:2:end) + ba(:,2:2:end)) / 2;
B = zeros(size(ra) .* [2 1]);
B(1:2:end,1:2:end) = ( rb(:,1:2:end) + rb(:,2:2:end)) / 2;
B(1:2:end,2:2:end) = (g1b(:,1:2:end) + g1b(:,2:2:end)) / 2;
B(2:2:end,1:2:end) = (g2b(:,1:2:end) + g2b(:,2:2:end)) / 2;
B(2:2:end,2:2:end) = ( bb(:,1:2:end) + bb(:,2:2:end)) / 2;
show_raw([A B], 4096, 8000)
imwrite(uint16(A), "a.pgm");
imwrite(uint16(B), "b.pgm");
prctile([A(:) B(:)], [0 1 10 50 90 99 100])
In [22]:
%%shell
pgm2dng a.pgm; exiftool a.DNG -BlackLevel=4096 -WhiteLevel=8000 -overwrite_original
pgm2dng b.pgm; exiftool b.DNG -BlackLevel=4096 -WhiteLevel=8000 -overwrite_original
dcraw -r 1.35 1 2.5 1 a.DNG b.DNG
convert -loop 0 -delay 100 a.ppm b.ppm flip5.gif
Sort of. Could have been better, but apparently requires a custom debayering algorithm (2 lines RGGB, 2 lines GBRG).
In this mode, LiveView runs at 120 FPS only while recording. In standby, it runs at 60 FPS (apparently using the 720p60 configuration, with slightly higher resolution - about 900 vs 768 lines).
The mystery is... why did we get a dual pixel DNG with the same code that retrieved regular raw data in other video modes?! This last example, captured with LVRV25D4.FIR, used the following code to enable the raw stream:
call('lv_save_raw', 1); // enable raw stream
call('lv_set_raw_wp', 2); // select [raw type #8](https://www.magiclantern.fm/forum/index.php?topic=18393.0)
msleep(1000); // wait for some new frames with updated settings
while previous examples used only the first and the last line, i.e. they saved the default raw type from Canon firmware.
In [23]:
f = fopen("181004/720p120/LV.RAW");
x = fread(f, "uint16");
fclose(f);
x = reshape(x(1:1824*920), [1824 920])';
[A, B] = dual_pixel_split(x);
imwrite(uint16(A), "a.pgm");
imwrite(uint16(B), "b.pgm");
prctile([A(:) B(:)], [0 1 10 50 90 99 100])
In [24]:
%%shell
pgm2dng a.pgm; exiftool a.DNG -BlackLevel=4096 -WhiteLevel=18000 -overwrite_original
pgm2dng b.pgm; exiftool b.DNG -BlackLevel=4096 -WhiteLevel=18000 -overwrite_original
dcraw -r 1.9 1 1.9 1 a.DNG b.DNG
convert -loop 0 -delay 100 a.ppm b.ppm flip4.gif
Dual pixel raw stream resolution is probably the same as in 1080p. There are more valid vertical pixels, compared to previous samples, but that's probably just because we dumped a larger size of the raw buffer. In other words, my initial guess for LiveView RAW buffer size was incorrect.