ISO Fudge Factor
Before I continue I want to see if I can explain one needs to take what a manufacturer says with a grain of salt as they say. Most camera manufacturers lie. Yeah, I said it. They are big fat liars. They overstate ISO values compared to actual ISO. In some cases drastically. Recently DXOMark.com did their A99 test and didn’t do so well in the “Sports (High ISO)” test and I was a bit skeptical of the results. I maintained the A99 was about a half to full stop better than the A900 and easily a full stop better than the A77. After digging a bit, I am coming to the realization the DXO Labs data is actually probably closer to the truth than I think.
The following is a screen shot of the DXO Labs comparison tool of the A99, A77 and A900 respectively. You can click on the screen shot to go to the actual comparison.
According to the DXO Labs data, the Sports (Low-Light ISO) of the A99 is almost a full stop better than the A77, which is slightly lower than I originally though and fractionally better than the A900 which is drastically lower than I originally thought.
Digging further into the DXO Labs data, you see the measurement chart for ISO. You can click on the screenshot to see more details.
The vertical measurement, Measured ISO, is what DXO Labs measured as the actual ISO for a given setting. The horizontal columns are the manufacturer ISO settings. In other words, when DXO Labs set the body to ISO 100, they got a value fractionally different than 100. On the DXOMark site, if you mouse over the little red, yellow, or orange dots, you can see the values. I put these in a spreadsheet and calculated the differences for your consumption.
A couple of takeaways from this chart. DXO Labs measured the A99 at ISO 48 when they set the body at 50. That’s pretty impressive. If you go to their site and look at other bodies at ISO 50, including some from Canon or Nikon, they usually measure higher than 50 at 50 as the A77 did.
You dig further and you can see a little better what’s really going on here. Below 50, and really above 100, the A99 has measured ISO consistently around 57% below the ISO setting and 54.9% below at ISO 800. So, if you take these settings to heart, Sony is adding almost a full stop “fudge factor” into the ISO calculation above 50. The average is 42.5%. Now, this may be required to compensate for the Translucent Mirror to ensure the body jives with the Sunny 16 rule and meters in an expected manner. But in terms of raw measurement, the A99 just doesn’t do as well as I thought. It still isn’t bad, don’t get me wrong, it just isn’t as good as I thought.
The values for the A77 above ISO 50 is between 79.4 and 84.6%, generically around a 20% “fudge factor”, with the average of measured values being 18.3%. Fairly consistent data. The A900 “fudge factor” above 100 is around 25% with measured ISO values 73.5-79% below Manufacturer Settings. The average measured value below ISO 100 for the A900 is 24.1% below.
It appears DXO Labs only measures full stops above ISO 50, except in the case where a body’s highest value is a fractional stop above one of the settings as the A77 is at ISO 16000. So I have taken the average “fudge factor” above the single anomaly (ISO 50 for the A77 and A99 and ISO 100 for the A900) and applied them to the third stop values to get an estimated ISO for the fractional stops and put them in the table below with complimentary columns reiterating the DXO Labs measured values.
As you can see, the estimated measured ISO values taking the “fudge factor” into account is fairly accurate compared to the DXO Labs measured ISO values. The values are linear. Looking at this chart, the value one would set on the A99 would be more than two-thirds of a full stop higher than the measured value. The A77 is a hair under a third of a stop higher. And the A99 is slightly over a third of a stop over.
So what does this tell me about the DXO Labs “Sports (Low-Light ISO)” score? By their definition the score is “The highest ISO setting for a camera that allows it to achieve an [Signal to Noise Ratio] of 30dB while keeping a good dynamic range of 9 EVs and a color depth of 18bits”. If they are true to the definition where “The highest ISO setting for a camera” implied the Manufacturer ISO, then the A99 and A900 have negligible difference at Manufacturer ISO 1600, where they would go under the defined settings of 30db SNR, 9EV’s of dynamic range, and 18 bits of color depth and the average quality would set the A900 at 8% lower than the A99. Flipping the numbers around, the A99 would be 9% better than the A900. It would also imply, taking the “fudge factor” into account, the measured ISO of the A900 would actually be almost a half stop better than the A99! This for a sensor 4 years older.
This is not consistent with what I have seen. I have a feeling the DXO Labs measurement definition reworded to “The highest measured ISO setting for a camera that allows it to achieve an [Signal to Noise Ratio] of 30dB while keeping a good dynamic range of 9 EVs and a color depth of 18bits” is closer to the truth. If this is the case, the A99 at a particular ISO setting would be a tad better than the A900 at an ISO setting a third of a stop lower. For instance, A99 at Manufacturer ISO 1600 (Measured 913) would be slightly better than the A900 at ISO 1250 (Estimated Measured at 948.8).
This would also imply to get similar SNR, Dynamic Range, and Color Depth with all three bodies around the 30dB/9EV/18bits threshold; where the A99 would need to be set to ISO 2500, and the A900 would need to be set to ISO 2000, the A77 would need to be set to ISO 1000. This is a little closer to what I have seen.
What about the Book Shelf?
OK, now we have this out of the way we can focus on the A99 and A77 test results. Since I messed up, I don’t have good A900 results to look at, hence the asterisk in the title.
A99 ISO 25600 JPEG Samples
I figured I would continue with the noisy high ISO samples. Here are the unmatched A99 ISO 25600 JPEG files straight off the body. Click on the photo to see the full-size shot.
A99 ISO 25600 f/8
A99 ISO 25600 f/5.6
A99 ISO 25600 Adjusted RAW
And here are the adjusted RAW files, where they have been uploaded to Adobe Lightroom, the Standard profiles from Maurizio Piraccini’s blog are applied, the frame is cropped to the top and bottom of the shelf, with the book The American Ways on the left side, and Visually Excel 2000 on the right side of the frame.
All things considered, I am fairly pleased with the out-of-body JPEG files at ISO 25600, especially after looking at the RAW files. Even if I apply Luminance to the RAW file in Lightroom, I don’t get results close to the JPEG files.
A99 vs. A77 ISO 16000 JPEG Comparison
The A77 has a high threshold manufacturer setting of ISO 16000, so I tested both the A77 and A99 at this setting. Here are the JPEG files straight off the bodies.
The in-body JPEGs are just plain nasty on the A77. I think the ISO 16000 setting on the A77 is basically the “In case of dire emergency” setting. If you have to get the shot, and you have no other choice in adding light or dropping shutter speed, use the setting. An alternative is the hand-held twilight preset which should reduce the amount of noise.
A99 vs. A77 ISO 16000 Adjusted RAW Comparison
Since I now presume the actual ISO on the A99 is around 9200 and the A77 is around 13417, and assuming the A99 is presumably a full stop better regardless, this would explain the vast difference between settings. This difference seems even greater with the adjusted RAW files.
As you can see, even in the 640 pixel wide shots, ISO 16000 on the A77 looks like someone tried to screen print onto a sweater and then threw it into the dryer. For how good the A77 is at ISO 50, it is not good at all at 16000.
Here are some comparison panels of selected areas of the shelf, first at f/8 and then at f/5.6.
This is what happens when you have a better sensor, a fuller frame (the A77 is further away), and then you add in the fudge factor. The A99 has less noise and retains much more detail. The A99 quality isn’t that great. The A77 quality is just plain awful.
A99 ISO 25600 vs. A77 ISO 16000 Adjusted RAW
To account for the fudge factor a bit here is a comparison with the A99 at ISO 25600 (measured at 14801) and the A77 at ISO 16000 (measured at 13417). All things considered, if the DXO Labs “Sports (High-ISO)” results are consistent and the assumption is made the measured ISO is used, the A99 ISO 25600 shot should be about 2/3 of a stop better than the A77.
If it isn’t apparent in the first example, look at the IBSN number of the blue book on the right panel compared to the left panel. I’m convinced the DXO Labs measurements are a lot closer to truth than I thought.
To Be Continued…
If you’ve read down this far, I commend you. Unfortunately, if I am going to try to post daily, I need to conclude this in a post tomorrow (if possible) as this took far too much time than I had hoped but I wanted to include the information regarding the DXO Labs Measured ISO vs. Manufacturer ISO. A side effect of this is I had a lot more A900-specific information for this post and I didn’t have to turn on the body to get it! I thank DXO Labs for doing all their hard work. I admit I may have been wrong about them in the past, though I still need to make sure I look at their data qualitatively, and use the data quantitatively by example as I did above. With electronic image sensors, I imagine there is a degree of variance between one sensor and another even with the same manufacturer or even body or sensor. I also imagine there is a small amount of variance between testing. The tests performed on the A900 in 2008 may have slight variance compared to the tests performed on the A99 in late 2012. We trust DXO Labs keeps it consistent enough. I look at it all like those folks who listen to music look at various recording mediums. The music is recorded using an instrument live, and recorded to a tape. An analog representation like vinyl has variances digital representations do not and vice-versa. Even the difference between CD and MP3 can’t be understated and then you have varied qualities of MP3. Most people can’t hear the difference from step to step, but trained ears can. You don’t have to convince anyone who uses film there is a difference between film and digital. One of the biggest things I have learned so far in testing these bodies is the slightest change in light actually makes a difference in the end. Whether it is a reflection off of your belt buckle, a little bit of light peering through the blinds, or the sun being at 40 degrees or 42 degrees. I can try to keep things the same by using the same light, making sure the blinds are shut or making sure I shoot at night. I can try to isolate the camera and subject. But in the end, there will still be a tiny variance.