OV5640 with STM32F429 (part II) - automatic gain and shutter control

 Once basic image capture and acquisition is working, it is time to start playing with some advanced configurations. This time I'll take a look at how to set up automatic gain control (AGC) and automatic exposure control (AEC) and how it performs. 

Starting from configuration blobs available in various ESP32/Arduino/Waveshare/Linux kernel drivers, I managed to set camera up to acquire compressed 5MP JPEGs at reasonable speeds that my MCU can handle. In datasheet table 2.1 it states that 5MP at 15fps requires pixel clock of 96MHz. STM32F4 DCMI interface supports up to 54MHz or HCLK/2.5, whichever comes first (at 180MHz we get 72MHz, so DCMI limit applies). Luckily, lowering frame rate also lowers the pixel clock, so reducing that to 8 fps should bring it down to ballpark 50MHz. Due to my lack of understanding how to properly set up clocks on camera module, I have it set up to something around 4.2fps (measured with oscilloscope off VSYNC signal).

OV5640 has AEC/AGC block, that is described in section 4.5. I'll paraphrase it a bit, as far as I have understood it. It has averaging window, that can be set up using registers 0x5680-0x5687. This window is divided into 16 equal sized sectors in 4 row 4 column configuration. It compares average of this window to preset target value in 0x3A0F-0x3A1F. If the value is far off, it uses "fast" increment, otherwise "slow". An extra feature is 50/60Hz compensation, where horizontal line with different brightness can appear on images in artificial lighting conditions. Enabling this compensation should pause integrating during these flicker periods thus eliminating these horizontal bands.

In theory it should increment gain only once integration time (shutter) period is maxed out. In practice it behaves, well, not that predictably. I would get too high gain (increased noise produces stripy images). The overall exposure level of subsequent images would vary slightly, but noticeably. Another issue was with taking images of forests with sky in frame - sometimes averaging would select either dark forest or bright sky, ending in overexposing one or underexposing the other. It might have been fine, if only the outcome would be predictable. Very seldom I would get some reasonable average out of it. Especially bad results I was getting from taking images of the sky and sun - those would overexpose and often end up being close to completely white. 

Since one of my project goals is to capture sky/sun images, I had to figure a  way of working around these issues. So I decided not to mess around with poorly documented inbuilt AGC/AEC block, but roll out my own. Basically, what I ended up with was using the average brightness values estimated by the camera ISP itself, as if it would use them for inbuilt AGC (registers 0x5691-0x56A0). Except I would read out these values, find the brightest 1/16th of the image and adjust exposure/gain until it falls close to some arbitrary value. "Arbitrary value" I selected to be 45 (out of 255 max) and "close" was deemed +-5. Logic is as follows:

1. Read out all 16 sector values;
2. find the brightest sector;
3. Crank the shutter up to max (VTS);
4. read brightest sector value;
5. compare it to the target, adjust gain;
6. wait 1 frame period;
7. repeat points 4-6 until changes in gain do not produce any meaningful impact;
8. reduce shutter to get closer to target value; read avg, reduce, rinse, repeat.

This seems to work reasonably well. It is rather slow, because after each gain/shutter update cycle I have to wait until camera module has captured another image (next VREF trigger) and estimated new averaging values. At 4fps with 5-6 adjustment steps that adds up to close to 2 seconds. In varying light conditions (clouds or some other movement) this can easily turn into 5-6 seconds. Room for improvement.

 For debugging purposes I use AFC (auto focus) DRAW WINDOW feature, that allows you to draw arbitrary windows on images using 0x5028-0x502F registers. I would draw sector that is used for gain control. This has allowed to observe following...

... issue with corner cases, where bright object (sun, lamp, etc) can be right in the middle between 2 sectors. Or even 4. Then initial estimate can select one sector, while object moves into next sector, that messes the averaging. I was hoping that I could change the averaging window size and location on the fly to zoom in and track the brightest location. Except average values won't change in the registers, if window location is changed. Tried resetting AEG/AGC peripherals, tried setting "new function" bits and everything else I could imagine, but it wouldn't work. A bit of a dead end, seems to me. Hope for eventual accidental discovery on how to activate that fsker. That would allow me to adjust window until I have centered averaging correctly or even home into 1/256th of the whole image for adjustments. This averaging stop is another reason why I didn't manage to try out using inbuilt AEC/AGC with custom window. There could be a slight chance that it works faster than manual changing.

Another issue is with the duration of the whole procedure, but I can't do anything about it until I have figured out how to control frame size and rates reliably. I could start with VGA resolution at 60/90fps for initial estimation and then do fine tuning in full-scale. But the OV5640 clock tree still remains a mis(t)ery beyond my understanding.