Camera on Galaxy S3
It's been a while since my last update, but much progress has been made: the Note 2 touchscreen will be landing in 4.16, and the actual device tree source files will be landing in 4.17 (hopefully along with support for the notification LED, if not more). But recently, I've been focusing on figuring out how to get the camera working.
Cameras, especially on mobile devices, are pretty complicated. On the Galaxy S3/Note 2, there are two cameras: the S5C73M3 on the back, which has its own firmware and uses SPI to talk to the SoC, and the S5K6A3 on the front, which is connected to a dedicated image processing subsystem, called "FIMC-IS". This is where things get complex.
Exynos 4412 has three distinct but interconnected parts of the SoC called FIMC - "Fully Integrated Mobile Camera". There's one called "FIMC", or sometimes "CAMIF" - found in older SoCs, which supports colour conversion, image rotation/flipping, and image scaling, and some image effects (sepia, negative, etc). These aren't restricted to being used for camera input, and can also use the display output as input (writeback), or just operate on in-memory buffers (m2m mode).
Next, comes FIMC-IS - the "Imaging Subsystem". The FIMC-IS is basically a SoC in and of itself: it consists of a Cortex-A5, with its own I2C/GPIO/SPI ports. FIMC-IS is found in all Exynos SoCs (to my knowledge) since Exynos4412, although each SoC has a newer, better revision of it. The FIMC-IS requires a firmware blob to run on the A5, and another one called a "setfile", which (presumably) contains configuration for the attached camera module[s]. The FIMC-IS supports all the normal camera things like dynamic range compression, face detection, and denoising.
Finally, FIMC-LITE (or "flite"). This basically just captures images, optionally crops them, and controls vsync/the pixel clock - as the name suggests, sort of a "FIMC-lite".
Additionally, on the Note 2 and S3, MIPI-CSI is used to transfer image packets from the camera, and this is controlled by the (surprisingly named) MIPI-CSI block.
Mainline support
Thanks to the Samsung Open Source group, there is already mainline support for all the components outlined above, including the two camera sensors found on the Note 2/S3. This should make life really easy, right?
Unfortunately not... (although it could be much, much harder!)
The drivers only seem to work properly when compiled into the kernel, and the firmware for FIMC-IS must be available at boot (either through a ramdisk or built-in to the kernel). Support for supplying buffers directly to and reading buffers directly from FIMC-IS doesn't work - attempting to open the v4l2 device will just hang. So, the Kconfig option for that must be disabled. Then, finally, you can almost start to consider the possibility of taking a picture.
Of course, it's not as simple as telling the camera to send you a frame and receiving it - there's all these fancy image-processing blocks in the SoC that we have to get to communicate with one another somehow...
Pipelining
Most of these IP blocks support arbitrary chaining: the limitations are not entirely clear from looking at the manual,
but, for instance, a typical arrangement for the camera 'pipeline' would be: CSI -> FIMC-LITE -> FIMC,
or CSI -> FIMC-LITE -> FIMC-IS -> FIMC. We can also (according to the manual) pull data out or supply data at
arbitrary steps - for things like image reprocessing later on (i.e. buffer -> FIMC-IS -> buffer), or
skipping the FIMC-IS/FIMC stages entirely (CSI -> FIMC-LITE -> buffer) - unfortunately, lots of these edge-cases
are not well tested and often contain bugs. On the bright side, the media-ctl
utility gives us a way to pretty easily configure these interconnections. We can inspect the default configuration
using media-ctl -p:
$ media-ctl -p Media controller API version 4.16.0 Media device information ------------------------ driver s5p-fimc-md model SAMSUNG S5P FIMC serial bus info hw revision 0x0 driver version 4.16.0 Device topology - entity 1: FIMC.0 (3 pads, 3 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:VYUY8_2X8/640x480 colorspace:jpeg crop.bounds:(0,0)/640x480 crop:(0,0)/640x480 compose.bounds:(0,0)/320x200 compose:(0,0)/320x200] <- "s5p-mipi-csis.0":1 [] pad1: Sink [fmt:YUV10_1X30/640x480 colorspace:jpeg crop.bounds:(0,0)/640x480 crop:(0,0)/640x480 compose.bounds:(0,0)/320x200 compose:(0,0)/320x200] <- "FIMC-IS-ISP":1 [] pad2: Source [fmt:VYUY8_2X8/640x480 colorspace:jpeg] -> "fimc.0.capture":0 [ENABLED,IMMUTABLE] - entity 5: fimc.0.m2m (0 pad, 0 link) type Node subtype V4L flags 0 device node name /dev/video0 - entity 8: fimc.0.capture (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video1 pad0: Sink <- "FIMC.0":2 [ENABLED,IMMUTABLE] - entity 12: FIMC.1 (3 pads, 3 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:YUYV8_2X8/640x480 colorspace:jpeg crop.bounds:(0,0)/0x0 crop:(0,0)/640x480 compose.bounds:(0,0)/640x480 compose:(0,0)/640x480] <- "s5p-mipi-csis.0":1 [] pad1: Sink [fmt:YUV10_1X30/640x480 colorspace:jpeg crop.bounds:(0,0)/0x0 crop:(0,0)/640x480 compose.bounds:(0,0)/640x480 compose:(0,0)/640x480] <- "FIMC-IS-ISP":1 [] pad2: Source [fmt:YUYV8_2X8/640x480 colorspace:jpeg] -> "fimc.1.capture":0 [ENABLED,IMMUTABLE] - entity 16: fimc.1.m2m (0 pad, 0 link) type Node subtype V4L flags 0 device node name /dev/video2 - entity 19: fimc.1.capture (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video3 pad0: Sink <- "FIMC.1":2 [ENABLED,IMMUTABLE] - entity 23: s5p-mipi-csis.0 (2 pads, 5 links) type Node subtype V4L flags 0 device node name /dev/v4l-subdev2 pad0: Sink <- "S5C73M3-OIF":2 [ENABLED,IMMUTABLE] pad1: Source -> "FIMC.0":0 [] -> "FIMC.1":0 [] -> "FIMC-LITE.0":0 [] -> "FIMC-LITE.1":0 [] - entity 26: s5p-mipi-csis.1 (2 pads, 3 links) type Node subtype V4L flags 0 device node name /dev/v4l-subdev3 pad0: Sink <- "S5K6A3 16-0010":0 [ENABLED,IMMUTABLE] pad1: Source -> "FIMC-LITE.0":0 [] -> "FIMC-LITE.1":0 [] - entity 29: FIMC-LITE.0 (3 pads, 4 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev4 pad0: Sink [fmt:YUYV8_2X8/640x480 colorspace:jpeg crop.bounds:(0,0)/640x480 crop:(0,0)/640x480] <- "s5p-mipi-csis.0":1 [] <- "s5p-mipi-csis.1":1 [] pad1: Source [fmt:YUYV8_2X8/640x480 colorspace:jpeg] -> "fimc-lite.0.capture":0 [] pad2: Source [fmt:YUYV8_2X8/640x480 colorspace:jpeg] -> "FIMC-IS-ISP":0 [] - entity 33: fimc-lite.0.capture (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video4 pad0: Sink <- "FIMC-LITE.0":1 [] - entity 37: FIMC-LITE.1 (3 pads, 4 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev5 pad0: Sink [fmt:YUYV8_2X8/640x480 colorspace:jpeg crop.bounds:(0,0)/640x480 crop:(0,0)/640x480] <- "s5p-mipi-csis.0":1 [] <- "s5p-mipi-csis.1":1 [] pad1: Source [fmt:YUYV8_2X8/640x480 colorspace:jpeg] -> "fimc-lite.1.capture":0 [] pad2: Source [fmt:YUYV8_2X8/640x480 colorspace:jpeg] -> "FIMC-IS-ISP":0 [] - entity 41: fimc-lite.1.capture (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video5 pad0: Sink <- "FIMC-LITE.1":1 [] - entity 45: FIMC-IS-ISP (3 pads, 4 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev6 pad0: Sink [fmt:SGRBG10_1X10/1296x732] <- "FIMC-LITE.0":2 [] <- "FIMC-LITE.1":2 [] pad1: Source [fmt:YUV10_1X30/1280x720 colorspace:jpeg] -> "FIMC.0":1 [] -> "FIMC.1":1 [] pad2: Source [fmt:SGRBG10_1X10/1280x720] - entity 49: S5C73M3-OIF (3 pads, 3 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev8 pad0: Sink [fmt:VYUY8_2X8/640x480 field:none colorspace:jpeg] <- "S5C73M3":0 [ENABLED,IMMUTABLE] pad1: Sink [fmt:S5C_UYVY_JPEG_1X8/800x450 field:none colorspace:jpeg] <- "S5C73M3":1 [ENABLED,IMMUTABLE] pad2: Source [fmt:VYUY8_2X8/640x480@33333/1000000 field:none colorspace:jpeg] -> "s5p-mipi-csis.0":0 [ENABLED,IMMUTABLE] - entity 53: S5C73M3 (2 pads, 2 links) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev7 pad0: Source [fmt:VYUY8_2X8/640x480 field:none colorspace:jpeg] -> "S5C73M3-OIF":0 [ENABLED,IMMUTABLE] pad1: Source [fmt:S5C_UYVY_JPEG_1X8/800x450 field:none colorspace:jpeg] -> "S5C73M3-OIF":1 [ENABLED,IMMUTABLE] - entity 60: S5K6A3 16-0010 (1 pad, 1 link) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev9 pad0: Source [fmt:SGRBG10_1X10/1296x732] -> "s5p-mipi-csis.1":0 [ENABLED,IMMUTABLE]
This tells us quite a bit about how we're able to configure the camera pipeline: links marked as
[ENABLED,IMMUTABLE] are physically defined and unchangable on-the-fly, but most links can be arbitrarily
changed: for instance, we can choose to have S5C73M3 go into FIMC-LITE and via FIMC-IS, or directly to
FIMC for capture. Also note that different endpoints have different colour formats - we need to reconcile
them so that we can receive buffers. Using media-ctl, we can configure colour formats and also
set up the links. For S5C73M3, we can use something like this:
$ media-ctl --links '"s5p-mipi-csis.0":1 -> "FIMC.0":0 [1]'
and then using v4l2-ctl we can dump a frame from the camera...
$ v4l2-ctl --device /dev/video1 --stream-mmap=3 --stream-to=image --count=1
Now we have a raw frame. We can look at the raw hex data using something like xxd, or
we can use something like gstreamer to show the (arguably more exciting) live camera preview:
$ gst-launch-1.0 v4l2src device=/dev/video1 ! videoconvert ! glimagesink
and then, there we have it:
Unfortunately, for now I haven't been able to get the S5K6A3 to send camera data - it powers on, but no image data is ever received from the camera (or maybe it gets lost somewhere in the pipeline - who knows?) But still, this is some pretty nice progress.