Debugging FFMPEG V4L2 on Android

So I've figured out what's wrong with V4L2 in the master branch: git://source.ffmpeg.org/ffmpeg

In the past few days I've been trying to figure out why compiling FFMPEG on Android didn't work properly with any Android/v4l2 webcams.

I activated the uvcvideo module's trace with this:

echo 2047 > /sys/module/uvcvideo/parameters/trace

And fired up another terminal with this so I can see the kernel messages in real time:

cat /proc/kmsg

Which produced a result similar to this:

<.>[.] uvcvideo: Queuing buffer 8.

<.>[.] uvcvideo: uvc_v4l2_ioctl(VIDIOC_DQBUF)

<.>[.] uvcvideo: Frame complete (EOF found).

<.>[.] uvcvideo: Dequeuing buffer 9 (4, 614400 bytes).

<.>[.] uvcvideo: uvc_v4l2_ioctl(VIDIOC_QBUF)

<.>[.] uvcvideo: Queuing buffer 9.

<.>[.] uvcvideo: uvc_v4l2_ioctl(VIDIOC_DQBUF)

Which looks ok, so no hint there.

Then I dug in uvccapture which seemed to work, but could only capture very slow framerates. so I've changed that (from sleep to usleep in the capture loop) and tried again - seemed fine to me, time to find the differences...

So I've compared v4l2uvc.c from uvccapture and v4l2.c from ffmpeg:

At first glance they seemed the same, except for the O_NONBLOCK flag. so I've read a little about it and then seen that line in v4l2.c:

while ((res = v4l2_ioctl(s->fd, VIDIOC_DQBUF, &buf)) < 0 && (errno == EINTR));

Whenever I see a while loop without a CPU yield it looks suspicious... 

So I've attempted to remove O_NONBLOCK and recompile.

Great, now ffmpeg is not crashing anymore, but the video file looks weird, like its not getting enough frames and the video freezes but still not the full 30fps potential.

Not knowing what's wrong, I've started reading through the v4l2 documentation, pausing once every to check the code in ffmpeg and uvccapture is actually following the specs.

Everything looked fine, but I've really wanted to see what I'm getting from the uvcvideo device, so I've added this right after the VIDIOC_DQBUF line:

av_log(ctx, AV_LOG_DEBUG, "VIDIOC_DQBUF index: %d type: %d bytesused: %d flags: %d field: %d\n",
buf.index, buf.type, buf.bytesused, buf.flags, buf.field);

Again, the results looked fine, but I really wanted to see if the data was the same or different for every packet, I've looked through the docs and found out about framecrc:

ffmpeg -loglevel debug -f v4l2 -i /dev/video0 -f framecrc -

The result was curious, apparently, only when buf.flags has V4L2_BUF_FLAG_MAPPED there's also a change in buffer contents, which puzzled me because everything else seemed the same and Camera ICS was getting high FPS and smooth video.

I've started going over the method calls again and looking at the compile log, which showed a warning about long incompatible with int64 which raised another warning light, so I've looked through the header files to see which function wasn't getting the correct parameters.

I've overlooked this before, but this time it drew my attention:

int (*open_f)(const char *file, int oflag, ...);

int (*close_f)(int fd);

int (*dup_f)(int fd);

int (*ioctl_f)(int fd, unsigned long int request, ...);

ssize_t (*read_f)(int fd, void *buffer, size_t n);

void *(*mmap_f)(void *start, size_t length, int prot, int flags, int fd, int64_t offset);

int (*munmap_f)(void *_start, size_t length);

And

#define v4l2_open   s->open_f

#define v4l2_close  s->close_f

#define v4l2_dup    s->dup_f

#define v4l2_ioctl  s->ioctl_f

#define v4l2_read   s->read_f

#define v4l2_mmap   s->mmap_f

#define v4l2_munmap s->munmap_f

and decided to modify it a bit to call the native functions:

#define v4l2_open   open
#define v4l2_close  close
#define v4l2_dup    dup
#define v4l2_ioctl  ioctl
#define v4l2_read   read
#define v4l2_mmap   mmap
#define v4l2_munmap munmap

Which won't work with libv4l2 but I just wanted to test and see if it could be it... said naaa a few times while it compiled and tried it again:

0,          0,          0,        1,   614400, 0xa7c24d33
0,          1,          1,        1,   614400, 0xa39f740e
0,          2,          2,        1,   614400, 0x4a917832
0,          3,          3,        1,   614400, 0x0a972f6f
0,          4,          4,        1,   614400, 0x0d7511f3
0,          5,          5,        1,   614400, 0xa6507276
0,          6,          6,        1,   614400, 0xb6fb52dd

Beautiful. It works. Tags: android, ffmpeg, O_NONBLOCK, uvcvideo, V4L2_BUF_FLAG_MAPPED, VIDIOC_DQBUF