Camera support - Simple example

Discussion in 'UDOO NEO' started by jas-mx, Feb 28, 2016.

  1. jas-mx

    jas-mx Active Member

    Joined:
    Dec 31, 2013
    Messages:
    407
    Likes Received:
    118
    I know there are a number threads questioning and requesting camera support. The i.mx6sx has very basic video processing capabilities one of which processing the output of a NTSC/PAL camera. On my blog I've have posted a short article and a link to a simple example of how you can read the camera input and process the output for rendering. Given the code is in 'C', I'm am assuming anyone who attempts this has a good understanding what needs to be done. For instance the simple example won't run on X11. If you want build this on the UDOObuntu 2 release see the instructions after the video.

    Even though the video processing capabilities are limited it can still be used to good effect. Below is video showing how we can blend graphics on top of the camera output. The amount of blending is determined by the tilt angle of the gyro, furthermore depending on the direction of the tilt, the camera output is flipped horizontally or vertically. The camera is actually a parking (reversing) camera (courtesy of the UDOO team), this is highlighted by the red/yellow/green lines and the blinking 'STOP' text which form part of the camera output. Given the automotive theme the graphics overlay represents a engine check sensor dash.



    Instructions to build this on the UDOObuntu 2 release

    1. I'm assuming your running against the hdmi display, so first we need to disable the X11 desktop from auto starting. Open a terminal window or from the serial console do:

    Code:
    sudo echo  "manual" | sudo tee -a /etc/init/lightdm.override
    sudo reboot
    2. After the reboot, the hdmi display should now show a console logon. Log on as 'udooer'
    3. I couldn't find the kernel headers on the UDOObuntu release, so you need to download the attached file 'camera.zip' and copy it to neo from your pc eg:

    Code:
    scp camera.zip udooer@udooneo:


    4. On the neo you need to extract the kernel headers into /usr/include as root user: eg

    Code:
    unzip camera.zip
    tar xvf camera.tar
    sudo su
    cd /usr
    tar xvf /home/udooer/headers.tar
    exit
    5. We also need to install the pxp library and header files:

    Code:
    sudo su
    cp /home/udooer/pxp_lib.h /usr/include
    cd /usr/lib
    cp /home/udooer/libpxp.a .
    cp /home/udooer/libpxp.so.0 .
    ln -s libpxp.so.0 libpxp.so
    exit


    7. git clone the sample code and build:

    Code:
    git clone https://github.com/mtx512/imx6sx_samples.git
    cd imx6sx_samples/
    git checkout -b udoo_neo origin/udoo_neo
    cd imx6sx_vadc_pxp/
    make


    8. Hopefully the build works and you can running it (unfortunately need to run it as root) :

    Code:
    sudo ./imx6sx_vdac_pxp


    To reenable the x11 desktop remove the file /etc/init/lightdm.override
     

    Attached Files:

    Last edited: Feb 29, 2016
    graugans and waltervl like this.
  2. waltervl

    waltervl UDOOer

    Joined:
    Dec 12, 2015
    Messages:
    2,314
    Likes Received:
    580
    Thanks jas-mx! How did you connect your camera to the Neo (which neo pins have what input?) and did you have to do something with the device tree editor to set the Neo pins to the correct settings?
     
  3. jas-mx

    jas-mx Active Member

    Joined:
    Dec 31, 2013
    Messages:
    407
    Likes Received:
    118
    Analogue camera connector is J10, which I think will support 2 cameras. You should be able to use a 3 pin JST connector with a 1.25 mm pitch, something like this to wire to a camera. Camera input requires 2 pins (VADC_IN0 and GND) :

    1. VADC_IN0 (the pin nearest to the ethernet connector)
    2. Middle pin is GND.
    3. VADC_IN1 (furtherest away from the ethernet connector)

    The camera used is like this one and you need to ensure it outputs NTSC (720x480) or PAL (720x576) otherwise it won't be decoded. You will also require an addition 12v power source to power that camera.

    The imx6sx-udoo-neo.dtsi should already have the definitions to enable camera support for VADC_IN0. You should see the csi1 and vadc definitions in this file. I haven't tested with the official udoo kernel, I'm currently using the FSL 3.14.52 kernel , however I'm expecting it to work.
     
    Last edited: Feb 28, 2016
    waltervl likes this.
  4. Andrea Rovai

    Andrea Rovai Well-Known Member

    Joined:
    Oct 27, 2014
    Messages:
    1,703
    Likes Received:
    240
    Dear @jas-mx, great! It would be cool if you helped us integrate it in our UDOObuntu!
     
  5. Hannes Bo

    Hannes Bo New Member

    Joined:
    Jan 14, 2016
    Messages:
    24
    Likes Received:
    6

    Dear @jas-mx, I also would like to use the kernel you mentioned. But up to now I never built a Linux kernel. Could you give me any sources for information how to do this? I already read your blog, but don't know where to start...

    I would be very thankful for any help!
     
  6. Andrea Rovai

    Andrea Rovai Well-Known Member

    Joined:
    Oct 27, 2014
    Messages:
    1,703
    Likes Received:
    240
  7. Hannes Bo

    Hannes Bo New Member

    Joined:
    Jan 14, 2016
    Messages:
    24
    Likes Received:
    6
  8. Andrea Rovai

    Andrea Rovai Well-Known Member

    Joined:
    Oct 27, 2014
    Messages:
    1,703
    Likes Received:
    240
  9. Hannes Bo

    Hannes Bo New Member

    Joined:
    Jan 14, 2016
    Messages:
    24
    Likes Received:
    6
  10. Andrea Rovai

    Andrea Rovai Well-Known Member

    Joined:
    Oct 27, 2014
    Messages:
    1,703
    Likes Received:
    240
    I'm sorry then.
     
  11. jas-mx

    jas-mx Active Member

    Joined:
    Dec 31, 2013
    Messages:
    407
    Likes Received:
    118
    Updated the original post with build instructions if you want to try it out on UDOObuntu 2 release. Hopefully it works !
     
    Last edited: Feb 29, 2016
    Hannes Bo and Andrea Rovai like this.
  12. Hannes Bo

    Hannes Bo New Member

    Joined:
    Jan 14, 2016
    Messages:
    24
    Likes Received:
    6
    @jas-mx, thank you very much for your instructions. I will follow them and give you a feedback as soon as it worked (or not =) )

    Edited:

    @jas-mx , I followed your steps and got the following output:

    sudo ./imx6sx_vdac_pxp
    imx6sx_vadc_pxp.c:206 Found v4l2 capture device /dev/video0.
    imx6sx_vadc_pxp.c:220 pixelformat (output by camera): YUV4
    imx6sx_vadc_pxp.c:285 VIDIOC_G_PARM failed
    imx6sx_vadc_pxp.c:289 WxH@fps = 720x480@30 Image size = 1382400
    imx6sx_vadc_pxp.c:476 opening this fb_dev - /dev/fb0
    imx6sx_vadc_pxp.c:565 imx6sx_vdac_pxp finished!



    I also got this message, when creating the symbolic link:

    root@udooneo:/usr/lib# ln -s libpxp.so.0 libpxp.so
    ln: failed to create symbolic link ‘libpxp.so’: File exists


    I am accessing the udoo via ssh from another ubuntu machine. Is it possible, that this is the problem? At the moment I don't have the possibility to connect a monitor via hdmi, but this evening I will. Perhaps it will work then...

     
    Last edited: Mar 1, 2016
    jas-mx likes this.
  13. jas-mx

    jas-mx Active Member

    Joined:
    Dec 31, 2013
    Messages:
    407
    Likes Received:
    118
    Can you check if libpxp.so and libpxp.so.0 are the same size, If not the same size, can you rename libpxp.so to something else and retry the symlink.

    That output looks like its working however you need the hdmi display to verify the camera output is being rendered.
     
    Hannes Bo likes this.
  14. Hannes Bo

    Hannes Bo New Member

    Joined:
    Jan 14, 2016
    Messages:
    24
    Likes Received:
    6
    There are the same size. I moved the libpxp.so to my home directory and then created the symbolic link. I think this should be fine?! Now ls -l gives the following output:

    ...
    -rw-r--r-- 1 root root 3852 Mar 1 22:09 libpxp.a
    lrwxrwxrwx 1 root root 11 Nov 11 19:56 libpxp.so -> libpxp.so.0
    -rw-r--r-- 1 root root 7076 Mar 1 22:09 libpxp.so.0

    ...

    Ok, I will do that this evening. Thanks for your help so far.
     
  15. Hannes Bo

    Hannes Bo New Member

    Joined:
    Jan 14, 2016
    Messages:
    24
    Likes Received:
    6
    @jas-mx : It worked with the hdmi display.

    I will now have a closer look at your sample code. I would like to use openh264 to encode the camera input and another thing I would like to achieve is to create a network stream. Could you give me any hints on that?

    Thanks a lot for your help!
     
  16. jas-mx

    jas-mx Active Member

    Joined:
    Dec 31, 2013
    Messages:
    407
    Likes Received:
    118
    Thanks for testing it!

    Regarding encoding, this example should give you an idea of how to encode. You need to change the PXP output format so something that openh264 can understand ie UYVY or VYUY formats.
     
    Hannes Bo likes this.
  17. Hannes Bo

    Hannes Bo New Member

    Joined:
    Jan 14, 2016
    Messages:
    24
    Likes Received:
    6
    Ok, I already built an ran this example. I still need to understand the details =)

    I will try to use the openh264 together with your sample. As soon as I have any result I will post it. Thanks again for your great support!!!
     
  18. Hannes Bo

    Hannes Bo New Member

    Joined:
    Jan 14, 2016
    Messages:
    24
    Likes Received:
    6
    @jas-mx and @Andrea Rovai : Here is my current status:

    Unfortunately, the source code of your example as well as of the code coming with openh264 repository is not as easy to understand and amend as I thought =)

    So I first tried to use the NEON-feature with openh264. The openh264 repository got successfully built on the UDOO NEO and the example works. However I am not so sure if NEON is supported (there are some NEON-regarding object files, so at least it recognized the NEON capability and compiled something), because whatever YUV-file (different duration and fps rate) I encoded (with the executable "h264enc"), the process takes always about 90% of the CPU resources and regardless of how I changed the config file, I did not manage to achieve a frame rate higher than ~3 fps. I have to state here, that I am surely not familiar with h264 and therefore the changes I made in the config files are simply trial-and-error. The YUV-files I encoded I generated of some mp4-files (720x480 30fps) using avconv. I also tried some mp4-files recorded with only 15 fps, but there was no improvement (see performance_test.sh, layerXX.cfg and encXX.cfg).

    I also tried to encode some YUV-files with avconv (ffmpeg / libav). There it says: "(...) using cpu capabilities: ARMv6 NEON" and here as well the maximum output frame rate was not higher than ~3 fps (many times even lower), so probably open264 also uses NEON. Here the cpu usage was also about 95% (see avconvtest.sh). For my project I would need a minimum of 4 I-frames (key frames) per second. Do you know if that is granted for my output? I am not sure what these 3 fps mean exactly...
    Regarding the general use of the VADC interface of the UDOO NEO I tried again gstreamer (see camrec.sh) and found some interesting command to get the device capabilities of a v4l2 video device (see getting_device_capabilities.sh) and tried different versions, but was not successful in recording or playing anything from the VADC...


    Perhaps anything of my attempts mean more to you or you could give me any hint on how to proceed now.

    Thank you very much!!

    P.S. Although I am still not successful with my aim to use the VADC, I really like the UDOO NEO and I think it has a great future as IoT device or as core of any other embedded system =)

     

    Attached Files:

  19. jas-mx

    jas-mx Active Member

    Joined:
    Dec 31, 2013
    Messages:
    407
    Likes Received:
    118
    Given the encoding frame rates your seeing ~3fps with ffmpeg/libav then your not your going to encode the VADC output to h264, a point I highlighted in the other post . The only valid use case for the VADC I can see is to output to a display (which can be h/w accelerated).
     
  20. Hannes Bo

    Hannes Bo New Member

    Joined:
    Jan 14, 2016
    Messages:
    24
    Likes Received:
    6


    I can see your point. Is there not even the possibility to capture at least 4 jpegs per second? If not with gstreamer, perhaps with any custom application?!

    @Andrea Rovai : Is anyone of the NEO team working on the VADC at the moment?
     
    NickZano likes this.

Share This Page