Filter results by

MIPI Camera

This guide describes how to capture still images or movies from a MIPI camera, as well as how to configure video streaming.

Environment:
– Image capture program: fswebcam
– Video recording and streaming program: ffmpeg
– Video streaming server program: ffserver
– Image preview program: gstreamer (gst-launch development tool)

Connect MIPI Camera Module

Prepare a MIPI camera module and connect it to the ARTIK board.

  • ARTIK 520 or 1020 board: S5K5EA
  • ARTIK 530 or 710 board: SIP-ASRNX004

Check video node information. The video node will be /dev/videoXX – for example:

# ls -al /dev/video6

crw-rw---- 1 root video 81, 5 Jan  1  2014 /dev/video6

The node number will be different between board types. For example:

  • ARTIK 520 MIPI is video100
  • ARTIK 530 or 710 MIPI is video6
  • ARTIK 1020 MIPI is video101

video6 is used in the examples in this section, but you will need to modify it for your system type.

You can use a sysfs feature to list possible video input nodes for your device. Go to /sys/class/video4linux and then run ls -al to get a listing of devices.

The video device that shows v4l2 (note that the third character is an L, not the number 1), which is the device you will be referencing in the commands below.

Choose and Install Programs

You've got the camera attached, and the board should have recognized it as noted above. If you have an ARTIK 530, 710, or 1020 and have an HDMI monitor and USB keyboard attached, you're nearly done. We suggest you start by installing ffmpeg, and then try the ffmpeg Display Preview example. You might be surprised at how quickly you'll see an image on the screen.

fswebcam

fswebcam is a simple utility program used to capture an image from a webcam. The captured image can be stored as one or more PNG or JPEG files.

  • Supported image resolution : VGA(640x480), HD(1280x720) and (5MP)2560x1920
  • Supported palette format of source image : YUYV

To install from the command line, enter:

dnf install fswebcam

ffmpeg

ffmpeg is widely used to encode and decode video streams from a device or file. It also supports grabbing an input stream from V4L2 (Video4Linux2) devices.

Supported resolutions and frame rates

  • YUYV
    – VGA(640x480): 15fps, 10fps and 5fps
    – HD(1280x720): 7fps and 5fps

  • MJPEG
    – Either resolution: 30fps, 24fps, 15fps, 10fps and 5fps

To install from the command line, enter:

dnf install ffmpeg

ffserver

You can stream a video with RTSP protocol over the network by using ffserver along with ffmpeg. ffserver is installed as part of the ffmpeg package.

gstreamer

The Gstreamer framework allows you to build video, audio, and other processing pipelines using Gstreamer "plug-ins".

Supported resolution and frame rates for preview:
– VGA(640x480) : 30fps, 24fps, 15fps, 10fps and 5fps

Additionally supported on ARTIK 7:
– HD(1280x720) : 15fps, 10fps and 5fps
– 5MP (2560x1920) : still

To install Gstreamer plug-ins from the command line, enter:

dnf install gstreamer*

Display Preview on LCD

To clear the LCD screen before displaying a new preview, enter on the command line:

# cat /dev/zero > /dev/fb0

ffmpeg Display Preview

To display a preview screen on the LCD with ffmpeg, this example

ffmpeg -f v4l2 -s 640x480 -r 30 -i /dev/video6 -pix_fmt bgra -f fbdev /dev/fb0

specifies:

  • -f v4l2 – use V4L2 capture device
  • -s 640x480 – resolution
  • -r 30 – frame rate is 30 fps
  • -i /dev/video6 – input device for recording (MIPI camera)
  • -pix_fmt bgra – pixel format of LCD screen
  • -f fbdev /dev/fb0 – output device is frame buffer device.

gstreamer Display Preview

You can display a preview screen on the LCD by using gst-launch with plug-ins appended. You can see a detailed description for each plug-in by executing

gst-inspect [plug-in name]

For example,

gst-launch -e v4l2src queue-size=16 device=/dev/video6 ! video/x-raw-yuv,framerate=30/1,width=640,height=480 ! ffmpegcolorspace ! fbdevsink device=/dev/fb0

specifies:

  • -e v4l2src queue-size=16 device=/dev/video6 – use v4l2src plugin
  • video/x-raw-yuv,framerate=30/1,width=640,height=480 – format, framerate, and size of video, respectively
  • ffmpegcolorspace – do color space conversion between yuv video and fbdevsink
  • fbdevsink device=/dev/fb0 – output device is frame buffer device.

Rotate the preview screen on the LCD by adding the “videoflip” plug-in to the Gstreamer pipeline, such as in

gst-launch -e v4l2src queue-size=16 device=/dev/video6 ! video/x-raw-yuv,framerate=30/1,width=640,height=480 ! videoflip method=counterclockwise ! ffmpegcolorspace ! fbdevsink device=/dev/fb0

where:

videoflip method=counterclockwise` 

specifies the rotation direction.

Capture Images or Movies

fswebcam Image Capture

Refer to online man page for detailed usage information. Typical command line usage:

fswebcam -d [device node] -r [Resolution] -p [palette format] [filename]

For example,

fswebcam -d /dev/video6 -r 2560x1920 -p YUYV -D 1 --no-banner image.jpg

specifies:

  • -d /dev/video6 – name of device node (MIPI camera)

  • -r 2560x1920 – resolution

  • -p YUYV – format of source image

  • -D 1 – pre-capture Delay time (seconds). Since a camera usually needs time to process details such as focus, exposure, and white balance, use at least a 1 second pre-capture delay in order to capture a clearer image.

  • --no-banner – to hide the banner on the JPEG image.

gstreamer Movie Recording (ARTIK 530/710)

The ARTIK 530 and 710 modules incorporate a hardware video codec. These examples take advantage of the hardware features to record .avi and .mp4 videos.

Example 1 – Record a video with no audio:

gst-launch-1.0 -e camerasrc camera-crop-width=1280 camera-crop-height=720 ! nxvideoenc bitrate=12000000 ! avimux ! filesink location=result.avi

Example 2 – Record a video with preview (using a 'tee' plugin to provide a 1-to-N pipe fitting):

gst-launch-1.0 -e camerasrc camera-crop-width=1280 camera-crop-height=720 ! tee name=t \
t. ! queue ! nxvideosink \
t. ! queue ! nxvideoenc bitrate=12000000 ! mp4mux ! filesink location=result.mp4

Example 3 – Record FHD movie by using scaler. Although maximum resolution of camera source is HD(1280x720), recording a FHD(1920x1080) movie using the hardware scaler is possible.

gst-launch-1.0 -e camerasrc camera-crop-width=1280 camera-crop-height=720 ! nxscaler scaler-crop-width=1280 scaler-crop-height=720 scaler-dst-width=1920 scaler-dst-height=1080 ! tee name=t \
t. ! queue ! nxvideosink \
t. ! queue ! nxvideoenc bitrate=17000000 ! avimux ! filesink location=result_fhd.avi

Example 4 – Record a movie by using camera source plugin with sound, with simultaneous display on LCD or HDMI screen (need to install additional gstreamer plugin for encoding sound) dnf install gstreamer1-plugins-ugly

gst-launch-1.0 -e camerasrc camera-crop-width=1280 camera-crop-height=720 ! tee name=t \
t. ! queue ! nxvideoenc bitrate=12000000 ! queue ! mux. \
autoaudiosrc ! queue max-size-buffers=1000 ! liveadder start-time-selection=2 start-time=600000000 ! lamemp3enc ! queue ! mux. \
t. ! nxvideosink \
mp4mux name=mux ! filesink location=result_a.mp4

ffmpeg Movie Recording

Refer to online man page for detailed usage information.

Record without Sound

To record video with no audio, use this example as a template for ARTIK 520 or 1020.

ffmpeg -y -f v4l2 -s 640x480 -r 15 -i /dev/video6 -b:v 2048k -vcodec mpeg4 test.avi


Record with Sound

To record a movie with sound, additionally specify audio channels (1:Mono, 2:Stereo), and card number from arecord results (ac: audio channel, hw: card number, -t duration). Refer to the Audio Guide for more examples of ALSA parameters.

For an ARTIK 520/1020 example,

ffmpeg -y -thread_queue_size 2048 -f alsa -ac 1 -i hw:0 -f v4l2 -s 1280x720 -r 7 -i /dev/video6 -b:v 3072k -vcodec mpeg4 -t 20 test.avi

specifies:

  • -y – overwrite output files
  • -thread_queue_size 2048 – maximum number of queued packets. Raising this value can avoid audio packet loss on low-latency and high-rate live streams.
  • -f alsa – use ALSA for audio
  • -ac 1 – audio channel is mono
  • -i hw:0 – input audio card is 0
  • -f v4l2 – use V4L2 capture device
  • -s 640x480 – resolution
  • -r 15 – frame rate is 15 fps
  • -i /dev/video6 – input device for recording (MIPI camera)
  • -b:v 2048k – target bitrate of recorded video : 2Mbps
  • -vcodec mpeg4 – use mpeg4 as a video codec
  • -t 20 – duration of recording is 20 sec.
  • test.avi – output file name.

For an ARTIK 530/710 example,

ffmpeg -y -itsoffset 0.15 -thread_queue_size 2048 -f alsa -i hw:0 -f v4l2 -s 1280x720 -r 15 -i /dev/video6 -b:v 3072k -vcodec mpeg4 -t 20 test.avi

additionally specifies:

  • -itsoffset 0.15 – synchronize between audio and video by specified audio offset. For example:
    – when audio is faster than video by 0.15 sec : -itsoffset 0.15
    – when audio is slower than video by 0.5 sec : -itsoffset -0.5

    Recommended itsoffset value to apply to .avi video is 0.15 (sec).


Record with 'vcodec copy' Option

When encoding video using a software codec, rate and resolution are limited by the CPU performance. Maximum frame rate of video encoding with the ARTIK 5 solution is approximately 7 fps for HD (1280x720), requiring use of the -r option to trade off frame rate for resolution.

In order to get full frame rate video with high resolution, it is necessary to bypass the software video codec as follows.

  1. Change the output format of the camera to mjpeg.
  2. Specify -vcodec copy to just copy the stream without encoding with the codec.
ffmpeg -y -thread_queue_size 2048 -f alsa -ac 1 -i hw:0 -f v4l2 -s 1280x720 -vcodec mjpeg -r 30 -i /dev/video6 -vcodec copy -t 20 -loglevel error test.avi

Stream Video

You can stream a video with RTSP protocol over the network by using ffmpeg and ffserver. ffserver receives FFM streams from the ffmpeg instance as input, then streams them over RTSP. ARTIK acts as the RTSP server.

Supported resolutions and frame rates for video streaming

  • VGA(640x480) : 15fps, 10fps and 5fps
  • HD(1280x720): 7fps, and 5fps
  1. Configure ffserver with a server configuration file, such as this ffserver.conf file for VGA 15fps streaming.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Port 8090
BindAddress 0.0.0.0

RTSPPort 8091
RTSPBindAddress 0.0.0.0

MaxClients 1000
MaxBandwidth 10000
CustomLog -

<Feed feed1.ffm>
File /tmp/feed1.ffm
ACL Allow 127.0.0.1
</Feed>

<Stream test1.mp4>
Feed feed1.ffm
Format rtp
VideoFrameRate 15
VideoBitRate 5000
VideoSize 640x480
VideoQMin 3
VideoQMax 31
NoAudio
NoDefaults
</Stream>
  1. Launch ffserver in the background.
    # ffserver -f ffserver.conf &
  2. Run a stream feed with ffmpeg on ARTIK device.
    # ffmpeg -f v4l2 -s 640x480 -r 15 -i /dev/video6 http://localhost:8090/feed1.ffm

Receive and Play Streaming Content

On your PC, you can use media players like mplayer or VLC as the RTSP client to receive and play video streams.

mplayer rtsp://[IP Address]:8091/test1.mp4

Note that you run mplayer on the Host PC, not on the ARTIK board, to receive and play the video stream on the host (although you could set up a second ARTIK 530, 710, or 1020 board with an HDMI monitor to act as host).




Last updated on: