↓

Filter results by

MIPI Camera

This guide describes how to capture still images or movies from a MIPI camera, as well as how to configure video streaming.

Environment:
– Image capture program: fswebcam
– Video recording and streaming program: ffmpeg
– Video streaming server program: ffserver
– Image preview program: gstreamer (gst-launch development tool)

Connect Camera Module

Acquire the proper MIPI camera module. For ARTIK 530 or 710 boards, you have two choices.

ARTIK 520 or 1020 boards require the S5K5EA camera module.

Connect ARTIK S5K4EC Camera

The procedure here shows the ARTIK 530/710 MIPI camera accessory board used with an ARTIK 530/710 development kit. For the ARTIK Eagleye board from Seeed, look here.

  1. Connect it to the ARTIK board. With power disconnected and both boards placed upside down, attach
    • the flat cable end marked "PLATFORM Board" to the MIPI Camera connector of the ARTIK 530/710 board
    • the other end to the MIPI camera.

    The connectors snap together precisely, so align them carefully before applying pressure.

  2. Connect the ARTIK board to a TV or HDMI monitor as described here and power up the system.

  3. Verify that you get a console login screen. (You will not be able to log in on that screen unless you have a USB keyboard attached to the ARTIK board.)

  4. Check video node information. The video node will be /dev/videoXX – for example:

    # ls -al /dev/video6

    crw-rw---- 1 root video 81, 5 Jan  1  2014 /dev/video6

    The node number will be different between board types. For example:

    • ARTIK 520 MIPI is video100
    • ARTIK 530 or 710 MIPI is video6
    • ARTIK 1020 MIPI is video101

    video6 is used in the examples in this section, but you will need to modify it for your system type.

    You can use a sysfs feature to list possible video input nodes for your device. Go to /sys/class/video4linux and run ls -al
    The device that shows v4l2 ("video for Linux" – the third character is an L) is the device you will be referencing in the commands below.

  5. Read the Choose and Install Programs section to learn about capture programs available. We suggest you start by installing ffmpeg.

  6. Follow the ffmpeg Display Preview topic. Simply copy and paste the specified line with no modifications.

You should immediately see a sharp, clear video image on the monitor.

Connect Leopard OV5640 Camera

The Leopard OV5640 MIPI camera can be used with the ARTIK "Eagleye" board from Seeed by attaching it directly, and with the ARTIK 530 and 710 development boards by using the included adapter (flat cable). Attachment to the development board is similar to that shown above.

Drivers for this model are present in ARTIK OS_18.05.00 and later source code software releases on GitHub. However, in the OS_18.05.00 release only, the driver is disabled by default. To enable it, you'll need to patch the standard source code and then build a new image from that source code.

First, click on the link for your module to obtain the correct patch, and copy the code.

Then, follow the patch instructions here to save the patch file and rebuild your code.

Choose and Install Programs

Many programs are available for capturing and displaying video streams. Some of the more popular ones are included in the default system firmware image, or can be installed as noted below if necessary.

fswebcam

fswebcam is a simple utility program used to capture an image from a webcam. The captured image can be stored as one or more PNG or JPEG files.

  • Supported image resolution, S5K4EC: VGA(640x480), HD(1280x720),
    5MP(2560x1920)
  • Supported image resolution, OV5640: VGA(640x480), HD(1280x720),
    5MP*(2592x1944), FHD(1920x1080)
  • Supported palette format of source image : YUYV

*Note the slight difference in 5MP resolution between the two cameras.

To install from the command line, enter:

Ubuntu:
apt install fswebcam
Fedora:
dnf install fswebcam

ffmpeg

ffmpeg is widely used to encode and decode video streams from a device or file. It also supports grabbing an input stream from V4L2 (Video4Linux2) devices.

Supported resolutions and frame rates

  • YUYV
    – VGA(640x480): 15fps, 10fps and 5fps
    – HD(1280x720): 7fps and 5fps

  • MJPEG
    – Either resolution: 30fps, 24fps, 15fps, 10fps and 5fps

    – OV5640: Limited to 15fps and 30fps only

    Resolution Frame Rate
    ARTIK 710    		 Frame Rate
    ARTIK 530
    VGA(640x480) 30fps, 15fps 30fps, 15fps
    HD(1280x720) 30fps, 15fps 15fps

To install from the command line, enter:

Ubuntu:
apt install ffmpeg
Fedora:
dnf install ffmpeg

ffserver

You can stream a video with RTSP protocol over the network by using ffserver along with ffmpeg. ffserver is installed as part of the ffmpeg package.

gstreamer

The Gstreamer framework allows you to build video, audio, and other processing pipelines using Gstreamer "plug-ins".

Supported resolution and frame rates for preview:
– VGA(640x480) : 30fps, 24fps, 15fps, 10fps and 5fps

Additionally supported on ARTIK 7 with S5K4EC:
– HD(1280x720) : 15fps, 10fps and 5fps
– 5MP(2560x1920) : still

Supported on OV5640:
– VGA, HD, FHD(1920x1080): 30fps

To install Gstreamer plug-ins from the command line, enter:

Ubuntu:
apt install gstreamer*
apt install gstreamer1.0-plugins-bad
Fedora:
dnf install gstreamer*

Display Preview on LCD

To clear the LCD screen before displaying a new preview, enter on the command line:

# cat /dev/zero > /dev/fb0

ffmpeg Display Preview

To display a preview screen on the LCD with ffmpeg, this example

ffmpeg -f v4l2 -s 640x480 -r 30 -i /dev/video6 -pix_fmt bgra -f fbdev /dev/fb0

specifies:

  • -f v4l2 – use V4L2 capture device
  • -s 640x480 – resolution
  • -r 30 – frame rate is 30 fps
  • -i /dev/video6 – input device for recording (MIPI camera)
  • -pix_fmt bgra – pixel format of LCD screen
  • -f fbdev /dev/fb0 – output device is frame buffer device.

gstreamer Display Preview

You can display a preview screen on the LCD by using gst-launch with plug-ins appended. You can see a detailed description for each plug-in by executing

gst-inspect [plug-in name]

For example,

gst-launch -e v4l2src queue-size=16 device=/dev/video6 ! video/x-raw-yuv,framerate=30/1,width=640,height=480 ! ffmpegcolorspace ! fbdevsink device=/dev/fb0

specifies:

  • -e v4l2src queue-size=16 device=/dev/video6 – use v4l2src plugin
  • video/x-raw-yuv,framerate=30/1,width=640,height=480 – format, framerate, and size of video, respectively
  • ffmpegcolorspace – do color space conversion between yuv video and fbdevsink
  • fbdevsink device=/dev/fb0 – output device is frame buffer device.

Rotate the preview screen on the LCD by adding the “videoflip” plug-in to the Gstreamer pipeline, such as in

gst-launch -e v4l2src queue-size=16 device=/dev/video6 ! video/x-raw-yuv,framerate=30/1,width=640,height=480 ! videoflip method=counterclockwise ! ffmpegcolorspace ! fbdevsink device=/dev/fb0

where:

videoflip method=counterclockwise`

specifies the rotation direction.

Capture Images or Movies

fswebcam Image Capture

Refer to online man page for detailed usage information. Typical command line usage:

fswebcam -d [device node] -r [Resolution] -p [palette format] [filename]

S5K5EA:
ARTIK 710 example

fswebcam -d /dev/video6 -r 2560x1920 -p YUYV -D 1 --no-banner image.jpg

ARTIK 530 example

fswebcam -d /dev/video6 -r 2560x1920 -p YUV420P --no-banner image.jpg

ARTIK 710 example

fswebcam -d /dev/video6 -r 2592x1944 -S 10 -p YUYV --no-banner image.jpg

ARTIK 530 example

fswebcam -d /dev/video6 -r 2592x1944 -S 10 -p YUV420P --no-banner image.jpg

specifies:

  • -d /dev/video6 – name of device node (MIPI camera)

  • -r 2560x1920 – resolution

  • -p YUYV or p YUV420P – format of source image

  • -D 1 – pre-capture Delay time (seconds). Since a camera usually needs time to process details such as focus, exposure, and white balance, use at least a 1 second pre-capture delay in order to capture a clearer image.

  • -S xx -- skip first 'xx' frames (OV5640 feature only)

  • --no-banner – to hide the banner on the JPEG image.

gstreamer Movie Recording (ARTIK 530/710)

The ARTIK 530 and 710 modules incorporate a hardware video codec. These examples take advantage of the hardware features to record .avi and .mp4 videos. Single image capture is not supported from the command line for OV5640 (you can still do so using the capture application to access the gstreamer library).

Example 1 – Record a video with no audio:

gst-launch-1.0 -e camerasrc camera-crop-width=1280 camera-crop-height=720 ! nxvideoenc bitrate=12000000 ! avimux ! filesink location=result.avi

Example 2 – Record a video with preview (using a 'tee' plugin to provide a 1-to-N pipe fitting):

gst-launch-1.0 -e camerasrc camera-crop-width=1280 camera-crop-height=720 ! tee name=t \
t. ! queue ! nxvideosink \
t. ! queue ! nxvideoenc bitrate=12000000 ! mp4mux ! filesink location=result.mp4

Example 3 – Record FHD movie by using scaler. Although maximum resolution of camera source is HD(1280x720), recording a FHD(1920x1080) movie using the hardware scaler is possible.

gst-launch-1.0 -e camerasrc camera-crop-width=1280 camera-crop-height=720 ! nxscaler scaler-crop-width=1280 scaler-crop-height=720 scaler-dst-width=1920 scaler-dst-height=1080 ! tee name=t \
t. ! queue ! nxvideosink \
t. ! queue ! nxvideoenc bitrate=17000000 ! avimux ! filesink location=result_fhd.avi

Example 4 – Record a movie by using camera source plugin with sound, with simultaneous display on LCD or HDMI screen (need to install additional gstreamer plugin for encoding sound)

Ubuntu:
apt install gstreamer1.0-plugins-ugly gstreamer1.0-plugins-bad
Fedora:
dnf install gstreamer1-plugins-ugly

gst-launch-1.0 -e camerasrc camera-crop-width=1280 camera-crop-height=720 ! tee name=t \
t. ! queue ! nxvideoenc bitrate=12000000 ! queue ! mux. \
autoaudiosrc ! queue max-size-buffers=1000 ! liveadder start-time-selection=2 start-time=600000000 ! lamemp3enc ! queue ! mux. \
t. ! nxvideosink \
mp4mux name=mux ! filesink location=result_a.mp4

ffmpeg Movie Recording

Refer to online man page for detailed usage information.

For OV5640:

– ARTIK 530 supports VGA 15fps recording only.

– ARTIK 710 supports VGA 15/30fps, HD 15fps.

Record without Sound

To record video with no audio, use this example as a template.

ffmpeg -y -f v4l2 -s 640x480 -r 15 -i /dev/video6 -b:v 2048k -vcodec mpeg4 test.avi

Record with Sound

To record a movie with sound, additionally specify audio channels (1:Mono, 2:Stereo), and card number from arecord results (ac: audio channel, hw: card number, -t duration). Refer to the Audio Guide for more examples of ALSA parameters.

For an ARTIK 520/1020 example,

ffmpeg -y -thread_queue_size 2048 -f alsa -ac 1 -i hw:0 -f v4l2 -s 1280x720 -r 7 -i /dev/video6 -b:v 3072k -vcodec mpeg4 -t 20 test.avi

specifies:

  • -y – overwrite output files
  • -thread_queue_size 2048 – maximum number of queued packets. Raising this value can avoid audio packet loss on low-latency and high-rate live streams.
  • -f alsa – use ALSA for audio
  • -ac 1 – audio channel is mono
  • -i hw:0 – input audio card is 0
  • -f v4l2 – use V4L2 capture device
  • -s 640x480 – resolution
  • -r 15 – frame rate is 15 fps
  • -i /dev/video6 – input device for recording (MIPI camera)
  • -b:v 2048k – target bitrate of recorded video : 2Mbps
  • -vcodec mpeg4 – use mpeg4 as a video codec
  • -t 20 – duration of recording is 20 sec.
  • test.avi – output file name.

For an ARTIK 530/710 example,

ffmpeg -y -itsoffset 0.15 -thread_queue_size 2048 -f alsa -i hw:0 -f v4l2 -s 1280x720 -r 15 -i /dev/video6 -b:v 3072k -vcodec mpeg4 -t 20 test.avi

additionally specifies:

  • -itsoffset 0.15 – synchronize between audio and video by specified audio offset. For example:
    – when audio is faster than video by 0.15 sec : -itsoffset 0.15
    – when audio is slower than video by 0.5 sec : -itsoffset -0.5

    Recommended itsoffset value to apply to .avi video is 0.15 (sec).

Since ffmpeg uses S/W codec for recording a video file, the tool may print DTS warning messages when the timestamp between the image source and encoded stream are not identical.

If you want to turn off verbose warning messages, you can change log level of ffmpeg by adding "-log level error" option as follows. Note that default loglevel is "info".

ffmpeg -y -itsoffset 0.15 -thread_queue_size 2048 -f alsa -i hw:0 -f v4l2 -s 1280x720 -r 15 -i /dev/video6 -b:v 3072k -vcodec mpeg4 -t 20 -loglevel error test.avi

Record with 'vcodec copy' Option

When encoding video using a software codec, rate and resolution are limited by the CPU performance. Maximum frame rate of video encoding with the ARTIK 5 solution is approximately 7 fps for HD (1280x720), requiring use of the -r option to trade off frame rate for resolution.

In order to get full frame rate video with high resolution, it is necessary to bypass the software video codec as follows.

  1. Change the output format of the camera to mjpeg.
  2. Specify -vcodec copy to just copy the stream without encoding with the codec.
ffmpeg -y -thread_queue_size 2048 -f alsa -ac 1 -i hw:0 -f v4l2 -s 1280x720 -vcodec mjpeg -r 30 -i /dev/video6 -vcodec copy -t 20 -loglevel error test.avi

Stream Video

Using ffmpeg and ffserver

You can stream a video with RTSP protocol over the network by using ffmpeg and ffserver. ffserver receives FFM streams from the ffmpeg instance as input, then streams them over RTSP. ARTIK acts as the RTSP server.

Supported resolutions and frame rates for video streaming

S5K5EA:

  • VGA (640x480) : 15fps, 10fps and 5fps
  • HD (1280x720): 7fps, and 5fps

OV5640:

  • VGA : 30fps, 15fps
  • HD: 15fps
  1. Configure ffserver with a server configuration file, such as this ffserver.conf file for VGA 15fps streaming.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Port 8090
BindAddress 0.0.0.0

RTSPPort 8091
RTSPBindAddress 0.0.0.0

MaxClients 1000
MaxBandwidth 10000
CustomLog -

<Feed feed1.ffm>
File /tmp/feed1.ffm
ACL Allow 127.0.0.1
</Feed>

<Stream test1.mp4>
Feed feed1.ffm
Format rtp
VideoFrameRate 15
VideoBitRate 5000
VideoSize 640x480
VideoQMin 3
VideoQMax 31
NoAudio
NoDefaults
</Stream>
  1. Launch ffserver in the background.
    # ffserver -f ffserver.conf &
  2. Run a stream feed with ffmpeg on ARTIK device.
    # ffmpeg -f v4l2 -s 640x480 -r 15 -i /dev/video6 http://localhost:8090/feed1.ffm

Receive and Play Streaming Content

On your PC, you can use media players like mplayer or VLC as the RTSP client to receive and play video streams.

mplayer rtsp://[IP Address]:8091/test1.mp4

Note that you run mplayer on the Host PC, not on the ARTIK board, to receive and play the video stream on the host (although you could set up a second ARTIK 530, 710, or 1020 board with an HDMI monitor to act as host).

Using Gstreamer

OV5640 – H/W Codec

You can stream a video over RTSP with gstreamer. Supported resolutions and frame rates for video streaming:

– VGA (640x480) : 30fps, 15fps
– HD (1280x720): 30fps, 15fps
– FHD (1920x1080): 30fps, 15fps

Install additional packages that are needed to build an rtsp application.

apt install build-essential
apt install gstreamer1.0-dev gstreamer-plugins-base1.0-dev
apt install libgstreamer-plugins-base1.0-dev
apt install libgstrtspserver-1.0-dev

Build the launcher program, which we will use instead of gst-launch-1.0.

gcc -o test-launch test-launch.c `pkg-config --cflags --libs gstreamer-1.0 gstreamer-rtsp-1.0 gstreamer-rtsp-server-1.0`

Refer to the code snippet of the launcher program below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
#include <gst/gst.h>
 
#include <gst/rtsp-server/rtsp-server.h>
 
#define DEFAULT_RTSP_PORT "8554"
 
static char *port = (char *) DEFAULT_RTSP_PORT;
 
static GOptionEntry entries[] = {
 
  {"port", 'p', 0, G_OPTION_ARG_STRING, &port,
 
      "Port to listen on (default: " DEFAULT_RTSP_PORT ")", "PORT"},
 
  {NULL}
 
};
 
int
 
main (int argc, char *argv[])
 
{
 
  GMainLoop *loop;
 
  GstRTSPServer *server;
 
  GstRTSPMountPoints *mounts;
 
  GstRTSPMediaFactory *factory;
 
  GOptionContext *optctx;
 
  GError *error = NULL;
 
  optctx = g_option_context_new ("<launch line> - Test RTSP Server, Launch\n\n"
 
      "Example: \"( videotestsrc ! x264enc ! rtph264pay name=pay0 pt=96 )\"");
 
  g_option_context_add_main_entries (optctx, entries, NULL);
 
  g_option_context_add_group (optctx, gst_init_get_option_group ());
 
  if (!g_option_context_parse (optctx, &argc, &argv, &error)) {
 
    g_printerr ("Error parsing options: %s\n", error->message);
 
    g_option_context_free (optctx);
 
    g_clear_error (&error);
 
    return -1;
 
  }
 
  g_option_context_free (optctx);
 
  loop = g_main_loop_new (NULL, FALSE);
 
  /* create a server instance */
 
  server = gst_rtsp_server_new ();
 
  g_object_set (server, "service", port, NULL);
 
  /* get the mount points for this server, every server has a default object
 
   * that be used to map uri mount points to media factories */
 
  mounts = gst_rtsp_server_get_mount_points (server);
 
  /* make a media factory for a test stream. The default media factory can use
 
   * gst-launch syntax to create pipelines.
 
   * any launch line works as long as it contains elements named pay%d. Each
 
   * element with pay%d names will be a stream */
 
  factory = gst_rtsp_media_factory_new ();
 
  gst_rtsp_media_factory_set_launch (factory, argv[1]);
 
  gst_rtsp_media_factory_set_shared (factory, TRUE);
 
  /* attach the test factory to the /test url */
 
  gst_rtsp_mount_points_add_factory (mounts, "/test", factory);
 
  /* don't need the ref to the mapper anymore */
 
  g_object_unref (mounts);
 
  /* attach the server to the default maincontext */
 
  gst_rtsp_server_attach (server, NULL);
 
  /* start serving */
 
  g_print ("stream ready at rtsp://127.0.0.1:%s/test\n", port);
 
  g_main_loop_run (loop);
 
  return 0;
 
}

Run the launch program to stream a video on ARTIK host.

Video Only

./test-launch "( v4l2src device=/dev/video6 ! video/x-raw,format=I420,framerate=30/1,width=1280,height=720 ! nxvideoenc ! rtph264pay name=pay0 )"

Video + Audio

./test-launch "( v4l2src device=/dev/video6 ! video/x-raw,format=I420,framerate=15/1,width=1280,height=720 ! nxvideoenc ! rtph264pay name=pay0 autoaudiosrc ! opusenc ! rtpopuspay name=pay1 )"

Audio Only

// opus codec

./test-launch "( autoaudiosrc ! opusenc ! rtpopuspay name=pay0 )" &nbsp;

//Law audio encoder ./test-launch "( autoaudiosrc ! alawenc ! rtppcmapay name=pay0 )"

Run the gst-launch-1.0 program to show the video stream on client artik. Make sure to connect an LCD or HDMI display.

gst-launch-1.0 -e rtspsrc location=rtsp://[host ip]:8554/test latency=0 ! rtph264depay ! h264parse ! nxvideodec ! nxvideosink
Last updated on: