MIPI Camera
This guide describes how to capture still images or movies from a MIPI camera, as well as how to configure video streaming.
Environment:
– Image capture program: fswebcam
– Video recording and streaming program: ffmpeg
– Video streaming server program: ffserver
– Image preview program: gstreamer
(gst-launch
development tool)
Connect Camera Module
Acquire the proper MIPI camera module. For ARTIK 530 or 710 boards, you have two choices.
ARTIK 520 or 1020 boards require the S5K5EA camera module.
Connect ARTIK S5K4EC Camera
The procedure here shows the ARTIK 530/710 MIPI camera accessory board used with an ARTIK 530/710 development kit. For the ARTIK Eagleye board from Seeed, look here.
- Connect it to the ARTIK board. With power disconnected and both boards placed upside down, attach
- the flat cable end marked "PLATFORM Board" to the MIPI Camera connector of the ARTIK 530/710 board
- the other end to the MIPI camera.
The connectors snap together precisely, so align them carefully before applying pressure.
-
Connect the ARTIK board to a TV or HDMI monitor as described here and power up the system.
-
Verify that you get a console login screen. (You will not be able to log in on that screen unless you have a USB keyboard attached to the ARTIK board.)
-
Check video node information. The video node will be
/dev/videoXX
– for example:#
ls -al /dev/video6
crw-rw---- 1 root video 81, 5 Jan 1 2014 /dev/video6
The node number will be different between board types. For example:
- ARTIK 520 MIPI is
video100
- ARTIK 530 or 710 MIPI is
video6
- ARTIK 1020 MIPI is
video101
video6
is used in the examples in this section, but you will need to modify it for your system type.You can use a
sysfs
feature to list possible video input nodes for your device. Go to/sys/class/video4linux
and runls -al
The device that showsv4l2
("video for Linux" – the third character is an L) is the device you will be referencing in the commands below. - ARTIK 520 MIPI is
-
Read the Choose and Install Programs section to learn about capture programs available. We suggest you start by installing
ffmpeg
. - Follow the ffmpeg Display Preview topic. Simply copy and paste the specified line with no modifications.
You should immediately see a sharp, clear video image on the monitor.
Connect Leopard OV5640 Camera
The Leopard OV5640 MIPI camera can be used with the ARTIK "Eagleye" board from Seeed by attaching it directly, and with the ARTIK 530 and 710 development boards by using the included adapter (flat cable). Attachment to the development board is similar to that shown above.
Drivers for this model are present in ARTIK OS_18.05.00 and later source code software releases on GitHub. However, in the OS_18.05.00 release only, the driver is disabled by default. To enable it, you'll need to patch the standard source code and then build a new image from that source code.
First, click on the link for your module to obtain the correct patch, and copy the code.
Then, follow the patch instructions here to save the patch file and rebuild your code.
Choose and Install Programs
Many programs are available for capturing and displaying video streams. Some of the more popular ones are included in the default system firmware image, or can be installed as noted below if necessary.
fswebcam
fswebcam
is a simple utility program used to capture an image from a webcam. The captured image can be stored as one or more PNG or JPEG files.
- Supported image resolution, S5K4EC: VGA(640x480), HD(1280x720),
5MP(2560x1920) - Supported image resolution, OV5640: VGA(640x480), HD(1280x720),
5MP*(2592x1944), FHD(1920x1080) - Supported palette format of source image : YUYV
*Note the slight difference in 5MP resolution between the two cameras.
To install from the command line, enter:
Ubuntu:
apt install fswebcam
Fedora:
dnf install fswebcam
ffmpeg
ffmpeg
is widely used to encode and decode video streams from a device or file. It also supports grabbing an input stream from V4L2 (Video4Linux2) devices.
Supported resolutions and frame rates
-
YUYV
– VGA(640x480): 15fps, 10fps and 5fps
– HD(1280x720): 7fps and 5fps -
MJPEG
– Either resolution: 30fps, 24fps, 15fps, 10fps and 5fps– OV5640: Limited to 15fps and 30fps only
Resolution Frame Rate
ARTIK 710Frame Rate
ARTIK 530VGA(640x480) 30fps, 15fps 30fps, 15fps HD(1280x720) 30fps, 15fps 15fps
To install from the command line, enter:
Ubuntu:
apt install ffmpeg
Fedora:
dnf install ffmpeg
ffserver
You can stream a video with RTSP protocol over the network by using ffserver
along with ffmpeg
. ffserver
is installed as part of the ffmpeg
package.
gstreamer
The Gstreamer framework allows you to build video, audio, and other processing pipelines using Gstreamer "plug-ins".
Supported resolution and frame rates for preview:
– VGA(640x480) : 30fps, 24fps, 15fps, 10fps and 5fps
Additionally supported on ARTIK 7 with S5K4EC:
– HD(1280x720) : 15fps, 10fps and 5fps
– 5MP(2560x1920) : still
Supported on OV5640:
– VGA, HD, FHD(1920x1080): 30fps
To install Gstreamer plug-ins from the command line, enter:
Ubuntu:
apt install gstreamer*
apt install gstreamer1.0-plugins-bad
Fedora:
dnf install gstreamer*
Display Preview on LCD
To clear the LCD screen before displaying a new preview, enter on the command line:
# cat /dev/zero > /dev/fb0
ffmpeg Display Preview
To display a preview screen on the LCD with ffmpeg
, this example
ffmpeg -f v4l2 -s 640x480 -r 30 -i /dev/video6 -pix_fmt bgra -f fbdev /dev/fb0
specifies:
-f v4l2
– use V4L2 capture device-s 640x480
– resolution-r 30
– frame rate is 30 fps-i /dev/video6
– input device for recording (MIPI camera)-pix_fmt bgra
– pixel format of LCD screen-f fbdev /dev/fb0
– output device is frame buffer device.
gstreamer Display Preview
You can display a preview screen on the LCD by using gst-launch
with plug-ins appended.
You can see a detailed description for each plug-in by executing
gst-inspect [plug-in name]
For example,
gst-launch -e v4l2src queue-size=16 device=/dev/video6 ! video/x-raw-yuv,framerate=30/1,width=640,height=480 ! ffmpegcolorspace ! fbdevsink device=/dev/fb0
specifies:
-e v4l2src queue-size=16 device=/dev/video6
– use v4l2src pluginvideo/x-raw-yuv,framerate=30/1,width=640,height=480
– format, framerate, and size of video, respectivelyffmpegcolorspace
– do color space conversion between yuv video and fbdevsinkfbdevsink device=/dev/fb0
– output device is frame buffer device.
Rotate the preview screen on the LCD by adding the “videoflip” plug-in to the Gstreamer pipeline, such as in
gst-launch -e v4l2src queue-size=16 device=/dev/video6 ! video/x-raw-yuv,framerate=30/1,width=640,height=480 ! videoflip method=counterclockwise ! ffmpegcolorspace ! fbdevsink device=/dev/fb0
where:
videoflip method=counterclockwise`
specifies the rotation direction.
Capture Images or Movies
fswebcam Image Capture
Refer to online man page for detailed usage information. Typical command line usage:
fswebcam -d [device node] -r [Resolution] -p [palette format] [filename]
S5K5EA:
ARTIK 710 example
fswebcam -d /dev/video6 -r 2560x1920 -p YUYV -D 1 --no-banner image.jpg
ARTIK 530 example
fswebcam -d /dev/video6 -r 2560x1920 -p YUV420P --no-banner image.jpg
ARTIK 710 example
fswebcam -d /dev/video6 -r 2592x1944 -S 10 -p YUYV --no-banner image.jpg
ARTIK 530 example
fswebcam -d /dev/video6 -r 2592x1944 -S 10 -p YUV420P --no-banner image.jpg
specifies:
-
-d /dev/video6
– name of device node (MIPI camera) -
-r 2560x1920
– resolution -
-p YUYV
orp YUV420P
– format of source image -
-D 1
– pre-capture Delay time (seconds). Since a camera usually needs time to process details such as focus, exposure, and white balance, use at least a 1 second pre-capture delay in order to capture a clearer image. -
-S xx -- skip first 'xx' frames
(OV5640 feature only) -
--no-banner
– to hide the banner on the JPEG image.
gstreamer Movie Recording (ARTIK 530/710)
The ARTIK 530 and 710 modules incorporate a hardware video codec. These examples take advantage of the hardware features to record .avi and .mp4 videos. Single image capture is not supported from the command line for OV5640 (you can still do so using the capture application to access the gstreamer library).
Example 1 – Record a video with no audio:
gst-launch-1.0 -e camerasrc camera-crop-width=1280 camera-crop-height=720 ! nxvideoenc bitrate=12000000 ! avimux ! filesink location=result.avi
Example 2 – Record a video with preview (using a 'tee' plugin to provide a 1-to-N pipe fitting):
gst-launch-1.0 -e camerasrc camera-crop-width=1280 camera-crop-height=720 ! tee name=t \ t. ! queue ! nxvideosink \ t. ! queue ! nxvideoenc bitrate=12000000 ! mp4mux ! filesink location=result.mp4
Example 3 – Record FHD movie by using scaler. Although maximum resolution of camera source is HD(1280x720), recording a FHD(1920x1080) movie using the hardware scaler is possible.
gst-launch-1.0 -e camerasrc camera-crop-width=1280 camera-crop-height=720 ! nxscaler scaler-crop-width=1280 scaler-crop-height=720 scaler-dst-width=1920 scaler-dst-height=1080 ! tee name=t \ t. ! queue ! nxvideosink \ t. ! queue ! nxvideoenc bitrate=17000000 ! avimux ! filesink location=result_fhd.avi
Example 4 – Record a movie by using camera source plugin with sound, with simultaneous display on LCD or HDMI screen (need to install additional gstreamer plugin for encoding sound)
Ubuntu:
apt install gstreamer1.0-plugins-ugly gstreamer1.0-plugins-bad
Fedora:
dnf install gstreamer1-plugins-ugly
gst-launch-1.0 -e camerasrc camera-crop-width=1280 camera-crop-height=720 ! tee name=t \ t. ! queue ! nxvideoenc bitrate=12000000 ! queue ! mux. \ autoaudiosrc ! queue max-size-buffers=1000 ! liveadder start-time-selection=2 start-time=600000000 ! lamemp3enc ! queue ! mux. \ t. ! nxvideosink \ mp4mux name=mux ! filesink location=result_a.mp4
ffmpeg Movie Recording
Refer to online man page for detailed usage information.
For OV5640:
– ARTIK 530 supports VGA 15fps recording only.
– ARTIK 710 supports VGA 15/30fps, HD 15fps.
Record without Sound
To record video with no audio, use this example as a template.
ffmpeg -y -f v4l2 -s 640x480 -r 15 -i /dev/video6 -b:v 2048k -vcodec mpeg4 test.avi
Record with Sound
To record a movie with sound, additionally specify audio channels (1:Mono, 2:Stereo), and card number from arecord
results (ac: audio channel, hw: card number, -t duration). Refer to the Audio Guide for more examples of ALSA parameters.
For an ARTIK 520/1020 example,
ffmpeg -y -thread_queue_size 2048 -f alsa -ac 1 -i hw:0 -f v4l2 -s 1280x720 -r 7 -i /dev/video6 -b:v 3072k -vcodec mpeg4 -t 20 test.avi
specifies:
-y
– overwrite output files-thread_queue_size 2048
– maximum number of queued packets. Raising this value can avoid audio packet loss on low-latency and high-rate live streams.-f alsa
– use ALSA for audio-ac 1
– audio channel is mono-i hw:0
– input audio card is 0-f v4l2
– use V4L2 capture device-s 640x480
– resolution-r 15
– frame rate is 15 fps-i /dev/video6
– input device for recording (MIPI camera)-b:v 2048k
– target bitrate of recorded video : 2Mbps-vcodec mpeg4
– use mpeg4 as a video codec-t 20
– duration of recording is 20 sec.test.avi
– output file name.
For an ARTIK 530/710 example,
ffmpeg -y -itsoffset 0.15 -thread_queue_size 2048 -f alsa -i hw:0 -f v4l2 -s 1280x720 -r 15 -i /dev/video6 -b:v 3072k -vcodec mpeg4 -t 20 test.avi
additionally specifies:
-
-itsoffset 0.15
– synchronize between audio and video by specified audio offset. For example:
– when audio is faster than video by 0.15 sec : -itsoffset 0.15
– when audio is slower than video by 0.5 sec : -itsoffset -0.5Recommended
itsoffset
value to apply to.avi
video is 0.15 (sec).
Since ffmpeg uses S/W codec for recording a video file, the tool may print DTS warning messages when the timestamp between the image source and encoded stream are not identical.
If you want to turn off verbose warning messages, you can change log level of ffmpeg by adding "-log level error" option as follows. Note that default loglevel is "info".
ffmpeg -y -itsoffset 0.15 -thread_queue_size 2048 -f alsa -i hw:0 -f v4l2 -s 1280x720 -r 15 -i /dev/video6 -b:v 3072k -vcodec mpeg4 -t 20 -loglevel error test.avi
Record with 'vcodec copy' Option
When encoding video using a software codec, rate and resolution are limited by the CPU performance. Maximum frame rate of video encoding with the ARTIK 5 solution is approximately 7 fps for HD (1280x720), requiring use of the -r
option to trade off frame rate for resolution.
In order to get full frame rate video with high resolution, it is necessary to bypass the software video codec as follows.
- Change the output format of the camera to
mjpeg
. - Specify
-vcodec copy
to just copy the stream without encoding with the codec.
ffmpeg -y -thread_queue_size 2048 -f alsa -ac 1 -i hw:0 -f v4l2 -s 1280x720 -vcodec mjpeg -r 30 -i /dev/video6 -vcodec copy -t 20 -loglevel error test.avi
Stream Video
Using ffmpeg and ffserver
You can stream a video with RTSP protocol over the network by using ffmpeg
and ffserver
. ffserver
receives FFM streams from the ffmpeg
instance as input, then streams them over RTSP. ARTIK acts as the RTSP server.
Supported resolutions and frame rates for video streaming
S5K5EA:
- VGA (640x480) : 15fps, 10fps and 5fps
- HD (1280x720): 7fps, and 5fps
OV5640:
- VGA : 30fps, 15fps
- HD: 15fps
- Configure
ffserver
with a server configuration file, such as thisffserver.conf
file for VGA 15fps streaming.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | Port 8090 BindAddress 0.0.0.0 RTSPPort 8091 RTSPBindAddress 0.0.0.0 MaxClients 1000 MaxBandwidth 10000 CustomLog - <Feed feed1.ffm> File /tmp/feed1.ffm ACL Allow 127.0.0.1 </Feed> <Stream test1.mp4> Feed feed1.ffm Format rtp VideoFrameRate 15 VideoBitRate 5000 VideoSize 640x480 VideoQMin 3 VideoQMax 31 NoAudio NoDefaults </Stream> |
-
Launch
ffserver
in the background.
#ffserver -f ffserver.conf &
-
Run a stream feed with
ffmpeg
on ARTIK device.
#ffmpeg -f v4l2 -s 640x480 -r 15 -i /dev/video6 http://localhost:8090/feed1.ffm
Receive and Play Streaming Content
On your PC, you can use media players like mplayer
or VLC as the RTSP client to receive and play video streams.
mplayer rtsp://[IP Address]:8091/test1.mp4
Note that you run mplayer
on the Host PC, not on the ARTIK board, to receive and play the video stream on the host (although you could set up a second ARTIK 530, 710, or 1020 board with an HDMI monitor to act as host).
Using Gstreamer
OV5640 – H/W Codec
You can stream a video over RTSP with gstreamer. Supported resolutions and frame rates for video streaming:
– VGA (640x480) : 30fps, 15fps
– HD (1280x720): 30fps, 15fps
– FHD (1920x1080): 30fps, 15fps
Install additional packages that are needed to build an rtsp application.
apt install build-essential
apt install gstreamer1.0-dev gstreamer-plugins-base1.0-dev
apt install libgstreamer-plugins-base1.0-dev
apt install libgstrtspserver-1.0-dev
Build the launcher program, which we will use instead of gst-launch-1.0.
gcc -o test-launch test-launch.c `pkg-config --cflags --libs gstreamer-1.0 gstreamer-rtsp-1.0 gstreamer-rtsp-server-1.0`
Refer to the code snippet of the launcher program below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 | #include <gst/gst.h> #include <gst/rtsp-server/rtsp-server.h> #define DEFAULT_RTSP_PORT "8554" static char *port = (char *) DEFAULT_RTSP_PORT; static GOptionEntry entries[] = { {"port", 'p', 0, G_OPTION_ARG_STRING, &port, "Port to listen on (default: " DEFAULT_RTSP_PORT ")", "PORT"}, {NULL} }; int main (int argc, char *argv[]) { GMainLoop *loop; GstRTSPServer *server; GstRTSPMountPoints *mounts; GstRTSPMediaFactory *factory; GOptionContext *optctx; GError *error = NULL; optctx = g_option_context_new ("<launch line> - Test RTSP Server, Launch\n\n" "Example: \"( videotestsrc ! x264enc ! rtph264pay name=pay0 pt=96 )\""); g_option_context_add_main_entries (optctx, entries, NULL); g_option_context_add_group (optctx, gst_init_get_option_group ()); if (!g_option_context_parse (optctx, &argc, &argv, &error)) { g_printerr ("Error parsing options: %s\n", error->message); g_option_context_free (optctx); g_clear_error (&error); return -1; } g_option_context_free (optctx); loop = g_main_loop_new (NULL, FALSE); /* create a server instance */ server = gst_rtsp_server_new (); g_object_set (server, "service", port, NULL); /* get the mount points for this server, every server has a default object * that be used to map uri mount points to media factories */ mounts = gst_rtsp_server_get_mount_points (server); /* make a media factory for a test stream. The default media factory can use * gst-launch syntax to create pipelines. * any launch line works as long as it contains elements named pay%d. Each * element with pay%d names will be a stream */ factory = gst_rtsp_media_factory_new (); gst_rtsp_media_factory_set_launch (factory, argv[1]); gst_rtsp_media_factory_set_shared (factory, TRUE); /* attach the test factory to the /test url */ gst_rtsp_mount_points_add_factory (mounts, "/test", factory); /* don't need the ref to the mapper anymore */ g_object_unref (mounts); /* attach the server to the default maincontext */ gst_rtsp_server_attach (server, NULL); /* start serving */ g_print ("stream ready at rtsp://127.0.0.1:%s/test\n", port); g_main_loop_run (loop); return 0; } |
Run the launch program to stream a video on ARTIK host.
Video Only
./test-launch "( v4l2src device=/dev/video6 ! video/x-raw,format=I420,framerate=30/1,width=1280,height=720 ! nxvideoenc ! rtph264pay name=pay0 )"
Video + Audio
./test-launch "( v4l2src device=/dev/video6 ! video/x-raw,format=I420,framerate=15/1,width=1280,height=720 ! nxvideoenc ! rtph264pay name=pay0 autoaudiosrc ! opusenc ! rtpopuspay name=pay1 )"
Audio Only
// opus codec
./test-launch "( autoaudiosrc ! opusenc ! rtpopuspay name=pay0 )"
//Law audio encoder
./test-launch "( autoaudiosrc ! alawenc ! rtppcmapay name=pay0 )"
Run the gst-launch-1.0 program to show the video stream on client artik. Make sure to connect an LCD or HDMI display.
gst-launch-1.0 -e rtspsrc location=rtsp://[host ip]:8554/test latency=0 ! rtph264depay ! h264parse ! nxvideodec ! nxvideosink