Está en la página 1de 5

Automatic Test Framework For Video Streaming

Quality Assessment
Dat Truong Tieu+, Binh Cong Nguyen+ , Linh Manh Le+, Quan Minh Pham+, Quang Tri Nguyen* and Dung Trung Vo+
+
Faculty of Electrical and Electronics Engineering
University of Technology, VNU-HCM
Ho Chi Minh City, Viet Nam
*
Personify Inc
Chicago, Illinois, USA

Abstract— To understand better what ha ppens when video streaming thus play a very important role in v ideo
streaming takes place, this paper introduces a framework to transmission.
simulate a real-time video streaming over wireless channels. The
system is divided into many modules and is simulated with In [1], the effect when blocks are missed in a single frame
different tools. DUMMYN ET is used for the network simulator during transmission is studied. That paper simu lates a single-
while FFMPEG is for coder/encoder an d sender/receiver module. hop wireless testbed with IEEE802.11n W LAN configuration.
The video quality is measured by spatial (SS IM and PS NR) If blocks in an intra-frame are lost due to one or more packages
metrics. To permit automatic characteristic, a system controller is lost, not only the quality of this intra-frame is affected but
with JENKINS is used to control the encoder, sender, network, also the quality of the inter-frames predicted based on this
receiver and decoder. A script at this server can be run to intra-frame is affected. This case is studied in [2] where this
simulate automaticly the system with many test cases for the error propagation is investigated with package loss. If the
practical network setting. The simulation results show that the artifact happens at fixed location for a long time, it becomes
simulated system can estimate accurately the quality of the very annoying. [3] also studies and predicts the visibility of
videos transmitted over wireless network. package loss for MPEG-2 videos. If both the original and
received videos are available, the quality metrics can be
Keywords— video streaming; quality assessment; FFMPEG;
calculated by a frame-based method. Metric with frame -based
DUMMYNET; JENKINS; PSNR; SSIM
methods are PSNR [4] and SSIM [5]. Not many metrics are
applied in real framework for video streaming yet. If the
I. INT RODUCTION original v ideos are not available at the decoder, optical flow [1]
Video streaming plays a very important role in our daily can be used to estimate the temporal quality of the videos.
lives and has become very popular these days. There exists a Based on a framework with video quality assessment, a
very diverse range of different video communication and specific solution to enhance their performances can be verified
streaming applications from video-on-demand channels like and be compared to other solutions. One application examp le is
YouTube, Netflix to live sports broadcasting or interactive on- that if the transmission bandwidth is dropped or is not stable,
line games and video conferencing like Skype. However, our the codec can reduce the speed by downsampling the frames or
knowledge about those kinds of applications’ behaviors in the increase the compression ratio. The video is thus in smaller
real world is quite not much. Therefore, to understand video bandwidth and is easier to be transmission over the channel
streaming better, it is necessary to build a framework for without any interruption. For these purposes, a novel
assessing their behaviors. framework is proposed in this paper to simulate the whole
Videos which are delivered to the users by transmission are process of video streaming in the real system. The system
suffered from the imperfect of the channel, such as package includes different modules which are implemented by specific
loss, delay, bandwidth drop… For v ideos, their quality is tools. Especially, the quality assessment module is done for
perceived by the viewers in a d ifferent scope than images’ spatial metrics.
quality. For images, the spatial quality is a concern. For The rest of the paper is organized as follow: section II
videos, the temporal quality of frames over time is also describes the architecture of the proposed framework for
considered. In a higher level, annoyance to video viewers can simulating the process of video streaming, section III, IV, V
be caused by one missing block or one low quality frame or and VI will describe in more details all the components of the
temporal quality inconsistent. Quality metric on v ideo framework. In section VII, measurement of the proposal
streaming is thus the key measurement not only to let viewers framework performances under some network conditions is
verifying the video quality but also to permit v ideo reported. Then, the conclusions and future works are
broadcasters enhancing their network. Quality metrics on video mentioned in section VIII.
II. PROPOSED FRAMEWORK ß Initialize the sender:
Fig. 1 describes the proposed framework, which has 5 main rtmp ://192.168.0.110/live/stream
components: Encoder/Sender module, Rece iver/Decoder
module, Network Emu lator module, Server module and Video ß Encode and send to server:
Quality Assessment module. ffmpeg -i input.mp 4 -c:v lib x264 -f flv
First, at the Encoder/Sender, the input video is encoded into rtmp ://localhost/live/stream
a bit stream. Then the bit stream is chopped into packets and ß Receive and decode:
sent to the Server. After that, the Server continues streaming
the packets through the Network Emulator, which emulates a ffmpeg -re -y -i rt mp ://localhost/live/stream live=1
plenty of network issues in the real world such as propagation output.flv

Fig. 1. Block diagram of the proposed framework

delay, packet loss, available bandwidth and multipath effect.


After going through the Network Emulator, the packets are IV. NETWORK EMULATOR
received and decoded into an output video by the There are plenty of tools and software for network
Receiver/Decoder. Then, the quality of the output video is emulat ion. In this paper, a non-charged but powerful tool
evaluated by comparing it to the input video at the Video called DUMM YNET [7] is selected due to its convenience and
Quality Assessment module. The System Controller will be in very simple usage along with the capability of running on a
charge of controlling the whole system by sending some trigger variety of operating systems.
signals to inform which component to start and when to start
and handling errors if occur. The basic concept of DUMM YNET is to intercept the
network stack with an object called "pipe". All packets going
Detail of each module in the proposed framework will be fro m the client to the server and vice versa have to enter these
exp lained in the following sections. Section III discusses about "pipes" before reaching the internet. With these "pipes", we can
the Encoder/Sender module and Receiver/Decoder module. implement a variety of network attributes, such as propagation
Section IV mentions about the Network Emulator module. delay, random packet loss, available bandwidth and mu ltipath
Next, section V describles the tool used for Server-System effect.
Controler module. Finally, section VI will detailize the Video
Quality Assessment Module. Two steps for setting up network environments using pipes:
ß Step 1: Create pipes with parameters
III. ENCODER/SENDER – RECEIVER/DECODER ipfw pipe <nu mber> config delay <var1>ms bw
FFMPEG [6] is used to create both the Encoder/Sender and <var2>Kbit/s plr <var3>
the Receiver/Decoder at the two ends of the streaming system.
The reason FFMPEG is chosen because in fact it is a non- <number>: pipe identification.
charged and powerful software with a cross-platform solution <var1>, <var2>, <var3>: input parameters setting up
to record, convert and stream audios/videos. Besides, FFMPEG for delay, bandwidth and packet loss rate, respectively.
supports many video formats and codecs as well as all the
common video streaming protocols such as: RTSP, ß Step 2: Add pipe into the network stack
RTP/RTCP, RTM P, HTTP ... ipfw add prob <var4> pipe <nu mber> proto
At the Encoder/Sender, the input video is first encoded into <protocol><down-up>
a bit stream. Then the bit stream is chunked into packets and <var4>: the probability for packet flows go through
sent out to the Server. At the Receiver/Decoder, similarly, this pipe.
FFMPEG does the inverse process of receiving the packets,
and decoding them into the video. <number>: pipe identification.
One example of the command to in itialize the stream; <protocol>: protocol in the transport layer, TCP, UDP
encode/send and receive/decode is or ICM P…
<down-up>: Dummynet defaults for both downlink run jobs. More details on JENKINS as well as on the
and uplink packets, or use in for incoming packets, out for framework establishment will be discussed in section VII.
uplink packets.
VI. VIDEO QUALITY A SSE SSMENT M ODULE
To remove pipe, the following co mmands can be used
Nu merous video quality assessment methods and metrics
ß Remove pipe <number> have been proposed over the past years with varying
ipfw –q pipe <nu mber> flush computational comp lexity and accuracy. Based on the
availability of the original and rece ived videos, for this study,
ß Remove all pipes: two traditional full-reference v ideo quality met rics are chosen:
ipfw –q flush PSNR (Peak Signal to Noise) and SSIM (Structural Similarity
Index Metric) for assessing the quality of received videos
Below is some examp les for network emu lator comparing to the original videos fro m the sender. Here we just
ß Delay: 50ms, TCP, downlink packets review the formu las for PSNR and SSIM co mputation, please
have a look at [10] and [11] for more details and specific
ipfw p ipe 1 config delay 50ms implementation.
ipfw add pipe 1 proto tcp in The formula
ß Loss: Packet loss rate: 0.1 %, UDP, uplin k packets
ipfw p ipe 2 config plr 0.1
ipfw add pipe 2 proto udp out
ß Bandwidth: Bandwidth management: 200Kbit/s, ICMP, where are the pixel values of two frames, one
both uplink and downlink packets fro m the input video and the other fro m the output video. M
ipfw p ipe 3 config bw 200Kbit/s and N are the horizontal size and the vertical size of the frame,
respectively. And n is the bit-depth, the number of b its
ipfw add pipe 3 proto icmp represent a pixel.
For two non-negative image signals x and y, the formu la
V. SERVER – SYST EM CONT ROLLER is:
Every video streaming system needs a streaming server.
For recent years, the Adobe Flash Media Server [8] has
emerged as a powerful and free software for providing where are the mean intensity of x, the mean
excellent functionality to create online video services. It intensity of y, the standard deviation of x, the standard
supports a plenty of live streaming protocols such as RTMP / deviation of y and the covariance between x and y,
RTMPE (Flash), HTTP DS (Flash), and HTTP LS (iOS).
respectively. are the calibrat ion constants.
Moreover, it also provides of high-performance mult i-format
content output to the users and runs of additional content VII. EXPERIMENT S
protection services. Because of those reasons, we decided to
In this section, the set-up of our proposed framework and
use the Adobe Flash Media Server as the Server in our report measured its performances under s ome network
proposed framework and RTMP as the streaming protocol.
conditions are described in details. Fig. 2 shows the proposed
The Real-Time Messaging Protocol (RTMP) is used for framework set-up.
streaming mu ltimedia content across a TCP/IP network. We
PC3
use ffmpeg to send and receive video.
Syntax o f a co mmand is:
rt mp://[username:password@]server[:port][/app][/
rt mp_live]
- Username: an optional username
- Password: an optional password
- Server: the address of the RTMP server.
- Port: the number of the TCP port to use (by
default is 1935). PC1 PC2
- App: the name of the application to access
- rt mp_live: live stream specification
For the System Controller, a tool ca lled JENKINS [9] is
used. JENKINS is an award-winning application that monitors Fig. 2. The proposed framework set-up
executions of repeated jobs, such as building/testing software
projects continuously and monitoring executions of e xternally-
A. Framework Set-up sending video from PC1 to PC3. The second one's task is to
Three co mputers are used for the set-up. PC1 plays the role activate The Network Emu lator in PC1. The third job is for
of the Encoder/Sender as sending the input video out to the PC2 to rece ive the video streaming fro m PC3 and then run the
Server. The Network Emu lator is also installed in PC1 instead VQA module afterward. All of the above jobs are managed
of PC2, not like the illustration in the first place in Figure 1 for and automatically triggered by JENKINS Server. Fig. 3 shows
getting some work load off PC2. the parameter setting for a mu ilti-job test and Fig. 4 shows the
JENKINS d ialogue for spatial test in the simu lated test frame.

Fig. 3. Parameter setting for a muilti-job test

Fig. 4. JENKINS dialogue for spatial test

The Network Emu lator, which is DUMM YNET, act as a


B. Framework Measurement Results
bridge between the Encoder/Sender and the streaming Server.
PC2 contains the module of Rece iver/Decoder receiv ing the Due to the main concern about the performance of video
incoming streaming v ideo fro m the Server. The Video Quality conferencing systems, many different test cases are simulated
Assessment (VQA) module is also set up in PC2. PC3 plays to investigate the performance of our frame work. The choice
two roles as both the Adobe Flash Media Server in charge of for the input videos is many head-and-shoulder videos
handling the streaming fro m PC1 and the JENKINS Server for encoded at two different bitrates. One for high quality
managing the whole system. JENKINS Server creates 3 jobs compression is with target bandwidth of 500kbps. The other
to conduct the above system. The first job is responsible for for lower quality co mpression is with target bandwidth of
100kbps. For network emulat ion, two cases are considered
with available bandwidth and propagation delay. For the VIII. CONCLUSIONS A ND FUTURE W ORKS
available bandwidth, two cases with bandwidth of 400kbps In this paper, a novel framework is proposed for simulating
and 200kbps are investigated. For the propagation delay, two the real-world v ideo streaming system. W ith this framework,
cases with propagation delay of 150ms and 50ms are it can simu late a v ideo streaming system in a real trans mission
considered. The results are shown in the Table I. belo w for system. This framework can also be used to evaluate the
both PSNR in d B and SSIM. performances of video-streaming-based applications.
In the future, more test cases should be examined on the
TABLE I. MEASUREMENT RESULTS OF THE P ROPOSED FRAMEWORK
framework with a great variety of cases on network issues and
different types of input videos as well as with other streaming
Video Quality Metrics
Video protocols. Besides, for this study, the VQA module is still
Network Test Cases Bitrate rather simp le. Only two popular full-reference metrics such as
PSNR SSIM PSNR and SSIM are imp lemented for evaluating the output
videos. In future, the VQA module will be extended with more
efficient and powerfu l metrics not only on spatial metrics but
500kbps 35.105 0.94443
on temporal metrics as well.
400kbps
100kbps 26.4432 0.80223 A CKNOWLEDGEMENT

Available This work was done during the time the authors were at
Bandw idth 500kbps 35.105 0.94443 Personify Inc.

200kbps REFERENCES
100kbps 26.4432 0.80223 [1] An (Jack) Chan, Amit Pande, Eilwoo Baik and Prasant Mohapatra,
“Temporal Quality Assessment for Mobile Videos”, MobiCom’12,
August 22–26, 2012, Istanbul, T urkey.
500kbps 35.105 0.94443 [2] Nick Feamster And Hari Balakrishnan, “Packet Loss Recovery For
Streaming Video”, 12th International Packet Video Workshop,
150ms Pittsburgh, PA, April 2002
100kbps 26.4432 0.80223
[3] Kanumuri, S.; Cosman, P.C.; Reibman, A.R.; Vaishampayan, V.A.,
Propagation "Modeling packet-loss visibility in MPEG-2 video," Multimedia, IEEE
Delay Transactions on , vol.8, no.2, pp.341,355, April 2006
500kbps 35.105 0.94443
doi: 10.1109/TMM.2005.864343
50ms [4] Q. Huynh-Thu and M. Ghanbari. Scope of validity of PSNR in
100kbps 26.4432 0.80223 image/video quality assessment. Electronics Letters, 44(13):800 –801,
19 2008.
[5] Z. Wang, L. Lu, and A. Bovik. Video quality assessment based on
structural distortion measurement. Signal Processing: Image
Apparently, based on Table I above, it can be seen that the Communication, 19(2):121–132, February 2004.
network characteristics does not affect the quality of the [6] https://www.ffmpeg.org/
output videos. All the PSNR values of the output videos with [7] http://info.iet.unipi.it/~luigi/dummynet/
the same bitrate are similar. The same thing happens to the [8] https://www.adobe.com/support/flashmediaserver/downloads_updaters.h
SSIM values. The reason is because of the chosen streaming tml
protocol, which is RTMP. RTM P is a TCP-based protocol, [9] https://jenkins-ci.org/
which obviously has flow control and erro r control. Therefore, [10] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “ Image
the output videos almost have its best form co mparing to the quality assessment: From error measurement to structural similarity,”
input videos. The only thing that affects the quality of service IEEE Trans. Image Processing, vol. 13, Jan. 2004.
here is just the video bitrate. Further check with the PSNR and [11] S. Chikkerur, V. Sundaram, M. Reisslein and L. J. Karam, “Objective
Video Quality Assessment Methods: A Classification, Review, and
SSIM at the sender verifies the robustness of the PSNR and Performance Comparison”, IEEE Trans. Broadcasting, vol. 57, No. 2,
SSIM values calculated at the receiver using the proposed June 2011.
Video Quality Assessment (VQA) module.

También podría gustarte