MOS Testing: A Robust Approach to Video Quality Evaluation

Introduction

Mean Opinion Score (MOS) is a standardized metric used to assess the perceived quality of multimedia content, particularly video communication. Originally, MOS was based on subjective human evaluations, where viewers would rate the quality of a video on a scale from 1 (Poor) to 5 (Excellent). Today, advanced algorithms simulate these subjective assessments, offering a more scalable and objective way to measure video quality.

Video MOS evaluates critical aspects of video content such as clarity, smoothness, resolution, frame rate, and the absence of visual artifacts like pixelation or blurring. By quantifying user perception, Video MOS provides a valuable benchmark for understanding and improving the quality of video communication in real-world conditions, making it an essential tool in industries like telecommunications, media, and gaming.

This blog will explore Video MOS and how it can be used to assess and optimize video call quality.

Understanding MOS: A Ground-Truth-Based Model

Why MOS?

MOS is crucial for understanding and enhancing the user experience in video communication across industries like media, telecommunications, and gaming. It helps identify service quality gaps and optimize platforms to deliver seamless, high-quality interactions.

By evaluating key factors such as visual clarity, frame consistency, and the absence of delays or distortions, MOS provides actionable insights into areas that require improvement. A high MOS score ensures smoother playback, better engagement, and an overall superior user experience, which is essential for retaining users in competitive digital environments.

‍● Importance of Perceived Quality

Perceived quality reflects how users experience video communication. MOS captures subjective human evaluations of video quality, emphasizing aspects like sharpness, frame rate, and smooth playback. It bridges the gap between technical performance and user satisfaction, focusing on aspects like sharpness, smooth playback, and the absence of delay. Addressing perceived quality ensures users remain engaged and have a positive interaction experience.

● Limitation of Reference-Based Approaches

Traditional reference-based approaches depend on ideal conditions or predefined benchmarks, which may not accurately reflect the real-world user experience. These methods often overlook dynamic factors such as varying network conditions, content complexity, or device capabilities. In video communication, reference-based evaluations may fail to capture the full spectrum of user perceptions, highlighting the need for adaptive, model-based approaches to account for these real-world variables.

Video MOS

Video MOS is determined through both subjective and objective methods. Historically, MOS was evaluated by human judges who scored the video quality based on their perceptions. Today, objective methods have been developed to predict MOS scores using algorithms that analyze video characteristics such as resolution, frame rate, and delay.

Modern Video MOS evaluations combine subjective human assessments with AI-driven models to predict video quality based on parameters like resolution, frame rate, and network latency. HeadSpin leverages these objective measurements to deliver accurate, scalable insights into video performance across devices.

A high video MOS score ensures better engagement during virtual interactions, making it an essential metric for modern communication systems.

Read: Testing Audio and Video on Real Devices - A Comprehensive Guide

Enhancing the MOS Model for Video Quality

The MOS model can be improved with advanced metrics and real-time analysis, incorporating machine learning algorithms to predict and optimize video quality under various network conditions. Real-world device testing across diverse scenarios ensures comprehensive and accurate evaluations.

Factors Influencing Video MOS Test Scores Vs. Factors Affecting Video Quality 

What Makes Video MOS Testing Effective

The Video MOS testing process evaluates key parameters to ensure a high-quality user experience. Regular testing and monitoring provide critical insights, enabling:

• Consistently clear and smooth video communication.
• Identification and resolution of performance gaps.
• Actionable strategies to enhance user satisfaction and engagement.

Network monitoring also plays a crucial role in assessing video call quality by providing end-to-end visibility into the network performance. It helps identify and resolve issues related to bandwidth, latency, and jitter, all of which can affect Video MOS scores.

By monitoring network performance, businesses can proactively address network issues, ensuring that video calls maintain a high MOS score and provide a seamless user experience.

How HeadSpin Implemented VMOS

Model Development Process

HeadSpin’s Video Quality Mean Opinion Score (VMOS) leverages advanced AI-based methodologies to evaluate reference-free video quality. The development process includes:

• Human-Labeled Data: Building a robust dataset with user-annotated subjective quality scores using the HeadSpin mobile annotation application.
• Diverse Video Content: Collecting real-world video streams from real devices to train models for better generalization.
• Validation Techniques: Employing statistical methods to eliminate user bias and validate quality scores using known ground-truth MOS data.
• Machine Learning Models: Integrating convolutional neural networks (CNNs) for extracting both high- and low-level visual features and tree-based algorithms to map these features to human-perceived MOS scores.

Issue Cards – Waterfall UI

The HeadSpin UI pairs VMOS outputs with Poor Video Quality Issue Cards to highlight problematic video stream segments.

• Visualization: A time-series representation identifies regions of perceptual video quality issues.
• Correlation Insights: Links video quality metrics with app-level performance to uncover root causes of quality degradation.
• Comparative Analysis: Provides tools to compare MOS across sessions or against reference videos for benchmarking and tracking improvements.

HeadSpin's AI Engine MOS Score

HeadSpin offers a sophisticated AI Engine that generates Mean Opinion Score (MOS) time series for videos captured directly on the HeadSpin Platform or supplied through an API. This algorithm estimates the MOS for each frame in the video, ranging from 1 (Very Poor) to 5 (Excellent) quality.

MOS Score Range

• MOS 1: Very Poor
• MOS 2: Poor
• MOS 3: Fair
• MOS 4: Good
• MOS 5: Excellent

iPhone X Video Examples

1. MOS Score: 3.54 [YouTube Video]

Comments: The Chinese and QR characters may need more clarity, but there are no visible blockiness issues.

2. MOS Score: 2 [NBA Video with Blur and Blockiness]

Comments: Noticeable blockiness affects the player on the video screen capture, while the scoreboard appears blurry.

3. MOS Score: 0.57 [YouTube Video with Severe Blockiness]

Comments: Severe blockiness renders the face of the person indistinguishable, impacting overall video clarity.

Also read: The Comprehensive Guide to Validating Audio-Visual Performances

Best Practices for Implementing Video MOS Testing with HeadSpin

Develop Distinct Experiments

To extract meaningful insights from the HeadSpin Video Quality MOS (Mean Opinion Score), it is crucial to design distinct experiments for each session or video upload. This will allow you to compare different test cases and ensure the statistical aggregate of the MOS time series is reflective of each scenario. For optimal results, ensure that each session is an independent test case representing real-world video content scenarios such as mobile streaming, gaming, or live events.

Leverage Rich Media Content

The HeadSpin MOS algorithm excels with rich media content. To get the most accurate results, focus on capturing video sessions or uploading content that mirrors the video types the system was trained on, such as streaming video, mobile gaming, console gaming, and live-streaming video. Videos with complex visual or temporal structures, such as fast-moving action scenes or live interactions, will ensure the AI provides an accurate MOS that reflects human perception.

Utilize the Full Power of the MOS Time Series

The HeadSpin Video Quality MOS generates a time series of quality scores for every frame of the video. To gain actionable insights, consider analyzing poor video quality regions by pairing the MOS time series with the Poor Video Quality Issue Card. This will highlight perceptual quality issues within specific time frames, helping you correlate video quality problems with other application metrics, such as network performance or device conditions.

Test with a Variety of Devices

Since the HeadSpin MOS algorithm is independent of device frame dimensions, it is important to test on a variety of devices to get a comprehensive view of how video quality is perceived across different hardware configurations. For videos with identical content, expect identical MOS time series results across different devices, ensuring consistent video quality measurement.

Compare MOS Scores Across Sessions

When a reference video or session is available, compare the MOS from new tests with previous ones. This will help you track video quality changes over time or across different conditions. Whether comparing the MOS from identical content captured on different devices or analyzing MOS distributions, such comparisons provide valuable insights into how various factors impact the viewer’s experience.

Optimize Your Experiment for the HeadSpin Platform

When testing video quality, ensure your content is optimized for the HeadSpin platform. The HeadSpin Video Quality MOS benefits from videos that have been captured in real-world conditions using HeadSpin’s device cloud, which offers diverse device configurations and network conditions. By capturing videos directly on HeadSpin devices or uploading them through an API, you can generate more reliable and representative MOS time series results.

By following these best practices, you can make the most of the HeadSpin Video Quality MOS and gain deeper insights into video quality that reflect the true user experience, regardless of device, network, or content type.

Conclusion

HeadSpin's expertise in network monitoring and real-device testing is crucial in optimizing video quality assessments. By testing video performance across real-world network conditions, HeadSpin ensures accurate insights into the factors that affect video quality. This comprehensive approach, combined with real-device testing, helps identify and resolve performance issues, improving overall call quality in video scenarios. As a result, teams can deliver seamless, high-quality video experiences for headspin users, regardless of device or network limitations.

Connect Now

FAQs

Q1. What role does the content delivery network (CDN) play in video streaming?

Ans: The content delivery network (CDN) is essential for video streaming as it brings content closer to viewers than the origin server. This proximity reduces round-trip time (RTT), enhancing streaming efficiency. Moreover, utilizing a CDN minimizes the risk of bandwidth-related delays, ensuring smoother live streams for viewers.

Q2. What methods are used for video quality testing?

Ans: Video quality testing employs various approaches:

1. Frame-Level Analysis: This technique is utilized for Set-Top Boxes (STBs) and yields specific Key Performance Indicators (KPIs) regardless of codecs or protocols. Leveraging the Media Processing Unit (MPU), it records and processes live or on-demand video using a non-referenced model.

2. IP-Level Analysis: KPI generation relies on bitstream analysis, which is compatible with encrypted and unencrypted transmission.

3. Application-Level Analysis: This method involves high-level KPIs and user-defined scripts, offering insights into the end-user experience provided by a video application.