Performance Testing of Mobile Chatbot Applications

In CAMAD 2018, a paper (PDF) that I co-authored introduces a new metric related to the Service Stability of mobile Chatbot Applications. The service stability metric is directly dependent to the following observed issues:

  • Image Loss,
  • Message Repetition,
  • Message Reorder and
  • Auxiliary Keyboard Loss.

Each of these observed issues has a different bias in the calculation formula of service stability. In the following equation the N represents the total number of steps in each scenario and m represents the number of successfully executed steps. The variable Ii is equal to 1 for each step that is observed the corresponding issue and equal to 0 elsewhere.

Each test score is normalized to the maximum score that can be achieved for each chatbot, so the best value for the service stability is 1 and the worst is 0. This metric provides an insight of the QoE that the user experiences under different reception conditions, quantifying the impact of the various impairments to the final service provision.

For the experimental needs of the paper, the proposed metric was validated based on experimental data retrieved by the 5G-TRIANGLE experimental testbed. Therefore, three different types of chatbots over Viber platform used for the execution of the experiment, each one having a different degree of complexity and requirements. The following Table summarizes the features of each chatbot.

Chatbot No. Features
Static Messages Database Communication API-based Communication
Chatbot #1
Chatbot #2
Chatbot #3

For the execution of the experiment it was mandatory to use the commercial application of Viber, via which the user would have access to the chatbot service. For the emulation of cellular network, the TRIANGLE testbed is using the UXM Wireless Test Platform device by Keysight. This device is capable of modifying a number of parameters of the wireless physical layer and thus emulate various network conditions. The parameters that modified in this experiment are:

  • the number of Downlink and Uplink Physical Resource Blocks (PRBS),
  • the number of Downlink and Uplink Subframes,
  • the Multipath Fading Propagation Conditions (EPA, EVA, ETU),
  • the antenna output power,
  • the mode, the type (AWGN) and the power of the environmental noise,
  • the max Doppler shift

Each mobile device was directly connected to the Keysight Source Management Unit instead of the battery and supplied with 5V DV voltage. This set-up was offering a flexible configuration to meet the power sourcing and analysis requirements.

Screenshot_1

For the orchestration of the experimental process as well as the configuration of the UXM and SMU devices the TRIANGLE testbed used the Keysight KS8400A Test Automation Platform (TAP). This software was enabling a powerful, flexible and extensible test sequence and test plan creation.

The mobile devices were controlled via the Quomation WebDriver, a test automation framework for use with native, hybrid and mobile web apps. Furthermore, the Quamotion Frontend provided a device monitoring and controlling interface. The mobile devices used in this experiment were the Samsung Galaxy S7 and Samsung Galaxy S4.

 

Advertisements

Special Issue for IEEE MultiMedia Magazine

I am co-guest editing this special issue. You are invited to submit your research work. Please do not hesitate to contact me if there are questions.

Harilaos

Special Issue for IEEE MultiMedia

Advancing Multimedia Distribution with SDN/NFV-based Systems

Submission deadline: September 1, 2016
Publication: July-September, 2017

 

Recently, software-enabled appliances that support network virtualization have seen wide adoption by data centers and service providers. The main advantage of Software Defined Networking (SDN) is that the control plane is separated from the data plane, thereby allowing for network visibility, dynamic resource provisioning, and flexible service deployments. Complementary to SDN, Network Function Virtualization (NFV) focuses on virtualization of software-based network node functions, such as firewalls and intrusion detection. The increased agility and flexibility in the SDN/NFV-based network infrastructure opens up endless possibilities for multimedia content delivery and distribution. For example, virtual content delivery nodes can be dynamically deployed at the edge of the network in an on-demand fashion, potentially reducing energy costs and improving user satisfaction. This special issue aims to showcase and disseminate original research works that explore how multimedia content distribution can be supported and enhanced over SDN/NFV-based networks. Topics of interest for this special issue include, but are not limited to:

  • SDN/NFV-related architectures for next generation multimedia networks and services
  • Dynamic resource provisioning and management for SDN/NFV-based multimedia systems
  • QoE assessment for multimedia delivery over SDN/NFV-based multimedia systems
  • QoE management with dynamic policy deployments over SDN/NFV-enabled networks
  • Algorithms for live analysis and monitoring of audiovisual streaming quality in that correlates with network events in SDN or NFV-based multimedia systems
  • Real-time video adaptation algorithms over SDN/NFV-enabled networks
  • HTTP-based adaptive video streaming over SDN/NFV-enabled networks
  • Interactions between SDN/NFV architectures with content delivery networks (CDNs)
  • Applicability of SDN/NFV in mobile networks for multimedia provision
  • Integration of SDN/NFV-enabled mobile and fixed networks for multimedia services
  • Composition and deployment of Service Function Chains (SFCs) of video and/or VoIP services

Submission Guidelines

See https://www.computer.org/web/peer-review/magazines for general author guidelines. Submissions should not exceed 6,500 words, including all text, the abstract, keywords, bibliography, biographies, and table text. Each table and figure counts for 200 words.

Manuscripts should be submitted electronically (https://mc.manuscriptcentral.com/mm-cs), selecting this special issue option.

 

Guest editors

Xiaoqing Zhu, Cisco Systems Inc., USA

Harilaos Koumaras, NCSR Demokritos, Greece

Mea Wang, University of Calgary, Canada

David Hausheer, Technische Universitaet Darmstadt, Germany

 

Improving QoS and QoE for Mobile Communications – Journal of Computer Networks and Communications (JCNC)

Improving QoS and QoE for Mobile Communications – Journal of Computer Networks and Communications (JCNC)

Background and Scope:
The number of mobile Internet users and the volume of traffic generated by them keep increasing every year. In this context, the proliferation of smartphones, tablets, and cheap mobile data tariffs along with the emergence of new media services and the increasing use of 3G and upcoming 4G cellular networks as a substitute for traditional fixed-line connectivity, creates a host of new challenges when it comes to achieving good network performance and a satisfactory user experience. Beyond traditional multimedia applications such as IPTV and video conferencing which have a fairly very well-known set of performance requirements from the network perspective, the increasing uptake of web- and cloud-based services also introduces a new set of requirements for network performance. Furthermore, the evolution of mobile networks towards an all-IP paradigm (e.g. LTE) introduces new challenges for traditionally circuit-switched services such as voice telephony, where it is critical for operators to guarantee minimum levels of performance and improve them where possible.

For these reasons, operators need to understand, measure, and manage both quality and performance of the 3G and 4G services they offer. This has to be fulfilled on the technical quality of service (QoS) as well as on the quality of experience (QoE) level, since ultimately it is the human end-user who decides whether the quality of a given service is inacceptable, sufficient, or even exceeding expectations.

Topics of Interest:
This special issue aims at bringing together the state-of-the-art research on evaluating and improving QoS and QoE in wireless 3GPP networks. We encourage authors to submit recent unpublished work focused on the following topics. Potential topics include, but are not limited to:
. New mobile applications with special requirements in terms of network performance . Methods for objective and subjective QoS and QoE assessment . QoE evaluation methodologies for novel emerging mobile services . Subjective QoE testing of mobile services in the lab and in the field . Measurement, simulation, and evaluation techniques, tools, and testbeds . Combining user-, application-, and network-level measurements on different platforms . Modeling and mapping relationships between QoS and QoE . QoS and QoE monitoring in 3G and 4G networks . QoS- and QoE-driven resource control and service optimization in 4G networks, in particular regarding cross-layer approaches, extensions of current standards, and multitechnology connectivity . Trade-offs in contrast to QoS or QoE requirements (e.g., energy-efficient mechanisms coping with limited battery capacity, security, and reliability issues)

Before submission authors should carefully read over the journal’s Author Guidelines, which are located at http://www.hindawi.com/journals/jcnc/guidelines/. Prospective authors should submit an electronic copy of their complete manuscript through the journal Manuscript Tracking System at http://mts.hindawi.com/ according to the following timetable:
Manuscript Due Friday, 7 Sept 2012
First Round of Reviews Friday, 30 Nov 2012
Publication Date Friday, 4 Jan 2013

Lead Guest Editor:
. Pedro Merino, University of Malaga, Teatinos Campus, 29071 Malaga, Spain

Guest Editors:
. Maria G. Martini, Faculty of Computing, Information Systems and Mathematics, Kingston University, Kingston Upon Thames, UK . Raimund Schatz, Telecommunications Research Center Vienna (FTW), Vienna, Austria . Lea Skorin-Kapov, Faculty of Electrical Engineering and Computing, University of Zagreb, Unska 3, Zagreb, Croatia . Martin Varela, VTT Technical Research Centre of Finland, P.O. 1100, 90571 Oulu, Finland

Online version of this CfP:
http://www.hindawi.com/journals/jcnc/si/960596/cfp