Performance Testing of Mobile Chatbot Applications

In CAMAD 2018, a paper (PDF) that I co-authored introduces a new metric related to the Service Stability of mobile Chatbot Applications. The service stability metric is directly dependent to the following observed issues:

  • Image Loss,
  • Message Repetition,
  • Message Reorder and
  • Auxiliary Keyboard Loss.

Each of these observed issues has a different bias in the calculation formula of service stability. In the following equation the N represents the total number of steps in each scenario and m represents the number of successfully executed steps. The variable Ii is equal to 1 for each step that is observed the corresponding issue and equal to 0 elsewhere.

Each test score is normalized to the maximum score that can be achieved for each chatbot, so the best value for the service stability is 1 and the worst is 0. This metric provides an insight of the QoE that the user experiences under different reception conditions, quantifying the impact of the various impairments to the final service provision.

For the experimental needs of the paper, the proposed metric was validated based on experimental data retrieved by the 5G-TRIANGLE experimental testbed. Therefore, three different types of chatbots over Viber platform used for the execution of the experiment, each one having a different degree of complexity and requirements. The following Table summarizes the features of each chatbot.

Chatbot No. Features
Static Messages Database Communication API-based Communication
Chatbot #1
Chatbot #2
Chatbot #3

For the execution of the experiment it was mandatory to use the commercial application of Viber, via which the user would have access to the chatbot service. For the emulation of cellular network, the TRIANGLE testbed is using the UXM Wireless Test Platform device by Keysight. This device is capable of modifying a number of parameters of the wireless physical layer and thus emulate various network conditions. The parameters that modified in this experiment are:

  • the number of Downlink and Uplink Physical Resource Blocks (PRBS),
  • the number of Downlink and Uplink Subframes,
  • the Multipath Fading Propagation Conditions (EPA, EVA, ETU),
  • the antenna output power,
  • the mode, the type (AWGN) and the power of the environmental noise,
  • the max Doppler shift

Each mobile device was directly connected to the Keysight Source Management Unit instead of the battery and supplied with 5V DV voltage. This set-up was offering a flexible configuration to meet the power sourcing and analysis requirements.

Screenshot_1

For the orchestration of the experimental process as well as the configuration of the UXM and SMU devices the TRIANGLE testbed used the Keysight KS8400A Test Automation Platform (TAP). This software was enabling a powerful, flexible and extensible test sequence and test plan creation.

The mobile devices were controlled via the Quomation WebDriver, a test automation framework for use with native, hybrid and mobile web apps. Furthermore, the Quamotion Frontend provided a device monitoring and controlling interface. The mobile devices used in this experiment were the Samsung Galaxy S7 and Samsung Galaxy S4.

 

Advertisements

Mobilware’11

Mobilware 2011

4th Int. ICST Conf. on MOBILe Wireless
MiddleWARE, Operating Systems, and Applications
June 22-24, 2011 – London, UK

ANNUAL THEME
Towards Mobile Middleware Enablers for Seamless Service Provisioning http://www.mobilware.org/index.shtml

Important Dates
Paper Submission Deadline: February 1, 2011
Notification of Acceptance: April 15, 2011
Camera Ready Deadline: May 15, 2011
Conference Date: June 22-24, 2011

SCOPE OF CONFERENCE
The advances in wireless communication technologies and the proliferation of mobile devices have enabled the realization of intelligent environments for people to communicate with each other, interact with
information-processing devices, and receive a wide range of mobile wireless services through
various types of networks and systems everywhere, anytime. A key enabler of these pervasive and
ubiquitous connectivity environments is the advancement of software technology in various
communication sectors, ranging from communication middleware and operating systems to networking protocols and applications.

TOPICS
The conference solicits original technical papers, not previously published and not currently under review for publication elsewhere.
Topics of interest include but are not limited to:
– New middleware concepts for mobile devices
– Cross platform application development
– Open terminal and network APIs
– End-to-end architectures for seamless service provisioning and deployment – Integration of heterogeneous wired and wireless networks and frequency bands – Interworking of mobile applications in mobile cloud environments – QoS awareness, adaptation, and fault-tolerance of mobile services – Opportunistic and delay-tolerant mobile and wireless networking – Location-aware and context-aware networking and computing
– Energy-efficient applications and services and OS air interface management – Mobility and handoff management
– Location and tracking supports and services
– Human-computer interface and portable 3D graphics for mobile devices – Novel applications and communication protocols
for wireless sensor networks, vehicular networks, and home networks – Modeling, simulation, and performance
evaluation of mobile wireless systems and services
– Trustworthiness, security, and privacy of mobile and wireless systems

PUBLICATION
Accepted papers will be published by
Springer-Verlag in the Lecture Notes of the
Institute for Computer Sciences, Social
Informatics and Telecommunications Engineering
(LNICST) series. The proceedings will be
available in both soft and hard copies by Springer.

SPECIAL ISSUES
Selected high-quality papers will be invited to a
Special Issue of a prestigious journal in the
area, probably Springer Mobile Networks and
Applications (MONET) and/or Elsevier Pervasive
and Mobile Computing (PMC) Journals approval pending.

Steering Committee
Paolo Bellavista
University of Bologna (Italy)
Carl Chang
Iowa State University (USA)
Imrich Chlamtac
Create-Net (Italy)
Thomas Magedanz
Technical University Berlin (Germany)

Organizing Committee
General Chairs
Vladimir Getov
University of Westminster (UK)
Martin Sauter
WirelessMoves (Germany)

TPC Chairs
Stephan Steglich
Fraunhofer FOKUS (Germany)
Nalini Venkatasubramanian
University of California, Irvine (USA)
Qinghua Zheng
Xian Jiaotong University (P.R. China)

Publicity Chairs
Carlos Becker Westphall
Federal University of Santa Catarina (Brazil)
Panagiotis Chountas
University of Westminster (UK)

Web Chair
Carlo Giannelli
University of Bologna (Italy)

Conference Coordinator
Gergely Nagy
ICST

Local Chair
Alexander Bolotov
University of Westminster (UK)

Technical Program Committee
Allott Nick
Wholesale Application Community (UK)
Bonnin Jean-Marie
Telecom Bretagne(France)
Borcea Cristian
New Jersey Institute of Technology (USA)
Boussard Mathieu
Alcatel Lucent Bell Labs France (France)
Bu Jiajun
Zhejiang University (China)
Budimir Djuradj
University of Westminster (UK)
Caceres Marcos
Opera (Norway)
Cai Ying
Iowa State University (USA)
Cao Jiannong
Hong Kong Polytechnic University (HongKong)
Cheng Liang
Lehigh University (USA)
Dan Grigoras
University College Cork (Republic of Ireland)
Demeure Isabelle
Telecom Paris (France)
Dixit Sudhir
HP (India)
Dragehorn Olaf
University Kassel (Germany)
Economou Daphne
University of Westminster (UK)
Faeth Matthias
TNO (The Netherlands)
Fasbender Andreas
Ericsson (Germany)
Fu Xinwen
University of Massachusetts Lowell (USA)
Fuhrhop Christian
Fraunhofer FOKUS (Germany)
Gavras Anastasius
Eurescom (Germany)
Georgalas Nektarios
British Telecom (UK)
Giambene Giovanni
Università degli Studi di Siena (Italy)
Gunes Mesut
FU Berlin (Germany)
H. Glitho Roch
Concordia University (Canada)
Hesselman Cristian
Novay (The Netherlands)
Hirschfeld Robert
Hasso-Plattner-Institut Potsdam (Germany)
Holtel Stefan
flensburg.school for Advanced Business Research (Germany)
Hong Feng
Ocean University of China (China)
Jaokar Ajit
Futuretext (UK)
Ji Yang
BUPT (China)
Kanter Theo
Mid Sweden University (Sweden)
Kernchen Ralf
University of Surrey (UK)
Koolwaaij Johan
Novay (The Netherlands)
Krussel Steffen
Deutsche Telekom (Germany)
Ku Wei-Shinn
Auburn University (USA)
Kurze Martin
Deutsche Telekom (Germany)
Lachmund Sven
DOCOMO Euro-Labs(Germany)
Langendoerfer Peter
IHP Microelectronics (Germany)
Li Xiuqi (Suze)
University of North Carolina (USA)
Li Yan
Huawei (China)
Li Xiaolin
University of Florida (USA)
Lohi Mahi
University of Westminster (UK)
Minerva Roberto
Telecom Italia – Future Centre (Italy)
Moessner Klaus
University of Surrey (UK)
Nadeem Tamer
Old Dominion University (USA)
Nord Christian
Sony Ericsson (Sweden)
Ohsaki Hiroyuki
Osaka University (Japan)
Paul Andre
Fraunhofer FOKUS (Germany)
Quantz Joachim
ART+COM (Germany)
Shikh-Bahaei Mohammad R.
King’s College London (UK)
Raggett Dave
W3C (UK)
Ravi Nishkam
NEC Labs (USA)
Roman Gruia-Catalin
Washington University in St. Louis (USA)
Salden Alfons
Almende (The Netherlands)
Schwaiger Roland
Deutsche Telekom (Germany)
Sun Weiwei
Fudan University (China)
Taheri Javid
The University of Sydney (Australia)
Tao Xianping
Nanjing University (China)
Tarczynski Andrzej
University of Westminster (UK)
Tripathi Anand
University of Minnesota (USA)
Wagner Matthias
DOCOMO Euro-Labs (Germany)
Wang Xiaodong
National University of Defence Technology (China)
Xue Guangtao
Shanghai Jiao Tong University (China)
Zhu Yanmin
Shanghai Jiao Tong University (China)
Zambonelli Franco
University of Modena and Reggio Emilia (Italy)

Future Network & MobileSummit 2011

Future Network & Mobile Summit 2011 Conference and Exhibition
15 – 17 June 2011, Warsaw, Poland

Future Network & MobileSummit 2011 (previously ICT-MobileSummit) is the twentieth in a series of Annual Conferences supported by the European Commission, which regularly attracts over 600 delegates from industry and research to share experiences and research results, identify future trends, discuss business opportunities and identify opportunities for international research collaboration under the ICT Theme of Framework Programme 7 (FP7). It will thus contribute to showcasing European research in the field, and position it within the multiplicity of related initiatives supported in other regions of the world.

Future Network & MobileSummit 2011 will address all the challenges of building the Future Internet, which will be based on mobile, wireless and fixed broadband communications infrastructures. It showcases European research in the field, and positions it within the multiplicity of related initiatives supported in other regions of the world.

The Scientific Programme incorporates all the key constituents of Future Communication Networks including the themes Radio Access and Spectrum, Converged and Optical Networks, and Future Internet Technologies.

Supported by the European Commission and eMobility, the Future Network & MobileSummit’s reputation is based on high quality networking, plenary sessions that discuss strategic issues in core thematic areas, and paper sessions that showcase original results in all areas of mobile, wireless and fixed broadband communications systems and networks.

Future Network & MobileSummit 2011 is Technically Co-Sponsored by IEEE Poland Communications Chapter Krakow, IEEE Poland Computer Chapter and IEEE UK/RI Computer Chapter.

While European funded research results are of particular interest, innovative research funded commercially or at national level is equally interesting from a community perspective. The participation of representatives of leading commercial, research and regulatory organizations in the Conference Community ensures a stimulating and high quality networking environment.

Call for Papers
Important Deadlines:
Submission deadline for full papers: 10 December 2010
Acceptance decision: 29 January 2011
Final Paper Submission: 11 March 2011

The scientific programme for Future Network & MobileSummit 2011 is based on an open Call for Papers and the conference language is English.
Interested presenters are encouraged to prepare a paper of 6 – 8 pages and submit online by 10 December.
All papers are double blind reviewed. Please ensure that there are no contact details within the text of your paper submission for review (Word document or PDF), the properties of the file or the file name prior to uploading it onto the server. Please enter all authors and co-authors details as part of the online submission form. Please ensure that the paper uploaded by 10 December is your final version to be reviewed as all papers will be assigned to the TPC following the paper submission deadline.
All submissions will be double blind reviewed by the Technical Programme Committee and authors will receive feedback by 29 January. Accepted authors will then be invited to submit a final paper taking account of feedback provided for inclusion in the conference proceedings by 11 March.

Core Thematic Priorities for Future Network & MobileSummit 2011
– Access and Spectrum
– Next-generation Radio System Architectures and Protocols
– Cooperative Wireless Networking
– Advanced Physical Layer Techniques
– Cognitive and Reconfigurable Radio Systems
– Radio Spectral Coexistence and Future Frequency Bands
– Radio Network and Terminal Self-configuration and -management

Converged and Optical Networks
– WDM and long-reach Passive Optical Networks
– Mobile Access Networks based on Free-space Optical Technology including Infrared and Visible Light
– Next-generation Active Optical (P2P) Networks
– Gigabit Home and In-Building Networks
– Fixed-mobile Convergence including Infrastructure Convergence, Centralisation in Radio-over-fibre, Femto-cells and Supporting Solutions

Future Internet Technologies
– Future Internet Architecture and Protocol Design
– Scalable Architectures for the Real World Internet
– Future Internet Self-Management Frameworks
– Network Overlay, Virtualisation and Federation
– Future Internet Experimental Facilities and Benchmarking
– Platforms and Middleware for Seamless Service and Content Delivery
– Security, Trust and Privacy in the Future Internet

More information is available at
www.futurenetworksummit.eu/2011/default.asp?page=c4p