In CAMAD 2018, a paper (PDF) that I co-authored introduces a new metric related to the Service Stability of mobile Chatbot Applications. The service stability metric is directly dependent to the following observed issues:
- Image Loss,
- Message Repetition,
- Message Reorder and
- Auxiliary Keyboard Loss.
Each of these observed issues has a different bias in the calculation formula of service stability. In the following equation the N represents the total number of steps in each scenario and m represents the number of successfully executed steps. The variable Ii is equal to 1 for each step that is observed the corresponding issue and equal to 0 elsewhere.
Each test score is normalized to the maximum score that can be achieved for each chatbot, so the best value for the service stability is 1 and the worst is 0. This metric provides an insight of the QoE that the user experiences under different reception conditions, quantifying the impact of the various impairments to the final service provision.
For the experimental needs of the paper, the proposed metric was validated based on experimental data retrieved by the 5G-TRIANGLE experimental testbed. Therefore, three different types of chatbots over Viber platform used for the execution of the experiment, each one having a different degree of complexity and requirements. The following Table summarizes the features of each chatbot.
|Static Messages||Database Communication||API-based Communication|
For the execution of the experiment it was mandatory to use the commercial application of Viber, via which the user would have access to the chatbot service. For the emulation of cellular network, the TRIANGLE testbed is using the UXM Wireless Test Platform device by Keysight. This device is capable of modifying a number of parameters of the wireless physical layer and thus emulate various network conditions. The parameters that modified in this experiment are:
- the number of Downlink and Uplink Physical Resource Blocks (PRBS),
- the number of Downlink and Uplink Subframes,
- the Multipath Fading Propagation Conditions (EPA, EVA, ETU),
- the antenna output power,
- the mode, the type (AWGN) and the power of the environmental noise,
- the max Doppler shift
Each mobile device was directly connected to the Keysight Source Management Unit instead of the battery and supplied with 5V DV voltage. This set-up was offering a flexible configuration to meet the power sourcing and analysis requirements.
For the orchestration of the experimental process as well as the configuration of the UXM and SMU devices the TRIANGLE testbed used the Keysight KS8400A Test Automation Platform (TAP). This software was enabling a powerful, flexible and extensible test sequence and test plan creation.
The mobile devices were controlled via the Quomation WebDriver, a test automation framework for use with native, hybrid and mobile web apps. Furthermore, the Quamotion Frontend provided a device monitoring and controlling interface. The mobile devices used in this experiment were the Samsung Galaxy S7 and Samsung Galaxy S4.