Generic filters
Exact matches only
Filter by content type

Interaction- and Communication-based Systems

Prof. Dr. Simon Mayer

In our research group, we explore interactions among devices and people in ubiquitous computing environments. Our focus is on the integration of physical things into the Web, on increasing the autonomy of Web-enabled devices, and on making interactions of connected devices intelligible for people.


Automatic Classification of High vs. Low Individual Nutrition Literacy Levels from Loyalty Card Data in Switzerland

A new paper from our group has been published at the International Workshop on Multimedia Assisted Dietary Management (MADiMa 2022): Automatic Classification of High vs. Low Individual Nutrition Literacy Levels from Loyalty Card Data in Switzerland

Abstract: The increasingly prevalent diet-related non-communicable diseases (NCDs) constitute a modern health pandemic. Higher nutrition literacy (NL) correlates with healthier diets, which in turn has favorable effects on NCDs. Assessing and classifying people’s NL is helpful in tailoring the level of education required for disease self-management/empowerment and adequate treatment strategy selection. With recently introduced regulation in the European Union and beyond, it has become easier to leverage loyalty card data and enrich it with nutrition information about bought products. We present a novel system that utilizes such data to classify individuals into high- and low- NL classes, using well-known machine learning (ML) models, thereby permitting for instance better targeting of educational measures to support the population-level management of NCDs. An online survey (n = 779) was conducted to assess individual NL levels and divide participants into high- and low- NL groups. Our results show that there are significant differences in NL between male and female, as well as between overweight and non-overweight individuals. No significant differences were found for other demographic parameters that were investigated. Next, the loyalty card data of participants (n = 11) was collected from two leading Swiss retailers with the consent of participants and a ML system was trained to predict high or low NL for these individuals. Our best ML model, which utilizes the XGBoost algorithm and monthly aggregated baskets, achieved a Macro-F1-score of .89 at classifying NL. We hence show the feasibility of identifying individual NL levels based on household loyalty card data leveraging ML models, however due to the small sample size, the results need to be further verified with a larger sample size.

Author: Simon Mayer

Date: 21. January 2023

Human-Like Movements of Industrial Robots Positively Impact Observer Perception

A new paper from our group and in collaboration with the Institute of Behavioral Science and Technology has been published in the International Journal of Social Robotics: Human-Like Movements of Industrial Robots Positively Impact Observer Perception

Abstract: The number of industrial robots and collaborative robots on manufacturing shopfloors has been rapidly increasing over the past decades. However, research on industrial robot perception and attributions toward them is scarce as related work has predominantly explored the effect of robot appearance, movement patterns, or human-likeness of humanoid robots. The current research specifically examines attributions and perceptions of industrial robots—specifically, articulated collaborative robots—and how the type of movements of such robots impact human perception and preference. We developed and empirically tested a novel model of robot movement behavior and demonstrate how altering the movement behavior of a robotic arm leads to differing attributions of the robot’s human-likeness. These findings have important implications for emerging research on the impact of robot movement on worker perception, preferences, and behavior in industrial settings.

Author: Simon Mayer

Date: 20. December 2022

BetterPlanet: Sustainability Feedback from Digital Receipts

A new paper from our group has been published at the International Conference on Advances in Mobile Computing and Multimedia Intelligence (MoMM 2022): BetterPlanet: Sustainability Feedback from Digital Receipts

Abstract: The global food system accounts for 25–30% of anthropogenic greenhouse gas emissions. A large share of these emissions is due to individual food shopping patterns. Despite the rising concern about the environment, many individuals fail to act upon it and change their food consumption. In this study, we attempt to motivate individuals to reduce their food-shopping-induced environmental footprint. To narrow the intention-behavior gap, we propose a novel technical system that gives automated near-term sustainability feedback on individuals’ food shopping recorded on digital receipts and communicates this feedback through the mobile application BetterPlanet. Based on a small sample (n = 8), we find a directional decrease in the overall CO2-Scores. Therefore, our study demonstrates the technical feasibility of automated sustainability feedback from digital receipts. The proposed energy-weighted CO2-Scoring Model contributes to the growing knowledge body of sustainability assessment.

Author: Simon Mayer

Date: 27. November 2022

Increasing the Intelligence of Low-Power Sensors with Autonomous Agents

A new paper from our group has been published and won the Best Paper Award at the 2022 Workshop on Challenges in Artificial Intelligence and Machine Learning for Internet of Things (in conjunction with ACM SenSys 2022): Increasing the Intelligence of Low-Power Sensors with Autonomous Agents

Abstract: Low-power sensors are becoming ever more powerful, increasing both their energy efficiency as well as their processing capabilities. Much work in recent years has focused on optimizing machine learning models to low-power systems, typically to locally process sensor data. Significantly less attention has been paid to other artificial intelligence fields such as knowledge representation and automated reasoning, which may contribute to building autonomous devices. In this work, we present a low-power sensor node with an autonomous belief-desire-intention agent. This kind of agent simplifies the implementation of both proactive and reactive behaviors, promoting autonomy in our target applications. It does so by locally perceiving and reasoning, and then wirelessly broadcasting an intention, which can be forwarded to an actuator. The capabilities of the autonomous agent are demonstrated with a light-control application. Experiments demonstrate the feasibility of running intelligent agents in low-power platforms with little overhead.

Author: Simon Mayer

Date: 26. November 2022

Solar Semantics: Every Thing under the Sun

A new paper from our group has been published at the Next-Generation Internet of Things Infrastructures Workshop of the 12th International Conference on the Internet of Things: Every Thing Under the Sun: How Web of Things and Semantic Data Brings Benefit to Small-Scale Photovoltaic Installations

Abstract: Small-scale photovoltaic (PV) systems of up to a few kilowatts capacities are becoming increasingly available and affordable for off-grid installations. However, in our experience with using PV energy in farming in India, we found that many installations had faults or were lying underutilized. Though integrating such systems into IoT applications is now practical, analyzing the system’s performance and utilization requires knowledge of the components and the system design. Off-band infusion of this knowledge into the software applications leads to tight coupling and vertical silos. To address this challenge, we have developed an ontology to describe small-scale PV installations, which enables us to represent subsystems and their components in the form of Web of Things (WoT) Thing Descriptions. We show that our approach results in technical and semantic integration of the PV system into IoT applications, allowing the development of reusable fault detection and optimization programs. This reduces the cost of developing solutions to monitor and optimize the usage of PV systems, thereby bringing benefits to the farming community by improving their livelihood.

Author: Simon Mayer

Date: 26. November 2022

Semantic OCR through Augmented Reality

A new paper from our group has been published at the 12th International Conference on the Internet of Things: Semantic OCR through Augmented Reality (SOCRAR)


To enable people to interact more efficiently with virtual and physical services in their surroundings, it would be beneficial if information could more fluently be passed across digital and non-digital spaces. To this end, we propose to combine semantic technologies with Optical Character Recognition on an Augmented Reality (AR) interface to enable the semantic integration of (written) information located in our everyday environments with Internet of Things devices. We hence present SOCRAR, a system that is able to detect written information from a user’s physical environment while contextualizing this data through a semantic backend. The SOCRAR system enables in-band semantic translation on an AR interface, permits semantic filtering and selection of appropriate device interfaces, and provides cognitive offloading by enabling users to store information for later use. We demonstrate the feasibility of SOCRAR through the implementation of three concrete scenarios.

Link to the full paper

Author: Simon Mayer

Date: 20. November 2022

FoodCoach: Automated e-coach for a healthier diet

FoodCoach is a study by the University of St.Gallen and the University Hospital of Bern, supported by Swiss National Science Foundation and Sanitas Management AG. Sanitas Management AG supports the recruiting of study participants without interfering with the study content.

Project Goal:

The objective of this research project is to investigate the effectiveness of improving people’s food shopping healthiness by providing automated food shopping recommendations.

Participants donate their grocery data recorded on the loyalty card programs “Migros Cumulus” and “Coop Supercard”. Based on these data, the study team has built a model to generate automated recommendations according to the Nutri-Score framework and expertise of dieticians at the University Hospital of Bern. The results of the study could possibly help improve the health status of the Swiss population and beyond in a low-cost, automated, and longitudinal manner.


Unhealthy diets are a major risk for the development of non-communicable diseases. Changing unhealthy dietary behavior can have a significant impact on this risk. Providing interventions towards individual food purchase behavior offers tremendous potential to reduce cardiometabolic morbidity and mortality by the promotion of a healthier diet. By using digital receipts, data on purchased food can be collected automatically. This provides the opportunity to analyze food choices according to different criteria and design automated, low-cost and longitudinal interventions.

What can I benefit and how can I participate?

With successful participation, you will receive personalized analysis and recommendations on your food shopping.

You need to be at least 18 years old and use Migros Cumulus and/or Coop Supercard to join the study. Registration consists of two steps and takes roughly 1o minutes.

Should you have any questions, please contact .

Scan the QR code and join now!

Author: Jing Wu

Date: 20. November 2022

Towards Semantic Content Negotiation

A new poster with involvement of our group has been published at the 23rd International Conference on Knowledge Engineering and Knowledge Management: A Step Toward Semantic Content Negotiation



Content negotiation aims at enabling a server to provide a client with a representation of a resource that meets its needs. However, client and server might desire to negotiate constraints that go beyond the media type or language of the alternative representation. This is especially true in the Semantic Web, as a resource can be described with a single media type, but with different vocabularies (FOAF,, etc.), and may match specific patterns. In this paper, we propose an approach to increase the flexibility when negotiating a representation between client and server. Our approach follows the goals of the World Wide Web and uses a set of existing technologies: SHACL and profile-based negotiation. We define the mechanism (in terms of protocol and algorithm) for clients to announce their expectations and for servers to react and respond to them. We then explain, through a use case, how the same approach could be used in Web-based Multi-Agent Systems to help autonomous agents achieve their goals on the Web.

Link to the full paper

Author: Simon Mayer

Date: 28. September 2022



Full Professor, Interaction and Communication based Systems


Research Assistant, Hypermedia Multi-agent Systems