The Artificial Intelligence and Machine Learning [AIML] chair headed by Prof. Damian Borth covers the area of deep neural networks. Our research focuses on representation learning through supervised and unsupervised approaches with applications to computer vision, remote sensing, text-to-speech synthesis, and financial time-series data. You can find more details about our research at www.hsg.ai.

News

Radio Interview with SRF4 about Climate Change Monitoring from Space

Prof. Dr. Damian Borth has been interviewed by Radio SRF4 News for the “4x4 podcast”. The World Climate Council published its report on the current state of climate change, including global warming. In this context, Damian Borth was asked what contribution Deep Neural Networks and satellite data can make. You can learn more about this in the Podcast (German) at minute 04:10.

Climate Pollutant Monitoring from Space

The University of St.Gallen (HSG) covered our recent research about Climate Pollutant Monitoring from Space. Since global CO2 emissions are a blind spot on the data map we are working. changing that. Using satellite images, we aim to locate all fossil fuel power plants worldwide to track down pollutant emitters. With the help of deep neural networks it is quite possible to identify the type of fos...

Panel Session at St. Gallen Symposium (50th anniversary)

“A Looming Digital Identity Crisis” Prof. Damian Borth was invited to the St.Gallen Symposium to participate in a panel discussion about digital trust. The discussion was forming about the role of fake news in demographic processes, and secure digital identities helping us to gain trust in a post-pandemic world relying more on digitalisation. This year, the St. Gallen Symposium celebrate...

Keynote – Presentation at the St.Gallische Naturwissenschaftliche Gesellschaft

Prof. Dr. Damian Borth has been invited to give a keynote presentation at the St.Gallische Naturwissenschaftliche Gesellschaft (NWG) as part of the HSG public lecture series. The NWG aims to bring scientific insights and new research results closer to its members as well as to the public. In his talk Damian Borth highlights what is behind Artificial Intelligence (AI) and how Deep Learning could ev...

Keynote: Urban Digital Talks at Stiftung Energie & Klimaschutz and Baden-Württemberg-Stiftung

Prof. Dr. Damian Borth was invited to give a keynote at the Urban Digital Talks organized by Stiftung Energie & Klimaschutz and Baden-Württemberg-Stiftung. The event was visited by 60 students, young professionals and interested parties who discussed opportunities for Artificial Intelligence (AI) and to tackle climate change. Damian Borth spoke about the use of Deep Neural Networks and a sate...

Invited Talk at Deep Fakes Workshop at Roman Herzog Institute

How much Deep Fakes can our society take? The fact that manipulated news, photos, videos and and podcasts go viral in the social media and appear credible also has to do with their technical perfection. Prof. Dr. Damian Borth explains that synthetically generated data can indeed be used as deep fakes on the one hand, but the other hand can be used for good. For example, to give patients thei...

Keynote at LANXESS “Digital Morning”

Prof. Dr. Damian Borth was invited to the LANXESS Digital Morning to give a keynote speech about Deep Neural Networks for the chemical industry. The keynote was livestreamed to the entire Lanxess corporation and watched by over 800 participants.

Interview at Top Online TV about AI

TOP FOKUS reporters met with Prof. Dr. Damian Borth to learn more about the current state of research in Artificial Intelligence (AI). In the interview Damian Borth outlines current AI research in general and mentions examples of his current research at HSG (for more details have a look at the second video, minute 01:51).

News Overview

Research

Representation Learning with Deep Neural Networks

One of our fundamental research direction is representation learning with deep neural networks. Being able to understand, to disentangle, and to control the latent factors of the underlying data would make deep learning more scalable and more trustworthy. It would allow us to gradually move from supervised to unsupervised learning and therefore render model training less dependent on large amounts of labels. Additionally, having control over the latent space of the data provides interpretability of model inference, increase robustness, and allows to characterise model parameter in the weight space.

Analysis and Synthesis of Multimodal Data

One of our research goals is to make the large amount of visual and audio content accessible. This includes besides the analysis of such data also the generation of synthetic data with natural characteristics. Currently we focus our research on neural Text-to-Speech (TTS) generation for the German language. In this scope we aim to develop novel approaches for speaker embedding, prosody control, and voice transfer.

Remote Sensing

We are interested in how we can utilize the vast amount of remote sensing data for earth observation. In particular, we aim to develop novel deep neural networks architectures able to consume the full ensemble of spectral bands from today’s satellites for applications such as climate change monitoring, natural disaster recovery, or changes in land use and land cover over time.

Financial Data

Our research in this area covers the analysis and forecasting of financial time-series data on the one hand and anomaly detection in transaction data on the other hand. Our research focuses in particular on financial markets and the audit domain.

Team Member

Prof. Dr. Damian Borth

Full Professor for Artificial Intelligence and Machine Learning

+41 71 224 26 27

Angelika Gräfingholt

Office Manager Prof. Borth

+41 71 224 26 27

Prof. Dr. Michael Mommert

Assistant Professor

+41 71 224 2615

Marco Schreyer

Researcher

+41 71 224 79 13

Hamed Hemati

Researcher

+41 71 224 2627

Konstantin Schürholt

Researcher

+41 71 224 34 54

Shijun Wang

Researcher

+41 71 224 73 70

Linus Mathias Scheibenreif

Researcher

+41 71 224 26 98

Joëlle Hanna

Researcher

+41 71 224 26 27

Timur Sattarov

Researcher (ext.)

+4915731585025

Diyar Taskiran

Intern

+41 71 224 26 27

Most recent Publications

Federated Continual Learning to Detect Accounting Anomalies in Financial Auditing

@ Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022)

get_appMarco Schreyer, Hamed Hemati, Damian Borth, Miklos A. Vasarhelyi

Conference or Workshop Item

The International Standards on Auditing (ISA) require auditors to collect reasonable assurance that financial statements are free of material misstatement. At the same time, a central objective of Continuous Assurance is the real-time assessment of digital accounting journal entries. Recently, driven by the advances in artificial intelligence, Deep Learning techniques have emerged in financial auditing to examine vast quantities of accounting data. However, learning highly adaptive audit models in decentralized and dynamic settings remains challenging. It requires the study of data distribution shifts over multiple clients and time periods. In this work, we propose a Federated Continual Learning framework enabling auditors to learn audit models from decentral clients continuously. We evaluate the framework's ability to detect accounting anomalies in common scenarios of organizational activity. Our empirical results, using real-world datasets and combined federated-continual learning strategies, demonstrate the learned model's ability to detect anomalies in audit settings of data distribution shifts.

More

Federated and Privacy-Preserving Learning of Accounting Data in Financial Statement Audits

@ 3rd ACM International Conference on Artificial Intelligence in Finance (ICAIF)

get_appMarco Schreyer, Timur Sattarov, Damian Borth

Conference or Workshop Item

The ongoing 'digital transformation' fundamentally changes audit evidence's nature, recording, and volume. Nowadays, the International Standards on Auditing (ISA) requires auditors to examine vast volumes of a financial statement's underlying digital accounting records. As a result, audit firms also `digitize' their analytical capabilities and invest in Deep Learning (DL), a successful sub-discipline of Machine Learning. The application of DL offers the ability to learn specialized audit models from data of multiple clients, e.g., organizations operating in the same industry or jurisdiction. In general, regulations require auditors to adhere to strict data confidentiality measures. At the same time, recent intriguing discoveries showed that large-scale DL models are vulnerable to leaking sensitive training data information. Today, it often remains unclear how audit firms can apply DL models while complying with data protection regulations. In this work, we propose a Federated Learning framework to train DL models on auditing relevant accounting data of multiple clients. The framework encompasses Differential Privacy and Split Learning capabilities to mitigate data confidentiality risks at model inference. Our results provide empirical evidence that auditors can benefit from DL models that accumulate knowledge from multiple sources of proprietary client data.

More

Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights

@ Conference on Neural Information Processing Systems

get_appKonstantin Schürholt, Boris Knyazev, Xavier Giro-i-Nieto, Damian Borth

Conference or Workshop Item

Learning representations of neural network weights given a model zoo is an emerg- ing and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation. Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation, which captures intrinsic and extrinsic properties of the models in the zoo. In this work, we ex- tend hyper-representations for generative use to sample new model weights. We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and several sampling methods based on the topology of hyper-representations. The models generated using our methods are diverse, per- formant and capable to outperform strong baselines as evaluated on several down- stream tasks: initialization, ensemble sampling and transfer learning. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.

More

Model Zoos: A Dataset of Diverse Populations of Neural Network Models

@ 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks.

get_appKonstantin Schürholt, Diyar Taskiran, Boris Knyazev, Xavier Giro-i-Nieto, Damian Borth

Conference or Workshop Item

In the last years, neural networks (NN) have evolved from laboratory environments to the state-of-the-art for many real-world problems. It was shown that NN models (i.e., their weights and biases) evolve on unique trajectories in weight space during training. Following, a population of such neural network models (referred to as model zoo) would form structures in weight space. We think that the geometry, curvature and smoothness of these structures contain information about the state of training and can reveal latent properties of individual models. With such model zoos, one could investigate novel approaches for (i) model analysis, (ii) discover unknown learning dynamics, (iii) learn rich representations of such populations, or (iv) exploit the model zoos for generative modelling of NN weights and biases. Unfortunately, the lack of standardized model zoos and available benchmarks significantly increases the friction for further research about populations of NNs. With this work, we publish a novel dataset of model zoos containing systematically generated and diverse populations of NN models for further research. In total the proposed model zoo dataset is based on eight image datasets, consists of 27 model zoos trained with varying hyperparameter combinations and includes 50’360 unique NN models as well as their sparsified twins, resulting in over 3’844’360 collected model states. Additionally, to the model zoo data we provide an in-depth analysis of the zoos and provide benchmarks for multiple downstream tasks. The dataset can be found at www.modelzoos.cc.

More

Estimation of Power Generation and CO2 Emissions Using Satellite Imagery

@ AI4EO Symposium

get_appJoëlle Hanna, Michael Mommert, Damian Borth

Conference or Workshop Item

Burning fossil fuels produces large amounts of carbon dioxide (CO2), a major Greenhouse Gas (GHG) and a main driver of Climate Change. Quantification of GHG emissions related to power plants is crucial for accurate predictions of climate effects and for achieving a successful energy transition (from fossil-fuel to carbon-free energy). The reporting of such emissions is only required in some countries, resulting in insufficient global coverage. In this work, we propose an end-to-end method to predict power generation rates for fossil fuel power plants from satellite images based on which we estimate GHG emission rates. We present a multitask deep learning approach able to simultaneously predict: (i) the pixel-area covered by plumes from a single satellite image of a power plant, (ii) the type of fired fuel, and (iii) the power generation rate. To ensure physically realistic predictions from our model we account for environmental conditions. We then convert the predicted power generation rate into estimates for the rate at which CO2 is being emitted, using fuel-dependent conversion factors.

More

Quantifying Traffic with Deep Learning from Earth Observation Data

@ Symposium of the International Future Lab AI4EO 2022

get_appMichael Mommert, Moritz Blattner, Leonardo Eicher, Damian Borth

Conference or Workshop Item

Transportation by means of internal combustion still constitutes the main mode of mobility. As such, it exerts adverse effects on human health and Earth's climate through the emission of exhaust gases and noise. We present two works that aim to quantify commercial vehicle traffic [1] and to estimate road traffic noise [2]. In both works, we rely on Deep Learning methods and freely available multi-band satellite imagery from Sentinel-2. We utilize ground-truth traffic rates and road noise estimates available for Switzerland to train and evaluate our approaches. We find that we can quantify commercial vehicle traffic rates with an RMSE down to 60 vehicles per hour and we can estimate road traffic noise levels with an RMSE down to 9 dBA for any location in Switzerland. Our studies show that satellite-based observations in combination with Deep Learning are sufficient to estimate commercial vehicle traffic rates and resulting road traffic noise and that this method may provide reasonable approximations in places where no ground-truth data is available. [1]: Blattner, Moritz; Mommert, Michael & Borth, Damian: Commercial Vehicle Traffic Detection from Satellite Imagery with Deep Learning. 2021. - ICML 2021 Workshop on Tackling Climate Change with Machine Learning Workshop. [2]: Eicher, Leonardo; Mommert, Michael & Borth, Damian: Traffic Noise Estimation from Satellite Imagery with Deep Learning. 2022. - IEEE Geoscience and Remote Sensing Symposium 2022. - Kuala Lumpur, Malaysia.

More

Multi-modal Self-supervised Learning for Earth Observation

@ AI4EO Symposium

get_appLinus Mathias Scheibenreif, Joëlle Hanna, Michael Mommert, Damian Borth

Conference or Workshop Item

An ever-increasing number of Earth Observation satellites continually captures massive amounts of remote sensing data. This wealth of data makes manual analysis of all images by human experts impossible. At the same time, the data lacks readily available labels which are necessary for training supervised machine learning models, including state-of-the art deep learning approaches. Techniques from self-supervised machine learning make it possible to leverage unlabeled data for the training of deep neural network models, and thus improve our Earth Observation capabilities across different tasks.

More

Traffic Noise Estimation from Satellite Imagery with Deep Learning

@ IEEE Geoscience and Remote Sensing Symposium 2022

get_appLeonardo Eicher, Michael Mommert, Damian Borth

Conference or Workshop Item

Road traffic noise represents a global health issue. Despite its importance, noise data are unavailable in many regions of the world. We therefore propose to approximate noise data from satellite imagery in an end-to-end Deep Learning approach. We train a U-Net segmentation model to estimate road noise based on freely available Sentinel-2 satellite imagery and existing road traffic noise estimates for Switzerland. We are able to achieve an RMSE of 8.8 dB(A) for day-time traffic noise and 7.6 dB(A) for nighttime traffic noise with a spatial resolution of 10 m. In addition to identifying major road networks, our model succeeds to predict the spatial propagation of noise. Our results suggest that this approach provides a pathway to estimating road traffic noise for areas for which no such measures are available

More

A Multimodal Approach for Event Detection: Study of UK Lockdowns in the Year 2020.

@ IEEE Geoscience and Remote Sensing Symposium 2022

get_appJoëlle Hanna, Linus Mathias Scheibenreif, Michael Mommert, Damian Borth

Conference or Workshop Item

Satellites allow spatially precise monitoring of the Earth, but provide only limited information on events of societal impact. Subjective societal impact, however, may be quantified at a high frequency by monitoring social media data. In this work, we propose a multi-modal data fusion framework to accurately identify periods of COVID-19-related lockdown in the United Kingdom using satellite observations (NO2 measurements from Sentinel-5P) and social media (textual content of tweets from Twitter) data. We show that the data fusion of the two modalities improves the event detection accuracy on a national level and for large cities such as London.

More

Self-supervised Vision Transformers for Land-cover Segmentation and Classification

get_appLinus Mathias Scheibenreif, Joëlle Hanna, Michael Mommert, Damian Borth

Conference or Workshop Item

Transformer models have recently approached or even surpassed the performance of ConvNets on computer vision tasks like classification and segmentation. To a large degree, these successes have been enabled by the use of large-scale labelled image datasets for supervised pre-training. This poses a significant challenge for the adaption of vision Transformers to domains where datasets with millions of labelled samples are not available. In this work, we bridge the gap between ConvNets and Transformers for Earth observation by self-supervised pre-training on large-scale unlabelled remote sensing data. We show that self-supervised pre-training yields latent task-agnostic representations that can be utilized for both land cover classification and segmentation tasks, where they significantly outperform the fully supervised baselines. Additionally, we find that subsequent fine-tuning of Transformers for specific downstream tasks performs on-par with commonly used ConvNet architectures. An ablation study further illustrates that the labelled dataset size can be reduced to one-tenth after self-supervised pre-training while still maintaining the performance of the fully supervised approach.

More

More publications

Collaborations

phone Your contact person

Artificial Intelligence & Machine Learning [AIML]

Prof. Dr. Damian Borth
Rosenbergstrasse 30
CH-9000 St. Gallen

+41 71 224 26 27
www.hsg.ai

News

Radio Interview with SRF4 about Climate Change Monitoring from Space

Climate Pollutant Monitoring from Space

Panel Session at St. Gallen Symposium (50th anniversary)

Keynote – Presentation at the St.Gallische Naturwissenschaftliche Gesellschaft

Keynote: Urban Digital Talks at Stiftung Energie & Klimaschutz and Baden-Württemberg-Stiftung

Invited Talk at Deep Fakes Workshop at Roman Herzog Institute

Keynote at LANXESS “Digital Morning”

Interview at Top Online TV about AI

Radio Interview with SRF4 about Climate Change Monitoring from Space

Climate Pollutant Monitoring from Space

Panel Session at St. Gallen Symposium (50th anniversary)

Keynote – Presentation at the St.Gallische Naturwissenschaftliche Gesellschaft

Keynote: Urban Digital Talks at Stiftung Energie & Klimaschutz and Baden-Württemberg-Stiftung

Invited Talk at Deep Fakes Workshop at Roman Herzog Institute

Keynote at LANXESS “Digital Morning”

Interview at Top Online TV about AI

Research

Team Member

Most recent Publications

Collaborations

phone Your contact person

Contact Us