Two Visual Intelligence papers accepted for prestigious AI conference

New information theories and divergences by Visual Intelligence have been developed and accepted for the prestigious International Conference on Learning Representations (ICLR) 2024. ICLR has an acceptance rate of approximately 30 percent.

By: Robert Jenssen, Director, Visual Intelligence

Developing new theories to reveal information in deep learning

Modern society is data-driven in the sense that sensors and observations provide measurements. Images are examples of measurement. Key to Visual Intelligence is to reveal and exploit important information from images automatically with deep neural networks to help decision makers. For instance, to reveal information about possible tumors by analysing medical images. In order to do that, it is key to be able to define and quantify information in a mathematical sense and to be able to exploit it.

For instance, it will very often be crucial to be able to quantify in some sense how much information one population of measurements (P) carries about another population of measurements (Q). This is illustrated in a simplified manner in the figure. The difference between P and Q is often called divergence.

‍

‍

Introduces a new measure of divergence

The first paper is entitled Cauchy-Schwarz Divergence Information Bottleneck for Regression. The paper’s authors are Shujian Yu, Sigurd Løkse, Robert Jenssen, and Jose Principe.

In this paper, the aim is to capture as much information as possible about input images while, at the same time, compressing the data representation through a so-called bottleneck. This is highly related to compression, which is crucial to any digital system. The paper develops a new and better way to do this by introducing a new divergence measure.

You may read the paper abstract at the lower portion of this article.

New ways of presenting a population

The second paper is titled MAP IT to Visualize Representations. The paper's author is Robert Jenssen.

In this paper, a ubiquitous challenge in machine learning is tackled. When dealing with data such as images, each observation (image) is often composed of millions of numbers (pixel values). This creates big problems since machine learning systems in general work better when observations are characterized by fewer numbers. It is also very challenging to visualize (“look at”) observations composed of millions of numbers. MAT IT proposes a new way to represent a population such that each observation in the population is composed of only two numbers. This enables plotting of the data set for visualization purposes and helps machine learning systems work better.  

A simplified example is shown below. Small images of handwritten digits are in this case 24 by 24 pixels which means that each image is composed of 576 numbers. MAP IT minimizes the divergence between the set of images with a representation of these images composed only of two numbers. The two numbers represent a dot in a plot. When printing the actual images on top of the dots representing the images, it is clear that the main structure is captured, in the sense that dots corresponding to 4s are separated from dots corresponding to 9s and furthermore separated from dots corresponding to 7s.

The paper abstract can be viewed at the lower portion of this article.

‍

Information about each paper

Cauchy-Schwarz Divergence Information Bottleneck for Regression

By authors Shujian Yu, Sigurd Løkse, Robert Jenssen, Jose Principe.

Open Review link: https://openreview.net/pdf?id=7wY67ZDQTE

Abstract

The information bottleneck (IB) approach is popular to improve the generalization, robustness and explainability of deep neural networks. Essentially, it aims to find a minimum sufficient representation by striking a trade-off between a compression term, which is usually characterized by mutual information I(x; t) where x refers to the input, and a prediction term usually characterized by I (y; t) where y is the desired response. Mutual information is for the IB for the most part expressed in terms of the Kullback-Leibler (KL) divergence, which in the regression case corresponds to prediction based on mean squared error (MSE) loss with Gaussian assumption and compression approximated by variational inference. In this paper, we study the IB principle for the regression problem and develop a new way to parameterize the IB with deep neural networks by exploiting favorable properties of the Cauchy-Schwarz (CS) divergence. By doing so, we move away from MSE-based regression and ease estimation by avoiding variational approximations or distributional assumptions. We investigate the improved generalization ability of our proposed CS-IB and demonstrate strong adversarial robustness guarantees. We demonstrate its superior performance on six real-world regression tasks over other popular deep IB approaches. Additionally, we observe that the solutions discovered by CS-IB always achieve the best trade-off between prediction accuracy and compression ratio in the information plane.

MAP IT to Visualize Representations

By author Robert Jenssen.

Open Review link: https://openreview.net/pdf?id=OKf6JtXtoy

Abstract

MAP IT visualizes representations by taking a fundamentally different approach to dimensionality reduction. MAP IT aligns distributions over discrete marginal probabilities in the input space versus the target space, thus capturing information in wider local regions, as opposed to current methods which align based on pairwise probabilities between states only. The MAP IT theory reveals that alignment based on a projective divergence avoids normalization of weights (to obtain true probabilities) entirely, and further reveals a dual viewpoint via continuous densities and kernel smoothing. MAP IT is shown to produce visualizations which capture class structure better than the current state of the art.