What does BERT dream of?

Abstract

Feature visualization has proven to be helpful when analyzing neural networks. In recent years, more and more insights are published based on these techniques. However, currently feature visualization is limited to image data, while at the same time neural networks that operate on text data are getting more and more important. Thus, we investigated how feature visualization can be adapted to such models, and conducted a series of experiments along this line. In this explainable, we show how we adapted the techniques of feature visualization to text models. We present the results of such experiments with BERT, a transformer-based text model. Through our interactive environment, users can explore how feature visualization for text can be implemented, see the results of different feature visualization based experiments, learn about the limitations and problems with feature visualization for text, and get ideas for why some of the image-based results might not be transferable to text.