Understanding Chromaesthesia by Strengthening Auditory-Visual-Emotional Associations

Can cross-modal associations between notes and colours and between notes and emotions be strengthened by using an ad-hoc designed computer program? In this paper we present an experiment that we conducted to see (H1) if participants' associations between the auditory and visual senses could be strengthened through immersion in a multi-sensory program and (H2) if there was any correlation between auditory-visual and the auditory-emotion associations. In our experiment, participants were asked to use the SoundStrokes program, an application that translates notes played on an analog music instrument (i.e., a guitar) into coloured graphical shapes (i.e., rectangles). Between several training sessions with the program, participants were asked to associate single music notes to colours and emotions. Applications of our work may vary from entertainment, to enrich in a systematic manner the user emotional experience in, for example, videogames or immersive experiences, to digital health, in which relatively simple tools can be developed, for example, to reduce anxiety through music.