Synesthesia: Where Senses Collide | Journey Through The Senses

Synaesthesia Panel - Venice Design Week [EP.62]

[00:00:00] Julia: About four percent of people have synesthesia. So we might think that Michael would be great for designing things for himself and for other synesthetes, but how could they possibly be useful for the other 96 percent of the population?

[00:00:14] And here's where it's really exciting, because they are useful for everybody, and here's why.

[00:00:19] [Ident]: [Afrika Design Ident]

[00:00:22] Lisa: Thank you very much for being here today to introduce Venice design week. The theme of this year is Synesthesia and we have here the measure expert that are speaking about this theme. I would like to invite Adrian of Nairobi Design Week to speak about how he organized this convention. Please.

[00:00:45] Adrian: Sure. Thank you, Lisa. It's a amazing pleasure to be in Venice. It's kind of surreal having arrived yesterday and really excited because as a designer and as a synesthete, we have some incredible minds that I really want to learn from in the room. So I'm Adrian Jankowiak. I'm a industrial designer by trade.

[00:01:07] I also run Nairobi Design, and annually we run Nairobi Design Week, an annual festival just much like Venice Design Week. So, I'd love to take a few minutes for people to introduce themselves. Should we start with Julia's, and then... the gentleman can take over. So I'm just going to share Julia's video here.

[00:01:30] Julia: Look around you. What does the world feel like? Imagine how it would feel if you could hear with your eyes or see with your ears. This is what it means to have synesthesia, a rare neurological trait that gives rise to a merging of the senses. My name is Julia Simner, I'm a professor of neuropsychology and I've spent the last 20 years studying the minds and brains of people with synesthesia.

[00:01:54] Now there are lots of different ways to have synesthesia. Some people see colors when they hear music. Other people with synesthesia feel tastes in their mouth when they read words. Others picture the months of the year in particular patterns in space, perhaps January's up here and December's down here.

[00:02:11] Other synesthetes hear sounds when they look at bright colors, or when they see silent moving objects.

[00:02:17] We think there are over a hundred different ways of experiencing synesthesia but only a relatively small number of people in the world who do this. So what's it like to have synesthesia? Now I'm reminded of a conversation I've had with a fellow scientist who himself is a synesthete. He sees colors from the sounds of different instruments.

[00:02:35] When he was a boy, his parents used to take him to the symphony in New York, and he thought the house lights dimmed before the performance started, just so people could see the colors better. For him, the colors hung over the orchestra, and he assumed they could be seen by everyone. So some synesthetes feel like their colors are out there in the world, but other synesthetes experience their colors like pictures in the mind's eye, and others still simply know what those colors must be.

[00:03:03] So how can someone see colors when they're not there? Modern neuroscience methods have allowed us to peer into the brains of people with synesthesia. And we've seen they have tiny differences in the wiring around the region of the brain that processes colors. So, this region in the brain, the color region, can spring into action, not just when they get a message through the eyes but also, when they get a message through the ears. So importantly, redness is not something on a rose, in fact. It's just something in your brain. It's a feeling that happens when your color regions light up. And this region just happens to be a little more interconnected in synesthetes. It's normal, it's just a bit rare.

[00:03:43] People with synesthesia sometimes have special talents like better memory, and they also score higher in tests of creativity. Sometimes this creativity reaches levels of genius, and we have good reason to think that synesthesia has driven some of the most creative and insightful leaps in mankind, from physicists like Richard Feynman, to poets like Arthur Gamble, and perhaps the composer Rimsky Korsakov.

[00:04:07] Sometimes it's difficult to know with certainty whether historical figures had synesthesia, or whether they were just using colour for metaphor. But when Rimsky Korsakov describes his colours, he does them in a way that a modern scientist like me would recognise. For example, for him, B major was a sad, deep blue with glints of steel.

[00:04:27] And it's this glint of iridescence that, for me, resonates with genuine synesthesia. Ask a non synesthete, they'd be far less specific and far more flat. We also know that people with music color synesthesia are significantly more likely to work as musicians and also sometimes to have perfect pitch.

[00:04:45] Now over the last 20 years, I've had the pleasure and privilege of talking to tens of thousands of people with synesthesia through different means through the internet and in person. Not all of them are different, not all of them are unique. Hearing their stories and studying their minds and their brains has helped me to understand my own sense of reality, because I now know that the world around me is filtered through the individuality of my brain, and because some brains differ in subtle ways.

[00:05:12] Adrian: Thank you, Julia. That was a really insightful and I'm really looking forward to this conversation because a few things already have clicked with me. James, I'd love to invite you next to introduce yourself.

[00:05:28] James: Oh, hello. Yeah. My name's James Wannerton. I've been involved in making or at least attempting to increase general awareness of Synesthesia in the UK and overseas in fact. Probably approaching Saber Junior actually probably ran about 20 years ago.

[00:05:42] It started back in the early 2000s. I do believe I remember correctly. The reason I became involved was because I'm in fact a synesthete myself. I experience primarily sound to taste although there are some elements of sound to colored shapes that appear especially loud ones. So I myself am a synesthete.

[00:06:02] Adrian: Thank you very much. Anton as well if you'd like to say a few words.

[00:06:07] Anton: Yes, thank you so much for inviting me, and it's a great pleasure to be in company of James and you, Adrian, and Julia, and Mikhail, who's joining us, I hope, soon. I am a psychologist and a linguist. These are my... both of my career and I'm degreed in Moscow and, I am vice President of the International Association of Scientists artists and synesthetes. And again, thank you for inviting me. Let's talk and let's discuss the issues.

[00:06:38] Adrian: Thank you so much. It is a real honor to be kind of among the synesthesia all stars here. Just a quick intro on myself because I've got synesthesia and the first time I realized was I was watching a documentary, maybe I was 17 or something like that, and it was perhaps one that someone here was involved in as well.

[00:06:58] And they said that, oh, well, some children see color when they see letters and numbers. And I thought, what do you mean, some children? I thought that was normal in all of us. And again, that's kind of I think a repeating pattern that happens with synesthetes. And I consider myself lucky to have found it at that stage.

[00:07:16] Because maybe I'd still be thinking it's normal for everyone. And so for me really, it's very special. The first kind I remember was really the colors and letters and numbers. Because I think as a child, you probably associate those things and you're taking in so much information.

[00:07:33] That you're finding patterns, and when those patterns occur to you, you're like, Aha, well I know A is always red, well therefore, it's easier for me to remember these words. And then recently, I guess I've always known I have music or sound synesthesia as well. And it kind of occurs to me. I was watching Yo Yo Ma perform. Very famous cellist in Nairobi.

[00:07:56] He played two and a half hours live and my senses went into overdrive. Really just so many things happening. Probably two hours out of that, I had my eyes closed because it was so enjoyable. And I didn't fall asleep because it was just happening all the time. And now I've also recently realized that there's something called mirror touch synesthesia, which again, I thought was something that happens to everyone.

[00:08:20] But even when, Julia, you said about the comparison of these notes to blue and steel, it kind of gave me tingles. But I think where my mirror touch synesthesia comes in is. If one of you, if someone taps you on the shoulder, that's obvious to me that I can feel it.

[00:08:36] It's just there, right? It's also kind of... I wonder, because of the 3D design aspects I do, I wonder whether it also helps me position things in 3D space and try to see things from your eyes. Because I'm looking at you, but I can kind of picture what's happening through your eyes. I don't know if that's a synesthesia or if that's more to do with 3D design aspect. So those are I'm trying to understand. Which parts and how they combine, but that that's kind of from me. I wonder if there's anything you'd like to add as we go into the conversation and we will have time for questions as well. So we'll invite everyone at the end of the panel.

[00:09:15] Yeah.

[00:09:16] Julia: I mean, I think I can speak immediately. Adrian about your idea that you have this great spatial awareness. So what I do is run experiments, non invasive experiments on people with synesthesia and children. And one thing that we've been able to show is that the children with synesthesia or adults with synesthesia do have superior spatial abilities.

[00:09:37] So here's what we did. We went into a large number of schools in the Southeast of England, and we screened three to 4, 000 children and every single child had an objective test for synesthesia and every single child had a battery of other tests, including a test of spatial abilities, and none of the children knew what we were looking for.

[00:09:59] And in fact, the experimenters didn't know who the synesthetes were because we were identifying the synesthetes at the same time we were testing them for other skills. And the spatial task showed a difference between the synesthetes. and the children who did not have synesthesia. So in this spatial task, you're given blocks and they're colored, they're red or white, or they're kind of half red and white, like a kind of diagonal block.

[00:10:24] And then they see this photograph of an arrangement of the blocks to make a spatial picture, an abstract spatial picture. And the task is simply take these blocks and match that picture. So turn them around in space, put them together until it matches this abstract picture. And we found that the children with synesthesia were significantly better in that task.

[00:10:43] So I'm not surprised. You have good spatial processing, Adrian.

[00:10:46] James: I was gonna say you with other synesthetes Julia, have you... ' cause the tasks you described there, I've actually done it. I, I did immense test many, many years ago and that used to be pretty fundamental part of it. That part I was absolutely useless at. Couldn't do it at all. And I still have problems now.

[00:11:03] I love cartography. I love maps, but looking at one and trying to relate to what that actually means. And, where I am on that map is so difficult for me.

[00:11:12] Julia: Thanks, James. I mean, there are two things to say there. First of all, one is that there are many different types of synesthesia and we were... I should have clarified that we were only looking at two types of synesthesia and not your type.

[00:11:24] I'm sorry. Yours is really very, very rare. So we looked at Adrian's type. So colored letters and numbers. We also looked at another kind of synesthesia where letters and numbers, again, trigger synesthesia. But instead of triggering, say, red or green or blue, the letters and numbers trigger a sense of a personality.

[00:11:43] So, for example, the synesthete might know that A is a benevolent mother, and B is her toddler son, and C and D are next door neighbors who gossip with each other over the fence, and so on. And these are really complex feelings. And just to clarify, all of these synesthesias can be seen in the brain. So we scanned the synesthetes, we can see they have different connections in the brain in social areas, and that triggers this particular social synesthesia.

[00:12:11] But the point really, James, is we didn't look at your form of synesthesia, I'm very sorry. And of course, synesthetes are also individuals, right? So, we never find the same pattern across every single one. We just find that across all of them, there's this trend to be significantly different.

[00:12:27] So you could also be just an outlier.

[00:12:30] James: Just wanted to put one more thing over as well, because I've been involved with synesthesia research for many, many years. The thing that's always intrigued me, and I've never asked, anyone is how did you actually convince people to take this up being it was such a subjective experience that didn't appear to cause any problems.

[00:12:47] I mean, how did you manage to do that?

[00:12:49] Julia: I mean, I'm not sure about Anton, but speaking for myself. Yeah, it was really tough in the early days. It was really tough being a synesthesia researcher before brain imaging had really come in to show all of these sort of objective truths about synesthesia. So, personally speaking, we met in 2002, and we were working with you and all of the conference presentations I gave were 100 percent about showing the genuineness of synesthesia.

[00:13:16] That was the entire content of the talks I would give. It's genuine because A, B, C, D, you know, all of this evidence. And then by huge relief, brain imaging kind of roared in and showed us, James, for example, we've scanned your brain. We know that your brain is different to other people, and we know it's different in the taste regions.

[00:13:34] And James, that's your synesthetic experience. Adrian, we know that your brain is probably different to most other people, this time in the color regions. We can see extra pockets of connectivity near color regions for people with color synesthesia. And as soon as we had that big sledgehammer of evidence, it was just something I would put on a single slide at the beginning of the talk and then we could talk about the more interesting things.

[00:13:58] But kudos to the British Academy because they were willing to fund me as a very young researcher in, gosh, 2001 ish, 2002, to study synesthesia. They had that faith. And so, yeah, that's really how it happened for me.

[00:14:12] Anton: Yeah, James, and the issue of persuading and showing the reality of synesthesia to people and to other scientists, colleagues.

[00:14:22] So, well, it may have some cultural aspects as well as this empirical aspect. So the empirical aspect is, maybe objective tests and the results of objective tests and consistency and all that we know the golden standards for synesthesia now, so you start developing this and Julia has contributed to this and all our colleagues who have done so much work on showing, on demonstrating that synesthesia is real and developing these golden rules of testing and objectifying and verifying synesthesia.

[00:14:56] But the other aspect is cultural, and this is mostly how people, in general, trust what's going on in your head. Generally, because subjectivity may be an issue from one culture to the other, a stronger or less, or, you know, consistent, persistent issue. In Russia, for instance, be different is slightly, well, not suspicious, but a kind of more, less transparent issue.

[00:15:23] Well, to use a mild word to be different is this kind of you become less transparent for the other people. So, you know, you start, like, saying that it's not something very much different. It's like a variety on the same topic, and everyone has it. So this motto in Russian culture, that everyone has it, works like an entrance ticket for synesthetes.

[00:15:46] So nothing extraordinary, nothing special. You're just one of the kind. In this case, you oil the cogs of acceptance.

[00:15:55] Adrian: Thank you. And I'd just like to welcome Michael Haverkamp as well, who's just joined us.

[00:16:01] Michael: Yes. Hi. Good morning. Sorry. I am a little bit late. I just arrived in a different venue. And then I was told that this will happen here in the morning. So I had the opportunity to have some nice walks and nice venues with a very perfect weather and it is really nice and gives you a lot of mighty sensory impressions.

[00:16:20] And so the day started really nice and thank you for the invitation for this panel discussion. I really love to be here and attend this event.

[00:16:30] Adrian: Thank you, Michael. Now, let's just jump straight into what kind of multisensory impressions Venice has given you.

[00:16:37] Michael: Yeah, at first, if you read the hotel critics. To read the negative issues, it was told, for example, that this water really smells bad.

[00:16:48] Here in Venice, but that was not my impression. Maybe sometimes it is, but in fact, I had a very nice impression by going with a boat over this lagoon here and water smells really nice. It's a little bit like sea and it's a little bit sweet. So it has a really characteristic smell that I did not experience another place in the world.

[00:17:10] So, that is very interesting that even it was in the dark. So, the visual impressions were a little bit reduced and I was listening to the sound of the water of the speed boats. We have a lot of speed boats here. And I was smelling the water and that gave me a very interesting initial impression about this very specific location here.

[00:17:32] And now in the daylight, of course, the visual impression is very fantastic. You have so many color contrasts and so many different shapes and you always have some water and stones on the pathway. Lots of very different people here. A lot of tourists, of course, but that increases the colorful impression of this very exciting place here. And so it's a really multisensory experience here.

[00:17:58] Adrian: How did you discover synesthesia and how has that integrated into your professional life?

[00:18:05] Michael: Integrate, that's a very good word. Yes, at first from my individual, uh, specific perception, I have colored tones, colored music. Music has all this different color. The colors depend on the number of the music instruments.

[00:18:18] So each tone is a specific color and music also has some shapes and I also have those typical colored characters. So if I remember a word, then the word has a specific color, in my mind, but that doesn't mean that the colors overlap the text. If I'm reading a book or so, so the colors of the letters always in the mind.

[00:18:39] But if I think about different names of people, then those different names have different colors. It depends on the most protruding colors and also depends on the color of the first letter. So yeah, but very interesting for me because I work as an acoustic engineer and acoustic researcher for many years are the other central experiences I have if I listening to something and so I was dealing not only with the perception of sound, but I found that it's very important for the perception of sound that other senses are involved. For example, if you hear a sound, you know, for example, a bell sound, then you imagine a bell that you can hear from the sound, whether it's a large bell or small bell.

[00:19:24] So all those different associations, which are also connected to the sound perception beside the, specific synesthetic things, they are also very important. So my specific colors for tones motivated me very much to be interested in all those connections between the senses, because all those connections of the senses are very important.

[00:19:44] Especially if only one sense is stimulated, for example, we are in a room without any light sitting in the room and we can very much concentrate on sound on music. In that case, we imagine or we can experience that the contribution to the other senses are very intensive.

[00:20:02] Otherwise, I think if a tone would only be a tone without any visual emotional associations or without any aspects of motion and so on would not be interesting for us. So there's the music would not be interesting for us if it would be perceived only as tones only. That is my belief.

[00:20:22] And I found that, in daily life, I worked then I developed methods, how to integrate the different senses into a design approaches. And then I found that those principles are found for associating other senses with the tones. Those principles also apply if you're dealing with some senses, whatever it is, even smell.

[00:20:44] Correlating smell with visual appearance of things. And so you can find principles which can be applied everywhere. And so I developed a concept for overall multisensory design. They can include all those different aspects for design of everything because all the things we are using have to be perceived. I cannot drink this water if I do not perceive the cup and I cannot grasp the cup and I do not feel that it's cold and smell whatever it is and so on.

[00:21:12] So, this multi sensory configuration of things is very important, but it's often neglected because traditional designs very often pure visual. But the other senses, are a little bit, don't play the important role that would be necessary to have perfect products.

[00:21:29] For example, sometimes you have a cup that doesn't feel right. That doesn't feel solid or so that must be solid. It must also be feel reliable. Even if you're a hot tea or coffee or water, then it must feel very reliable. And you have to be sure from your haptic experience by fetching the cup, how hot the coffee is.

[00:21:50] Can you drink it or? Can't you drink it? So this is a very important, but often the visual design is put in front. And then others, for example, big industrial companies, the others should decide on the material. And then maybe the material does not really fit into the visual imagination.

[00:22:08] The designer had, for example, we had cases in the automotive industry that they designed a very extraordinary new shape for the steering wheel and then gave it to the material people and the material people said, well, we cannot fix any leather material on this shape. So at first, and the 1st thinking about the concept, you have to include all the senses and it is not always the case.

[00:22:31] So that was my main focus on work in the automotive industry and for other things.

[00:22:38] Adrian: Julia, you're welcome to go 1st. You mentioned in your research, you had a lot of children participating and a lot of those children or all of those children showed some sort of responses on a sense level.

[00:22:51] So, to what extent do all people have synesthesia? And to what extent can we train or practice our synesthesia?

[00:23:01] Julia: Thanks, Adrian. Yeah, I'm really keen to respond to some of the really fascinating things that Michael has just revealed about being a synesthete involved in design. So, the point I'd really like to make is to explain why Michael, as an individual synesthete, might be in a great position to be a designer for products for everybody in the world. Now you might think... synesthesia it's only common about four percent of people have synesthesia. So we might think that Michael would be great for designing things for himself and for other synesthetes, but how could they possibly be useful for the other 96 percent of the population?

[00:23:38] And here's where it's really exciting, because they are useful for everybody, and here's why. So when synesthetes have their experiences, their synesthetic experiences, those aren't from Mars. They have those specific sensory experiences, multi sensory feelings, because they are human beings, and the synesthesia is scaffolded onto normal, typical brain functions.

[00:24:01] And what that means in a nutshell, is that what synesthetes experience explicitly, everybody else in the world experiences implicitly. And that connection between synesthetes and everybody else is what makes a synesthete a great designer for anybody who knows about synesthesia. So here's what I mean. So I know that around 96 percent of the audience will not have synesthesia.

[00:24:26] I don't have synesthesia but we all have synesthetic like associations that are similar to the synesthetes here. Here's an example. Imagine I have a piano, and I tinkle on the very, very top notes, and I crash down on the very, very low notes, and I say, which one of those was a pale light yellow, and which one was a deep dark purple?

[00:24:47] And everybody will agree that the high notes, the tinkly high notes, feel more light yellow, and the crashy low notes feel more dark purple. That's because all people have associations across their senses that follow certain rules. In this case, we have a rule, an internal subconscious rule, that says that high pitch is light in color and low pitch is dark in color.

[00:25:13] Now what's really interesting is that when we test synesthetes, who see colors in space, they have exactly the same rules going on. High pitched sounds are much more likely to be light in colour, and low pitched sounds are more likely to be dark in colour. And that's just one example. So we've also done this in multiple senses.

[00:25:33] We have given 200 students a candy to eat, or a food product to eat. And if that food product has a smooth surface, it feels sweeter in the mouth. If it has a rough surface, it feels more sour and bitter in the mouth. And really importantly, we split the students in two. They did not see each other's sweet.

[00:25:55] Everybody only ate one of them. But when we rated the people who had the smooth surface and the people who had the rough surface, we saw that smoothness is associated. It literally tasted sweeter if it was smooth and it literally tasted more bitter, sour, if it was rough.

[00:26:12] So there's a form of synesthesia where placing something in the mouth, a food in the mouth, triggers the sensation of shapes and textures. And that is also the unconscious rule, followed by synesthetes. All these rules are unconscious. Synesthetes are not able to verbalize what's going on.

[00:26:28] But when you take hundreds of synesthetes and you analyze them, you see these rules emerging and they're the same across all people. So if Michael is producing a design that feels right for him, you can bet your socks that it's going to feel right for the rest of the population. And that's how I come to do design consulting.

[00:26:44] So I work with design companies using what I know about synesthetes to help them design their products. So, for example, some of its propriety, I can't be specific, but I can give an abstract example. If I were working with a company that designed speakers. Then we would be able to show that making that speaker white, shiny, and angular would make the sound quality sound better.

[00:27:08] And so that's the kind of work I do. And I also now I'm working with another design company that create VR, and alternative realities and virtual realities. And we're using multi sensory overlays to kind of bring out certain qualities of the objects that can be seen. So, for example, if something's particularly aversive, we can use colors that we know are aversive colors and so on.

[00:27:31] So yeah, I think it's nice to hear Michael talking about his design work and then understanding from a neuroscientist perspective, why he would be so great at design.

[00:27:39] Michael: Thanks for that. I agree in some manner, and I don't agree in some manner.

[00:27:44] I agree in some manner that many synaesthetic perception that I have are common multisensory perceptions like, correspondences between the color brightness and the height of pitch or the brightness of a tone. But there are others, synthetic experience, there appears some phenomena which cannot be aligned with those common things. For example, I listening to a specific music, and then are bubbles coming and floating down through my field of view and it's only one, one person telling that and that cannot easily be aligned. So you always have to test whether it is a feature that can be used in a common manner or not.

[00:28:25] Even in design. If you want to have mass production for a larger number of customers, then you have to be sure that your design will fit in the expectations of many of those peoples and those bubbles will have some problems with those models.

[00:28:40] And if you have a larger number of test persons for your perceptual experiments, then the classical way is to do statistics on that. And that can cause single persons with single experiences to be pushed out of the number of test results you will exclude them by averaging. From my experience, you're absolutely right with the statement, but you always have to exclude or specifically handle those with very specific and singular experiences.

[00:29:08] Julia: Yeah, that's absolutely right. So thanks. Yeah, no, that's exactly what I was saying at the start of this session just before you arrived. That actually, synesthetes are individuals still, some of their experience are individual experiences, but what we're able to show with statistical tests is what the trend is, what most people do if you average them.

[00:29:26] And in fact, I test sort of thousands and thousands of people and then I do run statistical tests. And so the kind of rules I'm talking about emerges when testing hundreds of people. And actually you're right. Some of them are outliers. And James and I had a discussion earlier about a moment when James was an outlier in his study.

[00:29:43] So no, you're absolutely right. Yeah. We use statistics. It's really the way to do this.

[00:29:48] Adrian: Anton and James, would you like to add?

[00:29:50] Anton: Yeah. Let me just shift slightly our attention from synesthetes personalities and individual differences to the applicability of synesthesia principles to design. For instance, in neuroscience, analyzing memory and some principles, some overall generalized principles of memories were applied in AI, and archiving, etc.

[00:30:13] So, we know that our knowledge about memory is very applicable. So what kind of applicability can synesthesia offer us? And this is my idea. Well, if you analyze results from cognitive neuroscience psychology, so we see that the congenital synesthesia is like a closely related set of clustered phenomena.

[00:30:34] And these phenomena are very individual and unique, but on the other hand, they share important identical features. This is what we call synesthesia like an umbrella terms. It's like memory is also an umbrella terms for various phenomena and mechanisms in the human mind. So first, let's ask what induces synesthesia?

[00:30:53] And we see... and Julia started this, by the way, some decade ago. What entities and units trigger a synesthetic experience? And we see that from type to type, these triggers are either letters or words or names. So these are kind of basic, meaningful system elements. And these triggers can go beyond language to embrace other systems.

[00:31:17] Or such as music, well, mathematics, or time conceptualization, and so on. And still other triggers are meaningfully grouped elements of human reality, such as, recognizable sounds. Recognizable is key here because it's not just noise, it's recognizable sounds. Tastes clustered, impression of meals, identifiable smells, and such like.

[00:31:40] And these trigger systems, they have an important common feature. These are all elements of those systems that we can use endlessly to recombine and produce new meaning. So across all their individualities, the synesthesia trigger systems are outward representations of generative meaning making cognitive mechanisms.

[00:32:02] So music is not just separate elements, it's like a common semiotic system. In linguistics, we call this semiosis or meaning making process. And second, what do these systems do in synesthesia? Well, let me contemplate this and try to highlight the major aspect. These, systems, they transform perceptual reality in a very specific way, they do it.

[00:32:24] These meaning making systems, through their elements, trigger sensory qualities. So as you see, these perceptual transformations are very selective, because, well, we don't know how it happens, but we'll soon, I think, we'll get to know. This happens early in childhood. So, synesthetic brains early in childhood at the precognitive level, they decide that only these particular meaning making systems should provoke these particular sensory qualities.

[00:32:55] Thus, time units, trigger located shapes, or speech sounds, they become flavor coded. And smells generate color tags, which strikingly resembles the experiential setup of augmented reality. But that's just an aside. And it's important that these sensory qualities are not associations.

[00:33:16] Well, these sensory qualities are random or near random additions. Well, because in people with synesthesia, letters may have their own metaphoric meaning, apart from synesthesia. Or music can have very intimate invocations besides synesthetic experiences. And synesthetic sensory qualities do not preclude personal associations.

[00:33:39] Well, they are added up on top of that. So they are non associative, non referential additions. And well, just to recap this, so congenital synesthesia is like a meaning mediated redesign of perception, which is subjectively manifested as additional sensory coded qualities whose activation is dependent element wise, so element for element, on a certain meaning making generative system, language or mathematics or music, and there is a very important third aspect. Take this example. When the letter color synesthetes studies a second language, the new letters almost instantaneously acquire new colors with no other outside prompts, with no further associations.

[00:34:23] It's just transferable. And synesthesia is basically consistent. It's feedback based. It's sensory, it's adaptable, it's meaning based, and it's user centered. So, music synesthetes have colors when they hear melodies they've never heard before. So, again, it's beyond association. It's like tagging.

[00:34:44] I had an experiment when an imaginary eighth day of the week. It's like not Friday, not Thursday, some eighth day of the week, got an unexpected color in a synesthete who had the same colors in their own language for seven days. Even if synesthetic effects are imitated superficially in design, these defining features are very instrumental in end products of design.

[00:35:09] And we see that Michael described that and Julia has also worked with companies of design. So, we see that these effects, they are a design practitioners dream because they are so personal and so immersive. But what can be done further? When we know the differences of synesthetic brains, when they're better understood, we will be able to tap into synesthesia algorithms as we have already tapped into memories algorithm to make it more productive and more efficient for AI and low tech and high tech design.

[00:35:41] So it may happen through neurofeedback. Through transcranial magnetic stimulation or other brain machine interfaces. But even now, well, arguably, we haven't yet fully explored, from my point of view, synesthesia at the psychological level. Take for example, synesthesia's types. How come that they feel so real and immersive?

[00:36:03] So what makes the reality of synesthesia so close? Can we mimic this? How do they govern attention? Because in synesthetes, we know that attention is peculiarly specific but it's so close and so natural. So, again, which meaningful systems do synesthesia embed? Why is it music? Why specific elements of music?

[00:36:25] What makes it music? What music is special? And what elements can we just hook upon to make music more vivid. Is it just the tunes, separate pitches, separate notes, styles, harmonics, etc? How many elements could there be? So we've got 28 letters, but in music we have seven notes. Or if you think of genres, there are multiple genres, and colors may be different being attached to different genres.

[00:36:52] So, how does a culture specific system, like alphabets, get linked to a physically or physiologically fine tuned system, such as colors or tastes? Well, that's the enigma of synesthesia, on the other hand, but this enigma can be tapped into and be used for design and so on and so forth. Can we introduce this into design and try to imitate this and make design all embracive and multi sensory in the real sense of this work? So, this is my vision. It's a bit like overgeneralizing and overarching. But again, we started with memory. I mean, in neuroscience and cognitive science, and we try to apply the same principles.

[00:37:35] I suggest are starting with synesthesia and apply synesthesia overall principles in design. Well, this is what I think, and this is my position. Hope it helps.

[00:37:46] Adrian: Brilliant. I will let everyone speak just to say we've got about maybe 10, 15 minutes left of the discussion, and then we'll have a few questions. So prep your questions as well.

[00:37:56] So maybe there's two parts that I wanted to ask. The first part will be around the desirability of synesthesia. I found it really interesting that Julia said that some of these colors are already appealing or associated by people, even maybe non synesthetes could understand why some of those color choices, for example, have been made.

[00:38:17] So in design, are there examples? I can think of maybe some. Other examples that aren't of synesthesia when it's not desirable and the same goes in life. Are there examples of undesirable synesthesia? So we'll package that as one question and the last part I'd love for us to talk about kind of three key words of emotion, empathy, and memory, because I feel like there's a lot of connections between those words and we've mentioned all three. So let's deal with that first part first. Undesirable synesthesia, does it happen?

[00:38:52] James: One of the peculiar things I've found out about my own synesthesia, which is particularly intrusive type. I mean, it's sound to taste. And these things do affect my concentration. These synesthetic perceptions. For example, on our train systems, we've got quiet carriages.

[00:39:09] We've got rooms where you can go where it's total silence. Now, somebody like me, a quiet carriage or a quiet place is in reality, it's sensorially louder because it tends to emphasize all the tastes and flavors I get because there's nothing else interacting there to cause any destruction from it.

[00:39:27] it's really peculiar. So in my mind, there's quite a few situations where you can overdo this. Overdo the incorporating synesthetic perceptions. Good design because we're talking about design here. I mean, good sensory design to me is always actually it gets inside me. It becomes very, very personal.

[00:39:47] In effect, it's actually touching me. And the beauty of all this is, is I can touch it back because I've got syphesthesia. And a lot of what Julia says as well, though, Julia mentioned... is that not cross modality? Because a lot of ad campaigns and all sorts, and I use my synesthetic experiences, but I only use the obvious ones, I don't use, because I get a lot that I cannot match up.

[00:40:09] And if I mention them, you know, people would sort of lose it a bit. So I tend to under emphasize stuff like that. What I'm interested in is, I think we should introduce this stuff. And I think, although designers have been taking into account multi sensory design in the process for quite some time.

[00:40:26] I mean, I see it all over the place. Can we do more? Should we do more? Is it cost effective to do that? And is there tangible value to be added to a design by incorporating all these new understandings that people like Julia, Anton are coming up with, you know.

[00:40:42] Michael: May I just introduce another point of which is very interesting. It is named sustainability. And if we think about the classical definitions of synesthesia, I just said that they will not change very much or not at all over life. And we know that a lot of multisensory connections can be learned, can be changed, will modify day by day. And so on. So at first, if you're designing a product, of course, from a technical point of view, we know how to make it reliable, that the quality will last for a longer time but on the other side, what's about the customer's expectations? If there are so many aspects, which always change in mind. How can we be sure that the product sustainability has been a longer lasting acceptance in the customer's mind.

[00:41:30] And so it's a question, some things can be very much easily modified in the human perception, and some things are more longer lasting and constant. So this question would be interesting. It will be addressed by science. How can we be sure that a design will longer last in the customer's expectation.

[00:41:49] So, because if the customer will change their mind every day, and we have that always regarding clothing industry and so on, they're changing their mind every day and they're buying lots of things and things, more things, more things, sending a lot of things back. So, that is not sustainability and the sustainability is not in the product.

[00:42:09] But this missing sustainability is in the mind of the customer, just as a question.

[00:42:14] James: I've sort of helped push all sorts of products like yogurts and cheese and all sorts of weird foods and stuff, crisps. But there's always got to be that cross modal element that I put in there that people can relate to. So isn't it a case of taking all the research that's been done in cross modality, again, which we're very advanced at over here in the UK, and marrying that with what we know about how synesthesia affects influences and how they influence and affect each other. In other words, if you, if you marry all those three together, the cross modality, the synesthesia and our understanding of how it all works and how it all fits together, can we not come up with something that's more sustainable?

[00:42:50] Because I understand perfectly, Michael, what you're saying, because there are things to me that I can actually say they've been designed using the multisensory process, but they're too much.

[00:42:58] Julia: There's a whole lot of different ideas being thrown around. I think I'll probably jump back to the initial question about whether it's desirable. And I was glad that James spoke first because I know, I know James synesthesia really well. I've studied it for 20 years. When James hears words, his mouth is flooded with the sensation of taste, almost like somebody is putting a dropper of taste in his mouth.

[00:43:20] So this entire sentence for James has been an onslaught in his mouth for different tastes. And I know that some of those tastes are nice. So I know that the name Julia tastes nice, and Jeanette, and they have quite nice tastes. But I... I have a nickname. I have a shortened version of my name that's very common in the United Kingdom.

[00:43:40] Everybody I know in the United Kingdom calls me by a name other than Julia, a short version of my name. But I won't use that name here, and I won't say it, and I don't use it with James, because I know that my short version of my name tastes really unpleasant, and I hope you're not even thinking about it, James.

[00:43:56] So, there are moments when James does have really undesirable flavors, you know. He's a super polite guy and people come up to James often and say, what does my name taste like? And I know James really well. He's super polite. And if James says porridge, that to me is his polite way of saying it's something really unpleasant and it can be really unpleasant.

[00:44:15] He's told me that my nickname tastes of porridge. I kind of think I know what my nickname tastes of. I don't think it's porridge. It's not very nice. So yeah, there are these undesirable aspects of synesthesia. Also, more generally, as a scientist, when we screen hundreds or sometimes thousands of people for synesthesia and look at the synesthesia traits, we find two key things.

[00:44:35] We find that synesthetes are superstars in terms of creativity or sometimes academic things, spatial processing, and so on. But they also have this penalty in well being in that they are significantly more likely to suffer from anxiety and some kinds of mental health problems that are related to, you know, anxiety really.

[00:44:54] That's the key issue. So yeah, benefits in creativity, etc. Problems in, you know, some of these well being elements.

[00:45:03] Anton: Well, I've been working with synesthetes for 15 years now, and I have like a reflex, a knee reflex. I do not wear bright clothes, especially when I meet synesthetes, because meeting after meeting, I had some feedback that the colors of my clothes clash with synesthetes experience.

[00:45:20] And usually when you do experiments, you'd better, well, abstain from this. At least you take away this extra stimulation. So usually I wear either black, well it's an exception today, black or white for experiments and for communicating with synesthetes. Because I know this is kind of intrusive and well, that leads us to the ecological aspects of design, generally.

[00:45:44] Well, definitely all design pretends to be multi sensory. Even if it doesn't, well, do not say this. It's like an art. Art is all inclusive and all immersive. Even if it's just use single modality. So if anyway, use and exploit your imagination and try to submerge you into the reality of art or into the reality of design. So, this intrusion is the designer's responsibility. So, the ecological aspects of design, again, is the designer's responsibility. If this step is necessary, and if it's feasible and efficient and it's got the meaning because synesthesia is meaningful and multi sensory design also claims to be meaningful.

[00:46:28] Well, if it's meaningful, okay, the designer may use synesthesia and synesthetic effects, projecting new meaning into a new modality. But if you can keep away, as James looks for... if you can keep away from this intrusion, if this intrusion doesn't mean anything extra, there is no new meaning for the brand, equity for the constructed environment, so probably it's better to leave humans and users on their own. Do not touch if it's not touchable and do not carrying new meanings if it's meaningless. That's my answer to the issue of too much synesthesia when it's too much.

[00:47:08] Adrian: Thank you. Anton. I'd love for us at some stage because we do have limited time just to touch on the links between emotion, memory and empathy.

[00:47:17] Because personally, I just wanted to say, I'm a designer, but I'm a human centered designer. That means that I have to talk to people, I have to listen to them, and so on. And I've wondered whether there's a link to empathy. I know I met a nurse who has synesthesia, and she told me that she'd never gotten to use her synesthesia professionally, when to me it was unthinkable as a designer creative director.

[00:47:39] But then now it's got me thinking, even as I was preparing for this panel, I'd love to talk to her again, because maybe... Actually, she uses the synesthesia differently, and it doesn't even occur to her. I'd love to open up for any questions, and if there are none, then we can just have a few more minutes.

[00:47:57] So please feel free to put your hands up, and I'll repeat your questions so we get them into the mic. Any reflections what I just said there?

[00:48:06] Julia: Yeah, what jumps out to me straight away, Adrian, is that I do anticipate you would have higher empathy, not because you're a synesthete, but because you have one particular type of synesthesia that is connected to empathy. I don't know if you can guess which one that is but at the beginning, you told us you had colored letters and numbers.

[00:48:23] You had colored music, mirror-touch, meaning that if I scratch my face, you feel it on your face. And that's the form of synesthesia. Apologies that you felt that. That's the form of synesthesia that is associated to empathy, that somehow you understand me better because you can feel what I feel. And that's been shown through work from our neuroscience colleagues, here in England and elsewhere.

[00:48:48] Adrian: I've never had someone expose me like that before, but I did. And I have to say it's professional and personal pleasure. And someone asked me about pain. Yes. If someone stamps on your foot, I can feel it in my toes but I think that rather than the pain, it's maybe unhappiness or suffering, unhappiness and pleasure. I saw another article or some synesthete who was a masseuse, and I just feel that's absolutely what I'm talking about, right?

[00:49:18] If I can feel which pleasure I'm providing, then it's really kind of helping me understand it. Yeah.

[00:49:24] Anton: Okay, just a couple of things here. Probably one. We know that in synesthesia, this mediator in the brain is the limbic system. So the limbic system, which mediates the meaning of the perception of sounds and activation of colors. We don't know how it happens. We know that the kind of a patchy puzzle for us so far.

[00:49:45] We haven't put this together, but the limbic system is responsible for mediating the meaning this extra activation in this overlay, either perceptual or sensory. Colors or other experiences. This is it. The limbic system might be the key.

[00:50:01] Because all I was going to say was again, Julia, but probably could back me up on this. But with synesthesia, especially the very intrusive types. You tend to remember the feeling of disgust more than anything else. And certainly with mine, there's not a lot of middle ground. I either really like something or I don't.

[00:50:19] And certainly with the synesthetes I've spoken to, if they don't like something, they simply don't like it. If they don't like the color of your name, you're not going to make them like it. And it's a very strong, feeling and perception that stays with you. And it stays with you forever. I hate the things I hate the sound of when I was a child, you know, age five, I can still remember them.

[00:50:39] I still hate them in the same way today. Nothing ever changes. And it's a case of trying to mask it or to try and override it like I do constantly. We're drinking strong coffee because it's the only way I could stop this constant flow of flavors, which I don't mind, but they're very distracting but again, it's all I'm saying there is synesthesia for me is extreme.

[00:50:59] I either love it or I hate it. The middle bits, you know, the bits in between, they're very few and far between.

[00:51:05] Michael: My personal experience is with my perception that feelings are secondary. So if I see a color, which I don't like, or it can occur that I have negative feelings on a specific color, only very few colors I don't like, but I can have negative feelings. If I perceive a sound and this sound induces a color, which I don't like, then my feelings are the same.

[00:51:29] That's my personal experience but given the fact that maybe in the later processing states of the brain, my synesthetic colors are handled like colors perceived by the eyes, then it is reasonable that the emotions induced are the same and the feelings I have are the perceptions of those emotions and maybe similar.

[00:51:51] I know that some synesthetes have directly induced emotions by a specific stimuli. So some sort of emotional synesthesia but in my case it is just sensory connections and then the sensory inputs are handled in the same way. It doesn't depend whether it's a synesthetic phenomenon, or it's directly perceived by a sensor.

[00:52:16] Adrian: Thank you everyone. Any questions? In that case, we'll just go to kind of closing statements, and I'd love to invite everyone to sum up what we've been talking about. I'll be honest, this has just given me more questions for everyone, and I'm happy about that. But every time I have a synesthesia conversation, it just opens up my mind to more possibilities.

[00:52:37] So, thank you everyone, thank you for joining us, and yeah, let's sum up our thoughts.

[00:52:42] Julia: I think a summary here from my perspective, given my contributions is that synesthesia is rare, but it's normal. It's not from Mars. It's two sides of the same piece of paper with synesthetes on one side and everybody else sharing intuitive associations on the other.

[00:52:59] Synesthesia has, advantages and disadvantages. Advantages in creativity and design and being, you know, if you understand synesthesia and cross modality, or if you're a synesthete, then you do make a great design consultant. But on the other hand it can be intrusive. We should be very mindful.

[00:53:15] I hope I'm always mindful when I'm in James's company, not to say things that are outrageously bad tasting. And we should also be aware that people with synesthesia, you know, they have these other traits that also challenges, you know, anxiety and so on but thank you very much for inviting me here.

[00:53:30] I've really learned a lot. Every time I talk to synesthetes, I learn a lot.

[00:53:33] Adrian: Thank you, Julia. James?

[00:53:35] James: Yeah, it's just to echo that I think that multi sensory design has got a very, very important part to play. It's just that again, people that do use it have got to be careful. It's like having a weapon, you know, don't overuse it, don't underuse it.

[00:53:48] So certainly take it into consideration when you're designing stuff. It's like I said before, anybody would agree with me, the more senses you involve in something, the better connection you have with that something. It could be an object or an environment. And good sensory design, as I said earlier, you can feel it, and it can feel me, and there's a perfect match there. So it's definitely got a place in society.

[00:54:10] Adrian: Thank you, James. Anton?

[00:54:12] Anton: Oh, yes. For me, synesthetic design, whatever you understand this, is the next step of multisensory design. Because there is kind of a more to it, in terms of how semantics and meaning projects into the senses, how it transforms the sensory experience. And we know, well, for instance, that cinema exploits the sense of presence and there are lots of other natural, you know, phenomena that we're trying to catch and understand and exploit elsewhere and use elsewhere.

[00:54:45] And so I think that synesthesia, as we study this, I mean, congenital synesthesia, promises so much in terms of meaning to the sensory links. That we shouldn't miss this, and we should go both ways, studying synesthesia, studying design, studying arts. Somewhere it will click. And gives us like, more resonant synesthetic effect. So thank you for inviting me and being in such a wonderful, insightful company.

[00:55:13] Adrian: Thank you too, Anton.

[00:55:15] Michael: Yes, also, thank you very much for inviting me also, I enjoyed it very much. My resume regarding design is... so that's my feeling.... that we need to be very precise what we are doing, which connections between the senses we are handling, what is meaning, what are semantic effects alongside, what are iconic aspects, what are the common aspects, and what are the individual aspects.

[00:55:42] In the design process and all in the arts, we can use both. Sometimes we need the common aspects and on the other side we, make use of our specific, synesthetic experience, which can be very individual to make something more exciting and extraordinary but we have to be in the design process. We have to be very, very clear what we are doing at the moment to achieve a real multisensory result. Thank you.

[00:56:07] Adrian: Thank you. Yeah, and as I mentioned, I'm an industrial designer, creative director, so I've also had a lot of experiences where some synesthesia has contributed directly.

[00:56:16] Sometimes if a client would ask me for a logo but they don't have a color requirement, immediately I will know what color I'm going to start with or what kind of palette it might be forming. And as you were saying, first of all, we have an installation at Venice design week, and that's being set up today. So you'll be able to come and experience a representation of my synesthesia as you move from the yellows through the reds and greens and blues.

[00:56:44] You'll be experiencing a representation of some of the notes and sounds that it makes. As well, I was writing some of my first synesthesia experiences. And I highly recommend one of my favorite movies, Fantasia, by Disney. Because it features cartoons which is beautiful. I used to want to be an animator, but at the same time, there's lots of visual music.

[00:57:07] And it really brings that the first Fantasia, which is from 1946, right? Something like that. Maybe even earlier. And I'm also happy to discover today that design can benefit from synesthesia, not just art, because I've always thought that art is the artist's expression, but design has to be within a brief, right?

[00:57:28] It fits into a box. And I'm happy that box has kind of expanded for me today. I'm just really grateful. And all these people are doctors, professors, academics so I've taken the liberty to just call you by your first names today for the ease of the panel, but I really highly respect everyone's work in here.

[00:57:47] And, just wanted to say as well, thank you to journey through the senses and that's that's the name of the installation as well as the organization, which brings artists together scientists together through synesthesia to create experiences, and there have been some amazing ones all over the world. So thank you as well to journey through the senses, and I'm Adrian from Nairobi design, and thank you everyone.