Perception and Action

Overview

This research theme relates to how we extract information from the environment and use this information to guide our actions, and how such interactions result in learning and memory. Researchers use various behavioral measures (such recording hand and eye-movements), neuroimaging techniques (EEG and fMRI), as well as investigating patients with brain lesions and manipulating neural responses using Transcranial Magnetic Stimulation (TMS) and Direct Current Stimulation (DCS).

Below we describe some of the research questions that motivate our work in each of these themes.

Perception and attention

How do we perceive the world, and our position in it, from vision, proprioception, and audition? How do our brains organise and prioritise the enormous amounts of information available, so that we can make sense of the structure and events in the world?

Visually guided movement

The efficiency and fluidity of human movements belies the computational complexity required to control them. How are targets for movement identified and selected? How are properties of these target objects estimated to plan the movement? How is information in visual ‘co-ordinates’ transformed into an appropriate form to control complex effectors like the arm and hand (and even tools that we may be holding)? How are movements monitored and controlled ‘online’ to be maximally efficient?

Eye movements

How do we select targets we want to look at? How are target locations encoded? What role does our eye position sense play in our perception of where things are? Are eye movements affected by disorders such as attention-deficit hyperactivity disorder?

Emotion and motivation

How does emotion influence decision making? How does the motivational context influence the valuation of available choices? How can we measure self-beliefs and motivations?

Memory

Humans can store a vast amount of different information in their memories. They can remember events from their past and the knowledge about the world they obtained over their lifetime. In addition to this declarative memory, humans also show a variety of other forms of non-declarative memory. Non-declarative memory usually leads to changes in performance and can occur even when the original learning episode cannot be remembered. The various forms of memory tend to co-occur at any point in time. Therefore, at any given moment, several forms of memory may be active, and they might work independently of each other or interact. We investigate these dynamics of human memory with both behavioural methods as well as with event-related brain potentials. The aim is to explore and describe the flexibility in how different forms of memory are invoked and how they interact.

Neuropsychology

As well as studying the above topics in normal, healthy participants, the group carries out a large amount of research on the effects of different kinds of neurological impairment on these processes.

Object representation and recognition

Imagine that you are sitting in a room observing the scene around you. You see a cup, and a guitar, and you instantly recognise them. If you want to, you can reach out across space and pick them up. In fact, this is something that most of us can do effortlessly. But how do we do it? How does our visual system work? How can we perceive, recognise and interact with objects so easily? How do we know where particular objects are in space, and how to reach them? And what happens to our visual system when we are no longer able to recognise objects as is the case of some individuals following an injury to the brain?

Discoveries

Understanding human tool use: information from vision and touch is integrated

When we use tools our brains receive information about the sizes, locations, etc of objects from both vision and touch. An ‘optimal brain’ would combine or integrate these signals because that would allow us to estimate properties of the world with the greatest possible precision, over a wide range of situations. For ‘multisensory integration’ to be effective, however, the brain must integrate related signals, and not unrelated ones (imagine, for example, if the apparent size of the coffee cup in your hand was affected by irrelevant objects that you also happened to look at, like buildings, people and so on).

The brain could achieve this by considering the statistical similarity of estimates from vision and touch (haptics) in terms of magnitude, spatial location, time of occurrence, etc? This would be effective because in normal grasping visual and haptic signals arising at the same time/place etc. are in fact more likely to originate from the same object than more dissimilar signals. The problem is complicated by using tools, however, because they systematically perturb the relationship between (visual) object size and location, and the positioning and opening of the hand required to feel an object. Said another way, our visual estimates are derived from the object itself, but our hands can only feel the object via the handles of the tool. Our research investigated whether the brain is able to understand the resulting sensory remapping, and appropriately integrate information from vision and haptic touch when using tools.

We used a stereoscopic display, and force-production robots, to create a virtual environment in which we could independently control the visual and haptic properties of objects, and create ‘virtual tools’. Psychophysical results showed that the brain does integrate information from vision and haptics during tool use in a near-optimal way, suggesting that when we use tools, the haptic signal is treated as coming from the tool-tip, not the hand.

As well as extending our understanding of human tool use, these results have practical implications for the design of visual-haptic interfaces such as surgical robots. This work is published in the Journal of Vision.

Journal of Vision

Building 3D stereoscopic displays that better suit the human visual system

Stereoscopic 3D (S3D) displays are enjoying a huge surge in popularity. In addition to entertainment, they have become common in a range of specialist applications that includes operation of remote devices, medical imaging, scientific visualisation, surgery and surgical training, design, and virtual prototyping. There have been significant technological developments— the red-green glasses are long gone—but some fundamental problems remain. Specifically, S3D can induce significant discomfort and fatigue and give rise to unwanted distortions in-depth perception.

As S3D viewing goes from an occasional to a routine activity, and more of us become users of the technology, there is a pressing need to better understand how our visual system interacts with S3D media. An EPSRC-funded project in Simon Watt’s lab has investigated this issue. In the real world, when we move our eyes to look at near and far objects (vergence), muscles in the eye also change the shape of the lens (accommodation), to focus at the same distance. In S3D viewing, however, we have to look at ‘objects’ that can be in-front-of or behind the screen, while remaining focused at the screen surface (see picture). This ‘decoupling’ is a key cause of problems, and the researchers have constructed a novel display (initially developed with researchers in the United States) designed to address this by simulating a continuous range of focal distances, as experienced in the real world, using a small number of discrete focal planes.

The researchers have carried out extensive experiments to develop and evaluate this approach, including measuring eye movements and the focusing response of the eye. The results have provided information on how to build improved S3D displays, and produce effective S3D content. This has resulted in invited talks/workshops not only for specialists in the S3D industry, but also at more general display-industry venues, and international standards-setting bodies (the Society for Motion Picture and Television Engineers, or SMPTE).

This research also has a significant basic science component, and the results have provided new insights into the mechanisms underlying how our eyes focus.

Example of published work from this project:

Journal of Vision

Paralinguistic cues and social interaction

Social interactions involve more than "just" speech. Similarly important is a, perhaps more primitive, non-linguistic mode of communication. While what we say is usually carefully and consciously controlled, how we say things, i.e. the sound of our voice when we speak, is not. The sound of our voice may therefore be seen as a much more "honest" signal. The researchers are interested in the effects of these paralinguistic cues on social interaction and how these are processed in the brain. One such signal is vocal attractiveness, which is known to influence the speaker's success at job applications, elections or short-term sexual relations. In a series of experiments we were interested in what makes a voice attractive.

Dr. Patricia Bestlemeyer and colleagues found that the voices which were perceived as more attractive tended to be more similar to an average voice composite in terms of its formant frequencies and sound "smoother".

Next, she measured brain activity while participants listened to the same vocal sounds ("ah") and performed an unrelated task. Activity in voice-sensitive auditory cortex and inferior frontal regions were strongly correlated with implicitly perceived vocal attractiveness. While the involvement of auditory areas reflected the processing of acoustic contributors to vocal attractiveness (e.g. frequency composition and smoothness), activity in inferior prefrontal regions (traditionally involved in speech) reflected the overall perceived attractiveness of the voices despite their lack of linguistic content. Dr. Bestlemeyer's results provide an objective measure of the influence of hidden non-linguistic aspects of communication signals on cerebral activity.