Animals and humans process vast amounts of information but retain only the most relevant experiences, with emotions playing a key role in memory consolidation. Emotions, particularly stress, enhancing attention and arousal, influence brain regions like the hippocampus. In neuroscience, it is known that hippocampal reactivations help recall and consolidate experiences. In artificial intelligence (AI), strategies inspired by these reactivations, especially in reinforcement learning, have been shown to accelerate learning. When it comes to emotionally charged circumstances, the relationship between stress, its most prominent hormone, cortisol, and memory functions in the hippocampus, is complex. This project aims to explore the link between stress and hippocampal reactivations via a computational model, and test it in artificial agents. This bio-inspired approach could (a) reveal new insights into the generation of hippocampal replay mechanisms and (b) improve AI's learning efficiency.

Most animals and humans encounter an immense amount of information and experiences throughout their lives, yet they manage to retain and recall only the most relevant, or a partial, but meaningful representation of them. Not everything experienced is stored in long-term memory; emotions play a key role in this memory consolidation process [Bower (1983) Phil. Trans. Royal Soc. London. B, Bio. Sci. 302.1110:387-402]. Emotions strongly influence cognitive processes, by enhancing our level of arousal and attentional resources, modulating the activity of brain regions such as the amygdala, the prefrontal cortex, and the hippocampus [Tyng et al., (2017) Front. Psychol]. Since the work of Scoville and Milner [(1957) Journ. Neurosc. 9.8:2907-2918], the role of the hippocampus as a cognitive mapping center has been studied in neuroscience, particularly through neurophysiological studies on spatial tasks in rodents. This research highlighted the fact that certain patterns of sequential activation of hippocampal neurons, observed during task execution, are then replayed during sleep or periods of calm wakefulness in the animal. These activities have been called hippocampal reactivations and are recognized as a powerful mechanism used, particularly by place cells, to recall, organize, and consolidate past experiences and infer future ones.
From an artificial intelligence (AI) perspective, it is also known that offline replay and updating of values associated with an agent’s actions can accelerate learning after a small number of real interactions with rewarding or punishing events (for example, Lin [(1992) Mach. Learn. 8-293-321]). The great interest in implementing computational strategies inspired by hippocampal reactivations in AI lies in tasks where past experiences and acquired knowledge must be re-evaluated and refined to perform better in future decision-making steps. This is the case with reinforcement learning (RL) paradigms [Sutton and Barto (2018) MIT press], where initially, when no prior knowledge is usually available, the best strategy is to interact with an environment through trial and error, and only when the level of experience increases the agent can exploit its previous knowledge to reach a sequence of actions approaching optimal behavior. In mammals and rodents, this consolidation of knowledge does not depend solely on the animal performing the same actions in the same situations: memories, particularly targeted recalls of experiences, are fundamental for effective learning from a small set of accumulated real experiences. Since the first RL algorithms which exploit strategies inspired by hippocampal reactivations [Sutton (1990) Mach. learn. proceed.], several researchers have proposed RL-based computational models inspired by these neuroscientific findings that are capable of reproducing the major experimental results regarding the quantity and type of reactivations generated [Khamassi and Girard (2020) Bio. Cybern. 114.2:231-248].
One of the major forces that impact the emotional state is stress, induced by the interaction between the animal, other animals, and the environment, both perceived through the animal’s senses. The influences from an external stressful environment and positive social interactions have been recently modeled by Khan and Cañamero [(2022) Front. Rob. AI 9] as the interaction between two hormones, cortisol and oxytocin, and their homeostatic balance. Concerning the role of emotional states related to stress and cortisol, Cañamero’s research has for example modeled their influence on learning and adaptation [Hiolle, Lewis and Cañamero (2014) Front. Neurorob. 8], behavior development [Lones, Lewis and Cañamero (2017) IEEE Trans. Cogn. Devel. Sys. 10.2-445-454], pain perception [L’Haridon and Cañamero (2023) ACII] and the appearance of compulsive behavior in Obsessive-Compulsive Disorder [Lewis, Canamero and Fineberg (2019) Comp. Psych.; Lewis and Canamero (2019) ACII]. Regarding the role of anxiety and stress on the hippocampus memory mechanisms, many key issues still need to be investigated: it has been known from various animal and human studies that stress impairs many memory functions at the level of the hippocampus [Kim, Pellman, and Kim (2015) Learn. Mem. 22.9:411-416] but, recently, Sherman et al. [(2023) Journ. Neurosc. 43.43:7198-7212] found that cortisol could also enhance the hippocampal associative memory functions.
The thesis project aim at a better understanding of the relationship between stress and the generation of hippocampal reactivations by means of a computational model that could be systematically tested in simulation or eventually on a real robot. This will bring the great advantage of testing functional hypotheses we have about the relationship between stress and hippocampal replay, with our computational model, in a very controlled experimental set-up, when we could repetitively simulate different emotional profiles and sensitivities on our agents. An additional point to this research will be to observe and analyzed the proposed model embodied on a robotic platform with the aim to look systematically at what are the effects of such a model in a very controlled context that still present elements of stochasticity and unpredictability that make a robot interacting with the real world a step closer to animal experiments, compared to pure simulations.
As analyzed in Massi et al. [Front.Neurorob. 16], the adoption of RL techniques inspired by hippocampal reactivations for AI has just begun. After validating a strategy that combines offline reactivations generation through a model-based agent with reactivations generated by a model-free method, the question remains of how to optimize the timing of this offline reactivation generation and its quantity. So, the proposed project aims to link the generation of offline reactivations to the internal emotional state of an agent. The idea is that, by following the concept of homeostasis and the regulation of key emotion-related hormones, such as oxytocin and cortisol [Avila-Garcia and Cañamero (2004) SAB], in a learning task, the agent will change its internal emotional state in relation to (a) its performance in task completion (e.g., in a spatial navigation task, effectively avoiding punishments to quickly reach a reward state), (b) egocentric external stimuli (e.g., in a spatial navigation task, proximity to walls or other agents approaching), and (c) a combination of the above two elements. This emotional internal state will allow the agent to trigger reactivations during moments of intense stress, for example, and not at just any moment in the task, where they may prove unnecessary. With this bio-inspired approach to AI, we will test and validate optimal strategies for offline reactivation generation within RL algorithms, with the aim of improving and accelerating artificial learning and disclose new possible driving mechanisms for hippocampal reactivations in neuroscience.
So far, in RL, many strategies inspired by neuroscientific evidence on the hippocampus have been proposed and tested to have a spontaneous and optimal generation of different types of replay-like activities [Mattar and Daw (2018) Nat. Neuro. 21.11:1609-1617; Diekmann and Cheng (2023) ELife 12]. Still a model that bases the generation of such reactivations on emotions and more specifically on the internal emotional state of an agent is lacking. That’s why the proposed project aims to theorize and test such a model which could spontaneously enable the generation of RL-based replay to improve memory consolidation and learning processing over different tasks and help shedding light on our understanding on the functional relationship between emotional states (stress in particular) and hippocampal replay. This research objective can be accomplished also thanks to the validation of our results against experimental behavioural data from rodents that could be provided by our collaborators.

The human brain is a complex, self-organizing system where neurons transmit information via ion channels and communicate through synapses. Mathematical models have helped researchers explore how local population dynamics and brain network topology influence overall brain activity and synaptic/structural neuroplasticity as well as the brain’s ability to reorganize in response to various stimuli. This research aims to create a large-scale spiking neural network (SNN) model to investigate mechanisms of memory formation, storage, and retrieval, focusing on the hippocampus. Using a Hodgkin-Huxley-based model, the study will explore dopamine's effects on synaptic plasticity and test various connectivity patterns to find optimal configurations for memory processes. Additionally, the project will computationally assess the impact of emotional states on the excitation-inhibition balance in hippocampal circuits, aiming to enhance our understanding of neural mechanisms underlying memory.

The human brain is a complex, self-organizing system. Researchers aim to understand its core principles to improve brain models, explore disease neurobiology, and develop treatments. Neurons, the brain's basic units, transmit information via voltage-gated ion channels and communicate through chemical and electrical synapses. Neuronal networks generate brain functions such as perception, thought, and memory. Pathological activity patterns, linked to disorders like epilepsy and migraines, often result from genetic mutations affecting ion channels and synaptic function. [Russell et al. (2013) Annu. Rev. Neurosci. 36:25]. Neuronal activity in individual nodes is typically modelled by a system of ordinary differential equations. Such types of mathematical models provide a theoretical framework for investigating the impact of the localized population dynamics combined with the complex topology of the brain network on the overall brain activity [Deco et al. (2008) PLoS Comput Biol 4:e1000092; Sanz-Leon et al. (2015) NeuroImage 111:385) and its dynamics [Popovych et al. (2021) NeuroImage 236:118201; Manos et al. (2023) Front. Comput. Neurosci. 17:295395; Courson et al. (2024) Front. Comput. Neurosci. 18:1360009].
Neuroplasticity, also known as neural or brain plasticity, refers to the brain’s ability to adapt and reorganize in response to factors such as learning, environmental changes, practice, or stress [Voss et al. (2017) Front. Psychol. 8:1657]. This adaptability occurs through both synaptic plasticity, where neurons modify synaptic transmission, and structural plasticity, involving the formation or retraction of neural connections and changes in cell morphology [Kehayas & Holtmaat (2017) The Rewiring Brain 3-26]. In [Manos et al. (2021) Front. Physiol. 12:716556] the authors computationally explored long-term changes in structural plasticity triggered by external stimulation to optimize stimulation dosage while in [Anil et al. (2023) PLoS Comput. Biol. 19:e1011027] the authors simulated a recurrent neural network with homeostatic structural plasticity, revealing its importance in stimulation protocols.
Recent studies have underscored the hippocampus's susceptibility to neuroplastic changes in neurodegenerative diseases, impacting cognitive and emotional functions [Weerasinghe et al. (2022) Int. J. Mol. Sci. 23:3349]. Synaptic structural alterations, crucial for learning and memory, have been observed under high-resolution time-lapse imaging [Ma and Ta (2022) Semin. Cell Biol. 125:84]. In the prefrontal cortex, spiking activity patterns supporting cognitive tasks like working memory can be learned and replayed using biologically realistic plasticity rules [Sarazin et al. (2021) Front Neural Circuits 15:648538]. Models of hippocampal circuits, including CA1, CA3, and DG regions, provide insights into long-term memory mechanisms [Chua and Tan (2017) AAAI Tech. Rep. SS-17-07]. Additionally, silent cell assemblies contribute to memory persistence, influenced by neuromodulation that balances synaptic connectivity [Gallinaro et al. (2021) PLoS Comput. Biol. 17:e1009593; Fuchsberger & Paulsen (2022) Curr. Opin. Neurol. 75:102558; Tesler et al. (2024) Front Comput Neurosci. 18, 1432593].
The hippocampus has been recognised as a predominant brain structure for the consolidation of new information from short-term to long-term memory [O’Keefe and Dostrovsky (1971) Brain Res. 34:171]. Many studies have been conducted since, given the increasing evidence of the importance of hippocampal cell units for self-localisation and information recall. The current consensus is that the hippocampus displays a particular activity pattern, called sharp wave ripples (SWR), at a frequency of 150-200 Hz, which is temporally compressed with respect to the timescale of the neural activity happening during the real spatial experience, and thus enhances spike-timing dependent plasticity (STDP) [Dan and Poo (2004) Neuron 44:23]. Hippocampal SWR could encode temporally structured spatial patterns and drive the initial storage and the later retrieval of relevant experience [Pfeiffer (2020) Hipp. 30:6]. Models of memory formation, consolidation, and retrieval have been proposed at different levels of brain abstraction, from decision-making [Cazé et al. (2018) Journ. Neurophys. 120:2877] to spiking neural networks (SNN) [Tan et al. (2013) IEEE CIS], tested on artificial agents [Massi et al. (2022) Front. Neurorob. 16: 864380] and compared to animals’ neural recordings [Mattar and Daw (2018) Nat Neurosci. 21:1609]. Exogenous stimuli and the subsequent emotions play an important role in this, by modulating the activity of brain areas such as the amygdala, the prefrontal cortex and the hippocampus [Tyng et al. (2017) Front.Psychol. 8:1454]. In animal and human behavior, the two most studied conditioning stimuli concern sustenance and survival and they can be of opposite emotional strength, with two different neural circuits involved in the process [Yacubian et al. (2006) J Neurosci 26:9530].
From a decision-making and reinforcement learning (RL) perspective, the question was addressed recently in [Bryzgalov, Massi, et al., in preparation], and preliminary results show that higher learning rate and more intense replay activity are needed while learning to avoid than learning to approach. Brain circuits for encoding aversive and appetitive stimuli can be considered as orthogonal in the brain [Tye (2018) Neuron 100:436], but the same neurotransmitters, such as dopamine, could be implied in the learning process. In fact, the release of dopamine is linked to unexpected reward signals during learning, but it is shown that it can be linked to unexpected punishment too [Sands et al. (2023) Sci. Adv. 9:eadi4927]. Dopamine but also other neurotransmitters, has been proved to enhance synaptic consolidation and recall, but the precise dynamics of this mechanism have just started to be investigated [Lehr et al. (2022) Sci. Rep. 12.1:17772]. The dopaminergic influence in a SNN model of the striatum has been modeled for RL tasks [González-Redondo et al. (2023) Neurocomp. 548:126377] and, it has also been implemented as the joint action of short-latency excitation and long-latency inhibition for the modulation of both STDP and neuronal excitability [Chorley and Anil (2011) Front.Comp.Neuro. 5:21]. Finally, [Bono et al. (2023) Elife, 12:e80671] have used a mechanism inspired by hippocampal replay to learn successor representations and cognitive maps in a SNN.
The Research Objectives (ROs) of this proposal can be summarized as follows:
- RO1: Large scale SNN setup and implementation of synaptic/structural plasticity rules and neuromodulation features: Theoretical and computational understanding of the relevant combined relevant mechanisms for memory formation, storage and retrieval.
- RO2: Study the effects in the neural excitation-inhibition balance of positive and negative delivered emotional states in the hippocampus large scale SNN.
We will use a Hodgkin-Huxley-based spiking neuron model that supports a wide range of firing patterns, including tonic spiking, bursting, and seizure-like events [Bandyopadhyay et al. (2021) J. Comput. Neurosci. 50:33]. Such a SNN model allows us to implement and investigate dopamine effects via voltage-gated calcium and potassium channels while will allow us to design large neural networks to explore various structural and synaptic connectivity patterns, aiming to identify configurations that favor memory formation and retrieval. More details can be found in the “Feasibility of the project in 3 years - Specify the steps” section.
In summary, this research proposal will allow us to build a large-scale SNN to (i) explore different mechanisms associated with optimal memory storage and retrieval and (ii) computationally investigate the role of neural excitation-inhibition balance in these processes when positive and negative emotional states are delivered in the hippocampus.