Role of brain serotonin in modulating fish behavior.

The organization of the brain serotonergic system appears to have been highly conserved across the vertebrate subphylum. In fish as well as in other vertebrates, brain serotonin (5-HT), mainly acts as a neuromodulator with complex effects on multiple functions. It is becoming increasingly clear that acute and chronic increase in brain 5-HT neurotransmission have very different effects. An acute 5-HT activation, which is seen in both winners and losers of agonistic interactions, could be related to a general arousal effect, whereas the chronic activation observed in subordinate fish is clearly linked to the behavioral inhibition displayed by these individuals. Fish displaying divergent stress coping styles (proactive vs. reactive) differ in 5-HT functions. In teleost fish, brain monoaminergic function is also related to life history traits.

Introduction

Behavioral plasticity includes rapid transient modulation in behavior in response to changing social context, for example, the classical winner and loser effects (Hsu et al. 2006). Also, other environmental factors such as food availability and predation pressure may result in plastic behavioral responses. In many species, environmental factors, biotic, as well as abiotic experienced early in life appears to be especially important in shaping the behavior of an individual. Life history traits such as migration, age of sexual maturation, or even sex, could also be modified, especially through social factors (Oliveira 2012).

Our knowledge on mechanisms mediating behavioral plasticity is increasing but is still limited. However, it is clear that the brain monoaminergic systems constitute an important part of these mechanisms (Winberg and Nilsson 1993; Oliveira 2012; Teles et al. 2013). Especially the serotonergic system has attracted a lot of interest as a neuromodulator. Serotonin (5-hydroxytryptamine, 5-HT) release and turnover is responsive to multiple stressors, for example, social interaction, nutritional status, and immune challenges (Winberg and Nilsson 1993). Moreover, central 5-HT has been implicated in the control of agonistic behavior, stress responses, endocrine functions, as well as sex change in sequential hermaphrodites (Larson et al. 2003), such as the saddleback wrasse Thalassoma duperrey.

Here we will review the organization of the teleost 5-HT system, focusing on similarities and differences as compared to other vertebrates, especially mammals. Moreover, we will review how the teleost 5-HT system responds to various stressors and how that may modify the behavior of fish subjected to social stress. The brain 5-HT system is a neuromodulatory system that plays a central role in the processes of neuronal plasticity. In this review, we will present results clearly illustrating the divergent behavioral effects of acute and chronic activation of the brain 5-HT system. We will end by describing the role of the brain 5-HT system in divergent stress coping styles and how this may relate to life history traits and phenotypic plasticity.

The Teleost Brain Serotonergic System

The organization of the monoaminergic systems is evolutionarily well conserved throughout the vertebrate subphylum (Lillesaar 2011). As in mammals and other vertebrates, 5-HTergic cell bodies of the teleost brain are mainly localized to the hindbrain, in the raphe nucleus. However, in teleosts, and other non-placental vertebrates, 5-HTergic cell bodies are also found outside the raphe area, in the pretectum and basal forebrain (Lillesaar 2011). Interestingly, in zebrafish Danio rerio, various 5-HT populations express different paralogues of 5-HT specific genes, as for example, different forms of tryptophan hydroxylase (TPH), the rate-limiting enzyme in 5-HT biosynthesis. In zebrafish, TPH2 is being expressed in the raphe, whereas TPH1a and TPH1b are dominating in diencephalic and peripheral 5-HT cells, for example, along the gustatory tract (Anderson and Caio 2014). A fourth TPH isoform, TPH3, is being expressed in hypothalamic areas of the zebrafish (Ren et al. 2013). Information on the kinetics of teleost TPH isoforms is still lacking. However, the zebrafish TPH3 appears to show kinetics comparable to those of mouse TPH1 (Ren et al. 2013). In mammals, TPH1 has a lower Km for its substrate, L-tryptophan (Trp), than TPH2. In fact, the Km value for TPH2 is in the same range as the in vivo brain concentrations of Trp (McKinney et al. 2001, 2005). Thus, the rate of raphe 5-HT synthesis by TPH2 is strictly limited by brain Trp availability.

As a consequence of the fact that the rate of raphe 5-HT synthesis is restricted by Trp availability, the rate of 5-HT synthesis will be affected by the amino acid composition of the diet and possibly also sympathetic activation and immune responses (Russo et al. 2009). It has been hypothesized that Trp acts as a mechanism for the 5-HT system to monitor homeostatic challenges (Russo et al. 2009). Interestingly, in teleosts this may not be the case for hypothalamic 5-HT neurons expressing TPH1 and TPH3. However, as in mammals raphe 5-HT neurons of the teleost brain, expressing TPH2, project to hypothalamic areas (Lillesaar 2011). Thus, Trp availability may still affect hypothalamic 5-HT release, through its effects on raphe 5-HT biosynthesis, even in teleosts. The function of teleost-specific hypothalamic 5-HT cells, and to what extent hypothalamic 5-HT functions in teleosts differ from that of mammals is still not known.

Effects of Stress and Social Interaction on Brain Serotonergic Activity

The development of dominance-based social hierarchy is common among teleost fish. The social rank of individual fish within a hierarchy is usually determined by the outcome of dyadic aggressive encounters (Huntingford and Turner 1987). These aggressive interactions consist of behaviors such as displays, charging, nipping, and chasing. Fish losing aggressive interactions becoming subordinate will suppress further aggression and retreat from the dominant individual (Øverli et al. 1999; Larson et al. 2006). The winner, the dominant individual, on the other hand, remains highly active and aggressive. Behavioral effects of social defeat include appetite inhibition (Meerlo et al. 1997; Øverli et al. 1998; Kramer et al. 1999; Montero et al. 2009), reduced aggression (Blanchard et al. 1995; Höglund et al. 2001), and decreased reproductive behavior (D’Amato 1988; Perret 1992).

Aggressive interactions are stressful to all individuals involved and in dyadic interactions such that winners as well as losers show a rapid elevation in plasma cortisol along with an activation of the brain 5-HT system, as indicated by elevated brain concentrations of 5-hydroxyindoleacetic acid (5-HIAA) and 5-HIAA/5-HT ratios (Winberg and Nilsson 1993; Øverli et al. 1999). During this early phase of hierarchy formation there is also an increase in brain levels of Trp and in some cases also an increase in brain 5-HT concentrations (Winberg and Nilsson 1993). In the established dominance hierarchy brain 5-HIAA concentrations and 5-HIAA/5-HT ratios remain elevated in subordinate fish, whereas this indexes of elevated 5-HT activity return to baseline levels in dominant individuals. In fact, in groups of juvenile salmonids with established hierarchies there is a correlation between social rank and brain 5-HIAA/5-HT ratios, fish of lower rank showing higher 5-HIAA/5-HT ratios (Winberg and Nilsson 1993). However, in established dominance hierarchies brain Trp levels tend to decrease in subordinate fish even though brain 5-HT concentrations are usually unchanged. Thus, brain 5-HT synthesis is probably upregulated by increasing TPH activity, an effect that has been documented in mammals subjected to chronic stress (Chen and Miller 2012). Still, in some cases, brain 5-HT concentrations may decline in subordinate fish following long-term intense social interaction (Cubitt et al. 2008).

Social subordination results in drastic behavioral effects in fish. The subordinate fish displays a general behavioral inhibition, including suppression of aggressive behavior, feeding, and locomotion (Winberg and Nilsson 1993). A chronic activation of the brain serotonergic system seems to play an important role in mediating these behavioral effects (Winberg and Nilsson 1993; Winberg et al. 2001).

Interestingly, only long-term activation of the brain 5-HT system seems to have an inhibitory effect on aggressive behavior. During the formation of a dominant–subordinate relationship during dyadic interaction, both the winner, the future dominant, and the loser shows an activation of the brain 5-HT (Øverli et al. 1999). Still, the winner shows no signs of behavioral inhibition. Similarly, in rainbow trout Oncorhynchus mykiss elevated dietary Trp results in a rapid increase in brain 5-HT but decrease in aggressive behavior occur first after feeding the elevated Trp for 7 days (Winberg et al. 2001). Are there any immediate behavioral effects of elevated brain 5-HT activity?

Behavioral Effects of Acute Fluoxetine Treatment in Zebrafish

Recently zebrafish Danio rerio have been applied as a model to study effects of agonistic interaction on behavior, stress responses, and brain monoamines (Dahlbom et al. 2012; Teles et al. 2013). If kept in pairs or low-density groups, zebrafish show agonistic behavior similar to that of cichlids and juvenile salmonids, which are well known to be territorial and highly aggressive. Clear winner and loser effects have been observed in zebrafish with subordinate fish displaying a clear behavioral inhibition (Oliveira et al. 2011). Also, neuroendocrine effects of social interaction appear to be similar to those observed in juvenile salmonids and other more territorial teleosts. In zebrafish, agonistic behavior of males and females has been compared in sexually mature fish. Filby et al. (2010) reported that males were more aggressive than females, whereas Dahlbom et al. (2012) showed that males and females were both equally aggressive when interacting in iso-sexual pairs. These divergent results may be related to population differences.

In a recent experiment, we applied zebrafish as a model to study the behavioral effects of acute treatment with the selective serotonin re-uptake inhibitor (SSRI), fluoxetine. In this experiment, male and female zebrafish were immersed in 0.5 or 1.5 mg/L fluoxetine in water for 2 h. The effects on open field behavior, aggression and, brain 5-HT and 5-HIAA concentrations were quantified. In the open field test (34 cm length × 29 cm width × 14 cm height, filled with water to a depth of 6 cm), a video tracking system (Ethovision, Noldus, The Netherlands) was used to quantify the distance moved, duration in the center zone (defined as half the area of the open field arena), immobility duration, and number of zone transitions. The center zone of the open field arena is considered a more risky area and duration in the center zone is an indicator of boldness. In the open field test, fluoxetine had no significant effects on immobility duration, total distance moved, or zone transitions neither at 0.5 mg/L nor at the dose of 1.5 mg/L. However, there was an effect on time spent in the center zone (“in-center zone duration,” P = 0.021). Subsequent pairwise comparisons revealed that fish treated with the lower dose (0.5 mg/L fluoxetine) tended to spend more time in the center zone compared to fish treated with the higher dose (1.5 mg/L fluoxetine) (Pairwise comparison, adjusted significance P = 0.022, Figure 1). When comparing sexes regardless of treatment however, female fish showed significantly lower distance moved (P = 0.002), longer immobility duration (P = 0.021), and fewer zone transitions (P = 0.003) than male fish (Figure 1).

Figure 1.

Open field test. Four different behavioral parameters recorded for 20 min following 2 h of immersion in 2 different fluoxetine doses compared to controls (A–D). A Kruskal–Wallis analysis revealed an effect of fluoxetine on time spent in the center zone. Subsequent pairwise comparison reported an adjusted significance (P = 0.022) between the 0.5 mg/L and 1.5 mg/L group (B). Frequency of transitions between the center and outer zone (A), distance moved (C), and immobility duration (D) were all significantly decreased in females in comparison to males, Mann–Whitney U-test comparisons (see text for statistical analysis). All data are presented as mean ± SEM. *P < 0.05.

Treatment with fluoxetine had no effects on the number of attacks initiated at the doses tested (P = 0.126). In these tests, pairs consisting of 1 control and 1 fluoxetine-treated fish were formed from size-matched fish of the same sex. The fish were visually isolated and allowed to acclimate to experimental tanks for 3 days. Before dyadic social interaction fish were placed in treatment beakers and immersed in 0.5 or 1.5 mg/L fluoxetine in water for 2 h after which they were returned to the experimental tank where the wall that had kept the fish in a pair separated was carefully removed. Aggression was manually scored as attacks/chases during three 5 min sessions; during the first 5 min of interaction, and following 30 and 55 min of dyadic interaction (a total of 15 min).

To verify that we had effects of fluoxetine on the brain 5-HT system following 2 h of immersion exposure to 0.5 and 1.5 mg/L fluoxetine brain from exposed and control zebrafish were analyzed for concentrations of 5-HT and 5-HIAA. Fluoxetine is a SSRI blocking synaptic clearance of 5-HT (Homberg et al. 2010). Thus, treatment with fluoxetine should result in a decrease in brain tissue levels of 5-HIAA, since 5-HT is catabolized following re-uptake to the presynaptic neuron. The results from our experiment are in accordance with this prediction. Zebrafish treated with fluoxetine, 0.5 as well as 1.5 mg/L, show a significant reduction in brain 5-HIAA concentrations and brain 5-HIAA/5-HT ratios (Figure 2). Thus, fluoxetine had a clear effect on the 5-HT system, blocking 5-HT re-uptake and by that increasing synaptic concentration of 5-HT. Still, there was no effect on aggressive behavior in either male or female fish. However, there was a significant effect of the lower dose on open field behavior, fish treated with this dose of fluoxetine spending more time in the center zone. This could be interpreted as an anxiolytic effect making the fish bolder. Still, there was no effect on aggressive behavior or the outcome of dyadic fights for social dominance. Apparently, only long-term elevation of brain 5-HT activity results in the behavioral inhibition displayed by socially subordinate fish. The rapid elevation of brain 5-HT activity displayed by both winners and losers during the early phase of hierarchy formation could be related to an arousal effect of 5-HT. In fact, using calcium imaging, Yokogawa et al. (2012) provided evidence for a role of TPH2 expressing cells in the superior raphe in the arousal response of zebrafish larvae. The behavioral effects of long-term activation of the 5-HT system may be related to plastic changes in serotonergic function (Levinstein and Samuels 2014).

Figure 2.

Brain serotonin (5-HT) and 5-hydroxyindoleacetic acid (5-HIAA) concentrations following 2 h of immersion in 2 different doses of fluoxetine in comparison to controls. Whole brain 5-HIAA/5-HT ratio (A) and 5-HIAA concentration (B) were both significantly lower in treatment groups. All data are presented as mean ± SEM. *P < 0.05, **P < 0.01, ***P < 0.001 (2-way ANOVA).

Stress Coping Styles, Boldness, and Dominance

Divergent stress coping styles, referred to as proactive and reactive, has been described among teleost fish as well as other vertebrates (Koolhaas et al. 1999, 2010; Øverli et al. 2007; Schjolden and Winberg 2007).

When challenged by a stressor proactive individuals display an active response, for example, arousal, active avoidance, display, and aggression whereas reactive individuals when challenged by the same stressor show a passive response characterized by behavioral inhibition. These divergent stress coping styles are also reflected in autonomic, neuroendocrine, and physiological stress responses. For instance, in response to challenges proactive animals respond with a larger sympathetic activation but a more modest elevation of plasma glucocorticoid concentrations than reactive animals. Proactive animals are generally bolder and also easily form behavioral routines. Reactive ones, on the other hand, appear shy but are more plastic in their behavior, for example, they respond more to changes in environmental cues.

In fights for social dominance proactive fish appear to have an advantage. Rainbow trout strains selectively bred for high (HR) and low (LR) post-stress cortisol has been frequently used as a model for studies on stress coping styles in teleost fish (Øverli et al. 2007). The LR strain shows a proactive and the HR strain a reactive stress coping style. In fights for social dominance in size-matched pairs of juvenile HR and LR trout, the LR fish became socially dominant significantly more often than the HR fish (Pottinger and Carrick 2001). Similarly, in non-selected rainbow trout, fish showing a proactive stress coping style had a significantly higher chance to become dominant when interacting in size-matched pairs with fish showing a more reactive coping style (Schjolden et al. 2005). Dahlbom et al. (2011) used a series of behavioral tests to screen individual zebrafish for a behavioral profile. The results show that it is possible to predict the outcome of subsequent dyadic fights for social dominance from this behavioral profile with great certainty. Bold fish were significantly more likely to become dominant than fish being shy.

Stress Coping, Plasticity, and Serotonin

Individual stress coping styles are reflected in brain serotonin functions. For instance, studies on HR/LR trout have shown that the concentration of 5-HT in the brain stem increases in response to stress in HR trout, whereas in LR fish it does not (Øverli et al. 2001). Moreover, concentrations of 5-HIAA increases in the brain stem and optic tectum of HR fish following confinement stress, but not in LR fish. The simultaneous increase in the concentration of 5-HT and its metabolite suggests that both synthesis and metabolism of 5-HT is elevated after stress in HR trout. A different pattern is observed in the hypothalamus, however, where LR trout display higher levels of 5-HIAA than HR fish. There are also differences in the telencephalon between the two lines, in that a higher baseline ratio of 5-HIAA/5-HT is seen in LR fish, but no differences is observed between the stressed LR and HR trout. These differences in neurotransmitter metabolism likely reflect a true difference in functional transmitter release in the two strains, since there was no difference in brain monoamine oxidase (MAO) activity between the LR and HR fish (Schjolden et al. 2006). Moreover, the expression of 5HT1A receptors is higher in LR than in HR trout (Winberg and Thörnqvist, unpublished result). Taken together, these results clearly suggest that selection for stress coping style in rainbow trout is also associated with changes in the function of brain 5-HT function. Divergent brain 5-HT function has also been observed in Atlantic salmon Salmo salar with different stress coping styles (Thörnqvist et al. 2015).

Stress responsiveness is a heritable trait, and the fact that that it was possible to create strains of rainbow trout with divergent stress coping styles through selective breeding suggests that stress coping style is determined by genetic factors. However, as discussed above social experience sometimes has dramatic effects on the behavioral profile of an individual, for example, the winner/loser effect. To a large extent, behavioral effects of social interaction appear to be mediated by the brain 5-HT system, a neuromodulatory system that is also clearly important for the development of divergent stress coping styles. Thus, to what extent are stress coping styles plastic? Frost et al. (2007) showed that social experience affected boldness in rainbow trout. However, in their study no physiological parameters were included. Ruiz-Gomez et al. (2008) reported a temporary shift in the behavioral traits of HR and LR rainbow trout following transport from UK to Norway. These fish were of the third generation of these selectively bred trout strains. Following the transport, HR fish regained feeding faster when transferred to a novel environment than LR fish. They also became dominant in dyadic fights with size-matched LR fish. This is completely opposite to previous results, and opposite to HR and LR of the same generation not being transported to Norway. However, the hypothalamic–pituitary–interrenal (HPI) axis reactivity did not change, HR fish still showing higher post-stress cortisol than LR fish even after the transport. Moreover, 1 year after the transport behavioral traits shifted and the transported fish again showed behavioral traits similar to non-transported fish.

Stress Coping, Life History Traits, and Serotonin

At least in salmonids, stress coping style appears to be associated with life history traits, such as time of alevin emergence from the nest, growth, and smoltification (Metcalfe and Thorpe 1992; Metcalfe et al. 1995; Einum and Fleming 2000). Female salmonids usually bury their eggs in gravel nests on the stream bottom. After hatching, the larvae remain within the gravel nests and feed on their yolk until they emerge and start defending territories for exogenous feeding. Within the same spawning nest the timing of emergence may vary by several weeks among individuals (Mason and Chapman 1965; Brännäs 1988) and is related to behavioral trait composition. Early emerging fish display proactive traits, such as being more aggressive, bold, and socially dominant (Metcalfe and Thorpe 1992; Metcalfe et al. 1995). In contrast, late emerging fish are generally shy, less aggressive, and socially subordinate (Metcalfe and Thorpe 1992; Metcalfe et al. 1995), traits associated with a reactive stress coping style (Koolhaas et al. 1999). This relationship between timing of emergence and stress coping styles was further supported by Åberg-Andersson et al. (2013) who showed that LR rainbow trout generally emerged earlier than HR fish from artificial spawning nests. However, even though time of emergence is clearly related to several behavioral traits of proactive and reactive coping styles, time of emergence show no relation to HPI axis reactivity (Vaz Serrano et al. 2011; Thörnqvist et al. 2015). In a recent study, Thörnqvist et al. (2015) showed that time of emergence in salmon is also related to forebrain gene expression and hindbrain 5-HT turnover. The 5-HT system appeared more responsive to stress in late emerging than early emerging salmon, late emerging fish showing higher hindbrain 5-HIAA levels than early emerging fish following confinements stress. Moreover, late emerging fish responded to confinement stress by an elevation of forebrain 5-HT1A receptor mRNA levels, an effect not observed in early emerging fish. Thus, overall late emerging salmon, which were also shyer than then early emerging fish, appear to show a more reactive coping style than early emerging fish, even though this divergence in coping style was not reflected in HPI axis reactivity. These results suggest that early and late emergence represent different strategies controlled by genetic factors and reflected in divergent coping styles. However, it could not be excluded that early social experience occurring immediately after emergence is important for shaping stress coping styles of salmon.

Conclusions and Future Perspectives

Clearly the brain 5-HT system plays important roles in fish phenotypic plasticity. In response to social interaction and other stressors there is a rapid increase in brain 5-HT activity. Moreover, socially subordinate fish is characterized by a chronic elevation of brain 5-HT activity. It is becoming increasingly clear that acute and chronic increase in brain 5-HT neurotransmission have very different effects. An acute 5-HT activation, which is seen in both winners and losers of agonistic interactions, could be related to a general arousal effect, whereas the chronic activation observed in subordinate fish is clearly linked to the behavioral inhibition displayed by these individuals. However, our knowledge on the exact mechanisms mediating divergent behavioral effects of 5-HT is still very limited.

The organization of the central 5-HT system appears to be evolutionarily very well conserved. However, in contrast to mammals where 5-HT cell bodies are restricted to the raphe area of the hindbrain, the 5-HT system of the teleost brain is more diverse with 5-HT producing cells also in diencephalic brain regions. These diencephalic 5-HT cells differ from those found in the raphe area, for example, 5-HT synthesis in these cells does not appear to be restricted by Trp availability. The function of these teleost-specific 5-HT expressing cells is still not known.

Divergent stress coping styles has been demonstrated in several teleost species as well as in other vertebrates. The brain 5-HT system is likely to play an important role in mediating the specific behavioral and stress response profiles displayed by animals with divergent stress coping styles, proactive versus reactive. Interestingly, stress coping profiles appears to be related to metabolic rate with proactive animals living a faster life with higher metabolic rate than reactive animals. Control of metabolic rate has been suggested to be an evolutionary old function of the 5-HT system.

Research brain serotonergic functions in teleost fish have so far, to a large extent, focused on similarities to mammals and other vertebrates. Future research should also address differences and specific functions of the 5-HT system of the teleost brain. For instance, what is the role of the teleost 5-HT system in modifying stress coping and how is this related to life history traits like time of sexual maturation, migration, etc. However, again our knowledge on both ecological implications and mechanisms involved is limited.