Can Drosophila melanogaster tell who's who?

Drosophila melanogaster are known to live in a social but cryptic world of touch and odours, but the extent to which they can perceive and integrate static visual information is a hotly debated topic. Some researchers fixate on the limited resolution of D. melanogaster's optics, others on their seemingly identical appearance; yet there is evidence of individual recognition and surprising visual learning in flies. Here, we apply machine learning and show that individual D. melanogaster are visually distinct. We also use the striking similarity of Drosophila's visual system to current convolutional neural networks to theoretically investigate D. melanogaster's capacity for visual understanding. We find that, despite their limited optical resolution, D. melanogaster's neuronal architecture has the capability to extract and encode a rich feature set that allows flies to re-identify individual conspecifics with surprising accuracy. These experiments provide a proof of principle that Drosophila inhabit a much more complex visual world than previously appreciated.

Introduction

There is an increasing body of evidence that Drosophila melanogaster live in a surprisingly rich and complex world that includes group behaviour [1], communal learning [2], and recognition during aggressive behaviours [3]. This repertoire of social behaviour has often been assumed to be independent of visual recognition, as Drosophila’s compound eye was thought to have insufficient visual acuity to play a serious role. With approximately 850 lens units (ommatidia), each capturing a single point in space, Drosophila’s compound eye resolution is certainly low. Additionally, the level of detail, traditionally determined by the inter-ommatidial angle, renders anything but movement or regular patterns seemingly impossible to discern (Fig 1B).

Fig 1

Theoretical visual acuity of Drosophila melanogaster.

Image of Drosophila melanogaster represented after various theoretical bottlenecks. A: Image of a female D. melanogaster re-sized through a 32×32 bottleneck. B: The same image, but adjusted using AcuityView [4] for a viewing distance of 3 body lengths using the inter-ommatidial angle of 4.8° [5]. C: The same image and distance, but using a conservative estimate of the effective acuity determined by Juusola et al. [6] of approximately 1.5°.

However, recent physiological experiments [6] reveal that D. melanogaster can respond to details as fine as 1.16° as long as they are presented (to a tethered fly) at specific speeds. These speeds happen to coincide with D. melanogaster’s natural saccadic gait [7], which strongly suggests that naturally behaving D. melanogaster have much finer resolution than the inter-ommatidial angle of 4.8° [5] would imply. This hyper-acuity is found at the photoreceptor level (due to rhabdomere movement changing the angle of light reception), allowing most of the capacity of the visual network to be devoted to information processing. At this effective hyper-acuity, and at socially relevant distances, the number of ommatidia and not the inter-ommatidial angle becomes the limiting factor (Fig 1). This acuity potentially puts them in the same visual league (albeit with lower resolution) as Apis mellifera [8], which has been able to, among other visual feats, identify individual human faces [9].

This spatio-temporal coding and increased visual acuity potentially explain recent studies which have shown that D. melanogaster can not only resolve other flies, but can also decode social meaning using vision (e.g. female choice of male phenotypes [10] and exposure to parasitoids [11]). Combined, these results open up the possibility that Drosophila uses vision to a much greater extent in object recognition, perhaps even using it to discriminate between species or sex (supplementing other olfactory cues that are known to convey this information [12]).

Even with D. melanogaster’s photoreceptor hyper-acuity [6], the image that is received is only around 29×29 units (or pixels; Fig 1). We wanted to know if there is enough absolute information contained in this low-resolution image to identify individuals from each other. One approach is to task Deep Convolutional Networks (DCNs) to differentiate individual D. melanogaster, as DCNs are engineered to learn, extract, and use any useful features found in the images. If there is enough individual-level variation for highly engineered DCNs, we would want to investigate the possibility that D. melanogaster also take this low resolution image and extract meaningful information out of it. Should individual flies prove visually unique and D. melanogaster’s visual network have enough capacity, vision could potentially play a role in identifying beyond species or sex, perhaps aiding in determining familiar or non-familiar conspecifics in social situations [3].

How a fly’s visual system could extract meaning out of low resolution images is suggested by the highly structured and layered organization of Drosophila’s visual system (Fig 2C). At the input, the ommatidia are packed one by one, but their individually-tuned photoreceptors are arranged spatially to essentially convolve a 6-unit filter across the receptive field. The output of this photoreceptor filter is, in turn, the input for downstream medulla neurons that connect to several ‘columns’ of photoreceptor outputs. This filter-convolution, coupled with the use of the output from one filter as a ‘feature map’ for another layer is a hallmark of the engineered architectures of DCNs that dominate computer vision today (one such DCN is illustrated in Fig 2A). Just as DCNs can take low level image representations and encode them into a semantic representation, D. melanogaster’s visual system seems well-suited to similarly uncover semantic meaning in images.

Fig 2

Our fly-eye merges engineered and biological architectures.

Schematics of a ‘standard’ convolutional network, our fly-eye model, and a simplified visual connectome of Drosophila. A: Architecture of Zeiler and Fergus [13], receiving the original 181×181 pixel image of an individual Drosophila melanogaster. B: Our fly-eye model, receiving a 29×29 down-scaled image of an individual Drosophila, and showing connections between feature maps. The initial three feature maps are a custom 6-pixel convolutional filter (‘R1-R6’; black pathway) and two 1×1 convolutional filters (‘R7’ and ‘R8’; red pathway). All other convolutions are locally connected filters. See S1 Table for complete connectivity map. C: A simplified map of the fly visual circuit receiving the same down-scaled image of another D. melanogaster. The connections among the neurons implemented in our model are displayed, illustrating the connections and links within and between layers (adapted from [14] and [15]). See S2 Table for performance of these models on a traditional image-classification dataset.

In this work, we investigate whether D. melanogaster could theoretically categorize and recognize its complex visual environment. To determine how much absolute underlying visual variation is available for social behaviour in D. melanogaster, we examine the ability of humans and human-inspired deep convolutional models to re-identify individual D. melanogaster across multiple days. To see whether D. melanogaster is capable of using this individual-level visual variation between flies, we investigate a model of Drosophila’s visual system in a conspecific re-identification paradigm. This study builds upon the behavioural results of conspecific information and physiological evidence of hyper-acute vision, and provides a proof of principle to dispel the oft-touted argument that D. melanogaster’s visual ability is limited to low-level object- and pattern-detection. Here, we present evidence that D. melanogaster may likely see and live in a much richer social environment than has been appreciated.

Materials and methods

Simplified Drosophila model eye

We implemented a virtual fly visual system using standard deep learning libraries (Keras). Our implementation uses approximately 25,000 artificial neurons, whereas Drosophila have approximately 60,000 neurons in each visual hemisphere [16]. We purposefully did not model neurons that are structurally suggestive to respond to movement, and therefore we were specifically limited to ‘modular’ neurons (with 1 neuron/column) throughout the medulla. The connections between neuronal types were extracted from published connectomes [17]. We imposed artificial hierarchy on our model eliminating self-connections between neuron ‘subtypes’ (i.e. no connections between L1 and L1, or L1 and L2), and while we allowed initial layers to feed into multiple downstream layers, we eliminated ‘upstream’ connections. The final lobula-like artificial neurons were modelled after Wu et al. [15], where the layers were ordered according to their axon penetration deeper into the system. Our ability to model Drosophila’s visual system is further limited to the connectivity, ignoring the sign (excitatory or inhibitory), as well as the neurons’ intrinsic membrane properties. The ability to create more biologically realistic simulations will increase once these properties are discovered and integrated into the connectome. The model is illustrated in Fig 2B, beside the biological inspiration (Fig 2C). S1 Table depicts a complete connection map and hierarchy, and S2 Table shows comparative performance of this model on a traditional image-classification dataset. Additional details are provided in S1 Methods.

Fly data acquisition

D. melanogaster were housed at 25°C on a 12-12 Light-Dark cycle. 10 males and 10 females were collected 1-4 hours after eclosion and housed separately. On the third day, post-eclosion flies were individually mouth pipetted into a circular acrylic arena (60mm diameter, 2mm high). These flies were illuminated with standard overhead LED bulbs and filmed in grayscale with a GRAS-20S4M for 15 minutes, with 16 frames/second. This was repeated for three consecutive days in total, which resulted in 14,400 × 3 images per fly. Each filming session was within 2 hours of ZT 8. Three independent datasets were acquired of 20 flies each.

Fly data processing

Each video was tracked using CTRAX [18], and the tracked position and orientation was used to localize the flies and orient the images so that the flies were always centred and facing ‘up’. These images therefore contained all viewpoints of the flies during their behaviour in the arena: dorsal, ventral, and lateral. The training set for each biological replicate was constructed from days 1 and 2 equally, and consisted of the first 75% of the each fly’s recording (12,240 frames). The validation set was the final 15% of the recording (2,160 frames). The test dataset was the entire recording on day 3. Images were standardized by subtracting the mean and dividing by the standard deviation of the training set. For the ResNet18 [19] and Zeiler and Fergus [13] models, the original 181×181 image was either: (1) reduced to 33×33, centre-cropped to 29×29, and then re-sized to 224×224; or (2) re-sized to 256×256 and centre-cropped to 224×224 (effectively using the centre 158×158 pixels).

Human performance

A GUI program was written in MATLAB that presented a human observer with 3 viewpoints of an exemplar fly: dorsal, ventral, and sideways obtained from the first two days of filming. The observer was then asked to choose among 20 images (of the 20 flies) obtained from day 3, of which one belong to the exemplar (S3 and S4 Figs). Note that this is a compare/match setup rather than a learn/recall. The images were randomly re-sized through a 29×29 bottleneck.

Results

In this work, we wanted to see whether various architectures, both biologically rooted and not, could detect visual differences between flies across days (a notably non-human task). We acquired three rounds of images, each round having 10 males and 10 females, filmed for 3 consecutive days. Knowing that age/experience may slightly affect the morphology of flies, we experimented with training these networks on days 1 and 2, and testing their ability to re-identify the flies on day 3. We evaluated the efficient and high-capacity model (ResNet18 [19]), a model that rivals human representation (Zeiler and Fergus [13]), our fly-eye model, and human performance. These results are summarized in Table 1.

Table 1

Performance on D. melanogaster re-identification.

Model Name	Resolution1 (pixels)	Accuracy (F1 Score)2
ResNet18 [19]	158×158	0.9426 ± 0.0358
Zeiler and Fergus [13]	158×158	0.9373 ± 0.0365
Human Performance	158×158	0.1309
Zeiler and Fergus [13]	29×29	0.8549 ± 0.0778
ResNet18 [19]	29×29	0.8357 ± 0.0909
Our fly-eye	29×29	0.7548 ± 0.1141
Our fly-eyew/ random zoom3	29×29	0.5486 ± 0.1316
Human Performance	29×29	0.0829
Random Chance		0.05

Mean and standard deviation shown (n = 3 independent datasets)

1 For Zeiler and Fergus and ResNet18, the “resolution” was a bottleneck (see Methods).

2 An example of high precision / low recall can be seen in S4 Table as ID 10 is assigned to the right fly 94% of the time, but fly 10 is only correctly identified 37% of the time.

3 The zoom was applied randomly without preserving aspect ratio (see S2 Fig).

As a benchmark, we applied a ResNet18 architecture (see S1 Fig). This was the highest performing network architecture, achieving an F1-score of 0.94 (with high resolution over the three datasets). While the average performance is good, we note certain problematic individual flies that become difficult to accurately re-identify across days (e.g. in biological replicate 2, fly 10’s accuracy on day 3 was 37% with equal confusion between two other flies S4 Table). Forcing the images through a bottleneck (which ensures that the information content is similar to the reduced-resolution fly-eye model) decreased the F1-score by 0.11 for ResNet18. The Zeiler and Fergus [13] architecture was the most robust in terms of the bottleneck, only decreasing the F1-score by 0.08, but did not achieve as high a performance as ResNet18.

The fly-eye model achieves a relatively high F1 score of 0.75, which is not that much lower than the highly engineered ResNet18 (at low resolution). To eliminate the fly-eye model’s ability to measure absolute size and shape and force relative feature extraction, we randomly re-sized the images (both training and testing) by up to 25% without preserving proportion (see S2 Fig for examples). Our virtual fly-eye outperformed humans, even without absolute size measures, achieving an F1-score of 0.55 re-identifying conspecifics. We also note that the fly-eye model almost never mistakes a male for a female (F1-score exceeding 0.99 when the re-identification IDs were collapsed by sex; S5–S7 Tables).

To establish a human performance baseline, we used volunteers to attempt the re-identification of flies (S3 and S4 Figs). This is an especially challenging task given the range of motion and viewpoints that an individual fly has in an unconstrained arena. As this task falls outside normal visual object recognition, we restricted volunteers to be very experienced “fly-pushing” scientists. Human performance was predictably low, but varied, with an average F1-score of 0.11 (0.08 when the images were reduced to 29×29 pixels, and 0.13 when given the full resolution 181×181 images).

Discussion

Our results suggest that Drosophila have an innate capacity to extract semantic meaning from their visual surrounding, and even if we are still discovering how Drosophila could encode its world, we should not disregard its visual understanding.

The idea of recognizing broad semantic information decoded from a low resolution image is not new (for example, the success of CNNs on the 32×32 pixel CIFAR10 dataset and experiments in other downsampled datasets [20]). We also note that our fly-eye model performs relatively poorly on one such classification task (F1 Score of 0.54 on CIFAR10; see S2 Table), which contains a wide range of scale and positional variance of the objects. One possible explanation for the inability to cope with variability in scale and size is that, unlike other architectures, D. melanogaster’s visual system maintains the input dimensions (the columnar medulla neurons). DCN’s dimensionality reduction through ‘tricks’ like pooling layers and strided convolutions convey a larger position-invariance to low level feature detectors. Without them, our fly-eye model only does well when the distance to the object is fixed. It is tempting to posit that an individual can therefore have an innate and/or experience-dependent distance at which visual information is preferentially understood, and that this could be one of the determinants for social spacing and interaction distances [21, 22].

One prediction that arises from this model and its apparent ability to encode more than simple “looming” and “movement” visual cues is that the highest (furthest from input) level feature maps may correspond to semantically rich meaning in the visual system. These lobula neurons should then encode (among other information) complex object recognition categories and stimulating them should produce more than simple object-avoidance behaviours in Drosophila. While some lobula columnar neurons (like LC11) seem specialized for high-acuity small object motion detection [23], others appear to encode more complex information. These other LC neurons (like LC17), when stimulated, seem to provoke social-context dependent behaviours [15].

We are aware of other investigations using DCNs to classify insect species [24]. However, the most relevant study, which involves organism re-identification, only does so over short (<1 min) time frames (IDTracker2.0 [25]). Prior to this study, DCNs have only been effective on images that are temporally very close [25]. We observe a non-general loss of accuracy for specific flies, with some flies having an accuracy lower than 40% (S4 Table). This ability to re-identify flies across days opens experimental possibilities, especially considering that this performance was achieved with static images (16fps yields around a thousand estimates of ID per minute, allowing high confidence in the parsimonious correct identification). This is in contrast to the human ability to re-identify flies, which at low resolutions is barely better than chance.

Clearly, all models can learn to re-identify flies to some extent, underscoring the individual-level variation in D. melanogaster. Re-identifying flies is in fact easier for DCNs than CIFAR10 (at least with centred images of flies acquired at the same distance). Even the model that rivals, in some sense, the representational performance of humans [26] does ten times better than humans. Why humans can’t tell one fly from another is not clear. Regardless of whether it was evolutionarily beneficial to discriminate individual flies, humans do have incredible pattern detection abilities. It may simply be a lack of experience (although we attempted to address this by only using experienced Drosophila researchers as volunteers) or a more cryptic pattern-recognition ‘blind-spot’ of humans. In either case, these findings should spur new experiments to further understand the mechanisms of human vision and experience and how they fail in this case.

Machine learning practitioners are constantly pushing for deep networks to use more biologically-inspired design choices and training algorithms [27-29]. As they become more biologically realistic, neurobiologists could use these models to generate hypotheses in how information is processed in the visual system. We think D. melanogaster is well-suited to link these two fields to continue unravelling evolution’s solution to visual processing. This new area offers a simple, genetically- and experimentally-tractable organism to peer into the workings of the visual system, which will no doubt uncover not just how Drosophila, but all of us, see the world.

Conclusion

These results help explain recent, traditionally controversial, findings that Drosophila melanogaster can resolve relatively detailed visual semantic meaning (e.g. female choice of males [2] and parasitoids exposure [11]). We show here that each D. melanogaster has visually distinguishable features that persist across days. This fact, combined with their hyper acuity [6] and theoretical capacity of their visual network, provides a proof of principle against the traditional belief that Drosophila only see blurry motion. In fact, Drosophila could have the ability to see and discriminate a surprisingly diverse visual world, in some cases, possibly better than we can.

Additional details concerning the creation of the model (connectome), and the details/results of testing the model on a traditional image-classification dataset.

(PDF)

Click here for additional data file.

ResNet18 architecture.

Residual Networks with 18 ‘layers’ [19] were constructed as depicted, using the improved block scheme proposed by [19] (top left inset).

(PDF)

Click here for additional data file.

Examples of fly images.

On the left are examples of the various viewpoints of the fly. The images indicated were run through the random non-proportion-preserving zoom to generate three examples of each.

(PDF)

Click here for additional data file.

Human query: High resolution example.

Participants were asked to re-identify the exemplar fly (top left), which exhibited three viewpoints: ventral, dorsal, and angled. They could request new images of said fly as many times as needed. The exemplar images were manually sorted into viewpoint categories and then randomly selected from Days 1 and 2. The other fly images (queries) were selected from Day 3.

(PNG)

Click here for additional data file.

Human query: Low resolution example.

Similar to the high-resolution experiment described above (S3 Fig), except all pictures were down sampled to 29×29.

(PNG)

Click here for additional data file.

Model fly-eye connectome.

The table indicates the filter size for the locally-connected layers. No self connections were allowed (grey), or connections with ‘higher’ layers (blue). See S1 Methods for additional details.

(PDF)

Click here for additional data file.

Results of a simple vision task (CIFAR10).

See S1 Methods for additional information about data processing and results.

(PDF)

Click here for additional data file.

Confusion matrix for ResNet18 biological replicate 1.

Flies are ordered by sex (Purple = male, Yellow = Female), then by ascending size. Predictions are colour coded and weighted by percentage (correct predictions are indicated in orange, incorrect predictions are coloured cyan).

(PDF)

Click here for additional data file.

Confusion matrix for ResNet18 biological replicate 2.

Note that fly 10 has a recall accuracy of 37%, and is almost equally confused for flies 2 and 7. Flies are ordered by sex (Purple = male, Yellow = Female), then by ascending size. Predictions are colour coded and weighted by percentage (correct predictions are indicated in orange, incorrect predictions are coloured cyan).

(PDF)

Click here for additional data file.

Confusion matrix for ResNet18 biological replicate 3.

Flies are ordered by sex (Purple = male, Yellow = Female), then by ascending size. Predictions are colour coded and weighted by percentage (correct predictions are indicated in orange, incorrect predictions are coloured cyan).

(PDF)

Click here for additional data file.

Confusion matrix for fly-eye model biological replicate 1.

Flies are ordered by sex (Purple = male, Yellow = Female), then by ascending size. Predictions are colour coded and weighted by percentage (correct predictions are indicated in orange, incorrect predictions are coloured cyan).

(PDF)

Click here for additional data file.

Confusion matrix for fly-eye model biological replicate 2.

Flies are ordered by sex (Purple = male, Yellow = Female), then by ascending size. Predictions are colour coded and weighted by percentage (correct predictions are indicated in orange, incorrect predictions are coloured cyan).

(PDF)

Click here for additional data file.

Confusion matrix for fly-eye model biological replicate 3.

Flies are ordered by sex (Purple = male, Yellow = Female), then by ascending size. Predictions are colour coded and weighted by percentage (correct predictions are indicated in orange, incorrect predictions are coloured cyan).

(PDF)

Click here for additional data file.