The story of how visual speed measurement was discovered from fruit flies — A 10th anniversary celebration
Sometimes mysteries lie in plain sight, concealed by familiarity. Take driving down a straight road for instance. You’re cruising with your favorites beats on, while the scenery flows by. To the sides, everything is as fast as it should be. Looking straight ahead though, anything barely anything moves. And the thing that blows my mind is that, when you zoom in, everything looks much slower. Like natural slow motion.
So, which one is the right speed? The fast one or the slow one?
Well… Do not try to see speed, that’s impossible. Instead, only try to realize the truth… there is no speed. Then you’ll see that it is not speed that you see, only the way that you see.
The distorted Matrixian quote above intends to convey the fact that with our eyes we can’t actually see speed the way we usually measure it. When wanting to know how fast our car is going, how quickly we’re walking, or the maximum speed of a cheetah, we usually intend linear speed: 88 mph, 5 km/h, or 100 km/h.
What we see with our eyes is angular speed. Even if something is moving on a line, its image is projected on each of our eyes as the changing of an angle. That is necessarily so because we always see from a single point in space. Our eyes, our cameras, our telescopes, all stare out into the vast universe from a single, tiny point.
The everyday mystery of different apparent speeds has thus a simple explanation. The world around us and the way we move through it however make for a strong illusion. Straight buildings, roads, objects, even tree trunks,… all support the impression that we can see straight lines.
Actually reconstructing the linear geometry of the World around us is however a complex undertaking. It requires multiple points of view or different sensors (like LIDAR) and sophisticated processing.
…Or at least that is what is commonly believed. The humble fruit fly doesn’t agree and calculates linear speed using its tiny brain and even tinier visual resolution (10’000 neurons and 30x30 pixels).
Drosophilae, this their scientific name, are those small insects that fly annoyingly around bananas and other fruit in summer. They are often studied because of their characteristics, especially in genetics. Researching how they fly in a holodeck (a wind tunnel capable of virtual reality — a metaverse for flies) has shown that they’re able to measure their linear speed and how they do it. It is a fascinating story for biology, engineering, and computer science.
The fly’s method for measuring its own linear speed using angular speeds from its eyes has 3 parts: first, a combination of an assumption engrained into neural structures, then computation facilitated by structure, and last, a simple mechanism for prediction.
The fundamental assumption engrained in the fly’s brain is that it flies in straight, level lines. Known as saccadic behavior, stretches of straight flight are interrupted by abrupt quick turns. This means that flies most of the time see what we see when driving down a straight road: a set of angular speeds, slowest exactly in the the direction we’re headed towards.
However, flies don’t need to turn their head to see that to the sides the scenery flows by very rapidly. Their eyes are semi-spherical, allowing them to see most of the visual field at once. This structural feature provides a significant advantage when making comparisons between different areas. Instead of looking at one spot, memorizing, turning the head, looking at another part, and then making the comparison, flies can compute directly, at each instant.
Being able to compare is great, but what is there to compare from different parts of the visual field? Given that they fly mostly in straight lines, they can compare whether the signals in front are consistent with what they see on the sides. And even better: given that what is in front is likely to eventually reach the side, flies can predict what the signal will be and by when. Every angular speed measured in front provides a set of predicted angular speeds in different parts of the eye following a mathematical formula:
Intuitively, the process is similar to what Galileo observed in 1603, using a sphere rolling down an incline equipped with bells that ring at its passage. If the bells are placed at increasing distances matching the acceleration from gravity, the result will chime at constant intervals.
Increase the slope without moving the bells and the intervals will be shorter, albeit remain equal one to another. The time between chimes is thus always proportional to gravity, implying that it is possible to measure a linear acceleration using only time. Usually you’d need a clock and a rule or another way to measure distance. Not if the “sensors” are placed at the distances predicted by physical equations.
Now, this has an interesting opposite implication: if everything sounds the same, any difference is more easily detected. If multiple spheres run on the same slope, they all should chime at the same intervals. And if one does not, then there is a difference in acceleration. A difference, you have guessed it, simple to calculate because it is proportional to the others.
What does this tell us about biological computation principles? It tells us that if neural structures mimic the mathematical foundations of reality, the computations can become much easier and unlock independence from parameters which would be too hard to actually measure.
In the case of the fly, the neural structures of the brain are placed —similar to Galileo’s bells— at the proper intervals to be able to measure a difference in time that is proportional to its linear flight speed. 1000s and 1000s of flights in the fly metaverse provided evidence supporting this conclusion. The implications are two-fold: (1) fruit flies are able to measure their own linear speed and (2) they are able to create 3D maps of their immediate surroundings. Indeed, any measured speed signals generates predictions for the rest of the visual field and thus if there’s something that doesn’t fit the prediction, its measurement and comparison with the prediction corresponds to a value proportional the distance to that object.
The simplicity of the computational principle and its engineering unorthodoxy are among the reasons why we had founded VISSEE, at a time when automakers would ask us why we wanted to put a camera on a car with a fisheye lens and an FPGA.