Seonglae ChoMultimodal Neuron from OpenAI (2021, Gabriel Goh)
In 2005, a letter published in Nature described human neurons responding to specific people, such as Jennifer Aniston or Halle Berry. The exciting thing was that they did so regardless of whether they were shown photographs, drawings, or even images of the person’s name; The neurons were multimodal. You are looking at the far end of the transformation from metric, visual shapes to conceptual information.
Multimodal Neurons in Artificial Neural Networks
We report the existence of multimodal neurons in artificial neural networks, similar to those found in the human brain.
https://distill.pub/2021/multimodal-neurons/
Multimodal neurons in artificial neural networks
We’ve discovered neurons in CLIP that respond to the same concept whether presented literally, symbolically, or conceptually. This may explain CLIP’s accuracy in classifying surprising visual renditions of concepts, and is also an important step toward understanding the associations and biases that CLIP and similar models learn.
https://openai.com/index/multimodal-neurons/
