That fact is made clear from an exchange between ZDNet and the researchers. We are also not at a point in time when the neural network can construct one representation that combines all the different data types, so that the neural net is learning things in combination. We are not yet at a world where a neural net is trained with no sense whatsoever of the input data types. In that way, the multi-modal aspect of the network still relies on clues about the data, what the team refer to as "small modality-specific input encoders."Īlso: Google unveils 'Pathways', a next-gen AI that can be trained to multitask
Image, speech, and text are all prepared by pre-processing of the data. Mind you, data2vec's very general approach to a single neural net for multiple modalities still has a lot of information about the different data types. Jeff Dean, head of Google's AI efforts, in October teased about "Pathways," calling it a " next generation AI architecture" for multi-modal data processing. In the self-supervised approach, data2vec isn't using those labels it's just trying to reconstruct the network's internal representation of the data.Įven more ambitious efforts lie in the wings. The training of the Perceiver neural network is the more-standard process of producing an output that is the answer to a labeled, supervised task such as ImageNet. For example, last summer, Google's DeepMind unit offered up what it calls "Perceiver," its own multi-modal version of the Transformer. This averaging is different from other recent approaches to building One Network To Crunch All Data. The point is that every data type that goes in becomes the same challenge for the Student network of reconstructing something inside the neural network that the Teacher had composed.
They add, "We generally use the output of the FFN prior to the last residual connection in each block as target," where a "block" is the Transformer equivalent of a neural network layer. Rather, data2vec is grabbing a bunch of neural network layers that are inside the neural network, somewhere in the middle, that represent the data before it is produced as a final output.Īs the researchers write, "One of the main differences of our method other than performing masked prediction, is the use of targets which are based on averaging multiple layers from the teacher network." Specifically, "we regress multiple neural network layer representations instead of just the top layer," so that "data2vec predicts the latent representations of the input data."
Why is my internet so slow? 11 ways to speed up your connection
0 kommentar(er)
