Attention is all you need

[RohitDhankar]

LSTM is dead. Long Live Transformers!

• YOUTUBE VIDEO -- https://www.youtube.com/watch?v=S27pHKBEp30&t=568s

• Leo Dirac (@leopd) talks about how LSTM models for Natural Language Processing (NLP) have been practically replaced by transformer-based models. Basic background on NLP, and a brief history of supervised learning techniques on documents, from bag of words, through vanilla RNNs and LSTM. Then there's a technical deep dive into how Transformers work with multi-headed self-attention, and positional encoding. Includes sample code for applying these ideas to real-world projects.

• @8:50 -- LSTM - Transfer Learning not Ok

• @10:30- Attention is all you need -- Multi Head Attention Mechanism --

Published as a conference paper at ICLR 2021

AN IMAGE IS WORTH 16X16 WORDS:TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE -- Alexey Dosovitskiy∗,†, Lucas Beyer∗, Alexander Kolesnikov∗, Dirk Weissenborn∗, Xiaohua Zhai∗, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby∗,† ∗equal technical contribution, †equal advising Google Research, Brain Team {adosovitskiy, neilhoulsby}@google.com

• ABSTRACT - While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited ...

• https://arxiv.org/pdf/2010.11929.pdf

• Short Name -- Vision_Transformers__AlexeyDosovitskiy_2010.11929.pdf

Transformers in Vision: A Survey

• https://arxiv.org/pdf/2101.01169.pdf

A Survey of Transformers - TIANYANG LIN, YUXIN WANG, XIANGYANG LIU, and XIPENG QIU∗, School of Computer
Science, Fudan University, China and Shanghai Key Laboratory of Intelligent Information Processing, Fudan
University, China

• ABSTRACT -- Transformers have achieved great success in many artificial intelligence fields, such as natural language
  processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from
  academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers)
  have been proposed, however, a systematic and comprehensive literature review on these Transformer variants
  is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly
  introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the
  various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally,
  we outline some potential directions for future research.

• https://arxiv.org/pdf/2106.04554.pdf

• Transformer Attention Modules -- Query-Key-Value