Skip to content

Commit 22a4302

Browse files
RohitDhankarRohitDhankar
committed
Attention is all you need
1 parent 802c184 commit 22a4302

5 files changed

+96
-2
lines changed

Diff for: ‎readme/Arxiv_papers_README.md

+3-1
Original file line numberDiff line numberDiff line change
@@ -168,4 +168,6 @@ These q-distributions are normally parameterized for each individual data point
168168
169169
However, variational autoencoders use a neural network as an amortized approach to jointly optimize across data points. This neural network takes as input the data points themselves, and outputs parameters for the variational distribution. As it maps from a known input space to the low-dimensional latent space, it is called the encoder.
170170
171-
```
171+
```
172+
173+

Diff for: ‎readme/VIT___2101.01169.pdf

7.3 MB
Binary file not shown.

Diff for: ‎readme/Vision_Transformers__AlexeyDosovitskiy_2010.11929.pdf

3.57 MB
Binary file not shown.

Diff for: ‎readme/arxiv_paper_readme.md

+64
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
2+
> LSTM is dead. Long Live Transformers!
3+
- YOUTUBE VIDEO -- https://www.youtube.com/watch?v=S27pHKBEp30&t=568s
4+
- Leo Dirac (@leopd) talks about how LSTM models for Natural Language Processing (NLP) have been practically replaced by transformer-based models. Basic background on NLP, and a brief history of supervised learning techniques on documents, from bag of words, through vanilla RNNs and LSTM. Then there's a technical deep dive into how Transformers work with multi-headed self-attention, and positional encoding. Includes sample code for applying these ideas to real-world projects.
5+
6+
- @8:50 -- LSTM - Transfer Learning not Ok
7+
- [@10:30](https://www.youtube.com/watch?v=S27pHKBEp30&t=630s)- Attention is all you need -- Multi Head Attention Mechanism --
8+
-
9+
10+
#
11+
12+
<br/>
13+
14+
#
15+
16+
Published as a conference paper at ICLR 2021
17+
18+
> AN IMAGE IS WORTH 16X16 WORDS:TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE -- Alexey Dosovitskiy∗,†, Lucas Beyer∗, Alexander Kolesnikov∗, Dirk Weissenborn∗, Xiaohua Zhai∗, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby∗,† ∗equal technical contribution, †equal advising Google Research, Brain Team {adosovitskiy, neilhoulsby}@google.com
19+
20+
- ABSTRACT - While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited ...
21+
22+
- https://arxiv.org/pdf/2010.11929.pdf
23+
- Short Name -- Vision_Transformers__AlexeyDosovitskiy_2010.11929.pdf
24+
25+
#
26+
27+
<br/>
28+
29+
#
30+
31+
> Transformers in Vision: A Survey
32+
- https://arxiv.org/pdf/2101.01169.pdf
33+
34+
35+
#
36+
37+
<br/>
38+
39+
#
40+
41+
42+
> A Survey of Transformers - TIANYANG LIN, YUXIN WANG, XIANGYANG LIU, and XIPENG QIU∗, School of Computer
43+
Science, Fudan University, China and Shanghai Key Laboratory of Intelligent Information Processing, Fudan
44+
University, China
45+
46+
- ABSTRACT -- Transformers have achieved great success in many artificial intelligence fields, such as natural language
47+
processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from
48+
academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers)
49+
have been proposed, however, a systematic and comprehensive literature review on these Transformer variants
50+
is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly
51+
introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the
52+
various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally,
53+
we outline some potential directions for future research.
54+
55+
- https://arxiv.org/pdf/2106.04554.pdf
56+
57+
- Transformer Attention Modules -- Query-Key-Value
58+
59+
#
60+
61+
<br/>
62+
63+
#
64+

Diff for: ‎readme/todo_list_readme.md

+29-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11

22
-[pytorch_unet](https://github.com/RohitDhankar/PyTorch_1/blob/master/src/unet_pytorch_2.py)
3+
-[pytorch_unet_1](https://cs231n.github.io/convolutional-networks/)
4+
-[input_and_output_volume](https://cs231n.github.io/convolutional-networks/)
5+
-[receptive_field_OR_context](https://cs231n.github.io/convolutional-networks/)
6+
-[VISUALIZE_Network](https://github.com/microsoft/tensorwatch)
7+
-[VISUALIZE_receptive_field_OR_context](https://github.com/shelfwise/receptivefield)
8+
9+
-[VISUALIZE_NEURALNET_LAYER_OUTPUTS]
10+
-[XAI_Explainable_AI_HuggingFaceModels](https://jacobgil.github.io/pytorch-gradcam-book/HuggingFace.html)
11+
-[Class_Activation_Maps](https://jacobgil.github.io/pytorch-gradcam-book/Class%20Activation%20Maps%20for%20Object%20Detection%20With%20Faster%20RCNN.html)
12+
-[Gradient_Class_Activation_Maps](https://jacobgil.github.io/pytorch-gradcam-book/Class%20Activation%20Maps%20for%20Object%20Detection%20With%20Faster%20RCNN.html)
13+
14+
- https://jacobgil.github.io/pytorch-gradcam-book/Class%20Activation%20Maps%20for%20Object%20Detection%20With%20Faster%20RCNN.html
315

416

517
- [CycleGAN-pix2pix--pytorch-CycleGAN-and-pix2pix](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix)
@@ -117,7 +129,23 @@ with open("logreg_iris.onnx", "wb") as f:
117129

118130
```
119131

132+
#
133+
134+
120135
- [GEO_GIS_Intro_Init_UCLA](https://www.youtube.com/watch?v=gi4UdFsayoM)
121136
- [GEO_GIS__Census_Data_Analysis_Mapping](https://www.youtube.com/watch?v=rrGw6ct-Cbw)
122137
- [GEO_GIS__Spatial_Statistics_with_Python](https://www.youtube.com/watch?v=B_LHPRVEOvs)
123-
-
138+
-
139+
140+
#
141+
142+
- [Forecasting_TimeSeries_LinearModels]
143+
- [Forecasting_TimeSeries_SARIMAX]
144+
-
145+
146+
#
147+
148+
- [Forecasting_TimeSeries_DeepLearning]
149+
- [Corrformer_PyTorch](https://github.com/thuml/Corrformer)
150+
- [Anomaly_Transformer_Time_Series_Anomaly_Detection](https://github.com/thuml/Anomaly-Transformer)
151+
-

0 commit comments

Comments
 (0)