Skip to content

Commit 1536c6e

Browse files
vpjvpj
committed
links
1 parent a7a7a3b commit 1536c6e

File tree

8 files changed

+25
-17
lines changed

8 files changed

+25
-17
lines changed

‎docs/graphs/gat/index.html‎

Lines changed: 3 additions & 3 deletions
Large diffs are not rendered by default.

‎docs/graphs/gatv2/index.html‎

Lines changed: 3 additions & 3 deletions
Large diffs are not rendered by default.

‎docs/index.html‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ <h4>✨ <a href="transformers/index.html">Transformers</a></h4>
7878
<li><a href="transformers/xl/index.html">Transformer XL</a> </li>
7979
<li><a href="transformers/xl/relative_mha.html">Relative multi-headed attention</a> </li>
8080
<li><a href="transformers/rope/index.html">Rotary Positional Embeddings</a> </li>
81+
<li><a href="transformers/retro/index.html">RETRO</a> </li>
8182
<li><a href="transformers/compressive/index.html">Compressive Transformer</a> </li>
8283
<li><a href="transformers/gpt/index.html">GPT Architecture</a> </li>
8384
<li><a href="transformers/glu_variants/simple.html">GLU Variants</a> </li>

‎docs/sitemap.xml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -211,7 +211,7 @@
211211

212212
<url>
213213
<loc>https://nn.labml.ai/experiments/nlp_autoregression.html</loc>
214-
<lastmod>2022-03-06T16:30:00+00:00</lastmod>
214+
<lastmod>2022-03-12T16:30:00+00:00</lastmod>
215215
<priority>1.00</priority>
216216
</url>
217217

@@ -267,7 +267,7 @@
267267

268268
<url>
269269
<loc>https://nn.labml.ai/distillation/small.html</loc>
270-
<lastmod>2022-03-06T16:30:00+00:00</lastmod>
270+
<lastmod>2022-03-12T16:30:00+00:00</lastmod>
271271
<priority>1.00</priority>
272272
</url>
273273

@@ -582,7 +582,7 @@
582582

583583
<url>
584584
<loc>https://nn.labml.ai/transformers/rope/experiment.html</loc>
585-
<lastmod>2022-03-06T16:30:00+00:00</lastmod>
585+
<lastmod>2022-03-12T16:30:00+00:00</lastmod>
586586
<priority>1.00</priority>
587587
</url>
588588

@@ -596,7 +596,7 @@
596596

597597
<url>
598598
<loc>https://nn.labml.ai/transformers/basic/autoregressive_experiment.html</loc>
599-
<lastmod>2022-03-06T16:30:00+00:00</lastmod>
599+
<lastmod>2022-03-12T16:30:00+00:00</lastmod>
600600
<priority>1.00</priority>
601601
</url>
602602

@@ -722,7 +722,7 @@
722722

723723
<url>
724724
<loc>https://nn.labml.ai/transformers/retro/index.html</loc>
725-
<lastmod>2022-03-10T16:30:00+00:00</lastmod>
725+
<lastmod>2022-03-12T16:30:00+00:00</lastmod>
726726
<priority>1.00</priority>
727727
</url>
728728

@@ -883,7 +883,7 @@
883883

884884
<url>
885885
<loc>https://nn.labml.ai/graphs/gat/index.html</loc>
886-
<lastmod>2021-08-19T16:30:00+00:00</lastmod>
886+
<lastmod>2022-03-12T16:30:00+00:00</lastmod>
887887
<priority>1.00</priority>
888888
</url>
889889

@@ -897,7 +897,7 @@
897897

898898
<url>
899899
<loc>https://nn.labml.ai/graphs/gatv2/index.html</loc>
900-
<lastmod>2021-08-19T16:30:00+00:00</lastmod>
900+
<lastmod>2022-03-12T16:30:00+00:00</lastmod>
901901
<priority>1.00</priority>
902902
</url>
903903

‎docs/transformers/index.html‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,8 @@ <h2><a href="xl/index.html">Transformer XL</a></h2>
7878
<p>This implements Transformer XL model using <a href="xl/relative_mha.html">relative multi-head attention</a></p>
7979
<h2><a href="rope/index.html">Rotary Positional Embeddings</a></h2>
8080
<p>This implements Rotary Positional Embeddings (RoPE)</p>
81+
<h2><a href="retro/index.html">RETRO</a></h2>
82+
<p>This implements the Retrieval-Enhanced Transformer (RETRO).</p>
8183
<h2><a href="compressive/index.html">Compressive Transformer</a></h2>
8284
<p>This is an implementation of compressive transformer that extends upon <a href="xl/index.html">Transformer XL</a> by compressing oldest memories to give a longer attention span.</p>
8385
<h2><a href="gpt/index.html">GPT Architecture</a></h2>
@@ -111,10 +113,10 @@ <h2><a href="hour_glass/index.html">Hourglass</a></h2>
111113

112114
</div>
113115
<div class='code'>
114-
<div class="highlight"><pre><span class="lineno">106</span><span></span><span class="kn">from</span> <span class="nn">.configs</span> <span class="kn">import</span> <span class="n">TransformerConfigs</span>
115-
<span class="lineno">107</span><span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">TransformerLayer</span><span class="p">,</span> <span class="n">Encoder</span><span class="p">,</span> <span class="n">Decoder</span><span class="p">,</span> <span class="n">Generator</span><span class="p">,</span> <span class="n">EncoderDecoder</span>
116-
<span class="lineno">108</span><span class="kn">from</span> <span class="nn">.mha</span> <span class="kn">import</span> <span class="n">MultiHeadAttention</span>
117-
<span class="lineno">109</span><span class="kn">from</span> <span class="nn">labml_nn.transformers.xl.relative_mha</span> <span class="kn">import</span> <span class="n">RelativeMultiHeadAttention</span></pre></div>
116+
<div class="highlight"><pre><span class="lineno">109</span><span></span><span class="kn">from</span> <span class="nn">.configs</span> <span class="kn">import</span> <span class="n">TransformerConfigs</span>
117+
<span class="lineno">110</span><span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">TransformerLayer</span><span class="p">,</span> <span class="n">Encoder</span><span class="p">,</span> <span class="n">Decoder</span><span class="p">,</span> <span class="n">Generator</span><span class="p">,</span> <span class="n">EncoderDecoder</span>
118+
<span class="lineno">111</span><span class="kn">from</span> <span class="nn">.mha</span> <span class="kn">import</span> <span class="n">MultiHeadAttention</span>
119+
<span class="lineno">112</span><span class="kn">from</span> <span class="nn">labml_nn.transformers.xl.relative_mha</span> <span class="kn">import</span> <span class="n">RelativeMultiHeadAttention</span></pre></div>
118120
</div>
119121
</div>
120122
<div class='footer'>

‎labml_nn/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
* [Transformer XL](transformers/xl/index.html)
2424
* [Relative multi-headed attention](transformers/xl/relative_mha.html)
2525
* [Rotary Positional Embeddings](transformers/rope/index.html)
26+
* [RETRO](transformers/retro/index.html)
2627
* [Compressive Transformer](transformers/compressive/index.html)
2728
* [GPT Architecture](transformers/gpt/index.html)
2829
* [GLU Variants](transformers/glu_variants/simple.html)

‎labml_nn/transformers/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,9 @@
2525
## [Rotary Positional Embeddings](rope/index.html)
2626
This implements Rotary Positional Embeddings (RoPE)
2727
28+
## [RETRO](retro/index.html)
29+
This implements the Retrieval-Enhanced Transformer (RETRO).
30+
2831
## [Compressive Transformer](compressive/index.html)
2932
3033
This is an implementation of compressive transformer

‎readme.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ implementations almost weekly.
2525
* [Transformer XL](https://nn.labml.ai/transformers/xl/index.html)
2626
* [Relative multi-headed attention](https://nn.labml.ai/transformers/xl/relative_mha.html)
2727
* [Rotary Positional Embeddings](https://nn.labml.ai/transformers/rope/index.html)
28+
* [RETRO](https://nn.labml.ai/transformers/retro/index.html)
2829
* [Compressive Transformer](https://nn.labml.ai/transformers/compressive/index.html)
2930
* [GPT Architecture](https://nn.labml.ai/transformers/gpt/index.html)
3031
* [GLU Variants](https://nn.labml.ai/transformers/glu_variants/simple.html)

0 commit comments

Comments
 (0)