rope links

Author: vpj

labml_nn/__init__.py

@@ -22,6 +22,7 @@
 * [Transformer building blocks](transformers/models.html)
 * [Transformer XL](transformers/xl/index.html)
     * [Relative multi-headed attention](transformers/xl/relative_mha.html)
+* [Rotary Positional Embeddings](transformers/rope/index.html)
 * [Compressive Transformer](transformers/compressive/index.html)
 * [GPT Architecture](transformers/gpt/index.html)
 * [GLU Variants](transformers/glu_variants/simple.html)

labml_nn/transformers/__init__.py

@@ -22,6 +22,9 @@
 This implements Transformer XL model using
 [relative multi-head attention](xl/relative_mha.html)
 
+## [Rotary Positional Embeddings](rope/index.html)
+This implements Rotary Positional Embeddings (RoPE)
+
 ## [Compressive Transformer](compressive/index.html)
 
 This is an implementation of compressive transformer

readme.md

@@ -24,6 +24,7 @@ implementations almost weekly.
 * [Transformer building blocks](https://nn.labml.ai/transformers/models.html) 
 * [Transformer XL](https://nn.labml.ai/transformers/xl/index.html)
     * [Relative multi-headed attention](https://nn.labml.ai/transformers/xl/relative_mha.html)
+* [Rotary Positional Embeddings](https://nn.labml.ai/transformers/rope/index.html)
 * [Compressive Transformer](https://nn.labml.ai/transformers/compressive/index.html)
 * [GPT Architecture](https://nn.labml.ai/transformers/gpt/index.html)
 * [GLU Variants](https://nn.labml.ai/transformers/glu_variants/simple.html)