Merge branch 'site' into pytorch-2.0-subpage-fix

Author: ritaiglesias-96

_get_started/pytorch.md

@@ -197,7 +197,7 @@ out.sum().backward()
 optimizer.step()
 ```
 
-The first time you run the `optimized_model(x)`, it compiles the model.. Hence, it takes longer to run. Subsequent runs are fast.
+The first time you run the `compiled_model(x)`, it compiles the model. Hence, it takes longer to run. Subsequent runs are fast.
 
 ### Modes
 
@@ -269,7 +269,7 @@ This is in early stages of development. Catch the talk on Export Path at the PyT
 
 ### Debugging Issues
 
-A compiled mode is opaque and and hard to debug. You will have questions such as:
+A compiled mode is opaque and hard to debug. You will have questions such as:
 
 - why is my program crashing in compiled mode?
 - is compiled mode as accurate as eager mode?
@@ -464,7 +464,7 @@ After all, we can’t claim we’re created a breadth-first unless **YOUR** mode
     <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip3 install numpy --pre torch torchvision torchaudio --force-reinstall --extra-index-url https://download.pytorch.org/whl/nightly/cpu</code></pre></div></div>
   </li>
 
-  <li><b>Is 2.0 code backwards-compatible with 1.X? </b><br>
+  <li><b>Is 2.0 code backwards-compatible with 1.X?</b><br>
   Yes, using 2.0 will not require you to modify your PyTorch workflows. A single line of code <code class="language-plaintext highlighter-rouge">model = torch.compile(model)</code> can optimize your model to use the 2.0 stack, and smoothly run with the rest of your PyTorch code. This is completely opt-in, and you are not required to use the new compiler.
   </li>
 
@@ -497,7 +497,7 @@ def infer(model, input):
   That said, even with static-shaped workloads, we’re still building Compiled mode and there might be bugs. Disable Compiled mode for parts of your code that are crashing, and raise an <a href="https://github.com/pytorch/pytorch/issues" target="_blank">issue</a> (if it isn’t raised already).
   </li>
 
-  <li><b>What is my code doing differently when running PyTorch 2.0? </b>
+  <li><b>What is my code doing differently when running PyTorch 2.0?</b>
   Out of the box, PyTorch 2.0 is the same as PyTorch 1.x, your models run in eager-mode i.e. every line of Python is executed one after the other. <br>
 
   In 2.0, if you wrap your model in <code>model = torch.compile(model)</code>, your model goes through 3 steps before execution: <br><br>
@@ -524,20 +524,15 @@ def infer(model, input):
     </ul>
   </li>
 
-  <li><b> What compiler backends does 2.0 currently support?</b><br>
-    <p>The default backend the most complete backend is <a href="https://github.com/pytorch/pytorch/tree/master/torch/_inductor" target="_blank">TorchInductor</a>, but TorchDynamo has a growing list of backends that can be found by calling <code class="language-plaintext highlighter-rouge">torchdynamo.list_backends().</code>
+  <li><b> What compiler backends does 2.0 currently support?</b>
+    <p>The default and the most complete backend is <a href="https://github.com/pytorch/pytorch/tree/master/torch/_inductor" target="_blank">TorchInductor</a>, but TorchDynamo has a growing list of backends that can be found by calling <code class="language-plaintext highlighter-rouge">torchdynamo.list_backends().</code>
     </p>
   </li>
 
   <li><b>How does distributed training work with 2.0?</b>
     <p>DDP and FSDP in Compiled mode  can run up to 15% faster than Eager-Mode in FP32 and up to 80% faster in AMP precision. PT2.0 does some extra optimization to ensure DDP’s communication-computation overlap works well with Dynamo’s partial graph creation. Ensure you run DDP with static_graph=False. More details <a href="https://dev-discuss.pytorch.org/t/torchdynamo-update-9-making-ddp-work-with-torchdynamo/860" target="_blank">here</a>.</p>
   </li>
 
-  <li><b> How can I learn more about PT2.0 developments?</b>
-    <p>The most likely reason for performance hits is too many graph breaks. For instance, something innocuous as a print statement in your model’s forward triggers a graph break. We have ways to diagnose these - read more <a href="https://pytorch.org/docs/master/dynamo/faq.html#why-am-i-not-seeing-speedups" target="_blank">here</a>.</p>
-    <p>The most likely reason for performance hits is too many graph breaks. For instance, something innocuous as a print statement in your model’s forward triggers a graph break. We have ways to diagnose these - read more <a href=" https://pytorch.org/docs/master/dynamo/faq.html#why-is-my-code-crashing" target="_blank">here</a>.</p>
-  </li>
-
   <li><b>Help my code is running slower with 2.0’s Compiled Model</b>
     <p>The most likely reason for performance hits is too many graph breaks. For instance, something innocuous as a print statement in your model’s forward triggers a graph break. We have ways to diagnose these - read more <a href="https://pytorch.org/docs/master/dynamo/faq.html#why-am-i-not-seeing-speedups" target="_blank">here</a>.</p>
   </li>

_posts/2022-12-02-Accelerating-Hugging-Face-and-TIMM-models.md

@@ -5,7 +5,7 @@ author: Mark Saroufim
 featured-img: "assets/images/pytorch-2.0-feature-img.png"
 ---
 
-`torch.compile()` makes it easy to experiment with different compiler backends to make PyTorch code faster with a single line decorator `torch.compile()`. It works either directly over an nn.Module as a drop-in replacement for torch.jit.script() but without requiring you to make any source code changes. We expect this one line code change to provide you with between 30%-2x training time speedups on the vast majority of models that you’re already running.
+`torch.compile()` makes it easy to experiment with different compiler backends to make PyTorch code faster with a single line decorator `torch.compile()`. It works either directly over an nn.Module as a drop-in replacement for `torch.jit.script()` but without requiring you to make any source code changes. We expect this one line code change to provide you with between 30%-2x training time speedups on the vast majority of models that you’re already running.
 
 ```python
 
@@ -82,7 +82,7 @@ import torch
        a = torch.sin(x).cuda()
        b = torch.sin(y).cuda()
        return a + b
-   new_fn = torch.compile("inductor")(fn)
+   new_fn = torch.compile(fn, backend="inductor")
    input_tensor = torch.randn(10000).to(device="cuda:0")
    a = new_fn()
 ```
@@ -133,7 +133,7 @@ As a next step let’s try a real model like resnet50 from the PyTorch hub.
 ```python
 import torch
    model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet18', pretrained=True)
-   opt_model = torch.compile("inductor",model)
+   opt_model = torch.compile(model, backend="inductor")
    model(torch.randn(1,3,64,64))
 
 ```
@@ -173,7 +173,7 @@ Similarly let’s try out a TIMM example
 import timm
    import torch
    model = timm.create_model('resnext101_32x8d', pretrained=True, num_classes=2)
-   opt_model = torch.compile(model, "inductor")
+   opt_model = torch.compile(model, backend="inductor")
    opt_model(torch.randn(64,3,7,7))
 ```
 

docs/master/_dynamo.html

@@ -237,7 +237,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_images/RReLU.png

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/index.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/__config__.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/_dynamo.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 
@@ -585,7 +585,7 @@ <h1>Source code for torch._dynamo.eval_frame</h1><div class="highlight"><pre>
         <span class="k">yield</span>
         <span class="k">return</span>
     <span class="k">with</span> <span class="n">patch</span><span class="p">(</span><span class="s2">&quot;torch._dynamo.config.dynamic_shapes&quot;</span><span class="p">,</span> <span class="kc">True</span><span class="p">),</span> <span class="n">patch</span><span class="p">(</span>
-        <span class="s2">&quot;functorch._src.config.use_dynamic_shapes&quot;</span><span class="p">,</span> <span class="kc">True</span>
+        <span class="s2">&quot;torch._functorch.config.use_dynamic_shapes&quot;</span><span class="p">,</span> <span class="kc">True</span>
     <span class="p">):</span>
         <span class="k">yield</span>
 
@@ -1219,6 +1219,10 @@ <h1>Source code for torch._dynamo.eval_frame</h1><div class="highlight"><pre>
                 <span class="n">opt</span><span class="o">.</span><span class="n">_cuda_graph_capture_health_check</span>
             <span class="p">)</span>
             <span class="n">opt</span><span class="o">.</span><span class="n">zero_grad</span> <span class="o">=</span> <span class="n">disable</span><span class="p">(</span><span class="n">opt</span><span class="o">.</span><span class="n">zero_grad</span><span class="p">)</span>
+
+            <span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">opt</span><span class="p">,</span> <span class="s2">&quot;_init_group&quot;</span><span class="p">):</span>
+                <span class="n">opt</span><span class="o">.</span><span class="n">_init_group</span> <span class="o">=</span> <span class="n">disable</span><span class="p">(</span><span class="n">opt</span><span class="o">.</span><span class="n">_init_group</span><span class="p">)</span>
+
             <span class="c1"># disable any currently set hooks</span>
             <span class="c1"># Note: we only want to disable the profiling hook</span>
             <span class="c1"># which is the *last* hook applied, we want to keep the no_grad hook</span>

docs/master/_modules/torch/_dynamo/eval_frame.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/_jit_internal.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/_lobpcg.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/_lowrank.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/_tensor.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/_tensor_str.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 
@@ -815,6 +815,8 @@ <h1>Source code for torch._utils</h1><div class="highlight"><pre>
     <span class="k">return</span> <span class="n">param</span>
 
 
+<span class="c1"># TODO(kshitij12345): Support serializing nn.Parameter with Python Attributes.</span>
+<span class="c1"># NOTE: We are just defining it here now for future use.</span>
 <span class="k">def</span> <span class="nf">_rebuild_parameter_with_state</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">requires_grad</span><span class="p">,</span> <span class="n">backward_hooks</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span>
     <span class="n">param</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Parameter</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">requires_grad</span><span class="p">)</span>
     <span class="c1"># NB: This line exists only for backwards compatibility; the</span>

docs/master/_modules/torch/_utils.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/_vmap_internals.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/amp/autocast_mode.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/intrinsic/modules/fused.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/intrinsic/qat/modules/conv_fused.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/intrinsic/qat/modules/linear_relu.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/intrinsic/quantized/dynamic/modules/linear_relu.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/intrinsic/quantized/modules/bn_relu.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/intrinsic/quantized/modules/conv_relu.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/intrinsic/quantized/modules/linear_relu.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/qat/dynamic/modules/linear.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/qat/modules/conv.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/qat/modules/linear.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/quantizable/modules/activation.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/quantizable/modules/rnn.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/quantized/dynamic/modules/linear.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/quantized/dynamic/modules/rnn.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>
 
 

docs/master/_modules/torch/ao/nn/quantized/functional.html

@@ -235,7 +235,7 @@
           <div class="pytorch-left-menu-search">
 
     <div class="version">
-      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+gitf628f2e ) &#x25BC</a>
+      <a href='https://pytorch.org/docs/versions.html'>master (1.14.0a0+git4648baa ) &#x25BC</a>
     </div>