Updating to 2022 version

Author: phlippe

dl2021_cpu.yml

@@ -0,0 +1,23 @@
+name: dl2021
+channels:
+  - pytorch
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.10.6
+  - pip=22.2.2
+  - cpuonly=2.0
+  - pytorch=1.13.0
+  - torchvision=0.14.0
+  - torchaudio=0.13.0
+  - pip:
+    - pytorch-lightning==1.7.7
+    - tensorboard==2.10.1
+    - tabulate>=0.8.9
+    - tqdm>=4.62.3
+    - pillow>=8.0.1
+    - notebook>=6.4.5
+    - jupyterlab>=3.2.1
+    - matplotlib>=3.4.3
+    - seaborn>=0.11.2
+    - ipywidgets>=7.6.5

dl2021_gpu.yml

@@ -0,0 +1,24 @@
+name: dl2022
+channels:
+  - pytorch
+  - nvidia
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.10.6
+  - pip=22.2.2
+  - pytorch-cuda=11.7
+  - pytorch=1.13.0
+  - torchvision=0.14.0
+  - torchaudio=0.13.0
+  - pip:
+    - pytorch-lightning==1.7.7
+    - tensorboard==2.10.1
+    - tabulate>=0.8.9
+    - tqdm>=4.62.3
+    - pillow>=8.0.1
+    - notebook>=6.4.5
+    - jupyterlab>=3.2.1
+    - matplotlib>=3.4.3
+    - seaborn>=0.11.2
+    - ipywidgets>=7.6.5

dl2022_cpu.yml

@@ -88,14 +88,14 @@
     "module load Anaconda3/2021.05\n",
     "```\n",
     "\n",
-    "The CUDA and cuDNN libraries are already taken care of by installing the cudatoolkits in conda.\n",
+    "Note that there also exist a `2022` module with slightly newer package versions, but it is at the moment not functional. Hence, we stick with the `2021` module. The CUDA and cuDNN libraries are already taken care of by installing the cudatoolkits in conda.\n",
     "\n",
     "### Install the environment\n",
     "\n",
-    "To run the Deep Learning assignments and other code like the notebooks on Lisa, you need to install the [provided environment for Lisa](https://github.com/uvadlc/uvadlc_practicals_2021) (`dl2021_gpu.yml`). You can either download it locally and copy it to your Lisa account via rsync or scp as described before, or simply clone the [practicals github](https://github.com/uvadlc/uvadlc_practicals_2021) on Lisa: \n",
+    "To run the Deep Learning assignments and other code like the notebooks on Lisa, you need to install the [provided environment for Lisa](https://github.com/uvadlc/uvadlc_practicals_2022) (`dl2022_gpu.yml`). You can either download it locally and copy it to your Lisa account via rsync or scp as described before, or simply clone the [practicals github](https://github.com/uvadlc/uvadlc_practicals_2022) on Lisa: \n",
     "\n",
     "```bash\n",
-    "git clone https://github.com/uvadlc/uvadlc_practicals_2021.git\n",
+    "git clone https://github.com/uvadlc/uvadlc_practicals_2022.git\n",
     "```\n",
     "\n",
     "Lisa provides an Anaconda module, which you can load via `module load Anaconda3/2021.05` as mentioned before (remember to load the `2021` module beforehand). We recommend installing the package via a job file since the installation can take 20-30 minutes, and any command on the login node will be killed without warning after 15 minutes.\n",
@@ -118,8 +118,8 @@
     "module load 2021\n",
     "module load Anaconda3/2021.05\n",
     "\n",
-    "cd $HOME/uvadlc_practicals_2021/\n",
-    "conda env create -f dl2021_gpu.yml\n",
+    "cd $HOME/uvadlc_practicals_2022/\n",
+    "conda env create -f dl2022_gpu.yml\n",
     "```\n",
     "\n",
     "You can use e.g. `nano` to do that. If the environment file is not in the cloned repository or you haven't cloned the repo, change the cd statement to the directory where it is stored. Once the file is saved, start the job with the command `sbatch install_environment.job`. The installation process is started on a compute node with a time limit of 4 hours, which should be sufficiently long. Let's look at the next section to understand what we have actually done here with respect to 'job files'.\n",
@@ -129,17 +129,17 @@
     "If the installation via job file does not work, try to install the environment with the following command from the login node after navigating to the directory the environment file is in: \n",
     "\n",
     "```bash\n",
-    "conda env create -f dl2021_gpu.yml\n",
+    "conda env create -f dl2022_gpu.yml\n",
     "```\n",
-    "Note that the jobs on the login node on Lisa are limited to 15 minutes. This is often not enough to install the full environment. If the installation command is killed, you can simply restart it. If you get the error that a package is corrupted, go to `/home/lcur___/.conda/pkgs/` under your home directory and remove the directory of the corrupted package. If you get the error that the environment dl2021 already exists, go to `/home/lcur___/.conda/envs/`, and remove the folder 'dl2021'.\n",
+    "Note that the jobs on the login node on Lisa are limited to 15 minutes. This is often not enough to install the full environment. If the installation command is killed, you can simply restart it. If you get the error that a package is corrupted, go to `/home/lcur___/.conda/pkgs/` under your home directory and remove the directory of the corrupted package. If you get the error that the environment dl2022 already exists, go to `/home/lcur___/.conda/envs/`, and remove the folder 'dl2022'.\n",
     "\n",
     "If you experience issues with the Anaconda module, you can also install Anaconda yourself ([download link](https://docs.anaconda.com/anaconda/install/linux/)) or ask your TA for help.\n",
     "\n",
     "#### Verifying the installation\n",
     "\n",
-    "When the installation process is completed, you can check if the process was successful by activating your environment on the login node via `source activate dl2021` (remember to have loaded the anaconda module beforehand), and starting a python console with executing `python`. It should say `Python 3.9.7 | packaged by conda-forge`. If you see a different python version, you might not have activated the environment correctly. \n",
+    "When the installation process is completed, you can check if the process was successful by activating your environment on the login node via `source activate dl2022` (remember to have loaded the anaconda module beforehand), and starting a python console with executing `python`. It should say `Python 3.10.6 | packaged by conda-forge`. If you see a different python version, you might not have activated the environment correctly. \n",
     "\n",
-    "In the python console, try to import pytorch via `import torch` and check the version: `torch.__version__`. It should say `1.10.0`. Finally, check whether PyTorch can access the GPU: `torch.cuda.is_available()`. Note that in most cases, this will return `False` because most login-nodes on Lisa do not have GPUs. You can login to a GPU node via `ssh lcur___@login-gpu.lisa.surfsara.nl`, and on this node, you should see that the command returns `True`. If that is the case, you should be all set."
+    "In the python console, try to import pytorch via `import torch` and check the version: `torch.__version__`. It should say `1.13.0`. Finally, check whether PyTorch can access the GPU: `torch.cuda.is_available()`. Note that in most cases, this will return `False` because most login-nodes on Lisa do not have GPUs. You can login to a GPU node via `ssh lcur___@login-gpu.lisa.surfsara.nl`, and on this node, you should see that the command returns `True`. If that is the case, you should be all set."
    ]
   },
   {
@@ -173,7 +173,7 @@
     "# Your job starts in the directory where you call sbatch\n",
     "cd $HOME/...\n",
     "# Activate your environment\n",
-    "source activate dl2021\n",
+    "source activate dl2022\n",
     "# Run your code\n",
     "srun python -u ...\n",
     "```\n",
@@ -197,7 +197,7 @@
     "\n",
     "If you work with a lot of data, or a larger dataset, it is advised to copy your data to the `/scratch` directory of the node. Otherwise, the read/write operation might become a bottleneck of your job. To do this, simply use your copy operation of choice (`cp`, `rsync`, ...), and copy the data to the directory `$TMPDIR`. You should add this command to your job file before calling `srun ...`. Remember to point to this data when you are running your code. If you have a dataset that can be downloaded, you can also directly download it to the scratch (can sometimes be faster than copying). In case you also write something on the scratch, you need to copy it back to your home directory before finishing the job. \n",
     "\n",
-    "**Edit Dec. 6, 2021**: Due to internal changes to the filesystem of Lisa, it is required to **use the scratch for any dataset** such as CIFAR10. In PyTorch, the CIFAR10 dataset is structured into multiple large batches (usually 5), and only that batch is loaded which is currently needed. This is why during training, it requires a lot of reading operations on the disk which can slow down your training and constitutes a challenge to the Lisa system when hundreds of students share the same filesystem. Hence, you have to use the scratch for such datasets. For most parts in the assignment, you can do this by specifying `--data_dir $TMPDIR` on the python command in your job file (check if the argument parser has this argument, otherwise you can add it yourself). This will download the dataset to the scratch and only load it from there."
+    "**Edit Dec. 6, 2021**: Due to internal changes to the filesystem of Lisa, it is required to **use the scratch for any dataset** such as CIFAR10. In PyTorch, the CIFAR10 dataset is structured into multiple large batches (usually 5), and only that batch is loaded which is currently needed. This is why during training, it requires a lot of reading operations on the disk which can slow down your training and constitutes a challenge to the Lisa system when hundreds of students share the same filesystem. Hence, you have to use the scratch for such datasets. For most parts in the assignment, you can do this by specifying `--data_dir $TMPDIR` on the python command in your job file (check if the argument parser has this argument, otherwise you can add it yourself). This will download the dataset to the scratch and only load it from there. We recommend using this approach also for future course editions."
    ]
   },
   {
@@ -268,11 +268,15 @@
     "\n",
     "### PyTorch or other packages cannot be imported\n",
     "\n",
-    "If you run a job and see the python error message in the slurm output file that a package is missing although you have installed it in the environment, there are two things to check. Firstly, make sure to not have the environment activated on the login node when submitting the job. This can lead to an error in the anaconda module such that packages are not found on the compute node. Secondly, check that you activate the environment correctly. To verify that the correct python version is used, you can add the command `which python` before your training file. This prints out the path of the python that will be used, in which you should see the anaconda version in the dl2021 environment.\n",
+    "If you run a job and see the python error message in the slurm output file that a package is missing although you have installed it in the environment, there are two things to check. Firstly, make sure to not have the environment activated on the login node when submitting the job. This can lead to an error in the anaconda module such that packages are not found on the compute node. Secondly, check that you activate the environment correctly. To verify that the correct python version is used, you can add the command `which python` before your training file. This prints out the path of the python that will be used, in which you should see the anaconda version in the dl2022 environment.\n",
     "\n",
     "### My job runs very slow\n",
     "\n",
-    "If your job executes your script much slower than you expect, check for two things: (1) have you requested a GPU and are you using it, and (2) are you using the scratch for your dataset? Not using the scratch for your dataset can create a significant communication bottleneck, especially if multiple students do it at the same time. Make sure to download or copy your dataset to the scratch and load it from there."
+    "If your job executes your script much slower than you expect, check for two things: (1) have you requested a GPU and are you using it, and (2) are you using the scratch for your dataset? Not using the scratch for your dataset can create a significant communication bottleneck, especially if multiple students do it at the same time. Make sure to download or copy your dataset to the scratch and load it from there.\n",
+    "\n",
+    "### I am not able to use Lisa at all\n",
+    "\n",
+    "If during the course, there will be major issues with Lisa (e.g. cluster goes into maintenance for a long time, issues with the filesystem, etc.), you can make use of GoogleColab, as we do already for all notebook tutorials here. All assignments do not necessarily require large amount of compute, and can often be comfortably trained on a GPU provided by GoogleColab. For an introduction to GoogleColab, see this [tutorial](https://colab.research.google.com/)."
    ]
   },
   {
@@ -409,7 +413,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.2"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,