Merge pull request #175 from hvelab/issue_48_build_without_eb

Adding documentation on how to build and run on top of EESSI without EB

Author: ocaisa

docs/using_eessi/building_on_eessi.md

@@ -108,10 +108,50 @@ $ module av netcdf4-python/1.6.5-foss-2023b
    netcdf4-python/1.6.5-foss-2023b
 ```
 
-
 This means you'll _always_ need to load the `EESSI-extend` module if you want to use these modules (also, and particularly when you want to use them in a job script).
 
-## Manually building software op top of EESSI
-Building software on top of EESSI would require your linker to use the same system-dependencies as the software in EESSI does. In other words: it requires you to link against libraries from the compatibility layer, instead of from your host OS.
+## Manually building software op top of EESSI (without EasyBuild)
+
+!!! warning
+
+    We are working on a module file that should make building on top of EESSI (without using EasyBuild)
+    more straightforward, particularly when using `Autotools` or `CMake`. Right now, it is a little convoluted
+    and requires you to have a decent grasp of
+    * What a runtime dynamic linker (`ld-linux*.so`) is and does
+    * How to influence the behaviour of the runtime linker with `LD_LIBRARY_PATH`
+    * The difference between `LIBRARY_PATH` and `LD_LIBRARY_PATH`
+    
+    As such, this documentation is intended for "experts" in the runtime linker and it's behaviour,
+    and most cases are untested. Any feedback on this topic is highly appreciated. 
+    
+Building and running software on top of EESSI without EasyBuild is not straightforward and requires some considerations to take care of. 
+
+It is expected that you will have loaded all of your required dependencies as modules from the EESSI environment. Since EESSI sets
+`LIBRARY_PATH` for all of the modules and the `GCC` compiler is configured to use the compat layer, there should be no additional configuration
+required to execute a standard build process. On the other hand, EESSI does not set `LD_LIBRARY_PATH` so, _at runtime_, the executable will need help
+finding the libraries that it needs to actually execute. The easiest way to circumvent this requirement is by setting the environment variable `LD_RUN_PATH`
+during compile time as well. With `LD_RUN_PATH` set, the program will be able to tell the dynamic linker to search in those paths when the program is being
+executed. 
+
+EESSI uses a [compatibility layer](../compatibility_layer.md) to ensure that it takes as few libraries from the host as possible. The safest way to make sure
+all libraries will point to the required locations in the compatibility layer (and do not leak in from the host operating system) is starting an EESSI prefix
+shell before building. To do this: 
+
+* First of all, load the environment by starting an EESSI shell as described [here](https://www.eessi.io/docs/using_eessi/setting_up_environment). 
+* Load all dependencies you need to build your software. You must use at least a toolchain from EESSI to compile it (`foss` is a good option as it will also
+  include MPI with OpenMPI and math libraries via FlexiBLAS/FFTW). 
+* Set manually `LD_RUN_PATH` to resolve libraries at runtime. `LIBRARY_PATH` should contain all the paths we need, and we also need to include the path to
+  `libstdc++` from our GCC installation to avoid picking up the one from the host:
+  ```sh
+      export LD_RUN_PATH=$LIBRARY_PATH:$EBROOTGCCcore/lib64
+  ```
+* Compile and make sure the library resolution points to the EESSI stack. For this, `ldd` from compatibility layer and **not** `/usr/bin/ldd` should be used
+  when checking the binary.
+
+* Run! 
+
+
+!!! Note RPATH should never point to a compatibility layer directory, only to software layer ones, as all resolving is done via the runtime linker (`ld-linux*.so`)  that is shipped with EESSI, which automatically searches these locations.
+
+The biggest downside of this approach is that your executable becomes bound to the architecture you linked your libraries for, i.e., if you add to your executable RPATH a `libhdf5.so`compiled for `intel_avx512`, you will not be able to run that binary on a machine with a different architecture. If this is an issue for you, you should look into how EESSI itself organises the location of binaries and perhaps leverage the relevant environment variables (e.g., `EESSI_SOFTWARE_SUBDIR`).
 
-While we plan to support this in the future, manually building on top of EESSI is currently not supported yet in a trivial way.