Extract errors from shell commands

[lexming]

Related to #4252 and #1796

Next step in revamping the error reporting #4351 is to be able to extract error messages from the output of shell commands executed by EasyBuild and show those in the error reports on console. The goal would be to show the following information on those console reports:

• command of the error
• working directory
• path to log file with complete output
• snippet of the error message

However, extracting the actual error messages is not straightforward because we log (on purpose) stdout and stderr in a single file to keep the context of any error message.

One solution avoiding any pattern matching is to intercept all messages in stderr and prepend some tag to those before merging them in the unified output file. In bash this can be done with process substitution:

#!/bin/bash

function slowecho(){
    echo "$1"
    sleep 0.1
}

function cmd_output(){
    slowecho "Starting command foo..."
    slowecho "Doing step 1"
    slowecho "ERROR: something went wrong" >&2
    slowecho "Doing step 2"
    slowecho "INFO: all went fine"
}

cmd_output
# OUTPUT:
# Starting command foo...
# Doing step 1
# ERROR: something went wrong
# Doing step 2
# INFO: all went fine

cmd_output > >(sed 's/^/\[OUT\] /') 2> >(sed 's/^/[ERR] /' >&2)
# OUTPUT:
# [OUT] Starting command foo...
# [OUT] Doing step 1
# [ERR] ERROR: something went wrong
# [OUT] Doing step 2
# [OUT] INFO: all went fine

The main shortcoming is that stdout and stderr are parsed in different processes, which might result in the final message order to be mangled. Nonetheless, this is only an issue if the command prints multiple messaged to stderr and stdout very fast, which is a rare case.

The question now is (if possible) how to best implement this in Python to not rely on a bash shell.