Syntax tests broke between 11.0.0-alpha.6 and beta.1

[cknitt]

Alpha.6:

✅ multi printer api tests
✅ Parser make: initializes parser and checking offsets
✅ Parser handles LF correct
✅ Parser handles CRLF correct
✅ utf8 tests
~/projects/cca/rescript-compiler/jscomp/syntax ~/projects/cca/rescript-compiler
✅ No unstaged tests difference.
~/projects/cca/rescript-compiler
Checking code formatting...
✅ Code formatting ok.
make reanalyze
reanalyze.exe -set-exit-code -all-cmt _build/default/jscomp -suppress jscomp/syntax/testrunner -exclude-paths jscomp/outcome_printer,jscomp/ounit_tests,jscomp/ml,jscomp/js_parser,jscomp/frontend,jscomp/ext,jscomp/depends,jscomp/core,jscomp/common,jscomp/cmij,jscomp/bsb_helper,jscomp/bsb
  Analysis reported 0 issues
bash ./scripts/testok.sh
✅ All Tests Passed

Beta.1:

✅ multi printer api tests
Unknown error while trying to print outcome tree.
We don't display all the outcome type errors; try adding the new case to the `try` pattern match.
Fatal error: exception Not_found
~/projects/cca/rescript-compiler/jscomp/syntax ~/projects/cca/rescript-compiler
✅ No unstaged tests difference.
~/projects/cca/rescript-compiler
Checking code formatting...
✅ Code formatting ok.
make reanalyze
reanalyze.exe -set-exit-code -all-cmt _build/default/jscomp -suppress jscomp/syntax/testrunner -exclude-paths jscomp/outcome_printer,jscomp/ounit_tests,jscomp/ml,jscomp/js_parser,jscomp/frontend,jscomp/ext,jscomp/depends,jscomp/core,jscomp/common,jscomp/cmij,jscomp/bsb_helper,jscomp/bsb
  Analysis reported 0 issues
bash ./scripts/testok.sh
✅ All Tests Passed

I see three problems here:

1. What broke with the outcome printer?
2. Why do we still get "All Tests Passed" in spite of the exception?
3. We are not running the syntax tests in CI unless something in the syntax folder changed. We could run them always, but only on our Mac Mini M1, this should not affect total build times then.

[cristianoc]

How about just fix the tests.

[cknitt]

It seems the problem was introduced in #6249.

[cristianoc]

To me locally they run fine on master.
Is the script itself for running tests locally broken?

[cristianoc]

OK I see there's an exception.
OK sure the test script should catch that exception.

[cknitt]

I played a bit and it seemed the problem occurs in

https://github.com/rescript-lang/rescript-compiler/blob/4a3dff4026235e5a8f15116a3c8acd3844a9bd55/jscomp/syntax/testrunner/res_test.ml#L69

when processing

https://github.com/rescript-lang/rescript-compiler/blob/4a3dff4026235e5a8f15116a3c8acd3844a9bd55/jscomp/syntax/tests/oprint/oprint.res#L124

(and when commenting that out there are some other lines further down oprint.res that trigger it, too).

No idea yet why, or how this is related to #6249.

[cristianoc]

So it fails here (looked at stack trace):

        let lid:Longident.t = Ldot (Ldot (Lident "Js", "Internal"), name) in
        let (path, desc) = Env.lookup_value lid env in

[cristianoc]

The name it can't find is Js.Internal.opaque, which is used by uncurried.
I guess that's not available in the test runner.

[cristianoc]

standard_library:/Users/cristianocalcagno/GitHub/rescript-compiler/_build/install/default/bin/../lib/ocaml

[cristianoc]

let standard_library =
  let ( // ) = Filename.concat in
  Filename.dirname Sys.executable_name
  // Filename.parent_dir_name // "lib" // "ocaml"

[cristianoc]

So that's what's going on. The standard library is not found as it's looking for it in some non-existing path.

[cristianoc]

This does the job, needs fixing for Windows path, but that should be it:

module OutcomePrinterTests = struct
  let signatureToOutcome structure =
    Lazy.force Res_outcome_printer.setup;

    Clflags.include_dirs := "lib/ocaml" :: !Clflags.include_dirs; (* Added this line *)
    Res_compmisc.init_path ();