Fix playground build infra w/ ReScript 11

CONTRIBUTING.md

@@ -196,8 +196,6 @@ This is usually the file you want to create to test certain compile behavior wit
 
 ## Contribute to the ReScript Playground Bundle
 
-> Note: These instructions are designed for building the 4.06 based version of ReScript (ReScript v6).
-
 The "Playground bundle" is a JS version of the ReScript compiler; including all necessary dependency files (stdlib / belt etc). It is useful for building tools where you want to compile and execute arbitrary ReScript code in the browser.
 
 The ReScript source code is compiled with a tool called [JSOO (js_of_ocaml)](https://ocsigen.org/js_of_ocaml/4.0.0/manual/overview), which uses OCaml bytecode to compile to JavaScript and is part of the bigger OCaml ecosystem.
@@ -210,99 +208,64 @@ opam install js_of_ocaml.4.0.0
 
 ### Building the Bundle
 
-The entry point of the JSOO bundle is located in `jscomp/main/jsoo_playground_main.ml`, the code for packing the compiler into a single compiler file is located in `jscomp/snapshot.ninja`, and the script for running JSOO can be found in `scripts/repl.js`. A full clean build can be done like this:
-
-```
-# We create a target directory for storing the bundle / stdlib files
-mkdir playground && mkdir playground/stdlib
-
-# We build the ReScript source code and also the bytecode for the JSOO entrypoint
-node scripts/ninja.js config && node scripts/ninja.js build
-
-# Now we run the repl.js script which will create all the required artifacts in the `./playground` directory
-node scripts/repl.js
-```
-
-In case you want to build the project with our default third party packages (like `@rescript/react`), prepare the `playground-bundling` project and then run `repl.js` with `BUILD_THIRD_PARTY` enabled:
+The entry point of the JSOO bundle is located in `jscomp/jsoo/jsoo_playground_main.ml`, the compiler and its relevant runtime cmij files can be built via make:
 
-```
-# Prepare the `playground-bundling` project to allow building of the third party cmij packages
-npm link
-cd packages/playground-bundling
-npm install
-npm link rescript
-
-BUILD_THIRD_PARTY=true node scripts/repl.js
+```sh
+make playground
+make playground-cmijs
 ```
 
-_Troubleshooting: if ninja build step failed with `Error: cannot find file '+runtime.js'`, make sure `ocamlfind` is installed with `opam install ocamlfind`._
+Note that building the cmijs is based on the dependencies defined in `packages/playground-bundling/package.json`. In case you want to build some different version of e.g. `@rescript/react` or just want to add a new package, change the definition within the `package.json` file and run `make playground-cmijs` again.
 
 After a successful compilation, you will find following files in your project:
 
 - `playground/compiler.js` -> This is the ReScript compiler, which binds the ReScript API to the `window` object.
-- `playground/stdlib/*.js` -> All the ReScript runtime files.
 - `playground/packages` -> Contains third party deps with cmij.js files (as defined in `packages/playground-bundling/bsconfig.json`)
+- `playground/compilerCmij.js` -> The compiler base cmij containing all the relevant core modules (`Js`, `Belt`, `Pervasives`, etc.)
 
-You can now use the `compiler.js` file either directly by using a `<script src="/path/to/compiler.js"/>` inside a html file, use a browser bundler infrastructure to optimize it, or you can even use it with `nodejs`:
+You can now use the `compiler.js` file either directly by using a `<script src="/path/to/compiler.js"/>` and `<script src="/path/to/packages/compilerCmij.js"/>` inside a html file, use a browser bundler infrastructure to optimize it, or use `nodejs` to run it on a command line:
 
 ```
 $ node
 > require("./compiler.js");
+> require("./packages/compilerCmij.js")
 > let compiler = rescript_compiler.make()
 > let result = compiler.rescript.compile(`Js.log(Sys.ocaml_version)`);
 > eval(result.js_code);
 4.06.2+BS
 ```
 
-You can also run `node playground/playground_test.js` for a quick sanity check to see if all the build artifacts are working together correctly.
-
-### Playground JS bundle API
+### Testing the Playground bundle
 
-As soon as the bundle is loaded, you will get access to the functions exposed in [`jsoo_playground_main.ml`](jscomp/main/jsoo_playground_main.ml). Best way to check out the API is by inspecting a compiler instance it either in node, or in the browser:
-
-```
-$ node
-require('./compiler.js')
-
-> let compiler = rescript_compiler.make()
-> console.log(compiler)
-```
+Run `node playground/playground_test.js` for a quick sanity check to see if all the build artifacts are working together correctly. When releasing the playground bundle, the test will always be executed before publishing to catch regressions.
 
 ### Working on the Playground JS API
 
 Whenever you are modifying any files in the ReScript compiler, or in the `jsoo_playground_main.ml` file, you'll need to rebuild the source and recreate the JS bundle.
 
-```sh
-node scripts/ninja.js config && node scripts/ninja.js build
-node scripts/repl.js
 ```
+make playground
 
-**.cmj files in the Web**
-
-A `.cmj` file contains compile information and JS package information of ReScript build artifacts (your `.res / .ml` modules) and are generated on build (`scripts/ninja.js build`).
-
-A `.cmi` file is an [OCaml originated file extension](https://waleedkhan.name/blog/ocaml-file-extensions/) and contains all interface information of a certain module without any implementation.
-
-In this repo, these files usually sit right next to each compiled `.ml` / `.res` file. The structure of a `.cmj` file is defined in [js_cmj_format.ml](jscomp/core/js_cmj_format.ml). You can run a tool called `./jscomp/bin/cmjdump.exe [some-file.cmj]` to inspect the contents of given `.cmj` file.
-
-`.cmj` files are required to compile modules (this includes modules like RescriptReact). ReScript includes a subset of modules by default, which can be found in `jscomp/stdlib-406` and `jscomp/others`. You can also find those modules listed in the JSOO call in `scripts/repl.js`. As you probably noticed, the generated `playground` files are all plain `.js`, so how are the `cmj` / `cmi` files embedded?
-
-JSOO offers an `build-fs` subcommand that takes a list of `.cmi` and `.cmj` files and creates a `cmij.js` file that can be loaded by the JS runtime **after** the `compiler.js` bundle has been loaded (either via a `require()` call in Node, or via `<link/>` directive in an HTML file). Since we are shipping our playground with third party modules like `RescriptReact`, we created a utility directory `packages/playground-bundling` that comes with a utility script to do the `cmij.js` file creation for us. Check out `packages/playground-bundling/scripts/generate_cmijs.js` for details.
+# optionally run your test / arbitrary node script to verify your changes
+node playground/playground_test.js
+```
 
 ### Publishing the Playground Bundle on our KeyCDN
 
 > Note: If you want to publish from your local machine, make sure to set the `KEYCDN_USER` and `KEYCDN_PASSWORD` environment variables accordingly (credentials currently managed by @ryyppy). Our CI servers / GH Action servers are already pre-configured with the right env variable values.
 
 Our `compiler.js` and third-party packages bundles are hosted on [KeyCDN](https://www.keycdn.com) and uploaded via FTPS.
 
-After a successful bundle build, run our upload script to publish the build artifacts to our server:
+The full release can be executed with the following make script:
 
 ```
-playground/upload_bundle.sh
+make playground-release
 ```
 
 The script will automatically detect the ReScript version from the `compiler.js` bundle and automatically create the correct directory structure on our CDN ftp server.
 
+Note that there's currently still a manual step involved on [rescript-lang.org](https://rescript-lang.org) to make the uploaded playground version publicly available.
+
 ## Contribute to the API Reference
 
 The API reference is generated from doc comments in the source code. [Here](https://github.com/rescript-lang/rescript-compiler/blob/99650/jscomp/others/js_re.mli#L146-L161)'s a good example.

Makefile

@@ -52,6 +52,21 @@ lib: build node_modules/.bin/semver
 artifacts: lib
 	./scripts/makeArtifactList.js
 
+# Builds the core playground bundle (without the relevant cmijs files for the runtime)
+playground:
+	dune build --profile browser
+	cp ./_build/default/jscomp/jsoo/jsoo_playground_main.bc.js playground/compiler.js
+
+# Creates all the relevant core and third party cmij files to side-load together with the playground bundle
+playground-cmijs: artifacts
+	node packages/playground-bundling/scripts/generate_cmijs.js
+
+# Builds the playground, runs some e2e tests and releases the playground to the
+# CDN (requires KEYCDN_USER and KEYCDN_PASSWORD set in the env variables)
+playground-release: playground playground-cmijs
+	node playground/playground_test.js
+	sh playground/upload_bundle.sh
+
 format:
 	dune build @fmt --auto-promote
 
@@ -69,4 +84,4 @@ clean-all: clean clean-gentype
 
 .DEFAULT_GOAL := build
 
-.PHONY: build watch ninja bench dce test test-syntax test-syntax-roundtrip test-gentype test-all lib artifacts format checkformat clean-gentype clean clean-all
+.PHONY: build watch ninja bench dce test test-syntax test-syntax-roundtrip test-gentype test-all lib playground playground-cmijs playground-release artifacts format checkformat clean-gentype clean clean-all

jscomp/jsoo/dune

@@ -1,10 +1,10 @@
 ; Don't build the JS compiler by default as it slows down CI considerably.
 
 (executables
- (names jsoo_main jsoo_playground_main)
+ (names jsoo_playground_main)
  (modes js)
  (enabled_if
   (= %{profile} browser))
  (flags
   (:standard -w +a-4-9-40-42-44-45))
- (libraries core ml super_errors))
+ (libraries core syntax ml js_of_ocaml))

jscomp/jsoo/jsoo_playground_main.ml

@@ -44,12 +44,14 @@
  * and use the proper interfaces as stated by the apiVersion.
  *
  * -----------------------------
- * Version History:
- * v2: Remove refmt support (removes compiler.reason apis)
+ * Version History: * v2: Remove refmt support (removes compiler.reason apis)
+ * v3: Switched to Uncurried mode by default (requires third party packages
+ to be built with uncurried: true in bsconfig.json). Also added
+ `config.uncurried` to the BundleConfig.
  * *)
-let apiVersion = "2"
+let apiVersion = "3"
 
-module Js = Jsoo_common.Js
+module Js = Js_of_ocaml.Js
 
 let export (field : string) v =
   Js.Unsafe.set (Js.Unsafe.global) field v
@@ -73,12 +75,18 @@ module BundleConfig = struct
     mutable module_system: Js_packages_info.module_system;
     mutable filename: string option;
     mutable warn_flags: string;
+
+    (* This one can't be mutated since we only provide
+       third-party packages that were compiled for uncurried
+       mode *)
+    uncurried: bool;
   }
 
   let make () = {
     module_system=Js_packages_info.NodeJS;
     filename=None;
     warn_flags=Bsc_warnings.defaults_w;
+    uncurried=(!Config.uncurried = Uncurried);
   }
 
 
@@ -191,9 +199,10 @@ end
 (* One time setup for all relevant modules *)
 let () =
   Bs_conditional_initial.setup_env ();
+  (* From now on the default setting will be uncurried mode *)
+  Config.uncurried := Uncurried;
   Clflags.binary_annotations := false;
-  Misc.Color.setup (Some Always);
-  Lazy.force Super_main.setup;
+  Clflags.color := Some Always;
   Lazy.force Res_outcome_printer.setup
 
 let error_of_exn e =
@@ -230,15 +239,15 @@ module ResDriver = struct
     Res_parser.make ~mode src filename
 
   (* get full super error message *)
-  let diagnosticToString ~src (d: Res_diagnostics.t) =
+  let diagnosticToString ~(src: string) (d: Res_diagnostics.t) =
     let startPos = Res_diagnostics.getStartPos(d) in
     let endPos = Res_diagnostics.getEndPos(d) in
     let msg = Res_diagnostics.explain(d) in
     let loc = {loc_start = startPos; Location.loc_end=endPos; loc_ghost=false} in
     let err = { Location.loc; msg; sub=[]; if_highlight=""} in
-    Res_diagnostics_printing_utils.Super_location.super_error_reporter
+    Location.default_error_reporter
+      ~src:(Some src)
       Format.str_formatter
-      src
       err;
     Format.flush_str_formatter ()
 
@@ -317,11 +326,13 @@ module Compile = struct
     Buffer.reset warning_buffer;
     str
 
-  let super_warning_printer loc ppf w =
+  (* We need to overload the original warning printer to capture the warnings
+     as an array *)
+  let playground_warning_printer loc ppf w =
     match Warnings.report w with
       | `Inactive -> ()
       | `Active { Warnings. number; is_error; } ->
-        Super_location.super_warning_printer loc ppf w;
+        Location.default_warning_printer loc ppf w;
         let open LocWarnInfo in
         let fullMsg = flush_warning_buffer () in
         let shortMsg = Warnings.message w in
@@ -336,7 +347,7 @@ module Compile = struct
 
   let () =
     Location.formatter_for_warnings := warning_ppf;
-    Location.warning_printer := super_warning_printer
+    Location.warning_printer := playground_warning_printer
 
   let handle_err e =
     (match error_of_exn e with
@@ -450,7 +461,7 @@ module Compile = struct
     List.iter Iter.iter_structure_item structure.str_items;
     Js.array (!acc |> Array.of_list)
 
-  let implementation ~(config: BundleConfig.t) ~lang str  : Js.Unsafe.obj =
+  let implementation ~(config: BundleConfig.t) ~lang str =
     let {BundleConfig.module_system; warn_flags} = config in
     try
       reset_compiler ();
@@ -469,7 +480,8 @@ module Compile = struct
       let env = Res_compmisc.initial_env () in (* Question ?? *)
       (* let finalenv = ref Env.empty in *)
       let types_signature = ref [] in
-      Js_config.jsx_version := Some Js_config.Jsx_v3; (* default *)
+      Js_config.jsx_version := Some Js_config.Jsx_v4; (* default *)
+      Js_config.jsx_mode := Js_config.Automatic; (* default *)
       let ast = impl (str) in
       let ast = Ppx_entry.rewrite_implementation ast in
       let typed_tree =
@@ -660,6 +672,7 @@ module Export = struct
                              );
                              "warn_flags",
                              inject @@ (Js.string config.warn_flags);
+                             "uncurried", inject @@ (Js.bool config.uncurried);
                            |]))
           );
       |])

packages/playground-bundling/bsconfig.json

@@ -1,13 +1,14 @@
 {
   "name": "playground",
   "version": "0.1.0",
-  "bs-dependencies": ["@rescript/react"],
+  "bs-dependencies": ["@rescript/react", "@rescript/core"],
   "package-specs": {
     "module": "es6",
     "in-source": false
   },
   "sources": {
     "dir": "src",
     "subdirs": true
-  }
+  },
+  "uncurried": true
 }

packages/playground-bundling/package.json

@@ -11,6 +11,7 @@
   "author": "",
   "license": "ISC",
   "dependencies": {
-    "@rescript/react": "^0.10.2"
+    "@rescript/core": "^0.3.0",
+    "@rescript/react": "^0.11.0"
   }
 }

packages/playground-bundling/scripts/generate_cmijs.js

@@ -5,16 +5,12 @@
  * project. Or in other words: You need to build cmij files with the same
  * rescript version as the compiler bundle.
  *
- * This script extracts all cmi / cmj files of all dependencies listed in the
- * project root's bsconfig.json, creates cmij.js files for each library and
- * puts them in the compiler playground directory.
+ * This script extracts all cmi / cmj files of the rescript/lib/ocaml and all
+ * dependencies listed in the project root's bsconfig.json, creates cmij.js
+ * files for each library and puts them in the compiler playground directory.
  *
  * The cmij files are representing the marshaled dependencies that can be used with the ReScript
  * playground bundle.
- *
- * This script is intended to be called by the `repl.js` script, but it can be independently run
- * by either running it within the compiler repo, or by providing a proper `PLAYGROUND_DIR` environment
- * flag pointing to the target folder the artifacts will be created.
  */
 
 const child_process = require("child_process");
@@ -23,13 +19,20 @@ const path = require("path");
 
 const bsconfig = require("../bsconfig.json");
 
-const PLAYGROUND_DIR =
-  process.env.PLAYGROUND ||
-  path.join(__dirname, "..", "..", "..", "playground");
+const RESCRIPT_COMPILER_ROOT_DIR = path.join(__dirname, "..", "..", "..");
+const PLAYGROUND_DIR = path.join(RESCRIPT_COMPILER_ROOT_DIR, "playground");
 
+// The playground-bundling root dir
 const PROJECT_ROOT_DIR = path.join(__dirname, "..");
+
+// Final target output directory where all the cmijs will be stored
 const PACKAGES_DIR = path.join(PLAYGROUND_DIR, "packages");
 
+// Making sure this directory exists, since it's not checked in to git
+if (!fs.existsSync(PACKAGES_DIR)) {
+  fs.mkdirSync(PACKAGES_DIR, { recursive: true });
+}
+
 const config = {
   cwd: PROJECT_ROOT_DIR,
   encoding: "utf8",
@@ -43,37 +46,67 @@ function e(cmd) {
   console.log(`<<<<<<`);
 }
 
+e(`npm install`);
+e(`npm link ${RESCRIPT_COMPILER_ROOT_DIR}`);
+e(`npx rescript clean`);
+e(`npx rescript`);
+
 const packages = bsconfig["bs-dependencies"];
 
-packages.forEach(function installLib(package) {
-  const libOcamlFolder = path.join(
+// We need to build the compiler's builtin modules as a separate cmij.
+// Otherwise we can't use them for compilation within the playground.
+function buildCompilerCmij() {
+  const rescriptLibOcamlFolder = path.join(
     PROJECT_ROOT_DIR,
     "node_modules",
-    package,
+    "rescript",
     "lib",
     "ocaml"
   );
-  const libEs6Folder = path.join(
-    PROJECT_ROOT_DIR,
-    "node_modules",
-    package,
-    "lib",
-    "es6"
-  );
-  const outputFolder = path.join(PACKAGES_DIR, package);
 
-  const cmijFile = path.join(outputFolder, `cmij.js`);
+  const outputFolder = path.join(PACKAGES_DIR, "compiler-builtins");
 
-  if (!fs.existsSync(PLAYGROUND_DIR)) {
-    console.error(`PLAYGROUND_DIR "${PLAYGROUND_DIR}" does not exist`);
-    process.exit(1);
-  }
+  const cmijFile = path.join(outputFolder, `cmij.js`);
 
   if (!fs.existsSync(outputFolder)) {
     fs.mkdirSync(outputFolder, { recursive: true });
   }
-  e(`find ${libEs6Folder} -name '*.js' -exec cp {} ${outputFolder} \\;`);
+
   e(
-    `find ${libOcamlFolder} -name "*.cmi" -or -name "*.cmj" | xargs -n1 basename | xargs js_of_ocaml build-fs -o ${cmijFile} -I ${libOcamlFolder}`
+    `find ${rescriptLibOcamlFolder} -name "*.cmi" -or -name "*.cmj" | xargs -n1 basename | xargs js_of_ocaml build-fs -o ${cmijFile} -I ${rescriptLibOcamlFolder}`
   );
-});
+}
+
+function buildThirdPartyCmijs() {
+  packages.forEach(function installLib(package) {
+    const libOcamlFolder = path.join(
+      PROJECT_ROOT_DIR,
+      "node_modules",
+      package,
+      "lib",
+      "ocaml"
+    );
+    const libEs6Folder = path.join(
+      PROJECT_ROOT_DIR,
+      "node_modules",
+      package,
+      "lib",
+      "es6"
+    );
+    const outputFolder = path.join(PACKAGES_DIR, package);
+
+    const cmijFile = path.join(outputFolder, `cmij.js`);
+
+    if (!fs.existsSync(outputFolder)) {
+      fs.mkdirSync(outputFolder, { recursive: true });
+    }
+
+    e(`find ${libEs6Folder} -name '*.js' -exec cp {} ${outputFolder} \\;`);
+    e(
+      `find ${libOcamlFolder} -name "*.cmi" -or -name "*.cmj" | xargs -n1 basename | xargs js_of_ocaml build-fs -o ${cmijFile} -I ${libOcamlFolder}`
+    );
+  });
+}
+
+buildCompilerCmij();
+buildThirdPartyCmijs();

playground/playground_test.js

@@ -1,16 +1,69 @@
 require("./compiler.js")
+require("./packages/compiler-builtins/cmij.js")
 require("./packages/@rescript/react/cmij.js")
+require("./packages/@rescript/core/cmij.js")
 
 let compiler = rescript_compiler.make()
 
 let result = compiler.rescript.compile(`
-  let a = <div> {React.string("test")} </div>
+  @@jsxConfig({ version: 4, mode: "automatic" })
+
+  open RescriptCore
+
+  module A = {
+    @react.component
+    let make = (~a) => {
+      // This should yield a warning (unused variable state)
+      let state = React.useState(() => 0)
+      <div> {React.string(a)} </div>
+    }
+  }
+
+  module B = {
+    type props = { a: string }
+
+    let make = ({a}) => {
+      <A a/>
+    }
+  }
+
+  let sum = [1,2,3] 
+    ->Array.map(x => x * 2)
+    ->Array.reduce(0, (acc, item) => acc + item)
+
+
+  // Test uncurried behavior
+  let runFn = (f, x) => f(x)
+  runFn(x => x + 1, 1)->Console.log
+
+  Console.log("Hello world!")
+
+  let a = <B a="hello" />
 `);
 
 if(result.js_code != "") {
   console.log('-- Playground test output --');
   console.log(`ReScript version: ${compiler.rescript.version}`);
   console.log('----');
+  if(result.type === "unexpected_error") {
+    console.log("UNEXPECTED ERROR");
+    console.log(result);
+    process.exit(1);
+  }
+  if(result.errors && result.errors.length > 0) {
+    console.log("COMPILATION ERROR");
+    for(let error of result.errors) {
+      console.log(error.shortMsg);
+    }
+    process.exit(1);
+  }
+
+  if(result.warnings.length === 0) {
+    console.log("TEST FAILED");
+    console.log("The code should have at least one warning.");
+    process.exit(1);
+  }
+
   console.log(result.js_code);
   console.log('-- Playground test complete --');
 }

playground/upload_bundle.sh

@@ -29,7 +29,7 @@ if [ ! -f "${NETRC_FILE}" ]; then
   echo "machine ${KEYCDN_SRV} login $KEYCDN_USER password $KEYCDN_PASSWORD" > "${NETRC_FILE}"
 fi
 
-PACKAGES=( "@rescript/react")
+PACKAGES=( "compiler-builtins" "@rescript/react" "@rescript/core")
 
 echo "Uploading compiler.js file..."
 curl --ftp-create-dirs -T "${SCRIPT_DIR}/compiler.js" --ssl --netrc-file $NETRC_FILE ftp://${KEYCDN_SRV}/v${VERSION}/compiler.js