Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pinned packages support #4361

Closed
BlueHotDog opened this issue May 8, 2020 · 109 comments
Closed

Pinned packages support #4361

BlueHotDog opened this issue May 8, 2020 · 109 comments
Labels
priority:high

Comments

@BlueHotDog
Copy link

https://github.com/BlueHotDog/reason_monorepo

Monorepo support via common tooling(specifically in the above repo Lerna) is broken.
Monorepo support for JS projects is usually being done by using symlinks to support cross-project dependencies - That's the most basic and common workflow.
Another possible monorepo approach is by using Yarn workspaces, which utilizes symlinking even more by hoisting all dependencies to the "root".

Currently, bucklescript is unable to handle any kind of monorepo setup, which, after trying to bang my head, is impossible to solve and leads to a very weird workflow for any ambitious, project(e.g a modern SPA + Nodejs server in Reason).

The example project above illustrates all the problems, but just to summarize what i've encountered:

  • Bucklescript randomly spits "duplicate dependency" warning, when two packages depend on the same package. e.g: package A depends on ReFetch(v1 for example) and package B depends on ReFetch too(same version), and package B depends on package A - will result in this warning, which is incorrect.
  • Bucklescript will randomly spit an error that bsconfig.json is broken(it's not) when trying to compile package B.
  • Bucklescript will randomly spit a: incorrect assumption about interface(same case as the first scenario)
  • Watch is just broken completely for the above reasons
  • When compiling to Es6 even more errors happen(paths etc) :(

Links to previous discussions that died out and might be related:
#3521
#4047

As there's no roadmap for reasonml. any type of communication regarding stale issue would really help.
@jchavarri
Copy link
Contributor

Another issue that makes the monorepo experience challenging is #4084. An example use case that we've seen is the generation of .ml and .mli files from .atd files. Because there are BuckleScript libraries in the monorepo that depend on .atd files, and on have dependents as well, we have to run atdgen "off-bsb" with Dune, but this means that bsb doesn't know anything about how these targets are generated, so one has to run scripts for every change to the source .atd files.

@TheSpyder
Copy link
Contributor

Other references to monorepo requests, so we can centralise the discussion
#1950
#2750
#3528

@TheSpyder
Copy link
Contributor

I have forked the monorepo @BlueHotDog posted and rewritten it to show the style of monorepo I would like to use. I've written up all the things this is designed to achieve, and why it doesn't currently work, in the readme:
https://github.com/TheSpyder/reason_monorepo

This structure holds a lot of potential, I very nearly used it for my production project, but the watch-rebuild process is just slow enough to be annoying and the lack of compiler warnings really killed it for us.

@BlueHotDog
Copy link
Author

@TheSpyder that's awesome! thank you(hopefully BS gods will hear this)
One requirement that you didnt list which is needed for me is the ability to run each project separately:
i'm developing a chrome extension + front-end and need to be able to run bsb+webpack + etc from two different packages side by side.

The way i'm doing it now, e.g by having a single top level bs-config makes this impossible(you have to run a single build, so when working on the flow i've to:

  • work on the extension. close it.
  • open the front-end. and vice-versa
    )
    If BuckleScript emphasis is performance, i would argue this isnt performant way to write code :)

@TheSpyder
Copy link
Contributor

TheSpyder commented May 17, 2020 •
edited
Loading

@BlueHotDog you can still run each project separately if you want, even in my example 😁

Little-known fact about npm module resolution, it also supports searching for a parent package.json and node_modules folder. This is what makes yarn workspaces work so well.

reason_monorepo spyder$ yarn
<stuff here>
reason_monorepo spyder$ yarn build
<build output>
reason_monorepo spyder$ cd packages/a
a spyder$ bsb
[18/18] Building src/FetchedDogPictures/FetchedDogPictures-A.cmj

packages/b also works, but that's where I have the warning-as-error configured.

In fact, my setup might work for you today if you never use a top-level bsb.

@harigopal
Copy link

I actually work on a monorepo that uses Lerna + Yarn workspaces - Pupilfirst.

It's not a traditional workspaces-based monorepo, in that there is a top-level application, with other packages inside it; they're not all side-by-side like in the example @TheSpyder created.

And it works... most of the time.

Occasionally, the bsb compilation will start failing, complaining about the presence of duplicate .re files (files with the same name). At that point, one of the packages (app/javascript/packages/pf-icon) will have a lib/ocaml folder inside it that contains dupes of some Reason files. Deleting that folder gets the compilation going again... until seemingly at random, the lib/ocaml folder shows up again.

It's a bit annoying, but it happens rarely enough that it doesn't get in the way of work, and it was the only way I could set this up as a monorepo.

I haven't a clue why this happens - I haven't been able to replicate it reliably, but it probably has something to do with all packages (including root) having their own bsconfig.json files - and two of the packages depend on another -pf-icon... and it's always pf-icon that gets the weird lib/ocaml folder with the dupe files inside it.

@TheSpyder
Copy link
Contributor

@harigopal your top-level bsconfig.json also loads all the packages as source, which is the technique I ended up using in production, unfortunately that means the packages can't leverage namespace: true.

My guess about your lib/ocaml problem is the language server occasionally realises there is another bsconfig.json and compiles that folder.

@gaku-sei gaku-sei mentioned this issue May 22, 2020
Closed
@BlueHotDog
Copy link
Author

Seems that on BS 8 the situation is better and now i can work with Yarn Workspaces.(was this fixed? couldnt see anything in any communications nor was it on the roadmap.. but.. yeah)

@TheSpyder
Copy link
Contributor

TheSpyder commented Jul 10, 2020 •
edited
Loading

I'm not sure what changed, but I don't believe it has been fixed. I'm still struggling along without access to namespace:true.

If you don't want to keep this open, I'll open a new one 🤔

@BlueHotDog BlueHotDog reopened this Jul 23, 2020
@BlueHotDog
Copy link
Author

Yup, my mistake, wasn't fixed.
@bobzhang is this still planned for the next releases as documented in the roadmap?

@BlueHotDog
Copy link
Author

@bobzhang any update on this? is this planning to be getting some attention? unfortunately if this is a no-go i can't use Reasonml and need to move code back to typescript i would just love to know before i make any drastic changes.

@bobzhang
Copy link
Member

bobzhang commented Aug 10, 2020 via email

@BlueHotDog
Copy link
Author

Hopefully, if i could get some visibility into what's planned to ship(and a ballpark of when) i can make an educated decision with the team.

@bobzhang
Copy link
Member

bobzhang commented Aug 10, 2020 via email

@TomiS TomiS mentioned this issue Aug 12, 2020
Closed
@bobzhang
Copy link
Member

bobzhang commented Aug 13, 2020 via email

@BlueHotDog
Copy link
Author

Hey Bob,
That'll work. please feel free to reach out for input/tests any assistance. happy to spend time on making this as best as we can make it.

Thanks!

@TheSpyder
Copy link
Contributor

I'm also happy to help with thoughts on how changes will impact my attempts to develop in a monorepo

@itayadler
Copy link

Hey guys, any news on this issue?
I've been working heavily these past few weeks on a ReScript monorepo, and I experience pains with setting up a consistent watch flow, between 2 packages (app, common), it keeps breaking whenever I change something in the common package, and then I need to run a clean build on both packages in order to get it working

@bobzhang
Copy link
Member

bobzhang commented Sep 14, 2020 via email

@itayadler
Copy link

sweet, ty 🙇 would love to help anyway I can

@BlueHotDog
Copy link
Author

Hey, i see there's a lot of progress on 8.3, and i was eager to try and see if we get good Monorepo support.
Not sure if this release is intended to fix these issue, but it seems like the problems are still there - Is 8.3 intends to address better support for monorepo?

@bobzhang
Copy link
Member

bobzhang commented Sep 24, 2020 via email

@itayadler
Copy link

@bobzhang exciting stuff about 8.3! myself and the company I work for really love ReScript and this issue is affecting us on a daily basis, as we need to workaround it each time it occurs.
If there was a Patreon for the project myself and the company would be the first to signup to help and prioritise this issue somehow.

@TheSpyder
Copy link
Contributor

If there was a Patreon for the project

As far as I understand they're all facebook employees, I don't think they need patreon support 😂

@fakenickels
Copy link
Contributor

Really eager to see this happening! We have a pretty big monorepo here and we suffer in a daily basis with the problems pointed above. If there is any way we can help testing you can count with us @bobzhang. Thanks for your effort

@Freddy03h
Copy link

I'm transitioning to a monorepo too, to share code between the react-native and the react-native-web apps essentially.

For now I launch bsb watch in each packages I needed, with the same package-specs (module: es6, in-source: true), It work while working on implementation detail (example : doing styles in packages/design-system while working on the package/react-native-app), but doesn't if I change parameters of a componentd for example.

@TheSpyder
Copy link
Contributor

Hi, it would be helpful if you can make a reproducible repo, I am planning to cut a release early next week, thanks
…

yes, sorry, I've been busy but I'll give it a go tonight or tomorrow

@TheSpyder
Copy link
Contributor

I couldn't make my example reason monorepo reproduce the issue so I'll try making it more complex. In the meantime I switched back to my main repo and I was wrong about something:

Whether by accident or design, it's really nice that dev dependencies are able to disconnect what would normally be a circular dependency.

It is intentional, we want to cut dependency as much as we can.

I must have put my repository into a weird state to make that compile. The problem is that it's still a circular dependency:

  • bsb tries to build both regular and dev sources of library 1 at the same time
  • the tests of library 1 depend on library 2
  • library 2 depends on library 1

So unless the dev sources can be compiled in a separate pass I'll need to unwind those dependency links. On a clean repo it's giving me a module not found error compiling the tests of library 1, which is a bit misleading but oh well.

@TheSpyder
Copy link
Contributor

Looks like the weird state I was seeing where no changes caused a full rebuild were related to the cycle that wasn't really a cycle. With a clean checkout and no fake cycles it's fine. Sorry about that.

It's still unusable for large monorepo projects due to this thing you said earlier

If your dependency changes, it will trigger a lot of rebuild

But I'm going to log a new issue for that as it is a separate discussion. Pinned packages seem to be working as designed.

@Hongbo-bb
Copy link
Contributor

Hongbo-bb commented Dec 6, 2020 via email

@Coobaha
Copy link
Contributor

Coobaha commented Dec 6, 2020

@Hongbo-bb amazing! it seems to pass, i've noticed that linux dist seems to be incorrectly packed? https://github.com/Coobaha/bsb-smallest-monorepo-example/runs/1505808007?check_suite_focus=true

Build passes on mac and uses correct bsb :)

@BlueHotDog
Copy link
Author

Doesnt work :(

$ bsb -clean-world -make-world
--
11:52:11.055 | bsb: error: loading 'build.ninja': No such file or directory
11:52:11.056 | Failed
11:52:11.060 | bs-platform version mismatch Running bsb 8.4.0-dev.1 (/vercel/workpath0/node_modules/bs-platform) vs vendored 8.4.0 (/vercel/workpath0/node_modules/bs-platform)
11:52:11.076 | error Command failed with exit code 2.
11:52:11.076 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
11:52:11.096 | error Command failed with exit code 2.
11:52:11.096 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
11:52:11.105 | Error: Command "yarn build" exited with 2
11:52:13.238 | Done with "package.json"

@Hongbo-bb
Copy link
Contributor

Hongbo-bb commented Dec 6, 2020 via email

@BlueHotDog
Copy link
Author

Just tried again :(
I'm a bit lost between all the versions, but seems to not work still:
Still getting this weird error.

19:51:04.978 | [4/4] Building fresh packages...
-- | --
19:51:16.633 | Done in 112.46s.
19:51:17.028 | yarn run v1.22.4
19:51:17.058 | $ NODE_ENV=production yarn re:cleanbuild && NODE_ENV=production yarn js:build
19:51:17.264 | $ bsb -clean-world -make-world
19:51:17.336 | bsb: error: loading 'build.ninja': No such file or directory
19:51:17.336 | Failed
19:51:17.341 | bs-platform version mismatch Running bsb 8.4.0-dev.1 (/vercel/workpath0/node_modules/bs-platform) vs vendored 8.4.0 (/vercel/workpath0/node_modules/bs-platform)
19:51:17.352 | error Command failed with exit code 2.
19:51:17.352 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
19:51:17.369 | error Command failed with exit code 2.
19:51:17.369 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
19:51:17.377 | Error: Command "yarn build " exited with 2
19:51:19.526 | Done with "package.json"

@Hongbo-bb
Copy link
Contributor

Hongbo-bb commented Dec 7, 2020 via email

@BlueHotDog
Copy link
Author

i did, also made sure yarn.lock has 8.4.1.
it fails the same way on chromatic/vercel

@Hongbo-bb
Copy link
Contributor

Hongbo-bb commented Dec 7, 2020 via email

@BlueHotDog
Copy link
Author

Yeah, super weird. and it works locally on my OSX. the problem is, i dont have shell access to any of these platforms. pinging you on Discord to maybe give you some other way to look.

@Coobaha
Copy link
Contributor

Coobaha commented Dec 7, 2020

It seems to be passing on Linux, before it was failing with the same error as you have @BlueHotDog
https://github.com/Coobaha/bsb-smallest-monorepo-example/actions/runs/403993626

Can you reproduce this error with my repo and vercel build?

@BlueHotDog
Copy link
Author

It seems to be passing on Linux, before it was failing with the same error as you have @BlueHotDog
https://github.com/Coobaha/bsb-smallest-monorepo-example/actions/runs/403993626

Can you reproduce this error with my repo and vercel build?

Just tried, and seems to work. well. no idea what is happening.. and no idea what this error means :(
The thing is that it also fails on Chromatic and they dont share the same cache/build pipeline

@TheSpyder
Copy link
Contributor

But I'm going to log a new issue for that as it is a separate discussion

I won't have time to log the issue until tomorrow, probably, but I have added pinned packages support to my example monorepo (it's probably a good way to show how pinned packages work, actually) and there's a blurb in the readme about why it isn't quite good enough for large projects.

@Hongbo-bb
Copy link
Contributor

Hongbo-bb commented Dec 8, 2020 via email

@Hongbo-bb
Copy link
Contributor

Hongbo-bb commented Dec 8, 2020 via email

@TheSpyder
Copy link
Contributor

Thanks! I look forward to testing it, I do usually have more interfaces it was just an example

@Hongbo-bb
Copy link
Contributor

Hongbo-bb commented Dec 9, 2020 via email

@TheSpyder
Copy link
Contributor

TheSpyder commented Dec 9, 2020 •
edited
Loading

I published a version 8.4.2-dev.1 for Mac, let me know if it works for you

That's beautiful, thank you! 😁

So the change means that saving an implementation will no longer trigger dependent rebuilds, but saving an interface still does? I tested by swapping back and forth between pinned packages and a single flat bsconfig.json, then saving one of my .rei files (no changes but it causes the file to be built).

The single namespace is able to avoid dependent modules rebuilding in this scenario when the interface contents remains the same. As a pinned package, that same change causes all dependent modules to rebuild; presumably because the build system is operating in new contexts for dependencies and only checks the file timestamp not whether the interface actually changed.

This means it's still slower to develop on than the single namespace but it's vastly improved from 8.4.1 and I think that tips the balance such that team will overlook the speed difference to get the improved monorepo experience.

@TheSpyder
Copy link
Contributor

I have updated my example monorepo with 8.4.2-dev.1 and added interface files to every pinned dependency. The readme now lists just one example that triggers a forest of rebuilds.

@Hongbo-bb
Copy link
Contributor

Hongbo-bb commented Dec 10, 2020 via email

@TheSpyder
Copy link
Contributor

Yeah I can understand that’s the current design. It would be nice if pinned packages weren’t a black box but that would be hard with features like namespace: true (and it can be improved later if you find a way to do it).

@Freddy03h
Copy link

Hi! I read the blogpost but I don't understand how it work. I put one of my workspace bs-dependencies in pinned-dependencies in my bsconfig file with updated (8.4.2) bs-platform inside all my packages.
But … the behavior seems the same? Nothing change when I work on a file in the pinned-dep.
I still can launch a other bsb -make-world -w on the other yarn workspace package. It's still work for inner change like style, but still can't for a fonction definition for example.

I read that watcher is an other topic that isn't yet ready I think.

How can I benefit of the advantages of a monorepo with yarn workspace? Working on files in different packages without switching script run, clean and relaunch every time, just like if we work on separate repositories…

I should miss something I don't understand, or maybe it's not yet ready? (because of watcher)

@TheSpyder
Copy link
Contributor

@Freddy03h The test monorepo I used to provide feedback is a good example of how to run with pinned packages. I'll update it to the 8.4 release soon.
https://github.com/TheSpyder/reason_monorepo

@Freddy03h
Copy link

@TheSpyder Thank's for the answer, I finally got time to test it.
My monorepo is different because I don't have source code at the root. But anyway, I tested to add a bsconfig at the root and references all packages as bs-dependencies and pinned-dependencies on it. (my difference is that sources target an empty directory).
The compile work fine, but it's still the same probleme: I can't work on the packages files because it's not watched…

pinned-dependencies seems to be a step for the monorepo use case, but for now, without the watcher part, it's impossible to have a normal workflow on a monorepo with rescript.

@TheSpyder
Copy link
Contributor

Watching is the easiest part, I had actually stopped using bsb -w even before pinned packages. I discovered it uses nodejs watch not a native implementation. There are many good native watch tools, I'm using watchexec. Here's a quick example, similar to the one I use in my example repo:

watchexec -w modules -i lib -e re,rei npx bsb

This does lose the timing printout for each compile but I'm planning to add a little shell script to do that later (and then I'll post the whole thing on the forum).

@ryyppy
Copy link
Member

ryyppy commented Jan 3, 2021

@TheSpyder we wanted to create a dedicated doc page just for monorepo support (see related issue)... might be worth adding your findings on a reliable watch setup in userland there as well.

@TheSpyder
Copy link
Contributor

Will do, sorry I thought I would have it done by now but life has been a bit busy.

@Freddy03h Freddy03h mentioned this issue May 4, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority:high
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests