Raycaster component fails to follow camera in immersive mode (mobile)

[NNskelly]

Description:

• A-Frame Version: 1.0.4
• Platform / Device: Android Chrome 82 (currently canary 82.0.4081.2)
• Reproducible Code Snippet or URL: https://dev.nextnowagency.com/camera_ray_test/

Likely tangential to #4412 which is in turn tied to underlying THREE issue mrdoob/three.js#18448

On Android mobile chrome, in immersive WebXR mode (AR or VR), 'raycaster' components on the a-camera cease follow the camera.

In the linked demo,
Expected behavior:

• Outside of immersive mode, can look around with tilt/mouse/keyboard controls; reticle tracks along groundplane
• In immersive mode, can look around with tilt/tracked control; reticle tracks along groundplane

Observed behavior:

• Outside of immersive mode, on Android Chrome 80 (WebXR AR not yet available), can look around with tilt controls; reticle tracks accordingly. On desktop, can look around with mouse controls; reticle tracks accordingly. On AR-enabled Chrome 82 (canary) on Android, camera view rotation appears to twitch with device rotation and track on 1 axis with touch, but is otherwise fixed directly downwards.
• In VR immersive mode on chrome 80, and in VR or AR immersive mode on chrome 82, can look around, but reticle resets and stays stuck on or about the scene origin; hit tests no longer return the point the camera is looking at. Rotation of reticle still updates, so the data is accurate in the camera's rotation matrix, the A-Frame raycaster simply is not sourcing it effectively.

Short of diving into the A-Frame raycaster component source, I would guess it's using something akin to Object3D.getWorldDirection() for its ray generation, which would mean it will be broken due to the getWorld* immersive-mode THREE breakage until and unless it is rewritten to e.g. transform a ray independently through the available and demonstrably correct raw transform matrices.

Chrome 80, AR unavailable, expected behavior outside immersive mode

Chrome 82, AR available, stuck view outside immersive mode

Chrome 82, in AR immersive mode, hit test result stuck at origin

Chrome 82 in VR immersive mode; essentially identical to Chrome 80 in VR immersive mode, hit test result equally stuck at origin.

[dmarcos]

This is an example of a reticle style cursor that works on both VR / AR / 2D mode:

https://aframe.io/aframe/examples/test/cursor/
https://github.com/aframevr/aframe/blob/master/examples/test/cursor/index.html#L27

You can see that children follow the camera as expected.

I see calls to getWorld* functions in your code. Don't use those, don't work. It's a known THREE bug. Use suggested alternative calculations

[NNskelly]

I apologize for the confusion; I have removed the references to getWorld* from my example for exactly that reason, as we have already discussed at length.

I am having difficulty locating the raycaster scripting API calls in your example. a-frame-event-set-component.js appears to only contain event binding & unbinding logic, and all other interactions look to be set up by relatively passive virtual-click triggering.
I appreciate the existence of workarounds, but just to be clear, it is your present assessment that the raycaster component .intersectedEls array and .getIntersection() 100% work as intended in immersive modes on mobile when applied directly to the scene camera, and the behavior of my cited sample is what you would expect from the way I am using the component, notwithstanding the availability of alternative mechanisms?

It appears that the raycaster component does work independently in immersive modes, and behaves fine when applied to a child element of the camera as in your example and a new tweak to my example ( https://dev.nextnowagency.com/camera_ray_test/index3.html ), just not when applied directly to the camera. This observation does further strengthen my suspicion that the A-Frame raycaster component itself is using the problematic getWorld* calls.

[dmarcos]

Removed use of getWorldPosition from the raycaster component to work around THREE bug mrdoob/three.js#18448.

Does it solve your issue? You can try with master builds: https://cdn.jsdelivr.net/gh/aframevr/aframe@0edd9871b9d47ca60a305c5ea7369f8dbab34ead/dist/aframe-master.min.js

[NNskelly]

Afaict, that did it! ( https://dev.nextnowagency.com/camera_ray_test/index4.html ; raycaster on camera, but sourcing the master build aframe)

Thanks- I had honestly planned to just jump into the code today and see if it was that quick a fix to pull-request myself, but a catastrophic cascade failure of the whole office's version control server AND backup kinda derailed any development work. I am now willing to consider this closed. The current behavior of locking the camera downwards when outside of immersive mode remains a concern, and if it persists after chrome 82 reaches release (having shown up between 80 and 82), I may re-raise that as an independent issue, but I'm willing to leave that at the feet of the google devs for now until their feature updates are stable.