Oculus Quest 2 animations on trigger / grip / button press don't work

[diarmidmackenzie]

Description:

• A-Frame Version: 1.3.0
• Platform / Device: Oculus Quest 2
• Reproducible Code Snippet or URL: Any A-Frame Example that uses controllers. E.g. https://aframe.io/examples/showcase/responsiveui/

When I press the trigger or grip, the trigger & grip on the controller models as viewed in VR. Similarly, changes to button colors (as configured via buttonColor, buttonHighlightColor, buttonTouchColor) are not applied.

I hadn't realized this was expected function, but looking at the code in oculus-touch-controls there is code to animate the trigger & grip & change colors of buttons. It looks like it would work for the Oculus Quest 1, but hasn't been updated to work for Oculus Quest 2.

The key bit of code is here, which uses getObjectByName to access the relevant sub-components of the controller 3D model.

I've been looking at the controller models in a GLTF viewer.

the "gen2" model (used for Quest 1) has subcomponents with names that match the code.

the "v3" model (used for Quest 2) has a different set of names for the subcomponents (as well as a much more complicated hierarchy).

Because the names don't match, the calls to getObjectByName all fail, and the component has no references in buttonMeshes, for example here and here

I haven't looked into what's involved in fixing yet. It might be a matter of just using the alternate names on the calls to getObjectByName. Or it's possible that the details of exactly what rotations & position offsets need to be applied also need to be adjusted to reflect the different controller model.

I'd be happy to look at doing a PR to fix this. I have Quest 2, so I can test that. I don't have a Quest 1, so I won't be able to regression test live, but should be able to cover with Unit Tests. Just wanted to check first that you'd be happy to receive such a PR. And also that the fix should be in the oculus-touch-controls component (rather than e.g. fixing up the v3 model to match the gen2 naming conventions).

[diarmidmackenzie]

Looking at the structure of the v3 controller object, I'm guessing that the _max and _min Object3Ds are intended to provide guidelines for the transforms for the trigger, grip etc as they are pressed.

And therefore these sub-components should be animated by setting the matrix4 of the _value Object3D to a suitable interpolation between the _max and _min matrix4s (which I'm assuming should be treated as read-only).

Interpolation between matrix4s will need to be done by decomposing them into position & rotation (quaterion) and scale (though I assume scale won't change), interpolating each of these components using lerp for position and slerp for quaternions, and composing these into the new matrix4.

I'd propose a fix along those lines. The gen2 controller model, just uses hard-coded position & rotation updates, but if this additional info is embedded in the model for v3, it surely makes sense to use it (and hopefully this will future-proof the implementation for future controller models).

I don't suppose we know of any documentation for the controller models that could confirm my interpretation of the _max, _min and _value object3Ds?

[dmarcos]

Thanks so much. It seems the feature was lost as we incorporated more controllers. There are no docs for the models unfortunately.

[kylebakerio]

Nice work diarmid, I was wondering why animations weren't working and found this issue. Do we know what the source of these controller models are? Were they extracted from somewhere?

Would be cool to see either a new component that supports this controller model or to see an updated model that fits the design expected by the component.

[diarmidmackenzie]

I don't know about the source of these, but pretty sure the best path forwards it to update A-Frame to fit the new model format rather than trying to engineer back-compatible models.

I don't think it's a big job, but I've had other things on my plate. I can see myself getting to this within a couple of weeks. If you'd like to have a go at this before then, please do!

[kylebakerio]

I'm well outside of my realm of expertise, but I started looking through the model design and then took an experimental stab. Some interesting progress.

1. grip works perfectly out of the box utilizing the existing oculus-touch-controls code, just by grabbing the correct mesh.
2. trigger almost works same way, but the rocking is not quite ont he right axis.
3. in the console, working with the model directly, I am able to confirm that that 'max'' and 'min' values do indeed correspond to the correct range. You can use something like this to manually click the button on/off:

buttonMeshes[buttonName].setRotationFromEuler(buttonRanges[buttonName][ /*min || max */].rotation);

(this is assuming you use my updated code to grab the correct meshes and ranges, see below for some code you can lightly tweak to do so.)

This seems to me like the matrix4 interpolation shouldn't be necessary. Using this method for the grip also works, but more interestingly using it for hte trigger immediately produces a correct result (unlike the existing code). Just a matter of applying it proportionately to its application.

Thumbstick values are provided in the model as well and should be able to be applied, too, which is pretty cool. In the console, I was also able to move the thumbstick around using a very similar command to the trigger/grip buttons, as above. Attempts to do this 'properly', on a thumbstickmoved event, in the component, aren't working yet for some reason, though after flreshing out a proof of concept attempt first draft I didn't dig much beyond verifying that the code was running at the right time.

I also have some crude, not quite right a/b/x/y up/down functionality working that looks to be very close to right.

You can try the working demo here, which has a bunch of gross proof of concept code just to give a sandbox for me to experiment on the model and with how the component might be updated. Only the right hand has the experimental component: https://basic-tracked.glitch.me/

working way of adding the meshes and the provided ranges:

  
  onModelLoadedV3(evt) {
    var controllerObject3D = this.controllerObject3D = evt.detail.model;
    var buttonMeshes, buttonRanges;

    if (!this.data.model) { return; }    
    
    buttonMeshes = this.buttonMeshes = {};
    buttonRanges = this.buttonRanges = {};

    buttonMeshes.grip = controllerObject3D.getObjectByName('squeeze');
    buttonRanges.grip = {
      min: controllerObject3D.getObjectByName('xr_standard_squeeze_pressed_min'),
      max: controllerObject3D.getObjectByName('xr_standard_squeeze_pressed_max'),
    }
    
    this.originalXPositionGrip = buttonMeshes.grip && buttonMeshes.grip.position.x;
    
    buttonMeshes.thumbstick = controllerObject3D.getObjectByName('thumbstick'); // todo: this one is pretty complex, with x/y stuff, so verify this works
    buttonRanges.thumbstick = {
      min: controllerObject3D.getObjectByName('xr_standard_thumbstick_pressed_min'),
      max: controllerObject3D.getObjectByName('xr_standard_thumbstick_pressed_max'),
      x: {
        min: controllerObject3D.getObjectByName('xr_standard_thumbstick_xaxis_pressed_min'),
        max: controllerObject3D.getObjectByName('xr_standard_thumbstick_xaxis_pressed_max'),
      },
      y: {
        min: controllerObject3D.getObjectByName('xr_standard_thumbstick_yaxis_pressed_min'),
        max: controllerObject3D.getObjectByName('xr_standard_thumbstick_yaxis_pressed_max'),
      }
    }    
    
    buttonMeshes.trigger = controllerObject3D.getObjectByName('trigger');
    buttonRanges.trigger = {
      min: controllerObject3D.getObjectByName('xr_standard_trigger_pressed_min'),
      max: controllerObject3D.getObjectByName('xr_standard_trigger_pressed_max'),
    }
    
    this.originalXRotationTrigger = buttonMeshes.trigger && buttonMeshes.trigger.rotation.x;
    
    buttonMeshes.xbutton = controllerObject3D.getObjectByName('x_button');
    buttonRanges.xbutton = {
      min: controllerObject3D.getObjectByName('x_button_pressed_min'),
      max: controllerObject3D.getObjectByName('x_button_pressed_max'),
    }
    buttonMeshes.abutton = controllerObject3D.getObjectByName('a_button');
    buttonRanges.abutton = {
      min: controllerObject3D.getObjectByName('a_button_pressed_min'),
      max: controllerObject3D.getObjectByName('a_button_pressed_max'),
    }    
    buttonMeshes.ybutton = controllerObject3D.getObjectByName('y_button');
    buttonRanges.ybutton = {
      min: controllerObject3D.getObjectByName('y_button_pressed_min'),
      max: controllerObject3D.getObjectByName('y_button_pressed_max'),
    }    
    buttonMeshes.bbutton = controllerObject3D.getObjectByName('b_button');
    buttonRanges.bbutton = {
      min: controllerObject3D.getObjectByName('b_button_pressed_min'),
      max: controllerObject3D.getObjectByName('b_button_pressed_max'),
    }    
  },

[kylebakerio]

Everything is now working, except

1. applying thumbstick motion correctly
2. modifying button mesh colors

Looking very close!

[kylebakerio]

thumbsticks working!

[kylebakerio]

Nothing seems to color a single mesh, for some reason--applying color to any mesh seems to apply color to every mesh. Also of note: the V2 controller model also applied the color to every button, iirc, and I think another of the quest models does too.

This is not at all my specialty, but it does not seem this is a limitation of the model, as I am able to selectively color this node in babylon's sandbox model viewer:

here's the controller component in action:

quest2-controllers.mp4

In the meantime, I'll work on refactoring this prototype into something cleaner and making a pull request.

[diarmidmackenzie]

A threejs geometry can have multiple materials, but the A-Frame material component doesn't support this.

I suspect that's the issue you are hitting.

Whenever I've wanted to do multiple materials on an object I had to hand-code at the three.js level.

I did once look at extending the material component to support multiple materials per-geometry, but it wasn't at all straightforward, and I didn't manage to get a working solution.

[diarmidmackenzie]

Great work BTW. Looks like my initial analysis that this was fairly straightforward may have been some way off the mark!

[kylebakerio]

A threejs geometry can have multiple materials, but the A-Frame material component doesn't support this.

I suspect that's the issue you are hitting.

Whenever I've wanted to do multiple materials on an object I had to hand-code at the three.js level.

I did once look at extending the material component to support multiple materials per-geometry, but it wasn't at all straightforward, and I didn't manage to get a working solution.

I see, that's an interesting detail. Do you have a code sample? Honestly didn't even realize I was working with something A-Frame specific at that point.

[diarmidmackenzie]

See e.g. penguin component in this module.

https://github.com/diarmidmackenzie/christmas-scene/blob/main/src/christmas-utils.js

Geometry constructed with 3 groups, and then materials are provided as an array. A-Frame can't do this.

[kylebakerio]

Thanks for that tip. Got colors working! Now I'm tempted to go try and fix the A-Frame standard mesh material... Anyways, let's get this done first, haha.

[kylebakerio]

Current iteration also fixes colored buttons for all the quest controllers, which have all shared the characteristic of being broken and having all buttons light up when any button is pressed.

[kylebakerio]

Pull request submitted:

#5103