TF 2, both hands, Unity.

Author: vinnik-dmitry07

.gitignore

@@ -22,7 +22,6 @@ parts/
 sdist/
 var/
 wheels/
-pip-wheel-metadata/
 share/python-wheels/
 *.egg-info/
 .installed.cfg
@@ -52,6 +51,7 @@ coverage.xml
 *.py,cover
 .hypothesis/
 .pytest_cache/
+cover/
 
 # Translations
 *.mo
@@ -74,6 +74,7 @@ instance/
 docs/_build/
 
 # PyBuilder
+.pybuilder/
 target/
 
 # Jupyter Notebook
@@ -84,7 +85,9 @@ profile_default/
 ipython_config.py
 
 # pyenv
-.python-version
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
 
 # pipenv
 #   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
@@ -129,3 +132,81 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
+# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
+
+# User-specific stuff
+.idea/**/workspace.xml
+.idea/**/tasks.xml
+.idea/**/usage.statistics.xml
+.idea/**/dictionaries
+.idea/**/shelf
+
+# Generated files
+.idea/**/contentModel.xml
+
+# Sensitive or high-churn files
+.idea/**/dataSources/
+.idea/**/dataSources.ids
+.idea/**/dataSources.local.xml
+.idea/**/sqlDataSources.xml
+.idea/**/dynamic.xml
+.idea/**/uiDesigner.xml
+.idea/**/dbnavigator.xml
+
+# Gradle
+.idea/**/gradle.xml
+.idea/**/libraries
+
+# Gradle and Maven with auto-import
+# When using Gradle or Maven with auto-import, you should exclude module files,
+# since they will be recreated, and may cause churn.  Uncomment if using
+# auto-import.
+# .idea/artifacts
+# .idea/compiler.xml
+# .idea/jarRepositories.xml
+# .idea/modules.xml
+# .idea/*.iml
+# .idea/modules
+# *.iml
+# *.ipr
+
+# CMake
+cmake-build-*/
+
+# Mongo Explorer plugin
+.idea/**/mongoSettings.xml
+
+# File-based project format
+*.iws
+
+# IntelliJ
+out/
+
+# mpeltonen/sbt-idea plugin
+.idea_modules/
+
+# JIRA plugin
+atlassian-ide-plugin.xml
+
+# Cursive Clojure plugin
+.idea/replstate.xml
+
+# Crashlytics plugin (for Android Studio and IntelliJ)
+com_crashlytics_export_strings.xml
+crashlytics.properties
+crashlytics-build.properties
+fabric.properties
+
+# Editor-based Rest Client
+.idea/httpRequests
+
+# Android studio 3.1+ serialized cache file
+.idea/caches/build_file_checksums.ser

README.md

@@ -1,79 +1,19 @@
-# Minimal Hand
+# Unity Minimal Hand
 
-A minimal solution to hand motion capture from a single color camera at over 100fps.
-Easy to use, plug to run.
+A hand motion capture from a single color camera + Unity + a virtual webcam.
 
-![teaser](teaser.gif)
+![](media/screenshot.png)
 
-This project provides the core components for hand motion capture:
-1. estimating joint **locations** from a monocular RGB image (DetNet)
-1. estimating joint **rotations** from locations (IKNet)
+## [Demo](https://www.youtube.com/watch?v=q_P_4KuAIVw)
 
-We focus on:
-1. ease of use (all you need is a webcam)
-1. time efficiency (on our 1080Ti, 8.9ms for DetNet, 0.9ms for IKNet)
-1. robustness to occlusion, hand-object interaction, fast motion, changing scale and view point
+## How to use
 
-Some links:
-[\[video\]](https://youtu.be/OIRulRoBdL4)
-[\[paper\]](https://calciferzh.github.io/files/zhou2020monocular.pdf)
-[\[supp doc\]](https://calciferzh.github.io/files/zhou2020monocular_supp.pdf)
-[\[webpage\]](https://calciferzh.github.io/publications/zhou2020monocular)
+1) Install dependencies
 
-The author is too busy to collect the training code for release.
-On the other hand, it should not be difficult to implement the training part.
-Feel free to open an issue for any encountered problems.
+2) Run the *app1.py* script
 
-## Usage
+3) Run the *Main.unity* scene
 
-### Install dependencies
-Please check `requirements.txt`. All dependencies are available via pip and conda.
+## Known issues
 
-### Prepare MANO hand model
-1. Download MANO model from [here](https://mano.is.tue.mpg.de/) and unzip it.
-1. In `config.py`, set `OFFICIAL_MANO_PATH` to the **left hand** model.
-1. Run `python prepare_mano.py`, you will get the converted MANO model that is compatible with this project at `config.HAND_MESH_MODEL_PATH`.
-
-### Prepare pre-trained network models
-1. Download models from [here](https://drive.google.com/open?id=1ZnDYF9rHKbef27tiHkWrLznqe18le7ol).
-1. Put `detnet.ckpt.*` in `model/detnet`, and `iknet.ckpt.*` in `model/iknet`.
-1. Check `config.py`, make sure all required files are there.
-
-### Run the demo for webcam input
-1. `python app.py`
-1. Put your **right hand** in front of the camera. The pre-trained model is for left hand, but the input would be flipped internally.
-1. Press `ESC` to quit.
-1. Although the model is robust to variant scales, most ideally the image should be 1.3x larger than the hand bounding box. A good bounding box may result in better accuracy. You can track the bounding box with the 2D predictions of the model.
-
-We found that the model may fail on some "simple" poses. We think this is because such poses were no presented in the training data. We are working on a v2 version with further extended data to tackle this problem.
-
-### Use the models in your project
-Please check `wrappers.py`.
-
-## IKNet Alternative
-We also provide an optimization-based IK solver [here](https://github.com/CalciferZh/Minimal-IK).
-
-## Dataset
-The detection model is trained with following datasets:
-* [CMU HandDB](http://domedb.perception.cs.cmu.edu/handdb.html)
-* [Rendered Handpose Dataset](https://lmb.informatik.uni-freiburg.de/resources/datasets/RenderedHandposeDataset.en.html)
-* [GANerated Hands Dataset](https://handtracker.mpi-inf.mpg.de/projects/GANeratedHands/GANeratedDataset.htm)
-
-The IK model is trained with the poses shipped with [MANO](https://mano.is.tue.mpg.de/).
-
-## Citation
-
-This is the official implementation of the paper "Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data" (CVPR 2020).
-
-The quantitative numbers reported in the paper can be found in `plot.py`.
-
-If you find the project helpful, please consider citing us:
-```
-@inproceedings{zhou2020monocular,
-  title={Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data},
-  author={Zhou, Yuxiao and Habermann, Marc and Xu, Weipeng and Habibie, Ikhsanul and Theobalt, Christian and Xu, Feng},
-  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
-  pages={0--0},
-  year={2020}
-}
-```
+* 24 FPS on Nvidia GTX 1080 is rather few

UnityProject/.gitignore

@@ -0,0 +1,71 @@
+# This .gitignore file should be placed at the root of your Unity project directory
+#
+# Get latest from https://github.com/github/gitignore/blob/master/Unity.gitignore
+#
+/[Ll]ibrary/
+/[Tt]emp/
+/[Oo]bj/
+/[Bb]uild/
+/[Bb]uilds/
+/[Ll]ogs/
+/[Uu]ser[Ss]ettings/
+
+# MemoryCaptures can get excessive in size.
+# They also could contain extremely sensitive data
+/[Mm]emoryCaptures/
+
+# Asset meta data should only be ignored when the corresponding asset is also ignored
+!/[Aa]ssets/**/*.meta
+
+# Uncomment this line if you wish to ignore the asset store tools plugin
+# /[Aa]ssets/AssetStoreTools*
+
+# Autogenerated Jetbrains Rider plugin
+/[Aa]ssets/Plugins/Editor/JetBrains*
+
+# Visual Studio cache directory
+.vs/
+
+# Gradle cache directory
+.gradle/
+
+# Autogenerated VS/MD/Consulo solution and project files
+ExportedObj/
+.consulo/
+*.csproj
+*.unityproj
+*.sln
+*.suo
+*.tmp
+*.user
+*.userprefs
+*.pidb
+*.booproj
+*.svd
+*.pdb
+*.mdb
+*.opendb
+*.VC.db
+
+# Unity3D generated meta files
+*.pidb.meta
+*.pdb.meta
+*.mdb.meta
+
+# Unity3D generated file on crash reports
+sysinfo.txt
+
+# Builds
+*.apk
+*.aab
+*.unitypackage
+
+# Crashlytics generated file
+crashlytics-build.properties
+
+# Packed Addressables
+/[Aa]ssets/[Aa]ddressable[Aa]ssets[Dd]ata/*/*.bin*
+
+# Temporary auto-generated Android Assets
+/[Aa]ssets/[Ss]treamingAssets/aa.meta
+/[Aa]ssets/[Ss]treamingAssets/aa/*

UnityProject/.vsconfig

@@ -0,0 +1,6 @@
+{
+  "version": "1.0",
+  "components": [ 
+    "Microsoft.VisualStudio.Workload.ManagedGame"
+  ]
+}