feat: ship proot, DirectAudio, Xbox, Comet and hand tracking as CI-built add-on packs (#1198)

* perf: download hand tracking models and native libs on first use

* feat: build and publish add-on packs in CI, add Xbox and Comet packs

* feat: ship proot and DirectAudio drivers as CI-built packs

* test: hand tracking readiness requires the tracking pack

* ci: attach PR test-build packs to the PR release instead of Components
This commit is contained in:
Kurt Himebauch authored and GitHub committed 2026-10-20 11:18:44 -04:00
1 parent 0663593920
commit e3246b7dd5
72 files changed
+710 -112

No files matched your search

@@ -106,7 +106,7 @@ The optional Lite bundle can be rebuilt with `mediapipe==0.10.21` using:
python tools/build_hand_lite.py --palm palm_detection_lite.tflite --lite hand_landmark_lite.tflite
```
Source URLs and hashes are in `app/src/main/assets/models/hand_landmarker_lite.provenance.json`.
Source URLs and hashes are in `tools/packs/tracking/hand_landmarker_lite.provenance.json`.
These changes still need live moving-hand, closed-fist, gesture-continuity, and latency validation.
Earlier Lite/normalized-input validation passed 52 focused JVM tests, and bundle rebuild and label-archive reproducibility
checks passed. Desktop MediaPipe 0.10.21 inference succeeded on color hand fixtures. All six
+1 -1
View File
@@ -13,7 +13,7 @@ The base enables experimental hand tracking without requiring every user to repe
Use the existing optional `droiddeck.vitureSdk` local property with the external VITURE Android SDK 2.4.0. No vendor headers or libraries are checked into the repository. `-PvitureSdkDir=` explicitly builds without the SDK. The ordinary runtime freshness checks remain enabled. Rebuild the ARM64 VR components with `tools/vr/build_windows.py`; its manifest includes `spatialpose_shm.h`.
The hand pipeline pins MediaPipe Tasks Vision 1.0.0, OpenCV 4.13.0, and the model and SHA-256 recorded in `app/src/main/assets/models/hand_landmarker.provenance.json`. Both Gradle and the inference worker verify the model checksum. CPU inference, the full model, concurrent camera inference, scheduled hand discovery, and automatic input-rate adjustment are the defaults, including after app restarts, reconnects, and provider restarts. **Show diagnostics → GPU hand inference** temporarily enables GPU until the next tracking restart; GPU remains experimental because SteamVR restarts and Vulkan device loss were observed during testing. Diagnostics report the selection as `handGpuEnabled` and the initialized detectors as `handInferenceBackend`; `not initialized` means detector startup has not completed. Sustained performance acceptance remains unresolved.
The hand pipeline pins MediaPipe Tasks Vision 1.0.0, OpenCV 4.13.0, and the model and SHA-256 recorded in `tools/packs/tracking/hand_landmarker.provenance.json`. The model, OpenCV and MediaPipe native libraries are downloaded on first use as the tracking pack (`tools/packs`); the pack build and the inference worker verify the model checksum. CPU inference, the full model, concurrent camera inference, scheduled hand discovery, and automatic input-rate adjustment are the defaults, including after app restarts, reconnects, and provider restarts. **Show diagnostics → GPU hand inference** temporarily enables GPU until the next tracking restart; GPU remains experimental because SteamVR restarts and Vulkan device loss were observed during testing. Diagnostics report the selection as `handGpuEnabled` and the initialized detectors as `handInferenceBackend`; `not initialized` means detector startup has not completed. Sustained performance acceptance remains unresolved.
Hand input starts by copying 500 permille of stereo callbacks, approximately 12.5 pairs per second at the SDK's documented 25 Hz camera rate. On CPU, automatic adjustment can raise this to 600 or 800 permille (about 15 or 20 Hz) only after eight consecutive valid observations have sufficient processing and freshness headroom. Promotion requires average processing at most 55 ms and observation age at most 110 ms for 15 Hz, then at most 40 ms and 90 ms for 20 Hz. A moving average allows occasional scheduled discovery frames while preserving a sustained processing budget. An average above the selected period reduces the rate; any single processing cost of 80 ms or observation age of 140 ms immediately returns it to 500 permille. Tracking failure and detector resets also return it to 500. This policy requests more input only when processing can accommodate it; it does not guarantee that many completed or valid hand updates.