Add VITURE XR controls, head tracking, and SteamVR (#732)

* Add VITURE glasses support: hardware control, 3D, head tracking

Glasses get their own controller over the MCU (permission, 120 Hz default, brightness, volume, sunblock, 2D/3D) and the game moves onto them when plugged in. The IMU drives Cardboard head tracking and the SteamVR driver. 3D falls back to one eye's view on phones that can't send the 3840x1080 signal. Options open from a glasses button in game and on the Steam page.

* Drop the glasses settings like Steam's: Play-style button, Settings lip, tabs

* Hang the glasses Settings lip over the top of the button

* Space glasses MCU commands so the IMU rate sticks, fit fixed resolutions to the glasses

* Keep Steam alive with SteamVR registered, run SteamVR at the glasses' rate, keep the glasses lip on top

* Fit VR games on the glasses: full view, seated origin at the head, pad hands reach with the d-pad

* Let the pad hands reach a metre out so buttons with deep colliders press

* Show every VR frame in order at 120 fps: wait for SteamVR's GPU work, present only finished copies, skip the grace on idle shared counters

* Glasses: SteamVR starts and stops with them, closer view with a field of view setting, Recenter on the pad

* Glasses: SteamVR field of view defaults to 90°

* VR pad hands: sticks move and aim the hands, left stick click switches the sticks to the game, d-pad pulls a held hand

* Glasses: Play SteamVR beside the glasses button instead of starting SteamVR with them, pad hands drawn as Index controllers and off while other VR controllers are connected, glasses sheet tabs sized like Steam's

* Track glasses mode, refresh-aware glasses window, align Play SteamVR with Play Steam

* Fix 3D on the glasses: full-res SteamVR presenter, revert VR 3D at session end, coalesce vsync ticks, no session restart after end

* Address review: glasses output after installs and session end, VR revert retry, display fallback, presenter swapchain retry, agent hex checks, drawer bumpers, glasses translations, docs

* Harden VITURE stereo negotiation and output refresh tracking

* Localize stereo recovery feedback and record Astra reconnect tests

* Improve test failure diagnostics and refresh Astra firmware findings

* Handle VITURE docks separately and reject truncated USB frames

* Glasses: remember a 3D fallback per model, docks as display only, VR timing fixes

- A 3D attempt that ends at the 640x480 fallback is remembered for that glasses
  model, so Play SteamVR no longer blacks out the glasses after every restart;
  Retry clears it and "3D for VR games" stays reachable
- A VITURE Mobile Dock on its own no longer counts as glasses; through a dock
  Play SteamVR is hidden and the sheet shows the output and why there is no control
- HDR is probed on the glasses when the session starts there
- Glasses frames with a mismatched checksum are marked in the status frame log
- SteamVR layer and driver vblank counters stay monotonic across refresh changes;
  one-eye render scale applies from activation and on mono/3D switches
- Compositor binds announce the refresh clients were last told; the presenter
  retries a failed first swapchain
- Dead strings and helpers removed; docs updated

* Glasses: drop USB frames with a bad checksum

* Glasses: count frames dropped for a bad checksum

* Feat/viture xr (#907)

* Add RayNeo Air and GT support to the XR glasses pipeline

* Add RayNeo Air and GT support to the XR glasses pipeline

* Checkpoint glasses tracking, calibration, and SteamVR launch integration

* Checkpoint glasses tracking, calibration, and SteamVR launch integration

* Add XREAL 1S support and correct GT and Pro 2 tracking

* Add XREAL 1S support and correct GT and Pro 2 tracking

* Fix mirrored glasses tracking and recover SteamVR after device loss

* Fix mirrored glasses tracking and recover SteamVR after device loss

* Document confirmed XREAL 1S brightness control

* Document confirmed XREAL 1S brightness control

* Fix SteamVR glasses defaults and rendering controls

* Fix SteamVR glasses defaults and rendering controls

* chore: keep only the 3 newest session log folders (#852)

* chore: add agent comment rules (#748)

Co-authored-by: Kurt Himebauch <136133082+xXJSONDeruloXx@users.noreply.github.com>

* Fix SteamVR polling, VITURE checksums, and fence tracking

Keep manual SteamVR launches polled, discard corrupt VITURE packets, and retire Vulkan fences out of order without blocking submissions at a fixed capacity. Preserve the contiguous completion watermark used by the presenter.

Fix optional Windows launcher configuration and audited helper staging. Add Android, Windows, and native tracker regression coverage.

* Fix SteamVR polling, VITURE checksums, and fence tracking

Keep manual SteamVR launches polled, discard corrupt VITURE packets, and retire Vulkan fences out of order without blocking submissions at a fixed capacity. Preserve the contiguous completion watermark used by the presenter.

Fix optional Windows launcher configuration and audited helper staging. Add Android, Windows, and native tracker regression coverage.

* fix: wireless ADB page names the child-process switch the way Android shows it (#848)

* l10n: refresh stale and untranslated strings in es, fr, ko, ru (#850)

* l10n: refresh stale and untranslated strings in es, fr, ko, ru

- mode_added_art_hint: now mentions Steam then SteamGridDB (all four locales).
- wincomp_needs_installer / wincomp_installers_note: es and ko still described
  the old "coming soon" behaviour; align with the current English text.
- summary_remove_game: was still English in all four locales.

Machine-assisted translation; native-speaker review is welcome.

* l10n(es): say registro, not grabación, for the missing installer record

---------

Co-authored-by: ovurrsl <ovurrsl@users.noreply.github.com>

* fix: a failed cloud save recovery no longer ends the app (#846)

The cloud-recovery thread started at every app start had nothing to
catch an error, so one went to CrashHandler, which halts the process
with status 2. In unit tests the thread can outlive the test that
started it, fail on the torn-down preferences, and take the whole test
run down. The error is now logged and the thread ends.

Co-authored-by: Kirk <12691869+KirkPatrickJunsay@users.noreply.github.com>

* fix: an interrupted Linux runtime download resumes instead of starting again (#847)

* fix: an interrupted Linux runtime download resumes instead of starting again

installOnce() asked the downloader to resume but always deleted the archive in its finally block, so the next attempt began at zero. The archive is now named after the release version and checksum, a failed download keeps what it fetched, and archives of other releases (and the old fixed name) are removed. A checksum or extraction failure still deletes the file.

* test: Downloader resumes a cut-off download from the bytes it kept

Starts a socket server that promises the whole file, hangs up part way and then answers a Range request, and checks that the second downloadFile call resumes with the right offset and ends with identical bytes.

* fix: a resumed download discards a 206 that does not start where the kept bytes end

Downloader appended every 206 response to the partial file without
looking at Content-Range. A server or proxy that answers a Range request
with a different offset would leave a corrupt file. Check that the range
starts at the kept length; otherwise delete the partial file and start
again without a Range header.

Adds a test with a server that answers the Range request from offset 0.

---------

Co-authored-by: ovurrsl <ovurrsl@users.noreply.github.com>

* l10n: German, French, Spanish, Italian, Portuguese and Russian for the zRAM, reset-settings and store free-space strings (#849)

* l10n: German, French, Spanish, Italian, Portuguese and Russian for the zRAM, reset-settings and store free-space strings

Adds the strings values-de, values-it and values-pt were missing (14 each) and the six that values-fr, values-es and values-ru were missing. Machine-assisted drafts that follow each file's existing register and terms; native review is welcome.

* l10n: correct missing active swap translations

---------

Co-authored-by: ovurrsl <ovurrsl@users.noreply.github.com>
Co-authored-by: xXJsonDeruloXx <danielhimebauch@gmail.com>

* fix: droiddeckctl reaches variant-package builds (recover #842) (#854)

fix: droiddeckctl reaches builds shipped under a variant package

The provider authority and the start Activity's component were fixed to
com.droiddeck.launcher, so droiddeckctl could not talk to the antutu,
ludashi or pubg builds from tools/release/variants.txt. DROIDDECK_PACKAGE
now picks the package; the Activity's class name stays the same in every
variant.

Co-authored-by: angelrenefm <17342133+angelrenefm@users.noreply.github.com>

* feat: droiddeckctl reports on-screen pacing (recover #843) (#855)

feat: droiddeckctl pacing reports the frames that reached the screen

Reads SurfaceFlinger's timestats for the game's layer (or the session's
SurfaceView) over a capture: frames on screen, fps on screen, the
present-to-present histogram and how many intervals were off the usual
cadence. Records split by a refresh-rate change are summed. A game's own fps
counter counts frames handed over, so it can't show dropped or doubled frames.

Co-authored-by: angelrenefm <17342133+angelrenefm@users.noreply.github.com>

* fix: Steam can restart more than twice in a session (#754)

* fix: Steam can restart more than twice in a session

Every exit 42 counted toward the cap that stops an update which will
not install, and the count was never cleared, so the third restart of
a session ended it however far apart they were. A client that ran for
a minute before asking is not that loop: the count starts again.

* fix: measure the client's run time as it exits

The time spent collecting logs and checking FEX after the client exits
counted toward its run, so a slow cleanup could make a quick exit 42
look like a long one and clear the count.

---------

Co-authored-by: Kirk <12691869+KirkPatrickJunsay@users.noreply.github.com>
Co-authored-by: Kurt Himebauch <136133082+xXJSONDeruloXx@users.noreply.github.com>

* fix: the front end keeps its page when the Fold is folded or unfolded (#832)

* fix: the front end keeps its page when the Fold is folded or unfolded

MainActivity, FileManagerActivity, FilePickerActivity and StoreLoginActivity did not declare the screen-size configuration changes, so folding or unfolding a foldable recreated them and dropped their in-memory state. They now handle orientation, screenSize, screenLayout, smallestScreenSize and density themselves, like SessionActivity already does. uiMode is left out on purpose.

* fix: store login keeps rebuilding its toolbar when the display density changes

StoreLoginActivity builds its toolbar in code with dp-to-pixel values taken at creation time and has no onConfigurationChanged. It no longer declares density in configChanges, so a density change recreates it as before this PR. Size and layout changes are still handled in place.

---------

Co-authored-by: ovurrsl <ovurrsl@users.noreply.github.com>

* Release pointer buttons without replaying stale motion (#803)

* Fix cursor warping on native touch down

* Preserve input cleanup without replaying stale pointer motion

---------

Co-authored-by: siyuan.qsy <siyuan.qsy@alibaba-inc.com>

* l10n: translate the strings added since #284 into zh-CN, zh-HK, zh-TW and ja (#539)

* l10n: translate the 184 strings added since #284 (zh-CN/HK/TW, ja)

The stores, cloud saves, Windows components, Steam touch, desktop
removal and background-downloads strings landed after #284's
translation snapshot, so every one of them fell back to English.
Translate all 184 into Simplified Chinese, Traditional Chinese
(Taiwan and Hong Kong) and Japanese, keeping each locale file's own
vocabulary (工作階段/會話, 解除安裝/卸載, 元件/組件, 設定).

Also fix four Hong Kong strings that kept the simplified 着色器.

* l10n(zh): fix stale and ambiguous strings found in review

Severe: summary_remove_game was untranslated English in all three
files; mode_added_art_hint, wincomp_installers_note and
wincomp_needs_installer still described the pre-#480/#492 English.

Also: distinguish the Setup wizard from Android system settings in
zh-HK/zh-TW (初始設定 / 系統設定 / Android 設定), unify library as
收藏庫, component as 組件 in zh-HK, session as 會話 in zh-HK, account
as 帳戶 in zh-HK, Trending as 趋势/趨勢, Forget as 清除, reword the
re-applied-at-launch, action-failed and already-in-Games strings, and
play as 執行/開啟遊戲 in zh-TW/zh-HK.

* l10n(ja): fix stale and ambiguous strings found in review

---------

Co-authored-by: Dantalian <Dantalian-dev@users.noreply.github.com>
Co-authored-by: xXJsonDeruloXx <danielhimebauch@gmail.com>

* Rebuild and verify Windows VR runtime components

* Rebuild and verify Windows VR runtime components

* Add top, bottom, and centered picture positions for Steam sessions (#871)

* fix: dependency scan ignores folders above the game folder (#859)

* DroidDeck-Proton: get releases from the Droid-Deck org repo

DroidDeck-Proton is now built and released on Droid-Deck/DroidDeck-Proton
(11-9.1 is Latest there), so the Components page, the update check and the
guest installer read releases from the org repo instead of The412Banner's.

* feat: low-latency controller and touch toggles in the session drawer (#808)

* fix: SteamGridDB lookups no longer stick as "no match" after a failure; add a search to the art chooser (#876)

* feat: Steam home wall fills from the signed-in library instead of store picks (#878)

* test: cover GPU family and support classification (#880)

* Fix local builds on Linux failing after the first Comet build (#866)

* feat: sort the store library and installed tabs, and switch between card and list views (#798)

* feat: three-finger tap toggles Steam's on-screen keyboard in game mode (#782)

* fix: keep Quest-patched Turnip as the auto default (#853)

* Fix deploy_local.sh and droiddeckctl not finding a phone connected over wireless debugging (#864)

* fix: keep proot sessions alive when the kernel loses fork events (#457)

* fix(quest): correct desktop and Steam display corruption across sessions (#733)

fix(quest): stabilize desktop and Steam layer presentation

Keep Quest SDR scenes and the black backdrop off the Android Vulkan swapchain regardless of session order or HDR preferences. Only report the first frame after client content is submitted.

Add backdrop lifecycle and first-frame regression coverage. Record the user's successful Quest 3 Test 2 result.

Co-authored-by: xXJsonDeruloXx <danielhimebauch@gmail.com>

* fix: keep SteamVR syscall hooks and report Steam crashes (#905)

* fix: preserve SteamVR syscall hooks

* fix: report Steam crashes through gamescope

* fix: honor glasses pacing and resolution in SteamVR launches

---------

Co-authored-by: connor g <connorsspam1992@gmail.com>
Co-authored-by: Kurt Himebauch <136133082+xXJSONDeruloXx@users.noreply.github.com>
Co-authored-by: Yassine Gherbi <y.gherbi.dev@gmail.com>
Co-authored-by: Resul Övür <92828257+ovurrsl@users.noreply.github.com>
Co-authored-by: ovurrsl <ovurrsl@users.noreply.github.com>
Co-authored-by: Kirk <kirtoy.junsay@gmail.com>
Co-authored-by: Kirk <12691869+KirkPatrickJunsay@users.noreply.github.com>
Co-authored-by: xXJsonDeruloXx <danielhimebauch@gmail.com>
Co-authored-by: angelrenefm <17342133+angelrenefm@users.noreply.github.com>
Co-authored-by: asdf17128 <asdf17128@gmail.com>
Co-authored-by: siyuan.qsy <siyuan.qsy@alibaba-inc.com>
Co-authored-by: Dantalian <wsu091227@qq.com>
Co-authored-by: Dantalian <Dantalian-dev@users.noreply.github.com>
Co-authored-by: The412Banner <the412banner@users.noreply.github.com>
Co-authored-by: The412Banner <205237651+The412Banner@users.noreply.github.com>
Co-authored-by: FX1922 <120828496+FX192@users.noreply.github.com>
Co-authored-by: Muaz Kadan <32595066+muazkadan@users.noreply.github.com>
Co-authored-by: kingokksa <47888451+kingokksa@users.noreply.github.com>
Co-authored-by: Jamie Schenker <jam1esch3nk3r@gmail.com>
Co-authored-by: Jagger15 <jagger2333@live.com>

* Glasses: keep a pending 2D restore and the SteamVR recovery offer, show Play SteamVR only with a picture

* Translate the glasses calibration, XREAL and SteamVR recovery strings

* SteamVR recovery: give a finished teardown time to collect logs before restarting

* Credit the projects the XR glasses support builds on

* Joy-Cons: SteamVR hands from their gyro while SteamVR runs, one combined pad otherwise

* Headsets: settings badge on Play SteamVR instead of a top-right button, both shown only once connected

* Hand tracking: camera-tracked Joy-Cons and bare hands in SteamVR, drawn as white gloves

- Joy-Con movement from the accelerometer (report reordering, per-Joy-Con calibration, zero-velocity updates)
- MediaPipe pose and hand landmarks from the device camera, placement check, per-hand Kalman filter with the Joy-Con
- No controller connected: bare camera hands with Quest-style gestures, 1€ filtered, rendered as a white glove model built from a distance field
- SteamVR safe mode cleared before each start so one crash no longer blocks the driver

---------

Co-authored-by: MaxsTechReview <6025299+maxjivi05@users.noreply.github.com>
Co-authored-by: informalTechCode <informaltechyt@gmail.com>
Co-authored-by: connor g <connorsspam1992@gmail.com>
Co-authored-by: Kurt Himebauch <136133082+xXJSONDeruloXx@users.noreply.github.com>
Co-authored-by: Yassine Gherbi <y.gherbi.dev@gmail.com>
Co-authored-by: Resul Övür <92828257+ovurrsl@users.noreply.github.com>
Co-authored-by: ovurrsl <ovurrsl@users.noreply.github.com>
Co-authored-by: Kirk <kirtoy.junsay@gmail.com>
Co-authored-by: Kirk <12691869+KirkPatrickJunsay@users.noreply.github.com>
Co-authored-by: xXJsonDeruloXx <danielhimebauch@gmail.com>
Co-authored-by: angelrenefm <17342133+angelrenefm@users.noreply.github.com>
Co-authored-by: asdf17128 <asdf17128@gmail.com>
Co-authored-by: siyuan.qsy <siyuan.qsy@alibaba-inc.com>
Co-authored-by: Dantalian <wsu091227@qq.com>
Co-authored-by: Dantalian <Dantalian-dev@users.noreply.github.com>
Co-authored-by: The412Banner <the412banner@users.noreply.github.com>
Co-authored-by: The412Banner <205237651+The412Banner@users.noreply.github.com>
Co-authored-by: FX1922 <120828496+FX192@users.noreply.github.com>
Co-authored-by: Muaz Kadan <32595066+muazkadan@users.noreply.github.com>
Co-authored-by: kingokksa <47888451+kingokksa@users.noreply.github.com>
Co-authored-by: Jamie Schenker <jam1esch3nk3r@gmail.com>
Co-authored-by: Jagger15 <jagger2333@live.com>
This commit is contained in:
221 files changed
+24369 -491

No files matched your search

@@ -16,12 +16,17 @@ class App : Application() {
override fun onCreate() {
super.onCreate()
// The restart screen must not initialize USB, guest processes, or another graphics context.
val process = (getSystemService(ACTIVITY_SERVICE) as android.app.ActivityManager)
.runningAppProcesses?.firstOrNull { it.pid == android.os.Process.myPid() }?.processName
if (process == "$packageName:vr_recovery") return
CrashHandler.install(this)
com.droiddeck.launcher.session.QuestExperimental.initialize(this)
com.droiddeck.launcher.session.SessionPrefs.settleGpuDriverMode(this)
com.droiddeck.launcher.update.AppUpdates.init(this)
com.droiddeck.launcher.core.DownloadSource.init(this)
GpuClockPin.clearLeftover(this)
com.droiddeck.launcher.glasses.Glasses.init(this)
BwrapSpawner.start(this)
com.droiddeck.launcher.frontend.GameFileSync.start(this)
com.droiddeck.launcher.stores.StoresState.init(this)
@@ -192,6 +192,10 @@ class MainActivity : ComponentActivity() {
SessionPrefs.setWifiDiscoveryEnabled(this, WifiDiscovery.permissionGranted(this))
refreshWifiDiscovery()
}
private val cameraPermission = registerForActivityResult(ActivityResultContracts.RequestPermission()) {
com.droiddeck.launcher.handtracking.HandTracking.refreshPermission(this)
}
private var placementCheck by mutableStateOf<Boolean?>(null)
private val wifiPermissionSettings = registerForActivityResult(ActivityResultContracts.StartActivityForResult()) {
SessionPrefs.setWifiDiscoveryEnabled(this, WifiDiscovery.permissionGranted(this))
refreshWifiDiscovery()
@@ -507,19 +511,27 @@ class MainActivity : ComponentActivity() {
android.widget.Toast.makeText(this, R.string.game_link_session_busy, android.widget.Toast.LENGTH_LONG).show()
return true
}
if (!SessionService.launchGame(this, game.gameIdString)) {
val steamVr = game.gameIdString == com.droiddeck.launcher.gpu.SteamVr.APP_ID
if (!steamVr && !SessionService.launchGame(this, game.gameIdString)) {
android.widget.Toast.makeText(this, R.string.game_link_failed, android.widget.Toast.LENGTH_LONG).show()
return false
}
startActivity(Intent(this, SessionActivity::class.java).setAction(SessionService.ACTION_RESUME)
.putExtra(SessionActivity.EXTRA_STEAMVR, steamVr)
.putExtra(SessionActivity.EXTRA_RETURN_HOME, returnHome)
.addFlags(Intent.FLAG_ACTIVITY_CLEAR_TOP or Intent.FLAG_ACTIVITY_SINGLE_TOP))
return true
}
return startSession(Intent(this, SessionActivity::class.java)
.putExtra(SessionActivity.EXTRA_RETURN_HOME, returnHome)
.putExtra(SessionService.EXTRA_STEAM_URL,
com.droiddeck.launcher.frontend.GameLaunchLink.steamUrl(game.gameIdString)), steamSession = true)
.apply {
if (game.gameIdString == com.droiddeck.launcher.gpu.SteamVr.APP_ID) {
putExtra(SessionActivity.EXTRA_STEAMVR, true)
} else {
putExtra(SessionService.EXTRA_STEAM_URL,
com.droiddeck.launcher.frontend.GameLaunchLink.steamUrl(game.gameIdString))
}
}, steamSession = true)
}
private fun handleGameLink(games: List<Library.SteamGame>) {
@@ -636,6 +648,7 @@ class MainActivity : ComponentActivity() {
),
FrontEndActions(
onPlay = { playSteam() },
onPlaySteamVr = { playSteamVr() },
onSteamUrl = { url ->
if (!SessionState.running) startSession(
Intent(this, SessionActivity::class.java).putExtra(SessionService.EXTRA_STEAM_URL, url), steamSession = true,
@@ -886,6 +899,7 @@ class MainActivity : ComponentActivity() {
onQamButton = { on -> ControllerPrefs.setQamButton(this, on); refreshController() },
onKeyboardButton = { on -> ControllerPrefs.setKeyboardButton(this, on); refreshController() },
onSelectSteam = { on -> ControllerPrefs.setSelectSteam(this, on); refreshController() },
onJoyConVr = { on -> ControllerPrefs.setJoyConVr(this, on); refreshController() },
onEditLayout = { startActivity(Intent(this, ControllerEditorActivity::class.java)) },
onResetLayout = { ControllerPrefs.resetAllLayouts(this); refreshController() },
onMapping = { showPerformance = false; showComponents = false; showMapping = true },
@@ -928,6 +942,12 @@ class MainActivity : ComponentActivity() {
onDismiss = { lowSpace = null },
)
}
com.droiddeck.launcher.ui.HandCameraHost(
check = placementCheck,
onAskPermission = { cameraPermission.launch(Manifest.permission.CAMERA) },
onPlay = { placementCheck = null; startSteamSession(steamVr = true) },
onClose = { placementCheck = null },
)
if (showResetSettings) com.droiddeck.launcher.ui.ResetSettingsDialog(
onConfirm = { gameProfiles -> resetSettings(gameProfiles) },
onDismiss = { showResetSettings = false },
@@ -988,8 +1008,13 @@ class MainActivity : ComponentActivity() {
} else startSteamSession()
}
private fun startSteamSession() {
startSession(Intent(this, SessionActivity::class.java), steamSession = true)
private fun playSteamVr() {
if (com.droiddeck.launcher.handtracking.HandTracking.wanted(this)) placementCheck = true
else startSteamSession(steamVr = true)
}
private fun startSteamSession(steamVr: Boolean = false) {
startSession(Intent(this, SessionActivity::class.java).putExtra(SessionActivity.EXTRA_STEAMVR, steamVr), steamSession = true)
}
override fun onWindowFocusChanged(hasFocus: Boolean) {
@@ -26,7 +26,7 @@ class QuestXrActivity : NativeActivity() {
if (active) {
check(CompositorHost.isStarted) { "DroidDeck compositor is not running" }
SessionService.resume(this)
WaylandCompositor.nativeSetCardboard(false, 1, 0.064f, 3f, 4f, 0f, 0f, 90f, 0f, 0f)
WaylandCompositor.nativeSetCardboard(false, 1, 0.064f, 3f, 4f, 0f, 0f, 90f, 0f, 0f, false, 0f)
WaylandCompositor.nativeSetFrameGenArmed(false, 0, 0)
WaylandCompositor.nativeSetZeroCopy(false)
CompositorHost.attachXr()
@@ -52,6 +52,11 @@ import com.droiddeck.launcher.ui.SessionAchievements
import com.droiddeck.launcher.gpu.Cardboard
import com.droiddeck.launcher.gpu.FrameGen
import com.droiddeck.launcher.gpu.HeadTracker
import com.droiddeck.launcher.glasses.Glasses
import com.droiddeck.launcher.glasses.GlassesScreen
import com.droiddeck.launcher.glasses.GlassesRendering
import kotlinx.coroutines.cancel
import kotlinx.coroutines.launch
import com.droiddeck.launcher.gpu.SteamVr
import com.droiddeck.launcher.gpu.ScreenEffectLooks
import com.droiddeck.launcher.gpu.ScreenEffects
@@ -141,6 +146,8 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
private lateinit var loading: LoadingState
private lateinit var hud: PerfHud
private var padBridge: PadBridge? = null
private var joyConVr: com.droiddeck.launcher.input.JoyConVr? = null
private var handCamera: com.droiddeck.launcher.handtracking.HandCamera? = null
private var padMotion: com.droiddeck.launcher.input.PadMotion? = null
private var resumed = false
private val deckPadListener: () -> Unit = { uiHandler.post { updatePadMotion(); openDeckRing("deck pad changed"); applyUnfoldedControls() } }
@@ -278,6 +285,23 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
private var steamVrStatus by mutableStateOf<SteamVr.Status?>(null)
private var steamVrSwitchedPicture by mutableStateOf(false)
private var steamVrSeenRunning = false
private val steamVrRecovery by lazy { com.droiddeck.launcher.gpu.SteamVrRecovery(this) {
resumed && !ended && SessionState.running && !isInPictureInPictureMode
} }
private var activeCardboard = Cardboard.OFF
private var glasses by mutableStateOf(Glasses.state.value)
private var glassesSettings by mutableStateOf(Glasses.settings.value)
private var glassesScreen: GlassesScreen? = null
private var glassesOutputActive by mutableStateOf(false)
private var glasses3dForVr = false
private var ended = false
private var glassesPresent = false
private var steamVrAutoTried = false
private var steamVrInstallTried = false
private var headTracking by mutableStateOf(false)
private var glassesSlot by mutableStateOf(0)
private var glassesKey: List<Any?> = emptyList()
private val glassesScope = kotlinx.coroutines.MainScope()
private val hasRotationSensor by lazy {
(getSystemService(SENSOR_SERVICE) as android.hardware.SensorManager).let {
it.getDefaultSensor(android.hardware.Sensor.TYPE_GAME_ROTATION_VECTOR)
@@ -318,10 +342,10 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
private fun followPanelRefresh(displayId: Int) {
if (!CompositorHost.isStarted) return
val display = (if (Build.VERSION.SDK_INT >= 30) display else windowManager.defaultDisplay) ?: return
if (display.displayId != displayId) return
val hz = display.refreshRate
if (hz > 1f) WaylandCompositor.nativeSetOutputRefreshRate(hz)
val output = glassesScreen?.display
?: (if (Build.VERSION.SDK_INT >= 30) display else windowManager.defaultDisplay) ?: return
if (output.displayId != displayId) return
applyOutputPacing()
}
private var isHomeApp by mutableStateOf(false)
private var androidApps by mutableStateOf<List<HomeApp.LaunchableApp>>(emptyList())
@@ -329,17 +353,24 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
private val deviceListener = object : InputManager.InputDeviceListener {
override fun onInputDeviceAdded(deviceId: Int) {
logInputDevice("connected", deviceId)
joyConVr?.refresh()
padBridge?.syncConnectedControllers()
refreshControls()
updatePointerCapture()
updateHandCamera()
}
override fun onInputDeviceRemoved(deviceId: Int) {
Log.i(TAG, "input device $deviceId disconnected")
joyConVr?.refresh()
padBridge?.onDeviceRemoved(deviceId)
refreshControls()
updatePointerCapture()
updateHandCamera()
}
override fun onInputDeviceChanged(deviceId: Int) {
joyConVr?.refresh()
refreshControls()
}
override fun onInputDeviceChanged(deviceId: Int) = refreshControls()
}
private var runtimeRemovalBlocked = false
@@ -363,17 +394,21 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
SessionState.programArgs = intent.getStringArrayExtra(SessionService.EXTRA_PROGRAM_ARGS)?.toList().orEmpty()
SessionState.steamUi = intent.getStringExtra(SessionService.EXTRA_STEAM_UI)
SessionState.steamUrl = intent.getStringExtra(SessionService.EXTRA_STEAM_URL)
SessionState.steamVr = intent.getBooleanExtra(EXTRA_STEAMVR, false)
} else if (intent.getBooleanExtra(EXTRA_STEAMVR, false) && SessionState.mode == SessionService.MODE_STEAM) {
SessionState.steamVr = true
}
if (intent.action == SessionService.ACTION_AGENT_START) {
SessionEvents.record("agent.start_requested", mapOf("mode" to SessionState.mode))
}
com.droiddeck.launcher.gpu.SteamVrHealth.begin(this, SessionState.sessionId)
window.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
if (!SessionState.running) {
SessionState.fpsLimit = SessionPrefs.fpsLimit(
this, SessionPrefs.prefMode(intent.getStringExtra(SessionService.EXTRA_MODE) ?: SessionService.MODE_STEAM),
)
}
Log.i(TAG, "perf: " + PerfMode.apply(this, SessionState.fpsLimit))
applyOutputPacing()
if (!SessionState.running) SessionState.followScreen = followChoice()
applyFollowOrientation(resources.configuration)
goFullscreen()
@@ -403,6 +438,16 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
bridge.setSelectSteam(ControllerPrefs.read(this).selectSteam)
padBridge = bridge
SessionState.padBridge = bridge
if (Build.VERSION.SDK_INT >= 31) {
val vr = com.droiddeck.launcher.input.JoyConVr(this, File(filesDir, ".wayland-rt/${com.droiddeck.launcher.input.JoyConVr.FILE}")) { taken, released ->
if (joyConVr == null) return@JoyConVr
taken.forEach(bridge::onDeviceRemoved)
if (released.isNotEmpty()) bridge.syncConnectedControllers()
updateHandCamera()
}
bridge.setExcluded(vr::owns)
joyConVr = vr
}
SteamControllerBle.listener = { uiHandler.post { updatePadMotion(); refreshControls() } }
steamController = SteamControllerBle(this).also { it.start() }
padMotion = com.droiddeck.launcher.input.PadMotion(this) {
@@ -450,6 +495,21 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
sessionOverlay = createSessionOverlay()
root.addView(sessionOverlay)
setContentView(root)
glassesScope.launch { Glasses.state.collect { glasses = it; onGlassesChanged() } }
glassesScope.launch { Glasses.settings.collect { glassesSettings = it; onGlassesChanged() } }
glassesScope.launch { com.droiddeck.launcher.handtracking.HandTracking.settings.collect { updateHandCamera() } }
glassesScope.launch {
while (true) {
if (SessionState.running && SessionState.mode == SessionService.MODE_STEAM &&
(SessionState.steamVr || steamVrStatus == SteamVr.Status.RUNNING)) {
val lost = kotlinx.coroutines.withContext(kotlinx.coroutines.Dispatchers.IO) {
com.droiddeck.launcher.gpu.SteamVrHealth.deviceLost()
}
if (lost) steamVrRecovery.detected()
}
kotlinx.coroutines.delay(2000)
}
}
pip = com.droiddeck.launcher.session.SessionPipController(this, ::pipBounds, ::showPipUi)
surfaceView.addOnLayoutChangeListener { _, _, _, _, _, _, _, _, _ -> pip.refresh() }
pip.refresh()
@@ -496,6 +556,15 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
CursorOverlay(cursorPos, cursorVisible, resources.displayMetrics.density,
cursorImage, cursorHotX, cursorHotY, cursorImageScale)
if (hud.text.isNotEmpty()) HudText(hud.text)
if (glassesOutputActive && !loading.visible) com.droiddeck.launcher.ui.GlassesPlayingHint(glasses.model)
if (GlassesRendering.usesGlasses(glasses, glassesOutputActive, steamVrStatus == SteamVr.Status.RUNNING) &&
headTracking && !drawerOpen && !pcKeyboardOpen && !loading.visible) com.droiddeck.launcher.ui.GlassesRecenterButton(
controllerSettings?.tint ?: android.graphics.Color.rgb(90, 160, 255), controllerSettings?.opacity ?: 100) { headTracker?.recenter() }
if (glasses.present && !drawerOpen && !pcKeyboardOpen && !loading.visible) com.droiddeck.launcher.ui.GlassesCornerButton(glassesSlot,
controllerSettings?.tint ?: android.graphics.Color.rgb(90, 160, 255), controllerSettings?.opacity ?: 100) {
drawerPage = com.droiddeck.launcher.ui.SessionDrawerPage.GLASSES
drawerOpen = true
}
if (loading.visible) LoadingOverlay(
loading.step, loading.percent, loading.elapsed, loading.hint, loading.ended,
topic = loading.topic, readable = loading.readable,
@@ -514,7 +583,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
androidx.compose.runtime.LaunchedEffect(drawerOpen) {
if (drawerOpen) {
syncLossless()
padBridge?.releaseAll()
releasePads()
androidx.compose.runtime.withFrameNanos { }
val requested = sessionOverlay.requestFocus()
Log.i(TAG, "drawer root focus requested=$requested focused=${sessionOverlay.hasFocus()}")
@@ -522,8 +591,12 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
releaseDrawerDirection()
}
}
LaunchedEffect(cardboard.enabled, steamVrSwitchedPicture, drawerOpen, drawerPage) {
while (cardboard.enabled || steamVrSwitchedPicture || (drawerOpen && drawerPage == SessionDrawerPage.VR)) {
val glassesVr = glasses.usable && glasses.canControl
val steamVrRequested = SessionState.steamVr
val pollSteamVr = SteamVr.shouldPoll(steamVrStatus, cardboard.enabled,
steamVrSwitchedPicture, glassesVr, steamVrRequested) || (drawerOpen && drawerPage == SessionDrawerPage.VR)
LaunchedEffect(pollSteamVr) {
while (pollSteamVr) {
val now = kotlinx.coroutines.withContext(kotlinx.coroutines.Dispatchers.IO) {
SteamVr.status(this@SessionActivity)
}
@@ -540,7 +613,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
achievementsInFront = found
if (found == null) achievementsOpen = false
}
androidx.compose.runtime.LaunchedEffect(pcKeyboardOpen) { if (pcKeyboardOpen) padBridge?.releaseAll() }
androidx.compose.runtime.LaunchedEffect(pcKeyboardOpen) { if (pcKeyboardOpen) releasePads() }
if (pcKeyboardOpen) com.droiddeck.launcher.ui.PcKeyboard(
sendKey = { code, down -> if (CompositorHost.isStarted) sendGuestKey(code, down) },
onAndroidKeyboard = { pcKeyboardOpen = false; keyboard?.toggle() },
@@ -559,6 +632,9 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
cardboardSensor = hasRotationSensor,
onCardboard = { c -> SessionPrefs.setCardboard(this@SessionActivity, c); applyCardboard(c) },
onCardboardRecenter = { headTracker?.recenter() },
glassesPresent = glasses.present,
glassesOutput = glassesOutputActive,
onGlassesRecenter = { headTracker?.recenter() },
steamVr = steamVrStatus,
immersiveSteamVr = com.droiddeck.launcher.session.QuestExperimental.isSupported(),
onSteamVr = { action -> steamVrAction(action) },
@@ -747,7 +823,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
))
LaunchedEffect(achievementsOpen) {
if (achievementsOpen) {
padBridge?.releaseAll()
releasePads()
withFrameNanos { }
sessionOverlay.requestFocus()
}
@@ -1224,7 +1300,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
loading.percent = -1
loading.say(getString(R.string.session_starting_session), LoadingState.Topic.SESSION, readable = false)
if (surfaceView.holder.surface?.isValid == true) surfaceCreated(surfaceView.holder)
outputHolder().let { if (it.surface?.isValid == true) surfaceCreated(it) }
}
}, "runtime-install").start()
}
@@ -1288,7 +1364,8 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
override fun surfaceCreated(holder: SurfaceHolder) {
if (installingRuntime || SessionState.stopRequested) return
if (holder !== outputHolder()) return
if (installingRuntime || SessionState.stopRequested || ended) return
if (!SessionState.running) {
SessionEvents.transition(SessionPhase.STARTING_COMPOSITOR, "compositor.starting")
}
@@ -1304,7 +1381,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
val live = if (SessionState.followScreen && !isInPictureInPictureMode)
com.droiddeck.launcher.session.SessionDisplay.followSize(holder.surfaceFrame.width(), holder.surfaceFrame.height())
else null
val size = live ?: if (SessionState.running) previous else outputSize()
val size = live ?: if (SessionState.running) previous else glassesOutputSize() ?: outputSize()
SessionState.outputSize = size
if (!SessionState.running) SessionState.refreshHz = refreshHz()
val panel = panelBounds()
@@ -1317,7 +1394,8 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
WaylandCompositor.nativeSetUpscaler(SessionPrefs.upscaler(this))
WaylandCompositor.nativeSetUpscaleSharpness(SessionPrefs.upscaleSharpness(this))
SessionPrefs.screenEffects(this).push(SessionPrefs.upscaler(this))
panelSizeMm().let { (w, h) -> SessionPrefs.cardboard(this).push(w, h) }
activeCardboard = effectiveCardboard(SessionPrefs.cardboard(this))
pushCardboard(activeCardboard)
WaylandCompositor.nativeSetCardboardPoseFile(File(filesDir, ".wayland-rt/droiddeck-headpose").path)
if (!SessionState.running) {
val waylandLog = File(SessionPaths.beginOrCurrent(this), "wayland.log")
@@ -1330,7 +1408,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
if (!CompositorHost.isStarted) {
val mode = SessionPrefs.prefMode(intent.getStringExtra(SessionService.EXTRA_MODE) ?: SessionService.MODE_STEAM)
val probe = HdrSupport.probe(this)
val probe = HdrSupport.probe(this, glassesScreen?.display)
WaylandCompositor.nativeSetHdrDisplay(
probe.displayId, probe.name, probe.formats, probe.hdr10,
probe.maxLuminance, probe.maxAverageLuminance, probe.minLuminance,
@@ -1354,8 +1432,9 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
CompositorHost.startOrAttach(
holder.surface, runtimeDir.path,
driverId?.let { turnip.driverPath(it) }, driverId?.let { turnip.libraryName(it) },
applicationInfo.nativeLibraryDir, size.first, size.second, refreshHz(), SessionState.fpsLimit,
applicationInfo.nativeLibraryDir, size.first, size.second, refreshHz(), outputFpsLimit(),
)
onOutputAttached(holder)
if (SessionState.running && size != previous) {
Log.i(TAG, "follow screen: ${previous.first}x${previous.second} -> ${size.first}x${size.second} on a new surface")
WaylandCompositor.nativeResizeOutput(size.first, size.second)
@@ -1379,6 +1458,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
private var surfaceH = 0
override fun surfaceChanged(holder: SurfaceHolder, format: Int, width: Int, height: Int) {
if (holder !== outputHolder()) return
val resized = surfaceW != 0 && (width != surfaceW || height != surfaceH)
surfaceW = width
surfaceH = height
@@ -1422,8 +1502,12 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
override fun surfaceDestroyed(holder: SurfaceHolder) {
// Hiding the phone surface after a handoff delivers a late destruction
// callback. It must not reset the new output's size or attachment.
if (holder !== outputHolder()) return
surfaceW = 0
surfaceH = 0
glassesOutputActive = false
CompositorHost.detach(holder.surface)
}
@@ -1448,14 +1532,32 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
private fun refreshHz(): Float {
glassesScreen?.hz?.takeIf { it > 1f }?.let { return it }
val display = if (Build.VERSION.SDK_INT >= 30) display else windowManager.defaultDisplay
val wanted = window.attributes.preferredDisplayModeId
val hz = display?.supportedModes?.firstOrNull { wanted != 0 && it.modeId == wanted }?.refreshRate
?: display?.refreshRate ?: 60f
val capped = minOf(hz, PerfMode.peakRefreshCap(this))
val mirrorCap = mirrorRefreshLimit().takeIf { it > 0 }?.toFloat() ?: Float.POSITIVE_INFINITY
val capped = minOf(hz, PerfMode.peakRefreshCap(this), mirrorCap)
return if (capped > 1f) capped else 60f
}
private fun mirrorRefreshLimit(): Int =
GlassesRendering.mirrorRefreshLimit(glassesSettings, glasses, glassesScreen != null)
private fun outputFpsLimit(): Int = GlassesRendering.fpsLimit(SessionState.fpsLimit, mirrorRefreshLimit())
private fun applyOutputPacing() {
Log.i(TAG, "perf: " + PerfMode.apply(this, outputFpsLimit(), mirrorRefreshLimit()))
if (CompositorHost.isStarted) {
val hz = refreshHz()
val changed = kotlin.math.abs(SessionState.refreshHz - hz) > 0.1f
CompositorHost.updatePacing(hz, outputFpsLimit())
pushCardboard(activeCardboard)
if (changed && ::surfaceView.isInitialized) applyFrameGen()
}
}
private fun applyFrameGen() {
val hz = refreshHz()
Thread({
@@ -1514,11 +1616,14 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
private fun onSessionEnded(status: Int) {
ended = true
if (isInPictureInPictureMode) {
runOnUiThread { finish() }
return
}
closeSecondScreen(reset = true)
updateGlassesScreen()
updateHeadTracker()
if (status == 0) {
runOnUiThread {
if (isFinishing || isDestroyed) return@runOnUiThread
@@ -1560,9 +1665,14 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
focusEndedScreen()
}
private fun releasePads() {
padBridge?.releaseAll()
joyConVr?.releaseButtons()
}
private fun focusEndedScreen() {
drawerOpen = false
padBridge?.releaseAll()
releasePads()
uiHandler.post { sessionOverlay.requestFocus() }
}
@@ -1610,6 +1720,8 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
return super.dispatchKeyEvent(event)
}
if (pipUi) return true
val pad = com.droiddeck.launcher.input.JoyCon.normalize(event)
if (pad !== event) return dispatchKeyEvent(pad)
val fromController = PadBridge.isControllerKeyEvent(event)
if (fromController && event.action == KeyEvent.ACTION_DOWN) {
if (drawerOpen && !drawerControllerActive) sessionOverlay.requestFocus()
@@ -1632,7 +1744,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
if (drawerOpen) {
if (fromController && (event.keyCode == KeyEvent.KEYCODE_BUTTON_L1 || event.keyCode == KeyEvent.KEYCODE_BUTTON_R1)) {
if (event.action == KeyEvent.ACTION_DOWN && event.repeatCount == 0) {
drawerPage = sessionDrawerPages(SessionState.mode == SessionService.MODE_STEAM && secondScreenDisplays.isNotEmpty())
drawerPage = sessionDrawerPages(SessionState.mode == SessionService.MODE_STEAM && secondScreenDisplays.isNotEmpty(), glasses.present)
.step(drawerPage, if (event.keyCode == KeyEvent.KEYCODE_BUTTON_R1) 1 else -1)
releaseDrawerDirection()
}
@@ -1661,6 +1773,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
return true
}
if ((pcKeyboardOpen || achievementsOpen) && fromController) return super.dispatchKeyEvent(event)
if (joyConVr?.onKeyEvent(event) == true) return true
if (event.keyCode != KeyEvent.KEYCODE_BACK && padBridge?.onKeyEvent(event) == true) return true
if (CompositorHost.isStarted && event.keyCode != KeyEvent.KEYCODE_BACK && !fromController) {
val down = event.action == KeyEvent.ACTION_DOWN
@@ -1704,6 +1817,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
if (drawerDirectionKey != KeyEvent.KEYCODE_UNKNOWN) releaseDrawerDirection()
if ((pcKeyboardOpen || achievementsOpen) && PadBridge.isControllerMotionEvent(event)) return super.dispatchGenericMotionEvent(event)
if (joyConVr?.onMotionEvent(event) == true) return true
if (padBridge?.onMotionEvent(event) == true) return true
if (event.isFromSource(android.view.InputDevice.SOURCE_MOUSE) && !drawerOpen && onMouse(event)) return true
return super.dispatchGenericMotionEvent(event)
@@ -1868,6 +1982,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
if (!::displayManager.isInitialized) return
val primaryId = display?.displayId ?: windowManager.defaultDisplay.displayId
val candidates = SecondScreenDisplays.available(displayManager, primaryId)
.filter { glassesScreen == null || it.id != glasses.displayId }
val oldIds = secondScreenDisplays.map { it.id }.toSet()
secondScreenDisplays = candidates
val newIds = candidates.map { it.id }.toSet()
@@ -2219,6 +2334,17 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
private fun updateOnScreenControls() {
updateOnScreenPad()
val c = onScreenControls
glassesSlot = when {
c == null || c.visibility != View.VISIBLE -> 0
c.isButtonsOnly -> -1
controllerSettings?.keyboardButton != false -> 2
else -> 1
}
}
private fun updateOnScreenPad() {
val forced = File(Environment.getExternalStorageDirectory(), "Download/droiddeck-osc")
.takeIf { it.isFile }
?.let { FileUtils.readString(it)?.trim()?.lowercase() }
@@ -2301,7 +2427,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
pendingBackAction = null
releaseDrawerDirection()
onScreenControls?.releaseAll()
padBridge?.releaseAll()
releasePads()
touchpad.cancel()
guestKeysDown.toList().forEach { sendGuestKey(it, false) }
keyboard?.hide()
@@ -2337,6 +2463,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
if (::pip.isInitialized) pip.changed(inPip)
if (!inPip && surfaceW > 0) followWindow(surfaceW, surfaceH)
applyUnfoldedControls()
updateGlassesScreen()
}
override fun startActivityForResult(intent: Intent, requestCode: Int, options: Bundle?) {
@@ -2355,13 +2482,18 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
refreshSecondScreenDisplays()
if (secondScreenMode != SecondScreenMode.NONE && secondScreenPresentation == null) showSecondScreen(secondScreenMode)
SessionService.setActivityVisible(this, true)
Glasses.rescan()
updateGlassesScreen()
updateHandCamera()
}
override fun onStop() {
if (runtimeRemovalBlocked) { super.onStop(); return }
started = false
updateHandCamera()
displayManager.unregisterDisplayListener(displayListener)
closeSecondScreen(reset = false)
updateGlassesScreen()
SessionService.setActivityVisible(this, false)
super.onStop()
}
@@ -2379,6 +2511,10 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
if (this.intent.getBooleanExtra(EXTRA_RETURN_HOME, false)) intent.putExtra(EXTRA_RETURN_HOME, true)
setIntent(intent)
if (intent.getBooleanExtra(EXTRA_STEAMVR, false) && SessionState.mode == SessionService.MODE_STEAM) {
SessionState.steamVr = true
steamVrAutoTried = false
}
if (intent.action == SessionService.ACTION_HOME_GUIDE) {
handleHomeGuideIntent(intent)
} else if (intent.action == SessionService.ACTION_RESUME) {
@@ -2400,6 +2536,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
(getSystemService(INPUT_SERVICE) as? InputManager)
?.registerInputDeviceListener(deviceListener, Handler(Looper.getMainLooper()))
if (SessionState.guestPid > 1) padBridge?.syncConnectedControllers()
joyConVr?.refresh()
onScreenControls?.reload()
controllerSettings = ControllerPrefs.read(this)
resumed = true
@@ -2436,13 +2573,32 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
startActivity(android.content.Intent(this, QuestXrActivity::class.java))
return
}
if (action == "start") {
SessionState.steamVr = true
steamVrAutoTried = true
}
if (action == "stop") SessionState.steamVr = false
if (glasses.present) {
// Optional stereo starts only after RUNNING is observed. READY can
// arrive repeatedly while the launch worker is still starting VR.
if (action == "install") steamVrStatus = SteamVr.Status.INSTALLING
steamVrWorker.execute {
val ok = SteamVr.run(this, action)
val now = SteamVr.status(this)
uiHandler.post {
onSteamVrStatus(now)
if (!ok) Toast.makeText(this, getString(R.string.cardboard_steamvr_failed), Toast.LENGTH_LONG).show()
}
}
return
}
if (action == "start") {
switchPictureForSteamVr()
steamVrSeenRunning = false
}
if (action == "stop") restoreSteamVrPicture()
if (action == "install") steamVrStatus = SteamVr.Status.INSTALLING
Thread({
steamVrWorker.execute {
val ok = SteamVr.run(this, action)
val now = SteamVr.status(this)
uiHandler.post {
@@ -2450,7 +2606,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
onSteamVrStatus(now)
if (!ok) Toast.makeText(this, getString(R.string.cardboard_steamvr_failed), Toast.LENGTH_LONG).show()
}
}, "steamvr-$action").start()
}
}
private fun switchPictureForSteamVr() {
@@ -2463,8 +2619,41 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
steamVrSwitchedPicture = SessionPrefs.cardboardSourceBeforeSteamVr(this) >= 0
}
private fun updateHandCamera() {
val joyCons = joyConVr?.holding == true
val bare = !joyCons && !controllerConnected()
val want = started && steamVrStatus == SteamVr.Status.RUNNING && (joyCons || bare) &&
com.droiddeck.launcher.handtracking.HandTracking.wanted(this)
handCamera?.bareHands = bare
if (want == (handCamera != null)) return
if (want) {
handCamera = com.droiddeck.launcher.handtracking.HandCamera(this,
File(filesDir, ".wayland-rt/${com.droiddeck.launcher.handtracking.HandCameraShm.FILE}"), preview = false)
.also {
it.bareHands = bare
it.start(com.droiddeck.launcher.handtracking.HandTracking.settings.value.side)
}
} else {
handCamera?.stop()
handCamera = null
}
}
private fun controllerConnected(): Boolean = InputDevice.getDeviceIds().any { id ->
joyConVr?.owns(id) != true && com.droiddeck.launcher.input.PadBridge.isFromController(InputDevice.getDevice(id))
}
private fun onSteamVrStatus(now: SteamVr.Status) {
val was = steamVrStatus
steamVrStatus = now
joyConVr?.setActive(now == SteamVr.Status.RUNNING && ControllerPrefs.joyConVr(this))
updateHandCamera()
autoStartSteamVr(now)
if ((was == SteamVr.Status.RUNNING) != (now == SteamVr.Status.RUNNING)) applyCardboard(cardboard)
if (glasses.present) onGlassesSteamVr(now)
if (GlassesRendering.usesGlasses(glasses, glassesScreen != null, now == SteamVr.Status.RUNNING)) {
return
}
if (now == SteamVr.Status.RUNNING && cardboard.enabled && !steamVrSwitchedPicture &&
cardboard.source != Cardboard.SOURCE_SBS) {
switchPictureForSteamVr()
@@ -2487,23 +2676,172 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
private fun applyCardboard(c: Cardboard) {
val was = cardboard
val was = activeCardboard
cardboard = c
if (CompositorHost.isStarted) panelSizeMm().let { (w, h) -> c.push(w, h) }
if (c.enabled && !was.enabled) headTracker?.recenter()
val e = effectiveCardboard(c)
activeCardboard = e
if (CompositorHost.isStarted) pushCardboard(e)
if (e.enabled && !was.enabled) headTracker?.recenter()
updateHeadTracker()
}
private fun pushCardboard(c: Cardboard) {
val fov = when {
c.viewer != Cardboard.Viewer.VITURE.id -> 0f
c.source == Cardboard.SOURCE_SBS -> glassesSettings.vrFovDeg.toFloat()
else -> glasses.verticalFovDeg
}
panelSizeMm().let { (w, h) -> c.push(w, h, fov, refreshHz()) }
}
private fun effectiveCardboard(c: Cardboard): Cardboard = GlassesRendering.cardboard(
base = c,
settings = glassesSettings,
state = glasses,
outputActive = glassesScreen != null,
steamVrRunning = steamVrStatus == SteamVr.Status.RUNNING,
)
private fun updateHeadTracker() {
val want = resumed && cardboard.enabled && cardboard.headTracking
val onGlasses = GlassesRendering.usesGlasses(glasses, glassesScreen != null,
steamVrStatus == SteamVr.Status.RUNNING)
val c = activeCardboard
val want = !ended && (resumed || (glassesScreen != null && started)) && c.enabled && c.headTracking
val current = headTracker
if (current != null && (!want || current.usesGlasses != onGlasses)) {
current.stop()
headTracker = null
}
if (want) {
val t = headTracker ?: HeadTracker(this) {
val t = headTracker ?: HeadTracker(this, glasses = onGlasses) {
@Suppress("DEPRECATION")
(if (Build.VERSION.SDK_INT >= 30) display else windowManager.defaultDisplay)?.rotation ?: android.view.Surface.ROTATION_0
}.also { headTracker = it }
t.start()
} else {
headTracker?.stop()
}
headTracking = headTracker != null
}
private fun outputHolder(): SurfaceHolder = glassesScreen?.holder ?: surfaceView.holder
private fun onOutputAttached(holder: SurfaceHolder) {
if (holder !== outputHolder()) return
glassesOutputActive = glassesScreen?.holder === holder
// Keep the phone's layout for touch/controller coordinates, but remove
// its SurfaceView so it cannot retain the last game frame under the hint.
surfaceView.visibility = if (glassesOutputActive) View.INVISIBLE else View.VISIBLE
Log.i(TAG, "compositor output attached: " +
if (glassesOutputActive) "glasses display ${glassesScreen?.display?.displayId}" else "phone")
}
private fun restorePhoneOutput(screen: GlassesScreen) {
if (glassesScreen !== screen) return
glassesScreen = null
glassesOutputActive = false
surfaceW = 0
surfaceH = 0
CompositorHost.detach(screen.holder.surface)
surfaceView.visibility = View.VISIBLE
applyOutputPacing()
if (surfaceView.holder.surface?.isValid == true) surfaceCreated(surfaceView.holder)
}
private fun glassesOutputSize(): Pair<Int, Int>? {
if (glassesScreen == null || glasses.width <= 0 || glasses.height <= 0) return null
val panel = (if (glasses.sbs) glasses.width / 2 else glasses.width) to glasses.height
val mode = SessionPrefs.prefMode(intent.getStringExtra(SessionService.EXTRA_MODE) ?: SessionService.MODE_STEAM)
val display = com.droiddeck.launcher.session.SessionDisplay
val choice = SessionPrefs.resolutionChoice(this, mode, panel)
val size = display.resolveChoice(panel, choice)
val fixed = choice != display.MATCH_SCREEN && choice != display.FOLLOW_SCREEN && display.presetHeight(choice) == null
return if (fixed) display.resolve(panel, minOf(size.first, size.second), SessionPrefs.SHAPE_AUTO) else size
}
private fun onGlassesChanged() {
val g = glasses
val s = glassesSettings
val key = listOf(g.present, g.usable, g.displayId, g.sbs, g.sbs90, g.switching3d, g.canControl, g.width, g.height, g.hz, s)
if (key == glassesKey) return
glassesKey = key
updateGlassesScreen()
applyCardboard(cardboard)
if (!g.switching3d) {
if (glassesPresent && !g.present) onGlassesUnplugged()
glassesPresent = g.present
}
}
private fun onGlassesUnplugged() {
val hadVr = steamVrAutoTried
steamVrAutoTried = false
glasses3dForVr = false
if (hadVr && !cardboard.enabled) steamVrAction("stop")
}
private fun autoStartSteamVr(now: SteamVr.Status) {
if (now == SteamVr.Status.RUNNING) steamVrAutoTried = true
val action = SteamVr.autoAction(now, SessionState.steamVr, loading.visible,
SessionState.mode == SessionService.MODE_STEAM, steamVrAutoTried, steamVrInstallTried) ?: return
if (action == "start") steamVrAutoTried = true else steamVrInstallTried = true
Log.i(TAG, "requested SteamVR $action (glasses=${glasses.model}, output=$glassesOutputActive, control=${glasses.canControl})")
steamVrAction(action)
}
private fun updateGlassesScreen() {
if (!::surfaceView.isInitialized) return
val g = glasses
val want = started && !ended && !isInPictureInPictureMode && !runtimeRemovalBlocked && g.usable && glassesSettings.gameOnGlasses
val target = if (want) Glasses.glassesDisplay()?.takeIf { it.isValid } else null
val current = glassesScreen
if (current != null && current.display.displayId == target?.displayId) {
current.prefer(Glasses.outputHz())
applyOutputPacing()
return
}
if (current != null) {
restorePhoneOutput(current)
runCatching { current.dismiss() }
}
if (target != null) {
val screen = GlassesScreen(this, target, Glasses.outputHz(), this)
// show() can create the surface synchronously. Select its holder
// first, otherwise surfaceCreated() rejects the glasses callback.
glassesScreen = screen
applyOutputPacing()
surfaceW = 0
surfaceH = 0
screen.setOnDismissListener {
if (glassesScreen === screen) {
restorePhoneOutput(screen)
refreshSecondScreenDisplays()
applyCardboard(cardboard)
}
}
try {
screen.show()
} catch (e: Exception) {
restorePhoneOutput(screen)
runCatching { screen.dismiss() }
Log.w(TAG, "could not show the game on ${target.name}", e)
return
}
Log.i(TAG, "requested game output on ${g.model} (display ${target.displayId})")
}
applyOutputPacing()
refreshSecondScreenDisplays()
applyCardboard(cardboard)
}
private fun onGlassesSteamVr(now: SteamVr.Status) {
val g = glasses
if (now == SteamVr.Status.RUNNING) {
if (glassesSettings.auto3d && g.can3d && !g.sbs && !glasses3dForVr) {
glasses3dForVr = true
Glasses.set3dForVr(true)
}
} else if (glasses3dForVr && now != SteamVr.Status.UNAVAILABLE) {
glasses3dForVr = false
Glasses.set3dForVr(false)
}
}
@@ -2575,6 +2913,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
override fun onDestroy() {
steamVrRecovery.close()
uiHandler.removeCallbacks(steamTouchRetry)
if (runtimeRemovalBlocked) { super.onDestroy(); return }
watching = false
@@ -2583,7 +2922,14 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
pendingBackAction = null
if (::pip.isInitialized) pip.destroyed()
closeSecondScreen(reset = false)
glassesScope.cancel()
glassesScreen?.let { screen -> glassesScreen = null; runCatching { screen.dismiss() } }
headTracker?.stop()
if (::hud.isInitialized) hud.stop()
joyConVr?.close()
joyConVr = null
handCamera?.stop()
handCamera = null
padBridge?.stop()
steamController?.stop()
if (SessionState.padBridge === padBridge) SessionState.padBridge = null
@@ -2637,6 +2983,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
companion object {
const val EXTRA_RETURN_HOME = "returnHome"
const val EXTRA_STEAMVR = "steamVr"
private const val LEAVE_TIMEOUT_MS = 1_500L
private const val TAG = "SessionActivity"
private const val FEX_MISSING = "droiddeck-fex: x86 Linux programs need"
@@ -2650,6 +2997,8 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
private const val DRAWER_SLOW_REPEAT_FLOOR_MS = 240L
private const val DRAWER_FAST_REPEAT_INTERVAL_MS = 90L
private const val DRAWER_REPEAT_ACCELERATION_MS = 1_400L
private val steamVrWorker = java.util.concurrent.ThreadPoolExecutor(0, 1, 30, java.util.concurrent.TimeUnit.SECONDS,
java.util.concurrent.LinkedBlockingQueue()) { Thread(it, "steamvr") }
private const val SCALE_FIT = 1
private const val ALIGN_CENTER = 0
private const val ALIGN_PICTURE_TOP = 3
@@ -92,11 +92,75 @@ class AgentBridgeProvider : ContentProvider() {
AgentEnv.override(context, request.optString("name")).put("ok", true).put("command", "override")
}
"prefs" -> prefs(context, request)
"glasses" -> glasses(context, request)
"access" -> JSONObject().put("ok", true).put("commands", AgentAccess.commandsAllowed(context))
.put("debugBuild", BuildConfig.DEBUG).put("inbox", AgentEnv.inbox(context).absolutePath)
else -> error("UNKNOWN_COMMAND", "Unknown agent command '$method'")
}
private fun glasses(context: Context, request: JSONObject): JSONObject {
val g = com.droiddeck.launcher.glasses.Glasses
when (request.optString("op", "status")) {
"status" -> {}
"allow" -> g.requestControl()
"set" -> {
val key = request.optString("key")
val value = request.optString("value")
fun on() = value == "1" || value.equals("true", true) || value.equals("on", true)
val number = value.toIntOrNull()
val numeric = setOf("refreshHz", "sbsHz", "brightness", "volume", "picture3d", "screenWidthDm", "screenDistanceDm", "vrFovDeg", "sbsSupport")
if (key in numeric && number == null) return error("INVALID_VALUE", "'$key' takes a number")
when (key) {
"refreshHz" -> g.setRefreshHz(number!!)
"sbsHz" -> g.setSbsHz(number!!)
"brightness" -> g.setBrightness(number!!)
"volume" -> g.setVolume(number!!)
"sunblock" -> g.setSunblock(on())
"3d" -> g.set3d(on())
"gameOnGlasses" -> g.setGameOnGlasses(on())
"headTracking" -> g.setHeadTracking(on())
"auto3d" -> g.setAuto3d(on())
"picture3d" -> g.setPicture3d(number!!)
"screenWidthDm" -> g.setScreenWidthDm(number!!)
"screenDistanceDm" -> g.setScreenDistanceDm(number!!)
"vrFovDeg" -> g.setVrFovDeg(number!!)
"sbsSupport" -> g.setSbsSupport(number!!)
else -> return error("INVALID_KEY", "Unknown glasses setting '$key'")
}
}
"simulate" -> {
fun angle(key: String) = if (request.has(key)) request.optDouble(key).takeIf { it.isFinite() }?.coerceIn(-90.0, 90.0) else null
if (!g.simulate(request.optInt("seconds", 20).coerceIn(1, 600), angle("yaw"), angle("pitch"))) return error("BUSY", "Already simulating")
}
"stop-simulate" -> g.stopSimulation()
"calibrate" -> g.startCalibration()
"cancel-calibration" -> g.cancelCalibration()
"reset-calibration" -> g.resetCalibration()
"query" -> { g.query(); Thread.sleep(600) }
"mode" -> {
val mode = hex(request.optString("mode"), 0xFF) ?: return error("INVALID_VALUE", "mode must be one hex byte, e.g. 0x31")
g.sendMode(mode)
Thread.sleep(600)
}
"imu" -> return g.sampleImu(request.optLong("ms", 2000).coerceIn(200, 20_000)).put("ok", true).put("command", "glasses")
"raw" -> {
AgentAccess.requireCommands(context)
val msgId = hex(request.optString("msgId"), 0xFFFF) ?: return error("INVALID_VALUE", "msgId must be a hex number up to 0xFFFF")
val digits = request.optString("payload").removePrefix("0x")
if (digits.length % 2 != 0 || digits.any { Character.digit(it, 16) < 0 }) return error("INVALID_VALUE", "payload must be whole hex bytes")
val payload = digits.chunked(2).map { it.toInt(16).toByte() }.toByteArray()
if (!g.sendRaw(msgId, payload)) return error("NOT_CONNECTED", "The glasses' control link is not open")
Thread.sleep(300)
}
else -> return error("INVALID_OP", "Unknown glasses op")
}
if (request.optString("op") in setOf("set", "allow")) Thread.sleep(400)
return g.status().put("ok", true).put("command", "glasses").put("frames", g.recentFrames())
}
private fun hex(text: String, max: Int): Int? =
text.removePrefix("0x").removePrefix("0X").takeIf { it.isNotEmpty() }?.toIntOrNull(16)?.takeIf { it in 0..max }
private fun state(context: Context): JSONObject {
val runtimeVersion = LinuxRuntimeInstaller.installedVersion(context)
val dir = SessionState.logDirectory ?: SessionState.logFile?.parentFile ?: SessionPaths.sessionFolders(context).lastOrNull()
@@ -109,6 +173,7 @@ class AgentBridgeProvider : ContentProvider() {
.put("steamUi", SessionState.steamUi ?: JSONObject.NULL)
.put("steamUrl", SessionState.steamUrl ?: JSONObject.NULL)
.put("suspended", SessionState.suspended)
.put("steamVr", SessionState.steamVr)
.put("pip", SessionState.pipActive)
.put("firstFrame", SessionState.firstFrameSeen)
.put("output", JSONArray().put(SessionState.outputSize.first).put(SessionState.outputSize.second))
@@ -97,6 +97,7 @@ class AgentStartActivity : Activity() {
request.optString("steamUrl").takeIf { it.startsWith("steam://") }?.let {
putExtra(SessionService.EXTRA_STEAM_URL, it)
}
if (request.optBoolean("steamVr")) putExtra(SessionActivity.EXTRA_STEAMVR, true)
}
}
startActivity(sessionIntent)
@@ -0,0 +1,862 @@
package com.droiddeck.launcher.glasses
import android.annotation.SuppressLint
import android.app.PendingIntent
import android.content.BroadcastReceiver
import android.content.Context
import android.content.Intent
import android.content.IntentFilter
import android.hardware.display.DisplayManager
import android.hardware.usb.UsbDevice
import android.hardware.usb.UsbManager
import android.os.Build
import android.os.Handler
import android.os.HandlerThread
import android.os.SystemClock
import android.util.Log
import android.view.Display
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import org.json.JSONArray
import org.json.JSONObject
import kotlin.math.abs
import kotlin.math.sin
import com.droiddeck.launcher.R
import java.io.File
data class GlassesState(
val present: Boolean = false,
val model: String = "VITURE",
val controllable: Boolean = false,
val permission: Glasses.Permission = Glasses.Permission.NONE,
val connected: Boolean = false,
val displayId: Int = Display.INVALID_DISPLAY,
val width: Int = 0,
val height: Int = 0,
val hz: Float = 0f,
val sbs: Boolean = false,
val switching3d: Boolean = false,
val brightness: Int = -1,
val volume: Int = -1,
val brightnessMax: Int = 8,
val volumeMax: Int = 8,
val sunblock: Boolean = false,
val hardwareControls: Boolean = true,
val hasSunblock: Boolean = true,
val controlError: String? = null,
val tracking: Boolean = false,
val diagonalFovDeg: Float = 46f,
val noSignal: Boolean = false,
val sbsSupport: Int = SBS_UNKNOWN,
val sbs90: Boolean = false,
val experimental: Boolean = false,
) {
val hasDisplay: Boolean get() = displayId != Display.INVALID_DISPLAY
val usable: Boolean get() = hasDisplay && !noSignal
val canControl: Boolean get() = connected && controllable
val can3d: Boolean get() = canControl && !noSignal && sbsSupport != SBS_FAILS
val verticalFovDeg: Float get() {
val half = Math.toRadians(diagonalFovDeg / 2.0)
val aspect = if (width > 0 && height > 0)
width.toDouble() / (if (sbs) 2 else 1) / height else 16.0 / 9.0
val v = kotlin.math.atan(kotlin.math.tan(half) / kotlin.math.sqrt(1 + aspect * aspect))
return Math.toDegrees(2 * v).toFloat()
}
companion object {
const val SBS_UNKNOWN = 0
const val SBS_WORKS = 1
const val SBS_FAILS = -1
}
}
data class GlassesSettings(
val refreshHz: Int = 120,
val sbsHz: Int = 60,
val sunblock: Boolean? = null,
val gameOnGlasses: Boolean = true,
val headTracking: Boolean = true,
val auto3d: Boolean = false,
val picture3d: Int = PICTURE_AUTO,
val screenWidthDm: Int = 20,
val screenDistanceDm: Int = 30,
val vrFovDeg: Int = VR_FOV_DEFAULT,
) {
companion object {
const val VR_FOV_MIN = 24
const val VR_FOV_MAX = 90
const val VR_FOV_DEFAULT = 90
const val PICTURE_AUTO = 0
const val PICTURE_SCREEN = 1
const val PICTURE_SBS = 2
}
}
@SuppressLint("StaticFieldLeak")
object Glasses {
enum class Permission { NONE, ASKING, DENIED, GRANTED }
fun interface PoseSink {
fun onPose(x: Float, y: Float, z: Float, w: Float, timeNs: Long)
}
private const val TAG = "Glasses"
private const val ACTION_PERMISSION = "com.droiddeck.launcher.glasses.USB_PERMISSION"
private const val PREFS = "glasses"
private const val KEY_SBS_FAILED = "sbsFailed"
private const val SBS_CONFIRM_MS = 20_000L
private val _state = MutableStateFlow(GlassesState())
val state: StateFlow<GlassesState> = _state.asStateFlow()
private val _settings = MutableStateFlow(GlassesSettings())
val settings: StateFlow<GlassesSettings> = _settings.asStateFlow()
private val _calibration = MutableStateFlow(GlassesCalibrationState())
val calibration: StateFlow<GlassesCalibrationState> = _calibration.asStateFlow()
private lateinit var calibrationStore: RayNeoCalibrationStore
private var calibrationKey: String? = null
private var calibrationRevision = 0L
private lateinit var app: Context
private lateinit var handler: Handler
private lateinit var usb: UsbManager
private lateinit var displays: DisplayManager
@Volatile private var link: GlassesLink? = null
@Volatile private var linkGeneration = 0L
private var linkDevice: UsbDevice? = null
private var model: GlassesModel? = null
private var asked = false
private var lastModeSentAt = 0L
private val modeState = GlassesModeState(SBS_CONFIRM_MS)
private val confirmMode = Runnable { refresh() }
private var glassesMode = 0
private var autoModeSent = false
@Volatile private var sink: PoseSink? = null
@Volatile private var probe: ((ByteArray?, FloatArray?) -> Unit)? = null
@Volatile private var latest: FloatArray? = null
@Volatile private var imuPackets = 0L
private val simulating = java.util.concurrent.atomic.AtomicReference<Thread?>(null)
private val frames = ArrayDeque<String>()
fun init(context: Context) {
if (::handler.isInitialized) return
app = context.applicationContext
usb = app.getSystemService(Context.USB_SERVICE) as UsbManager
displays = app.getSystemService(Context.DISPLAY_SERVICE) as DisplayManager
_settings.value = load()
calibrationStore = RayNeoCalibrationStore(app.getSharedPreferences(PREFS, Context.MODE_PRIVATE))
handler = Handler(HandlerThread("droiddeck-glasses").apply { start() }.looper)
handler.post {
val filter = IntentFilter(ACTION_PERMISSION).apply {
addAction(UsbManager.ACTION_USB_DEVICE_ATTACHED)
addAction(UsbManager.ACTION_USB_DEVICE_DETACHED)
}
if (Build.VERSION.SDK_INT >= 33) app.registerReceiver(receiver, filter, null, handler, Context.RECEIVER_NOT_EXPORTED)
else app.registerReceiver(receiver, filter, null, handler)
displays.registerDisplayListener(displayListener, handler)
refresh()
}
}
fun rescan() = post { refresh() }
fun startCalibration() = post {
val l = link as? RayNeoLink ?: return@post
if (!l.ready || _calibration.value.running || simulating.get() != null) return@post
_calibration.value = _calibration.value.copy(phase = CalibrationPhase.SETTLING, progress = 0)
l.startCalibration(++calibrationRevision)
updateImu()
}
fun cancelCalibration() = post {
(link as? RayNeoLink)?.cancelCalibration(++calibrationRevision, calibrationKey?.let(calibrationStore::load))
_calibration.value = _calibration.value.copy(phase = CalibrationPhase.IDLE, progress = 0)
updateImu()
}
fun resetCalibration() = post {
(link as? RayNeoLink)?.resetCalibration(++calibrationRevision)
calibrationKey?.let { calibrationStore.save(it, null) }
_calibration.value = _calibration.value.copy(saved = false, phase = CalibrationPhase.IDLE, progress = 0)
updateImu()
}
fun query() = post {
val l = link ?: return@post
listOf(VitureProtocol.MSG_GET_DISPLAY, VitureProtocol.MSG_GET_BRIGHTNESS, VitureProtocol.MSG_GET_VOLUME, VitureProtocol.MSG_GET_IMU).forEachIndexed { i, msg ->
handler.postDelayed({ if (link === l) l.send(msg) }, VitureLink.COMMAND_GAP_MS * i)
}
}
fun sendMode(mode: Int) = post {
if (!_state.value.canControl) return@post
if (link?.send(VitureProtocol.MSG_DISPLAY, byteArrayOf(mode.toByte())) == true) lastModeSentAt = SystemClock.uptimeMillis()
}
fun requestControl() = post {
asked = false
refresh(ask = true)
}
fun setRefreshHz(hz: Int) = post {
update { it.copy(refreshHz = hz.coerceIn(60, 120)) }
if (!_state.value.sbs) applyMode(force = true)
}
fun setSbsHz(hz: Int) = post {
update { it.copy(sbsHz = if (hz >= 90) 90 else 60) }
if (_state.value.sbs && _state.value.can3d) sendMode(true)
}
fun set3d(on: Boolean) = post {
val s = _state.value
if (!s.canControl || (on && s.noSignal)) return@post
modeState.manual()
if (on) {
markSbsFailed(linkDevice?.productId, false)
rememberSbs(GlassesState.SBS_UNKNOWN)
}
sendMode(on)
}
fun set3dForVr(on: Boolean) = post {
val s = _state.value
if (on) {
if (!s.can3d || s.sbs || modeState.pendingStereo) return@post
if (sendMode(true)) modeState.startVr()
} else if (modeState.stopVr() && s.canControl) {
sendMode(false)
}
}
fun setBrightness(level: Int) = post {
val s = _state.value
if (!s.canControl || s.brightnessMax == 0) return@post
val v = level.coerceIn(0, s.brightnessMax)
if (link?.send(VitureProtocol.MSG_BRIGHTNESS, byteArrayOf(v.toByte(), 0)) == true && link !is XrealLink)
publish { it.copy(brightness = v) }
}
fun setVolume(level: Int) = post {
val s = _state.value
if (!s.canControl || s.volumeMax == 0) return@post
val v = level.coerceIn(0, s.volumeMax)
if (link?.send(VitureProtocol.MSG_VOLUME, byteArrayOf(v.toByte(), 0)) == true) publish { it.copy(volume = v) }
}
fun setSunblock(on: Boolean) = post {
update { it.copy(sunblock = on) }
sendFilm(on)
}
fun setSbsSupport(support: Int) = post {
val s = support.coerceIn(GlassesState.SBS_FAILS, GlassesState.SBS_WORKS)
markSbsFailed(linkDevice?.productId, s == GlassesState.SBS_FAILS)
rememberSbs(s)
}
fun setGameOnGlasses(on: Boolean) = post { update { it.copy(gameOnGlasses = on) } }
fun setHeadTracking(on: Boolean) = post { update { it.copy(headTracking = on) } }
fun setAuto3d(on: Boolean) = post { update { it.copy(auto3d = on) } }
fun setPicture3d(picture: Int) = post { update { it.copy(picture3d = picture.coerceIn(0, 2)) } }
fun setScreenWidthDm(dm: Int) = post { update { it.copy(screenWidthDm = dm.coerceIn(5, 200)) } }
fun setScreenDistanceDm(dm: Int) = post { update { it.copy(screenDistanceDm = dm.coerceIn(5, 200)) } }
fun setVrFovDeg(deg: Int) = post { update { it.copy(vrFovDeg = deg.coerceIn(GlassesSettings.VR_FOV_MIN, GlassesSettings.VR_FOV_MAX)) } }
fun outputHz(): Int = GlassesRendering.outputHz(_settings.value, _state.value)
fun glassesDisplay(): Display? = _state.value.displayId.takeIf { it != Display.INVALID_DISPLAY }?.let { displays.getDisplay(it) }
fun setPoseSink(s: PoseSink) {
synchronized(this) { sink = s }
post { updateImu() }
}
fun clearPoseSink(s: PoseSink) {
synchronized(this) { if (sink === s) sink = null }
post { updateImu() }
}
fun simulate(seconds: Int, holdYaw: Double? = null, holdPitch: Double? = null): Boolean {
val hold = holdYaw != null || holdPitch != null
val t = Thread({
val start = SystemClock.elapsedRealtimeNanos()
val end = start + seconds * 1_000_000_000L
while (SystemClock.elapsedRealtimeNanos() < end && simulating.get() === Thread.currentThread()) {
val now = SystemClock.elapsedRealtimeNanos()
val t = (now - start) / 1e9
val yaw = if (hold) holdYaw ?: 0.0 else 40.0 * sin(t * 2 * Math.PI / 6.0)
val pitch = if (hold) holdPitch ?: 0.0 else 15.0 * sin(t * 2 * Math.PI / 4.0)
val roll = if (hold) 0.0 else 5.0 * sin(t * 2 * Math.PI / 5.0)
val q = VitureProtocol.eulerZxy(roll.toFloat(), pitch.toFloat(), yaw.toFloat())
deliver(q, now)
SystemClock.sleep(4)
}
simulating.compareAndSet(Thread.currentThread(), null)
}, "glasses-simulate")
if (!simulating.compareAndSet(null, t)) return false
t.start()
return true
}
fun stopSimulation() {
simulating.set(null)
}
fun status(): JSONObject {
val s = _state.value
val g = _settings.value
val q = latest
return JSONObject()
.put("present", s.present).put("model", s.model).put("controllable", s.controllable)
.put("permission", s.permission.name).put("connected", s.connected)
.put("controlError", s.controlError ?: JSONObject.NULL)
.put("experimental", s.experimental)
.put("display", if (s.hasDisplay) JSONObject().put("id", s.displayId).put("width", s.width).put("height", s.height).put("hz", s.hz.toDouble()) else JSONObject.NULL)
.put("sbs", s.sbs).put("deviceMode", glassesMode).put("switching3d", s.switching3d)
.put("brightness", s.brightness).put("volume", s.volume)
.put("brightnessMax", s.brightnessMax).put("volumeMax", s.volumeMax).put("sunblock", s.sunblock)
.put("tracking", s.tracking).put("simulating", simulating.get() != null).put("imuPackets", imuPackets)
.put("calibration", JSONObject().put("supported", _calibration.value.supported)
.put("saved", _calibration.value.saved).put("phase", _calibration.value.phase.name)
.put("progress", _calibration.value.progress))
.put("badFrames", link?.badFrames ?: 0L)
.put("pose", q?.let { JSONArray().put(it[0].toDouble()).put(it[1].toDouble()).put(it[2].toDouble()).put(it[3].toDouble()) } ?: JSONObject.NULL)
.put("fovDeg", s.diagonalFovDeg.toDouble()).put("noSignal", s.noSignal).put("sbsSupport", s.sbsSupport)
.put("settings", JSONObject().put("refreshHz", g.refreshHz).put("sbsHz", g.sbsHz).put("sunblock", g.sunblock ?: JSONObject.NULL)
.put("gameOnGlasses", g.gameOnGlasses).put("headTracking", g.headTracking).put("auto3d", g.auto3d)
.put("picture3d", g.picture3d).put("screenWidthDm", g.screenWidthDm).put("screenDistanceDm", g.screenDistanceDm)
.put("vrFovDeg", g.vrFovDeg))
}
fun recentFrames(): JSONArray = synchronized(frames) { JSONArray(frames.toList()) }
fun sendRaw(msgId: Int, payload: ByteArray): Boolean {
val l = link ?: return false
return l.send(msgId, payload)
}
private fun post(block: () -> Unit) {
if (!::handler.isInitialized) return
handler.post(block)
}
private fun publish(transform: (GlassesState) -> GlassesState) {
_state.value = transform(_state.value)
}
private fun update(transform: (GlassesSettings) -> GlassesSettings) {
val next = transform(_settings.value)
_settings.value = next
save(next)
}
private val receiver = object : BroadcastReceiver() {
override fun onReceive(context: Context, intent: Intent) {
when (intent.action) {
ACTION_PERMISSION -> {
val granted = intent.getBooleanExtra(UsbManager.EXTRA_PERMISSION_GRANTED, false)
Log.i(TAG, "USB permission ${if (granted) "granted" else "denied"}")
if (!granted) publish { it.copy(permission = Permission.DENIED) }
refresh()
}
UsbManager.ACTION_USB_DEVICE_ATTACHED -> {
if (intent.device()?.let { isGlassesVendor(it.vendorId) } == true) asked = false
refresh(ask = true)
}
UsbManager.ACTION_USB_DEVICE_DETACHED -> if (intent.device()?.let { isGlassesVendor(it.vendorId) } == true) {
refresh()
}
}
}
}
private val displayListener = object : DisplayManager.DisplayListener {
override fun onDisplayAdded(displayId: Int) = refresh()
override fun onDisplayRemoved(displayId: Int) = refresh()
override fun onDisplayChanged(displayId: Int) {
if (displayId == _state.value.displayId || displayId != Display.DEFAULT_DISPLAY) refresh()
}
}
private fun refresh(ask: Boolean = false) {
val devices = usb.deviceList.values.filter {
it.vendorId == VitureProtocol.VENDOR_ID || deviceModel(it) != null
}
val known = devices.mapNotNull { device ->
deviceModel(device)?.let { device to it }
}
val selected = known.firstOrNull { canOpenModel(it.second) } ?: known.firstOrNull()
val device = selected?.first
val m = selected?.second
val controllable = m?.let(::canOpenModel) == true
// XREAL uses Android's Ethernet sockets, without opening or claiming USB interfaces.
val permitted = controllable && device != null && (m === XrealModel || usb.hasPermission(device))
val identity = device?.let {
GlassesModeState.Device(it.productId, if (permitted) runCatching { it.serialNumber }.getOrNull() else null)
}
if (modeState.updateDevice(identity, SystemClock.uptimeMillis())) {
rememberSbs(if (identity != null && sbsFailed(identity.productId)) GlassesState.SBS_FAILS else GlassesState.SBS_UNKNOWN)
}
if (device == null || (linkDevice != null && linkDevice?.deviceName != device.deviceName)) closeLink()
model = m
val display = findDisplay(devices.isNotEmpty())
val mode = display?.mode
val width = mode?.physicalWidth ?: 0
val height = mode?.physicalHeight ?: 0
val sbsByDisplay = width >= 3000
val noSignal = display != null && width in 1 until 1280
when (modeState.observe(sbsByDisplay, SystemClock.uptimeMillis())) {
GlassesModeState.Confirmation.FAILED -> {
Log.w(TAG, "3D signal was not confirmed at ${width}x$height; restoring 2D")
if (noSignal) markSbsFailed(device?.productId, true)
rememberSbs(GlassesState.SBS_FAILS)
}
else -> Unit
}
if (sbsByDisplay) {
markSbsFailed(device?.productId, false)
rememberSbs(GlassesState.SBS_WORKS)
}
if (controllable && device != null && !permitted && (ask || !asked) && _state.value.permission != Permission.ASKING) askPermission(device)
if (device != null && permitted && link == null) openLink(device)
publish {
it.copy(
present = device != null || display != null,
model = m?.name ?: display?.let(::displayName) ?: "VITURE",
controllable = controllable,
permission = when {
!controllable -> Permission.NONE
permitted -> Permission.GRANTED
it.permission == Permission.ASKING || it.permission == Permission.DENIED -> it.permission
else -> Permission.NONE
},
connected = link?.ready == true,
controlError = when {
m is VitureProtocol.Model && m.needsSdk && !VitureSdk.available -> app.getString(R.string.glasses_driver_missing)
device == null || link?.ready == true -> null
else -> it.controlError
},
displayId = display?.displayId ?: Display.INVALID_DISPLAY,
width = width, height = height, hz = mode?.refreshRate ?: 0f,
sbs = sbsByDisplay,
switching3d = modeState.pendingStereo,
brightnessMax = m?.brightnessMax ?: 8, volumeMax = m?.volumeMax ?: 8,
sunblock = if (m === XrealModel) it.sunblock else _settings.value.sunblock ?: it.sunblock,
hardwareControls = device?.vendorId == VitureProtocol.VENDOR_ID || m === XrealModel,
hasSunblock = m?.hasSunblock == true,
diagonalFovDeg = m?.diagonalFovDeg ?: 46f,
tracking = it.tracking && link != null,
noSignal = noSignal,
sbs90 = m?.sbs90 == true,
experimental = m === XrealModel,
)
}
if (link != null) applyMode(force = false)
scheduleModeCheck()
}
internal fun isGlassesVendor(vid: Int) = vid == VitureProtocol.VENDOR_ID || RayNeoModel.isVendor(vid) || vid == XrealModel.VENDOR_ID
private fun canOpenModel(m: GlassesModel): Boolean = m.controllable &&
(m !is VitureProtocol.Model || !m.needsSdk || VitureSdk.available)
internal fun deviceModel(device: UsbDevice): GlassesModel? {
val name = runCatching { device.productName }.getOrNull()
return if (device.vendorId == VitureProtocol.VENDOR_ID) {
VitureSdk.marketName(device.productId)?.let(VitureProtocol::sdkModel)
?: VitureProtocol.model(device.productId, name)
}
else if (XrealModel.matches(device.vendorId, device.productId)) XrealModel
else RayNeoModel.model(device.vendorId, device.productId, name)
}
internal fun isGlassesDisplayName(name: String) =
name.contains("VITURE", ignoreCase = true) || RayNeoModel.isDisplay(name) || XrealModel.isDisplay(name)
private fun findDisplay(usbPresent: Boolean): Display? {
return selectGlassesDisplay(displays.getDisplays(DisplayManager.DISPLAY_CATEGORY_PRESENTATION).toList(), usbPresent)
}
internal fun selectGlassesDisplay(displays: List<Display>, usbPresent: Boolean): Display? {
val candidates = displays
.filter { it.displayId != Display.DEFAULT_DISPLAY && it.flags and Display.FLAG_PRESENTATION != 0 }
// Android can expose the glasses name on either the logical display or
// its product metadata. A generic label on one must not hide the other.
return candidates.firstOrNull { display ->
isGlassesDisplayName(display.name) ||
(Build.VERSION.SDK_INT >= 31 && display.deviceProductInfo?.let { product ->
product.name?.let(::isGlassesDisplayName) == true ||
RayNeoModel.isDisplayProduct(product.name, product.manufacturerPnpId, product.productId)
} == true)
}
?: candidates.singleOrNull()?.takeIf { usbPresent && unnamedPanel(it) }
}
private fun unnamedPanel(d: Display): Boolean {
if (Build.VERSION.SDK_INT < 31) return true
return d.deviceProductInfo?.name.isNullOrBlank()
}
private fun displayName(d: Display): String? {
if (Build.VERSION.SDK_INT >= 31) d.deviceProductInfo?.name?.takeIf { it.isNotBlank() }?.let { return it }
return d.name
}
private fun askPermission(device: UsbDevice) {
asked = true
publish { it.copy(permission = Permission.ASKING) }
val flags = if (Build.VERSION.SDK_INT >= 31) PendingIntent.FLAG_MUTABLE else 0
val pi = PendingIntent.getBroadcast(app, 0, Intent(ACTION_PERMISSION).setPackage(app.packageName), flags)
runCatching { usb.requestPermission(device, pi) }.onFailure { Log.w(TAG, "requestPermission", it) }
}
private fun openLink(device: UsbDevice) {
val generation = ++linkGeneration
calibrationKey = calibrationStore.key(device.vendorId, device.productId,
runCatching { device.serialNumber }.getOrNull(), model?.name.orEmpty())
val l = if (device.vendorId == VitureProtocol.VENDOR_ID) {
if ((model as? VitureProtocol.Model)?.needsSdk == true) {
VitureSdkLink.open(usb, device, File(app.cacheDir, "viture"), sdkListener(generation),
(model as? VitureProtocol.Model)?.family)
} else VitureLink.open(usb, device, linkListener(generation))
} else if (model === XrealModel) {
XrealLink.open(app, xrealListener(generation))
} else {
val baseKey = calibrationKey!!
val gtUsb = RayNeoProtocol.isGt(device.vendorId, device.productId)
RayNeoLink.open(usb, device, rayNeoListener(generation)) { board ->
calibrationStore.load(RayNeoCalibrationStore.profileKey(baseKey, gtUsb, board))
}
}
val opened = l ?: run {
linkGeneration++
Log.w(TAG, "could not open ${device.deviceName}")
publish { it.copy(controlError = app.getString(R.string.glasses_connection_failed)) }
return
}
link = opened
_calibration.value = GlassesCalibrationState(supported = opened is RayNeoLink,
saved = opened is RayNeoLink && calibrationKey?.let(calibrationStore::load) != null)
linkDevice = device
glassesMode = 0
autoModeSent = false
Log.i(TAG, "opened ${model?.name} pid=0x%04X imu=%s".format(device.productId, opened.hasImu))
publish { it.copy(connected = opened.ready, controlError = null) }
listOf(VitureProtocol.MSG_GET_BRIGHTNESS, VitureProtocol.MSG_GET_VOLUME, VitureProtocol.MSG_GET_DISPLAY).forEachIndexed { i, msg ->
handler.postDelayed({ if (link === opened) opened.send(msg) }, VitureLink.COMMAND_GAP_MS * (i + 1))
}
handler.postDelayed({
if (link !== opened) return@postDelayed
_settings.value.sunblock?.let { sendFilm(it) }
updateImu()
applyMode(force = false)
}, 300)
}
private fun xrealListener(generation: Long) = object : XrealLink.Listener {
override fun onReady() = post {
if (generation != linkGeneration) return@post
publish { it.copy(connected = true, controlError = null) }
autoModeSent = false
updateImu()
applyMode(force = false)
}
override fun onValue(message: Int, value: Int) = post {
if (generation != linkGeneration) return@post
when (message) {
VitureProtocol.MSG_GET_DISPLAY -> glassesMode = value
VitureProtocol.MSG_BRIGHTNESS -> publish { it.copy(brightness = value) }
VitureProtocol.MSG_FILM -> publish { it.copy(sunblock = value != 0) }
}
}
override fun onPose(raw: ByteArray, pose: FloatArray, timeNs: Long) {
if (generation != linkGeneration) return
imuPackets++
probe?.invoke(raw, pose)
if (simulating.get() == null) deliver(pose, timeNs)
}
override fun onUnavailable(networkMissing: Boolean) = post {
if (generation != linkGeneration) return@post
publish { it.copy(connected = false, tracking = false,
controlError = app.getString(if (networkMissing) R.string.glasses_xreal_network_missing else R.string.glasses_xreal_connection_failed)) }
}
}
private fun sdkListener(generation: Long) = object : VitureSdkLink.Listener {
override fun onValue(msgId: Int, value: Int) = post {
if (generation != linkGeneration) return@post
when (msgId) {
VitureProtocol.MSG_GET_DISPLAY -> glassesMode = value
VitureProtocol.MSG_GET_BRIGHTNESS -> publish { it.copy(brightness = value) }
VitureProtocol.MSG_GET_VOLUME -> publish { it.copy(volume = value) }
VitureProtocol.MSG_FILM -> publish { it.copy(sunblock = value != 0) }
}
}
override fun onPose(q: FloatArray, timeNs: Long) {
if (generation != linkGeneration) return
imuPackets++
probe?.invoke(null, q)
if (simulating.get() == null) deliver(q, timeNs)
}
override fun onLost() = post {
if (generation != linkGeneration) return@post
closeLink()
publish { it.copy(controlError = app.getString(R.string.glasses_connection_failed)) }
// Let a reconnect or explicit Allow retry restart the SDK, not an endless failure loop.
}
}
private fun rayNeoListener(generation: Long) = object : RayNeoLink.Listener {
override fun onDeviceInfo(boardId: Int, calibrationSaved: Boolean) = post {
if (generation != linkGeneration) return@post
val device = linkDevice ?: return@post
val base = calibrationStore.key(device.vendorId, device.productId,
runCatching { device.serialNumber }.getOrNull(), model?.name.orEmpty())
calibrationKey = RayNeoCalibrationStore.profileKey(base,
RayNeoProtocol.isGt(device.vendorId, device.productId), boardId)
_calibration.value = _calibration.value.copy(saved = calibrationSaved)
}
override fun onCalibration(revision: Long, phase: CalibrationPhase, progress: Int) = post {
if (generation != linkGeneration || revision != calibrationRevision) return@post
if (phase != CalibrationPhase.IDLE && !_calibration.value.running) return@post
_calibration.value = _calibration.value.copy(phase = phase, progress = progress)
if (!_calibration.value.running) updateImu()
}
override fun onCalibrationSaved(revision: Long, value: RayNeoCalibration?) = post {
if (generation != linkGeneration || revision != calibrationRevision) return@post
if (value != null && !_calibration.value.running) return@post
calibrationKey?.let { calibrationStore.save(it, value) }
_calibration.value = _calibration.value.copy(saved = value != null)
}
override fun onReady() = post {
if (generation == linkGeneration) {
publish { it.copy(connected = true) }
updateImu()
applyMode(force = false)
}
}
override fun onPose(raw: ByteArray, xyzw: FloatArray, timeNs: Long) {
if (generation != linkGeneration) return
imuPackets++
probe?.invoke(raw, xyzw)
if (simulating.get() == null) deliver(xyzw, timeNs)
}
override fun onLost() = post {
if (generation != linkGeneration) return@post
closeLink()
refresh()
}
}
private fun closeLink() {
val l = link ?: return
linkGeneration++
link = null
linkDevice = null
calibrationKey = null
_calibration.value = GlassesCalibrationState(phase = if (_calibration.value.running) CalibrationPhase.FAILED else CalibrationPhase.IDLE)
glassesMode = 0
l.close()
publish { it.copy(connected = false, tracking = false) }
Log.i(TAG, "link closed")
}
private fun rememberSbs(support: Int) {
if (_state.value.sbsSupport == support) return
publish { it.copy(sbsSupport = support) }
}
private fun sbsFailed(productId: Int): Boolean =
app.getSharedPreferences(PREFS, Context.MODE_PRIVATE).getStringSet(KEY_SBS_FAILED, null)?.contains(productId.toString(16)) == true
private fun markSbsFailed(productId: Int?, failed: Boolean) {
if (productId == null || sbsFailed(productId) == failed) return
val prefs = app.getSharedPreferences(PREFS, Context.MODE_PRIVATE)
val ids = prefs.getStringSet(KEY_SBS_FAILED, null).orEmpty()
val id = productId.toString(16)
prefs.edit().putStringSet(KEY_SBS_FAILED, if (failed) ids + id else ids - id).apply()
}
private fun applyMode(force: Boolean) {
val s = _state.value
val m = model ?: return
if (!s.canControl) return
if (modeState.restorePending) {
sendMode(false)
return
}
if (!s.usable || s.sbs || modeState.pendingStereo) return
val hz = _settings.value.refreshHz
if (glassesMode == m.mode(hz, false)) return
if (!force) {
if (autoModeSent || SystemClock.uptimeMillis() - lastModeSentAt < 8_000) return
if (glassesMode == 0 && abs(s.hz - hz) < 1.5f) return
autoModeSent = true
}
sendMode(false)
}
private fun sendMode(sbs: Boolean): Boolean {
val m = model ?: return false
val g = _settings.value
val mode = m.mode(if (sbs) g.sbsHz else g.refreshHz, sbs)
if (link?.send(VitureProtocol.MSG_DISPLAY, byteArrayOf(mode.toByte())) == true) {
if (link is RayNeoLink) glassesMode = mode
lastModeSentAt = SystemClock.uptimeMillis()
modeState.sent(sbs, lastModeSentAt)
publish { it.copy(switching3d = modeState.pendingStereo) }
scheduleModeCheck()
return true
}
return false
}
private fun scheduleModeCheck() {
handler.removeCallbacks(confirmMode)
modeState.nextCheckAt?.let { at ->
handler.postDelayed(confirmMode, (at - SystemClock.uptimeMillis()).coerceAtLeast(0))
}
}
private fun sendFilm(on: Boolean) {
if (!_state.value.canControl || !_state.value.hardwareControls || !_state.value.hasSunblock) return
if (link?.send(VitureProtocol.MSG_FILM, byteArrayOf(if (on) 1 else 0)) == true && link !is XrealLink)
publish { it.copy(sunblock = on) }
}
private fun updateImu() {
val l = link ?: return
if (!l.ready) return
val want = (sink != null || probe != null || _calibration.value.running) && l.hasImu
if (want == _state.value.tracking) return
val accepted = l.send(VitureProtocol.MSG_IMU, byteArrayOf(if (want) 1 else 0))
if (want && !accepted) return
if (want) l.send(VitureProtocol.MSG_IMU_RATE, byteArrayOf(VitureProtocol.IMU_RATE_240.toByte()))
publish { it.copy(tracking = want) }
}
private fun linkListener(generation: Long) = object : VitureLink.Listener {
override fun onFrame(frame: VitureProtocol.Frame) = post {
if (generation != linkGeneration) return@post
val line = "%s 0x%04X %s".format(
when (frame.magic) { VitureProtocol.MAGIC_REPORT -> "FD"; VitureProtocol.MAGIC_COMMAND -> "FE"; else -> "FC" },
frame.msgId, frame.payload.joinToString("") { "%02X".format(it) },
)
synchronized(frames) {
frames.addLast(line)
while (frames.size > 64) frames.removeFirst()
}
Log.i(TAG, "frame $line")
onReport(frame)
}
override fun onImu(payload: ByteArray, timeNs: Long) {
if (generation != linkGeneration) return
imuPackets++
probe?.invoke(payload, null)
if (simulating.get() != null) return
VitureProtocol.imuQuaternion(payload)?.let { deliver(it, timeNs) }
}
override fun onLost() = post {
if (generation != linkGeneration) return@post
closeLink()
refresh()
}
}
private fun deliver(q: FloatArray, timeNs: Long) {
latest = q
sink?.onPose(q[0], q[1], q[2], q[3], timeNs)
}
private fun onReport(frame: VitureProtocol.Frame) {
val p = frame.payload
val event = p.firstOrNull()?.toInt()?.and(0xFF) ?: return
val answer = if (p.size >= 2 && event == 0) p[1].toInt() and 0xFF else null
when (frame.msgId) {
VitureProtocol.EVENT_DISPLAY -> {
Log.i(TAG, "glasses switched to mode 0x%02X".format(event))
glassesMode = event
}
VitureProtocol.EVENT_DISPLAY_STEP -> model?.modeForStep(event)?.let { mode ->
Log.i(TAG, "glasses report mode 0x%02X".format(mode))
glassesMode = mode
}
VitureProtocol.EVENT_BRIGHTNESS -> publish { it.copy(brightness = event) }
VitureProtocol.EVENT_VOLUME -> publish { it.copy(volume = event) }
VitureProtocol.MSG_GET_BRIGHTNESS, VitureProtocol.MSG_BRIGHTNESS -> answer?.let { v -> publish { it.copy(brightness = v) } }
VitureProtocol.MSG_GET_VOLUME, VitureProtocol.MSG_VOLUME -> answer?.let { v -> publish { it.copy(volume = v) } }
VitureProtocol.MSG_DISPLAY -> if (event != 0) {
Log.w(TAG, "glasses refused display mode 0x%02X".format(p.getOrNull(1)?.toInt()?.and(0xFF) ?: 0))
modeState.cancelPending()
publish { it.copy(switching3d = false) }
scheduleModeCheck()
} else answer?.let { glassesMode = it }
VitureProtocol.MSG_GET_DISPLAY -> answer?.let { v ->
glassesMode = v
}
}
}
fun sampleImu(ms: Long): JSONObject {
val raw = java.util.Collections.synchronizedList(mutableListOf<Pair<ByteArray?, FloatArray?>>())
val counter = java.util.concurrent.atomic.AtomicInteger()
val began = SystemClock.elapsedRealtime()
probe = { payload, pose -> val n = counter.incrementAndGet(); if (raw.size < 4 || n % 60 == 0) raw.add(payload?.copyOf() to pose?.copyOf()) }
post { updateImu() }
SystemClock.sleep(ms)
probe = null
post { updateImu() }
val seconds = (SystemClock.elapsedRealtime() - began) / 1000.0
val count = counter.get()
val samples = JSONArray()
synchronized(raw) {
raw.forEach { (b, pose) ->
val rayneo = b != null && RayNeoProtocol.classifyPacket(b) is RayNeoPacket.Sensor
val q = pose ?: if (b == null || rayneo) null else VitureProtocol.imuQuaternion(b)
samples.put(JSONObject()
.put("hex", b?.joinToString("") { "%02X".format(it) } ?: JSONObject.NULL)
.put("euler", if (b == null || b.size < 12 || rayneo || pose != null) JSONObject.NULL else JSONArray().put(VitureProtocol.beFloat(b, 0).toDouble()).put(VitureProtocol.beFloat(b, 4).toDouble()).put(VitureProtocol.beFloat(b, 8).toDouble()))
.put("quat", q?.let { JSONArray().put(it[0].toDouble()).put(it[1].toDouble()).put(it[2].toDouble()).put(it[3].toDouble()) } ?: JSONObject.NULL))
}
}
return JSONObject().put("packets", count).put("rateHz", count / seconds).put("samples", samples)
}
private fun Intent.device(): UsbDevice? =
if (Build.VERSION.SDK_INT >= 33) getParcelableExtra(UsbManager.EXTRA_DEVICE, UsbDevice::class.java)
else @Suppress("DEPRECATION") getParcelableExtra(UsbManager.EXTRA_DEVICE)
private fun load(): GlassesSettings {
val p = app.getSharedPreferences(PREFS, Context.MODE_PRIVATE)
return GlassesSettings(
refreshHz = p.getInt("refreshHz", 120),
sbsHz = p.getInt("sbsHz", 60),
sunblock = if (p.contains("sunblock")) p.getBoolean("sunblock", false) else null,
gameOnGlasses = p.getBoolean("gameOnGlasses", true),
headTracking = p.getBoolean("headTracking", true),
auto3d = p.getBoolean("auto3d", GlassesSettings().auto3d),
picture3d = p.getInt("picture3d", GlassesSettings.PICTURE_AUTO),
screenWidthDm = p.getInt("screenWidthDm", 20),
screenDistanceDm = p.getInt("screenDistanceDm", 30),
vrFovDeg = p.getInt("vrFovDeg", GlassesSettings.VR_FOV_DEFAULT).coerceIn(GlassesSettings.VR_FOV_MIN, GlassesSettings.VR_FOV_MAX),
)
}
private fun save(s: GlassesSettings) {
app.getSharedPreferences(PREFS, Context.MODE_PRIVATE).edit().apply {
putInt("refreshHz", s.refreshHz)
putInt("sbsHz", s.sbsHz)
s.sunblock?.let { putBoolean("sunblock", it) } ?: remove("sunblock")
putBoolean("gameOnGlasses", s.gameOnGlasses)
putBoolean("headTracking", s.headTracking)
putBoolean("auto3d", s.auto3d)
putInt("picture3d", s.picture3d)
putInt("screenWidthDm", s.screenWidthDm)
putInt("screenDistanceDm", s.screenDistanceDm)
putInt("vrFovDeg", s.vrFovDeg)
}.apply()
}
}
@@ -0,0 +1,25 @@
package com.droiddeck.launcher.glasses
/** Hardware properties; display refresh is selected from Android's advertised modes. */
interface GlassesModel {
val name: String
val controllable: Boolean
val brightnessMax: Int
val volumeMax: Int
val diagonalFovDeg: Float
val sbs90: Boolean
val hasSunblock: Boolean get() = true
fun mode(hz: Int, sbs: Boolean): Int
fun modeForStep(step: Int): Int?
}
/** The command IDs here are the existing glasses controller's internal vocabulary.
* Each transport translates them to its own wire protocol; unsupported controls return false.
*/
internal interface GlassesLink {
val hasImu: Boolean
val badFrames: Long get() = 0L
val ready: Boolean get() = true
fun send(msgId: Int, payload: ByteArray = ByteArray(0)): Boolean
fun close()
}
@@ -0,0 +1,84 @@
package com.droiddeck.launcher.glasses
internal class GlassesModeState(private val timeoutMs: Long) {
enum class Confirmation { NONE, WAITING, CONFIRMED, FAILED }
data class Device(val productId: Int, val serial: String?)
private var deadline: Long? = null
private var device: Device? = null
private var attached = false
private var reconnectDeadline: Long? = null
private var vrOwned = false
var restorePending = false
private set
val pendingStereo: Boolean get() = deadline != null
val nextCheckAt: Long? get() = listOfNotNull(deadline, reconnectDeadline).minOrNull()
fun updateDevice(current: Device?, now: Long): Boolean {
if (reconnectDeadline?.let { now >= it } == true) abandon()
if (current == null) {
if (attached) reconnectDeadline = now + timeoutMs
attached = false
return false
}
val previous = device
val changed = previous == null || previous.productId != current.productId ||
(previous.serial != null && current.serial != null && previous.serial != current.serial)
if (changed) abandon()
device = if (!changed && current.serial == null) previous else current
attached = true
reconnectDeadline = null
return changed
}
private fun abandon() {
deadline = null
reconnectDeadline = null
vrOwned = false
restorePending = false
}
fun manual() {
vrOwned = false
restorePending = false
}
fun startVr() {
vrOwned = true
restorePending = false
}
fun stopVr(): Boolean {
if (vrOwned) {
vrOwned = false
restorePending = true
}
return restorePending
}
fun sent(stereo: Boolean, now: Long) {
deadline = if (stereo) now + timeoutMs else null
if (!stereo) restorePending = false
}
fun cancelPending() {
deadline = null
}
fun observe(stereo: Boolean, now: Long): Confirmation {
val until = deadline ?: return Confirmation.NONE
if (stereo) {
deadline = null
return Confirmation.CONFIRMED
}
if (now < until) return Confirmation.WAITING
if (!attached) {
abandon()
return Confirmation.NONE
}
deadline = null
vrOwned = false
restorePending = true
return Confirmation.FAILED
}
}
@@ -0,0 +1,67 @@
package com.droiddeck.launcher.glasses
import com.droiddeck.launcher.gpu.Cardboard
/** Share the actual picture mode with settings so controls match the rendered view. */
internal object GlassesRendering {
enum class Picture { FLAT, VIRTUAL_SCREEN, STEREO }
fun outputHz(settings: GlassesSettings, state: GlassesState): Int =
if (state.sbs) if (state.sbs90) settings.sbsHz else 60 else settings.refreshHz
fun mirrorRefreshLimit(settings: GlassesSettings, state: GlassesState, outputActive: Boolean): Int =
if (state.usable && !outputActive) outputHz(settings, state) else 0
fun fpsLimit(requested: Int, mirrorRefreshLimit: Int): Int = when {
mirrorRefreshLimit <= 0 -> requested
requested <= 0 -> mirrorRefreshLimit
else -> minOf(requested, mirrorRefreshLimit)
}
fun usesGlasses(state: GlassesState, outputActive: Boolean, steamVrRunning: Boolean): Boolean =
outputActive || (steamVrRunning && state.usable)
fun picture(settings: GlassesSettings, stereoDisplay: Boolean, steamVrRunning: Boolean): Picture =
when (settings.picture3d) {
GlassesSettings.PICTURE_SCREEN -> Picture.VIRTUAL_SCREEN
GlassesSettings.PICTURE_SBS -> Picture.STEREO
else -> when {
steamVrRunning -> Picture.STEREO
stereoDisplay -> Picture.VIRTUAL_SCREEN
else -> Picture.FLAT
}
}
fun cardboard(
base: Cardboard,
settings: GlassesSettings,
state: GlassesState,
outputActive: Boolean,
steamVrRunning: Boolean,
): Cardboard {
// Mirror output still uses the glasses' picture and sensors during VR.
if (!usesGlasses(state, outputActive, steamVrRunning)) {
// SteamVR uses the renderer's enabled flag as headset proximity.
// A flat phone preview still needs an active HMD and one SBS eye;
// otherwise SteamVR enters standby and leaves a frozen frame.
return if (steamVrRunning && !base.enabled) base.copy(
enabled = true,
source = Cardboard.SOURCE_SBS,
viewer = Cardboard.Viewer.NONE.id,
mono = true,
) else base
}
val picture = picture(settings, state.sbs, steamVrRunning)
if (picture == Picture.FLAT) return Cardboard.OFF
return Cardboard(
enabled = true,
source = if (picture == Picture.STEREO) Cardboard.SOURCE_SBS else Cardboard.SOURCE_SCREEN,
viewer = Cardboard.Viewer.VITURE.id,
ipdMm = base.ipdMm,
screenWidthDm = settings.screenWidthDm,
screenDistanceDm = settings.screenDistanceDm,
headTracking = settings.headTracking && state.canControl,
mono = !state.sbs,
)
}
}
@@ -0,0 +1,51 @@
package com.droiddeck.launcher.glasses
import android.app.Presentation
import android.content.Context
import android.graphics.Color
import android.os.Bundle
import android.util.Log
import android.view.Display
import android.view.SurfaceHolder
import android.view.SurfaceView
import android.view.ViewGroup
import android.view.WindowManager
import android.widget.FrameLayout
import kotlin.math.abs
class GlassesScreen(context: Context, display: Display, refreshHz: Int, private val callback: SurfaceHolder.Callback) : Presentation(context, display) {
val surfaceView = SurfaceView(getContext())
val holder: SurfaceHolder get() = surfaceView.holder
private var wantHz = refreshHz
val hz: Float get() = display.mode.refreshRate
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
val root = FrameLayout(context).apply { setBackgroundColor(Color.BLACK) }
root.addView(surfaceView, FrameLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.MATCH_PARENT))
setContentView(root)
window?.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
applyMode()
surfaceView.holder.addCallback(callback)
}
fun prefer(hz: Int) {
wantHz = hz
// A 2D/SBS transition can replace the active mode without changing Hz.
applyMode()
}
private fun applyMode() {
val w = window ?: return
val current = display.mode
val best = display.supportedModes.filter {
it.physicalWidth == current.physicalWidth && it.physicalHeight == current.physicalHeight
}.minByOrNull { abs(it.refreshRate - wantHz) } ?: return
if (w.attributes.preferredDisplayModeId != best.modeId) {
Log.i("GlassesScreen", "display ${display.displayId}: prefer mode ${best.modeId} " +
"${best.physicalWidth}x${best.physicalHeight}@${best.refreshRate} (requested $wantHz Hz)")
w.attributes = w.attributes.apply { preferredDisplayModeId = best.modeId }
}
}
}
@@ -0,0 +1,105 @@
package com.droiddeck.launcher.glasses
import android.content.SharedPreferences
import org.json.JSONArray
import org.json.JSONObject
enum class CalibrationPhase { IDLE, SETTLING, COLLECTING, MOVING, TURN_OVER, SAVED, FAILED }
data class GlassesCalibrationState(
val supported: Boolean = false,
val saved: Boolean = false,
val phase: CalibrationPhase = CalibrationPhase.IDLE,
val progress: Int = 0,
) {
val running: Boolean get() = phase in setOf(CalibrationPhase.SETTLING, CalibrationPhase.COLLECTING,
CalibrationPhase.MOVING, CalibrationPhase.TURN_OVER)
}
internal data class RayNeoCalibration(val gyroBiasDps: Vec3, val level: Quaternion) {
fun encode(): String = JSONObject().put("version", 1)
.put("bias", JSONArray(listOf(gyroBiasDps.x, gyroBiasDps.y, gyroBiasDps.z)))
.put("level", JSONArray(listOf(level.w, level.x, level.y, level.z))).toString()
companion object {
fun decode(json: String?): RayNeoCalibration? = runCatching {
val obj = JSONObject(json ?: return null)
require(obj.getInt("version") == 1)
val b = obj.getJSONArray("bias"); val q = obj.getJSONArray("level")
require(b.length() == 3 && q.length() == 4)
val bias = Vec3(b.getDouble(0).toFloat(), b.getDouble(1).toFloat(), b.getDouble(2).toFloat())
val level = Quaternion(q.getDouble(0).toFloat(), q.getDouble(1).toFloat(), q.getDouble(2).toFloat(), q.getDouble(3).toFloat())
require(listOf(bias.x, bias.y, bias.z, level.w, level.x, level.y, level.z).all { it.isFinite() })
require(bias.length() <= 6f && level.w in 0.96f..1.001f)
val norm = level.w * level.w + level.x * level.x + level.y * level.y + level.z * level.z
require(norm in 0.99f..1.01f)
RayNeoCalibration(bias, level.normalized())
}.getOrNull()
}
}
internal class RayNeoCalibrationStore(private val prefs: SharedPreferences) {
companion object {
// GT and GT Max can share USB identity fields. Keep the non-Max mounting
// correction separate from existing Max profiles and their legacy keys.
fun profileKey(base: String, gtUsb: Boolean, boardId: Int): String =
if (gtUsb && boardId == RayNeoProtocol.BOARD_GT) "$base:gt-standard-v1" else base
}
// A serial isolates physical units. Devices without one share their model profile.
fun key(vid: Int, pid: Int, serial: String?, modelName: String = ""): String =
"rayneo-cal-v1:$vid:$pid:${serial?.takeIf { it.isNotBlank() } ?: "model:$modelName"}"
fun load(key: String): RayNeoCalibration? = RayNeoCalibration.decode(prefs.getString(key, null))
fun save(key: String, value: RayNeoCalibration?) {
if (value == null) prefs.edit().remove(key).apply()
else prefs.edit().putString(key, value.encode()).apply()
}
}
/** One upside-down stationary pose estimates gyro bias and tilt, never yaw or accelerometer scale. */
internal class RayNeoCalibrationCollector(private val axes: RayNeoSensorAxes, private val beganMs: Long) {
var phase = CalibrationPhase.SETTLING; private set
var progress = 0; private set
private var windowStart = -1L
private var lastMs = -1L
private var count = 0
private var meanA = Vec3(0f, 0f, 0f)
private var meanG = Vec3(0f, 0f, 0f)
private var aVariance = 0f
private var gVariance = 0f
private fun clear(next: CalibrationPhase) {
phase = next; progress = 0; windowStart = -1; count = 0
meanA = Vec3(0f, 0f, 0f); meanG = meanA; aVariance = 0f; gVariance = 0f
}
fun update(sample: RayNeoSensorSample, nowMs: Long): RayNeoCalibration? {
if (phase == CalibrationPhase.FAILED || phase == CalibrationPhase.SAVED) return null
if (nowMs - beganMs >= 30_000) { clear(CalibrationPhase.FAILED); return null }
val gap = lastMs >= 0 && nowMs - lastMs > 250
lastMs = nowMs
if (nowMs - beganMs < 3000) return null
val a = axes.map(Vec3(sample.accelMps2[0], sample.accelMps2[1], sample.accelMps2[2]))
val g = Vec3(sample.gyroDps[0], sample.gyroDps[1], sample.gyroDps[2])
if (a.length() !in 8.5f..11f || !g.length().isFinite() || g.length() > 6f || gap) {
clear(CalibrationPhase.MOVING); return null
}
if (a.normalized().y > -0.94f) { clear(CalibrationPhase.TURN_OVER); return null }
if (windowStart < 0) windowStart = nowMs
phase = CalibrationPhase.COLLECTING
count++
val da = a - meanA; val dg = g - meanG
meanA += da * (1f / count); meanG += dg * (1f / count)
fun dot(x: Vec3, y: Vec3) = x.x * y.x + x.y * y.y + x.z * y.z
aVariance += dot(da, a - meanA); gVariance += dot(dg, g - meanG)
if (count > 25 && (aVariance / count > 0.04f || gVariance / count > 2.25f)) {
clear(CalibrationPhase.MOVING); return null
}
progress = ((nowMs - windowStart) * 100 / 5000).toInt().coerceIn(0, 99)
if (nowMs - windowStart < 5000 || count < 200) return null
val down = meanA.normalized()
val axis = down.cross(Vec3(0f, -1f, 0f))
val result = RayNeoCalibration(meanG, Quaternion(1f - down.y, axis.x, axis.y, axis.z).normalized())
phase = CalibrationPhase.SAVED; progress = 100
return result
}
}
@@ -0,0 +1,202 @@
package com.droiddeck.launcher.glasses
import android.hardware.usb.UsbDevice
import android.hardware.usb.UsbDeviceConnection
import android.hardware.usb.UsbManager
import android.os.SystemClock
import android.util.Log
/** RayNeo command/response and IMU share one endpoint. Keep reads on a single worker. */
internal class RayNeoLink private constructor(
private val connection: UsbDeviceConnection,
private val endpoints: RayNeoEndpointSelection,
private val listener: Listener,
private val gt: Boolean,
private val loadCalibration: (Int) -> RayNeoCalibration?,
) : GlassesLink {
interface Listener {
fun onReady()
fun onPose(raw: ByteArray, xyzw: FloatArray, timeNs: Long)
fun onLost()
fun onCalibration(revision: Long, phase: CalibrationPhase, progress: Int) {}
fun onCalibrationSaved(revision: Long, value: RayNeoCalibration?) {}
fun onDeviceInfo(boardId: Int, calibrationSaved: Boolean) {}
}
@Volatile private var running = true
@Volatile override var ready = false
private set
@Volatile private var wantImu = false
private val writeLock = Any()
private val assembler = RayNeoPacketAssembler()
private val readBuffer = ByteArray(maxOf(endpoints.inEndpoint.maxPacketSize, 64))
private var failures = 0
private lateinit var worker: Thread
private var calibration: RayNeoCalibration? = null
private data class CalibrationCommand(val revision: Long, val action: Int, val saved: RayNeoCalibration? = null)
private val calibrationCommand = java.util.concurrent.atomic.AtomicReference<CalibrationCommand?>(null)
fun startCalibration(revision: Long) { calibrationCommand.set(CalibrationCommand(revision, 1)) }
fun cancelCalibration(revision: Long, saved: RayNeoCalibration?) { calibrationCommand.set(CalibrationCommand(revision, 2, saved)) }
fun resetCalibration(revision: Long) { calibrationCommand.set(CalibrationCommand(revision, 3)) }
override val hasImu = true
override fun send(msgId: Int, payload: ByteArray): Boolean {
if (!running) return false
return when (msgId) {
VitureProtocol.MSG_IMU -> {
wantImu = payload.firstOrNull() == 1.toByte()
true
}
VitureProtocol.MSG_DISPLAY -> {
val mode = payload.firstOrNull()?.toInt()?.and(0xff)
ready && (mode == RayNeoProtocol.CMD_SWITCH_TO_3D || mode == RayNeoProtocol.CMD_SWITCH_TO_2D) &&
command(mode)
}
// No VITURE brightness, volume, film, rate or raw command is sent to RayNeo.
else -> false
}
}
private fun command(cmd: Int): Boolean = synchronized(writeLock) {
if (!running) return false
val packet = RayNeoProtocol.buildCommandPacket(cmd, 0, endpoints.outEndpoint.maxPacketSize)
connection.bulkTransfer(endpoints.outEndpoint, packet, packet.size, 250) == packet.size
}
private fun read(timeoutMs: Int): ByteArray? {
val deadline = SystemClock.elapsedRealtime() + timeoutMs
while (running && SystemClock.elapsedRealtime() < deadline) {
assembler.nextPacket()?.let { return it }
val began = SystemClock.elapsedRealtime()
val count = connection.bulkTransfer(endpoints.inEndpoint, readBuffer, readBuffer.size, 50)
if (count > 0) {
failures = 0
assembler.append(readBuffer, count)
} else if (count < 0 && SystemClock.elapsedRealtime() - began < 25) {
if (++failures >= 50) error("RayNeo USB link lost")
SystemClock.sleep(5)
}
}
return null
}
private fun response(cmd: Int, timeoutMs: Int): RayNeoPacket.Response? {
val deadline = SystemClock.elapsedRealtime() + timeoutMs
while (running && SystemClock.elapsedRealtime() < deadline) {
val packet = read(100)?.let(RayNeoProtocol::classifyPacket)
if (packet is RayNeoPacket.Response && packet.cmd == cmd) return packet
}
return null
}
private fun run() {
var streaming = false
try {
check(command(RayNeoProtocol.CMD_ACQUIRE_DEVICE_INFO)) { "RayNeo device-info write failed" }
val info = response(RayNeoProtocol.CMD_ACQUIRE_DEVICE_INFO, 2500)
?.let { RayNeoProtocol.parseDeviceInfo(it.raw) } ?: error("RayNeo device-info timed out")
val profile = RayNeoSensorProfile.forDevice(gt, info.boardId)
val rotation = profile.rotationXDeg
val gtMaxAxes = profile.gtMaxAxes
calibration = loadCalibration(info.boardId)
listener.onDeviceInfo(info.boardId, calibration != null)
Log.i("RayNeoLink", "Device board=0x%02x gt=%s".format(info.boardId, gt))
var filter = RayNeoOrientationFilter(rotation, gtMaxAxes, calibration)
var collector: RayNeoCalibrationCollector? = null
var calibrationRevision = 0L
var lastCalibrationUpdate = 0L
var warmupUntil = 0L
var lastSensor = 0L
ready = true
listener.onReady()
while (running) {
val request = calibrationCommand.getAndSet(null)
if (request != null) calibrationRevision = request.revision
when (request?.action) {
1 -> collector = RayNeoCalibrationCollector(RayNeoSensorAxes(rotation, gtMaxAxes), SystemClock.elapsedRealtime())
2 -> {
collector = null
if (calibration != request.saved) {
calibration = request.saved
filter = RayNeoOrientationFilter(rotation, gtMaxAxes, calibration)
}
listener.onCalibration(calibrationRevision, CalibrationPhase.IDLE, 0)
}
3 -> {
collector = null; calibration = null
filter = RayNeoOrientationFilter(rotation, gtMaxAxes)
listener.onCalibrationSaved(calibrationRevision, null)
listener.onCalibration(calibrationRevision, CalibrationPhase.IDLE, 0)
}
}
if (wantImu != streaming) {
val cmd = if (wantImu) RayNeoProtocol.CMD_OPEN_IMU else RayNeoProtocol.CMD_CLOSE_IMU
check(command(cmd) && response(cmd, 1500) != null) { "RayNeo IMU command timed out" }
streaming = cmd == RayNeoProtocol.CMD_OPEN_IMU
filter = RayNeoOrientationFilter(rotation, gtMaxAxes, calibration)
lastSensor = SystemClock.elapsedRealtime()
warmupUntil = lastSensor + 2000
}
val raw = read(100)
if (streaming && SystemClock.elapsedRealtime() - lastSensor > 4000) error("RayNeo IMU stream stalled")
if (raw == null) continue
val packet = RayNeoProtocol.classifyPacket(raw)
if (streaming && packet is RayNeoPacket.Sensor) {
lastSensor = SystemClock.elapsedRealtime()
collector?.let { c ->
val result = c.update(packet.sample, lastSensor)
if (result != null && calibrationCommand.get() == null) {
calibration = result
filter = RayNeoOrientationFilter(rotation, gtMaxAxes, result)
warmupUntil = lastSensor + 2000
listener.onCalibrationSaved(calibrationRevision, result)
}
if (lastSensor - lastCalibrationUpdate >= 200 || result != null || c.phase == CalibrationPhase.FAILED) {
listener.onCalibration(calibrationRevision, c.phase, c.progress)
lastCalibrationUpdate = lastSensor
}
if (result != null || c.phase == CalibrationPhase.FAILED) collector = null
}
val wxyz = filter.update(packet.sample)
if (wantImu && collector == null && lastSensor >= warmupUntil) {
// Normalize basis and component order at the hardware boundary.
listener.onPose(raw, RayNeoOrientationFilter.glassesPose(wxyz),
SystemClock.elapsedRealtimeNanos())
}
}
}
} catch (e: Exception) {
if (running) Log.w("RayNeoLink", "RayNeo stream failed", e)
} finally {
ready = false
if (running) listener.onLost()
}
}
override fun close() {
synchronized(writeLock) {
if (!running) return
// Best effort: stop the sensor before releasing the claimed interface.
if (ready) runCatching { command(RayNeoProtocol.CMD_CLOSE_IMU) }
running = false
}
if (Thread.currentThread() !== worker) worker.join(500)
runCatching { connection.releaseInterface(endpoints.usbInterface) }
connection.close()
}
companion object {
fun open(usb: UsbManager, device: UsbDevice, listener: Listener, loadCalibration: (Int) -> RayNeoCalibration? = { null }): RayNeoLink? {
val endpoints = RayNeoProtocol.selectEndpoints(device) ?: return null
val conn = usb.openDevice(device) ?: return null
if (!conn.claimInterface(endpoints.usbInterface, true)) {
conn.close()
return null
}
return RayNeoLink(conn, endpoints, listener, RayNeoProtocol.isGt(device.vendorId, device.productId), loadCalibration).also { link ->
link.worker = Thread({ link.run() }, "rayneo-imu").apply { isDaemon = true }
link.worker.start()
}
}
}
}
@@ -0,0 +1,42 @@
package com.droiddeck.launcher.glasses
object RayNeoModel {
private val modelName = Regex("(?i)(ray\\s*neo|tcl).*\\b(air|gt)(?=\\b|[0-9])")
fun isVendor(vid: Int) = vid == RayNeoProtocol.VENDOR_ID ||
vid == RayNeoProtocol.GT_VENDOR_ID || vid == 0x22d9
fun model(vid: Int, pid: Int, name: String?): GlassesModel? {
if (!isVendor(vid)) return null
// GT's identity is confirmed by verncat's SDK; do not claim other products
// from that vendor just because their name resembles a supported model.
val gt = RayNeoProtocol.isGt(vid, pid)
if (vid == RayNeoProtocol.GT_VENDOR_ID && !gt) return null
if (!(vid == RayNeoProtocol.VENDOR_ID && pid == RayNeoProtocol.PRODUCT_ID) &&
!gt &&
!modelName.containsMatchIn(name.orEmpty())) return null
return Model(name?.trim()?.takeIf { modelName.containsMatchIn(it) }
?: if (gt) "RayNeo GT series" else "RayNeo Air")
}
fun isDisplay(name: String): Boolean =
name.contains("rayneo", ignoreCase = true) || name.contains("ray neo", ignoreCase = true) ||
modelName.containsMatchIn(name)
// Observed on GT Max: Android exposes an unbranded EDID monitor name.
// Match its product metadata, not every display called SmartGlasses/TCL.
fun isDisplayProduct(name: String?, manufacturer: String?, productId: String?): Boolean =
name?.trim().equals("SmartGlasses", ignoreCase = true) &&
manufacturer.equals("TCL", ignoreCase = true) && productId == "980"
private class Model(override val name: String) : GlassesModel {
override val controllable = true
override val brightnessMax = 0
override val volumeMax = 0
override val diagonalFovDeg = 46f // Conservative existing default, not a calibrated model profile.
override val sbs90 = false
override fun mode(hz: Int, sbs: Boolean) =
if (sbs) RayNeoProtocol.CMD_SWITCH_TO_3D else RayNeoProtocol.CMD_SWITCH_TO_2D
override fun modeForStep(step: Int): Int? = null
}
}
@@ -0,0 +1,190 @@
// Adapted from informalTechCode/RayNeo-Air-Series-3DoF-Scaffold (Apache-2.0).
// See docs/third-party/rayneo/. Returns scaffold quaternions in (w, x, y, z) order.
package com.droiddeck.launcher.glasses
import kotlin.math.cos
import kotlin.math.sin
import kotlin.math.sqrt
internal data class Vec3(val x: Float, val y: Float, val z: Float) {
fun length(): Float = sqrt(x * x + y * y + z * z)
fun normalized(): Vec3 {
val n = length()
if (n < 1e-6f) {
return Vec3(0f, 0f, 0f)
}
return Vec3(x / n, y / n, z / n)
}
fun cross(other: Vec3): Vec3 {
return Vec3(
y * other.z - z * other.y,
z * other.x - x * other.z,
x * other.y - y * other.x
)
}
operator fun plus(other: Vec3): Vec3 = Vec3(x + other.x, y + other.y, z + other.z)
operator fun minus(other: Vec3): Vec3 = Vec3(x - other.x, y - other.y, z - other.z)
operator fun times(scale: Float): Vec3 = Vec3(x * scale, y * scale, z * scale)
}
internal data class Quaternion(val w: Float, val x: Float, val y: Float, val z: Float) {
operator fun times(other: Quaternion): Quaternion {
return Quaternion(
w = w * other.w - x * other.x - y * other.y - z * other.z,
x = w * other.x + x * other.w + y * other.z - z * other.y,
y = w * other.y - x * other.z + y * other.w + z * other.x,
z = w * other.z + x * other.y - y * other.x + z * other.w
)
}
operator fun times(scale: Float): Quaternion {
return Quaternion(w * scale, x * scale, y * scale, z * scale)
}
fun normalized(): Quaternion {
val n = sqrt(w * w + x * x + y * y + z * z)
if (n < 1e-6f) {
return IDENTITY
}
return Quaternion(w / n, x / n, y / n, z / n)
}
fun conjugate(): Quaternion = Quaternion(w, -x, -y, -z)
fun rotate(v: Vec3): Vec3 {
val qv = Quaternion(0f, v.x, v.y, v.z)
val r = this * qv * conjugate()
return Vec3(r.x, r.y, r.z)
}
fun integrateBodyRate(omegaRadPerSec: Vec3, dtSec: Float): Quaternion {
val omega = Quaternion(0f, omegaRadPerSec.x, omegaRadPerSec.y, omegaRadPerSec.z)
val qDot = (this * omega) * 0.5f
return Quaternion(
w + qDot.w * dtSec,
x + qDot.x * dtSec,
y + qDot.y * dtSec,
z + qDot.z * dtSec
)
}
companion object {
val IDENTITY = Quaternion(1f, 0f, 0f, 0f)
fun fromAxisAngle(axis: Vec3, angleRad: Float): Quaternion {
val nAxis = axis.normalized()
val half = angleRad * 0.5f
val s = sin(half)
val c = cos(half)
return Quaternion(c, nAxis.x * s, nAxis.y * s, nAxis.z * s).normalized()
}
}
}
internal data class RayNeoSensorProfile(val rotationXDeg: Float, val gtMaxAxes: Boolean) {
companion object {
fun forDevice(gtUsb: Boolean, boardId: Int) = when {
gtUsb && boardId == RayNeoProtocol.BOARD_GT -> RayNeoSensorProfile(-25f, false)
gtUsb -> RayNeoSensorProfile(35f, true)
boardId == RayNeoProtocol.BOARD_AIR_4_PRO -> RayNeoSensorProfile(-20f, false)
else -> RayNeoSensorProfile(0f, false)
}
}
}
internal class RayNeoSensorAxes(imuRotationXDeg: Float, private val gt: Boolean) {
private val rotation = Quaternion.fromAxisAngle(Vec3(1f, 0f, 0f),
Math.toRadians((if (gt) 35f else imuRotationXDeg).toDouble()).toFloat())
fun map(v: Vec3): Vec3 = rotation.rotate(if (gt) Vec3(-v.y, v.z, -v.x) else v)
}
internal class RayNeoOrientationFilter(
imuRotationXDeg: Float,
gt: Boolean = false,
private val calibration: RayNeoCalibration? = null,
) {
private val worldUp = Vec3(0f, 1f, 0f)
private val accelGain = 1.5f
private val dpsToRad = (Math.PI.toFloat() / 180f)
private val gravityMps2 = 9.81f
private val stationaryAccelTol = 1.25f
private val stationaryGyroRadPerSec = 0.18f
private val gyroBiasUpdateHz = 0.5f
private val axes = RayNeoSensorAxes(imuRotationXDeg, gt)
private var q = Quaternion.IDENTITY
private var gravityInitialized = false
private var gyroBias = Vec3(0f, 0f, 0f)
private var lastTick100us: Long = -1L
private var lastRealtimeNs: Long = 0L
companion object {
/** Scaffold: right +X, up +Y, back +Z. VITURE: right -Y, up +Z, back -X.
* Conjugate by HeadPose.GLASSES_AXES before entering the unchanged VITURE pose path.
* This is a basis conversion, not a projection or seated-origin adjustment.
*/
fun glassesPose(wxyz: FloatArray): FloatArray =
floatArrayOf(-wxyz[3], -wxyz[1], wxyz[2], wxyz[0])
}
fun update(sample: RayNeoSensorSample): FloatArray {
var accel = Vec3(sample.accelMps2[0], sample.accelMps2[1], sample.accelMps2[2])
var gyro = Vec3(sample.gyroDps[0], sample.gyroDps[1], sample.gyroDps[2])
calibration?.let { gyro -= it.gyroBiasDps }
accel = axes.map(accel)
gyro = axes.map(gyro) * dpsToRad
calibration?.let {
accel = it.level.rotate(accel)
gyro = it.level.rotate(gyro)
}
val nowNs = System.nanoTime()
var dtSec = 0.01f
if (lastTick100us >= 0L && sample.deviceTick100us != lastTick100us) {
dtSec = (((sample.deviceTick100us - lastTick100us) and 0xffffffffL).toFloat() * 1e-4f)
} else if (lastRealtimeNs > 0L && nowNs > lastRealtimeNs) {
dtSec = ((nowNs - lastRealtimeNs).toFloat() * 1e-9f)
}
if (dtSec < 0.001f || dtSec > 0.1f) {
dtSec = 0.01f
}
lastTick100us = sample.deviceTick100us
lastRealtimeNs = nowNs
val accelNorm = accel.length()
val stationary =
kotlin.math.abs(accelNorm - gravityMps2) < stationaryAccelTol &&
gyro.length() < stationaryGyroRadPerSec
if (stationary) {
val alpha = minOf(1f, dtSec * gyroBiasUpdateHz)
gyroBias = gyroBias * (1f - alpha) + gyro * alpha
}
gyro -= gyroBias
if (accelNorm > 1e-3f) {
val measuredUp = accel.normalized()
if (!gravityInitialized) {
// Align gravity before publishing the first pose. Starting at
// identity and settling after recenter can rotate the horizon.
val axis = measuredUp.cross(worldUp)
q = if (measuredUp.y < -0.9999f) {
Quaternion.fromAxisAngle(Vec3(1f, 0f, 0f), Math.PI.toFloat())
} else {
Quaternion(1f + measuredUp.y, axis.x, axis.y, axis.z).normalized()
}
gravityInitialized = true
}
val predictedUp = q.conjugate().rotate(worldUp).normalized()
// q maps body to world: measured x predicted reduces gravity error.
val correction = measuredUp.cross(predictedUp)
gyro += correction * accelGain
}
q = q.integrateBodyRate(gyro, dtSec).normalized()
return floatArrayOf(q.w, q.x, q.y, q.z)
}
}
@@ -0,0 +1,270 @@
// Adapted from informalTechCode/RayNeo-Air-Series-3DoF-Scaffold (Apache-2.0).
// See docs/third-party/rayneo/ for license and protocol attribution.
package com.droiddeck.launcher.glasses
import android.hardware.usb.UsbConstants
import android.hardware.usb.UsbDevice
import android.hardware.usb.UsbEndpoint
import android.hardware.usb.UsbInterface
import java.util.Arrays
private fun Byte.u(): Int = toInt() and 0xFF
data class RayNeoSensorSample(
val accelMps2: FloatArray,
val gyroDps: FloatArray,
val temperatureC: Float,
val magnet: FloatArray,
val proximity: Float,
val light: Float,
val deviceTick100us: Long
)
data class RayNeoDeviceInfo(
val boardId: Int,
val sideBySideEnabled: Boolean
)
sealed interface RayNeoPacket {
data class Sensor(val sample: RayNeoSensorSample) : RayNeoPacket
data class Response(val cmd: Int, val raw: ByteArray) : RayNeoPacket
data object Unknown : RayNeoPacket
}
data class RayNeoEndpointSelection(
val usbInterface: UsbInterface,
val inEndpoint: UsbEndpoint,
val outEndpoint: UsbEndpoint
)
object RayNeoProtocol {
const val VENDOR_ID = 0x1BBB
const val PRODUCT_ID = 0xAF50
// verncat/RayNeo-Air-3S-Pro-OpenVR PR #3 (MIT), tested on GT hardware.
const val GT_VENDOR_ID = 0x3941
private const val GT_CONTROL_INTERFACE = 5
fun isGt(vid: Int, pid: Int) = vid == GT_VENDOR_ID && pid == PRODUCT_ID
const val BOARD_AIR_4_PRO = 0x3A
const val BOARD_GT = 0x40 // Observed on the user's non-Max GT.
const val PKT_MAGIC = 0x99
const val PKT_SENSOR = 0x65
const val PKT_RESPONSE = 0xC8
const val CMD_ACQUIRE_DEVICE_INFO = 0
const val CMD_OPEN_IMU = 1
const val CMD_CLOSE_IMU = 2
const val CMD_SWITCH_TO_3D = 6
const val CMD_SWITCH_TO_2D = 7
private const val RESPONSE_CMD_OFFSET = 8
private const val ACCEL_OFFSET = 4
private const val GYRO_OFFSET = 16
private const val TEMP_OFFSET = 28
private const val MAG_X_OFFSET = 32
private const val TICK_OFFSET = 40
private const val PSENSOR_OFFSET = 44
private const val LSENSOR_OFFSET = 48
private const val MAG_Z_OFFSET = 52
private const val DEVINFO_BOARD_ID_OFFSET = 21
private const val DEVINFO_SIDE_BY_SIDE_OFFSET = 43
fun buildCommandPacket(cmd: Int, arg: Int, packetSize: Int): ByteArray {
val len = maxOf(packetSize, 64)
val pkt = ByteArray(len)
pkt[0] = 0x66
pkt[1] = cmd.toByte()
pkt[2] = arg.toByte()
return pkt
}
fun classifyPacket(raw: ByteArray): RayNeoPacket {
if (raw.size < 4 || raw[0].u() != PKT_MAGIC || raw[2].u() != raw.size) {
return RayNeoPacket.Unknown
}
return when (raw[1].u()) {
PKT_SENSOR -> {
val sample = parseSensorSample(raw)
if (sample != null) RayNeoPacket.Sensor(sample) else RayNeoPacket.Unknown
}
PKT_RESPONSE -> {
if (raw.size <= RESPONSE_CMD_OFFSET) {
RayNeoPacket.Unknown
} else {
RayNeoPacket.Response(raw[RESPONSE_CMD_OFFSET].u(), raw)
}
}
else -> RayNeoPacket.Unknown
}
}
fun parseDeviceInfo(raw: ByteArray): RayNeoDeviceInfo? {
if (raw.size <= DEVINFO_SIDE_BY_SIDE_OFFSET || raw[2].u() != raw.size || raw[0].u() != PKT_MAGIC ||
raw[1].u() != PKT_RESPONSE || raw[RESPONSE_CMD_OFFSET].u() != CMD_ACQUIRE_DEVICE_INFO) {
return null
}
return RayNeoDeviceInfo(
boardId = raw[DEVINFO_BOARD_ID_OFFSET].u(),
sideBySideEnabled = raw[DEVINFO_SIDE_BY_SIDE_OFFSET].u() != 0
)
}
fun selectEndpoints(device: UsbDevice): RayNeoEndpointSelection? {
val gt = isGt(device.vendorId, device.productId)
val gtCandidates = mutableListOf<RayNeoEndpointSelection>()
var fallback: RayNeoEndpointSelection? = null
for (i in 0 until device.interfaceCount) {
val usbInterface = device.getInterface(i)
var inEndpoint: UsbEndpoint? = null
var outEndpoint: UsbEndpoint? = null
for (j in 0 until usbInterface.endpointCount) {
val endpoint = usbInterface.getEndpoint(j)
if (endpoint.type != UsbConstants.USB_ENDPOINT_XFER_INT &&
endpoint.type != UsbConstants.USB_ENDPOINT_XFER_BULK) {
continue
}
// Exclude small button reports from GT's protocol transport.
if (gt && endpoint.maxPacketSize < 64) continue
if (endpoint.direction == UsbConstants.USB_DIR_IN && inEndpoint == null) {
inEndpoint = endpoint
} else if (endpoint.direction == UsbConstants.USB_DIR_OUT && outEndpoint == null) {
outEndpoint = endpoint
}
}
if (inEndpoint != null && outEndpoint != null) {
val selection = RayNeoEndpointSelection(usbInterface, inEndpoint, outEndpoint)
if (gt) {
if (usbInterface.id == GT_CONTROL_INTERFACE) return selection
gtCandidates += selection
continue
}
if (inEndpoint.type == UsbConstants.USB_ENDPOINT_XFER_INT &&
outEndpoint.type == UsbConstants.USB_ENDPOINT_XFER_INT
) {
return selection
}
if (fallback == null) {
fallback = selection
}
}
}
// Interface numbers can change with firmware. Use a unique protocol-sized
// pair when interface 5 is absent, never an arbitrary composite interface.
return if (gt) gtCandidates.singleOrNull() else fallback
}
private fun parseSensorSample(raw: ByteArray): RayNeoSensorSample? {
if (raw.size < MAG_Z_OFFSET + 4 ||
(ACCEL_OFFSET until TEMP_OFFSET step 4).any { !readF32LE(raw, it).isFinite() }) {
return null
}
return RayNeoSensorSample(
accelMps2 = floatArrayOf(
readF32LE(raw, ACCEL_OFFSET + 0),
readF32LE(raw, ACCEL_OFFSET + 4),
readF32LE(raw, ACCEL_OFFSET + 8)
),
gyroDps = floatArrayOf(
readF32LE(raw, GYRO_OFFSET + 0),
readF32LE(raw, GYRO_OFFSET + 4),
readF32LE(raw, GYRO_OFFSET + 8)
),
temperatureC = readF32LE(raw, TEMP_OFFSET),
magnet = floatArrayOf(
readF32LE(raw, MAG_X_OFFSET),
readF32LE(raw, MAG_X_OFFSET + 4),
readF32LE(raw, MAG_Z_OFFSET)
),
proximity = readF32LE(raw, PSENSOR_OFFSET),
light = readF32LE(raw, LSENSOR_OFFSET),
deviceTick100us = readU32LE(raw, TICK_OFFSET).toLong() and 0xffffffffL
)
}
private fun readU32LE(raw: ByteArray, offset: Int): Int {
val b0 = raw[offset].u()
val b1 = raw[offset + 1].u()
val b2 = raw[offset + 2].u()
val b3 = raw[offset + 3].u()
return b0 or (b1 shl 8) or (b2 shl 16) or (b3 shl 24)
}
private fun readF32LE(raw: ByteArray, offset: Int): Float {
return Float.fromBits(readU32LE(raw, offset))
}
}
class RayNeoPacketAssembler {
private val stash = ByteArray(4096)
private var len = 0
fun append(data: ByteArray, count: Int) {
require(count in 0..data.size)
if (count <= 0) {
return
}
if (count >= stash.size || len + count > stash.size) {
len = 0
}
val copyLen = minOf(count, stash.size - len)
System.arraycopy(data, 0, stash, len, copyLen)
len += copyLen
}
fun nextPacket(): ByteArray? {
var offset = 0
while (offset + 4 <= len) {
if (stash[offset].u() != RayNeoProtocol.PKT_MAGIC) {
offset++
continue
}
val packetLen = stash[offset + 2].u()
if (packetLen < 4 || packetLen > 64 ||
stash[offset + 1].u() !in listOf(RayNeoProtocol.PKT_SENSOR, RayNeoProtocol.PKT_RESPONSE)) {
offset++
continue
}
if (offset + packetLen > len) {
break
}
val packet = Arrays.copyOfRange(stash, offset, offset + packetLen)
compact(offset + packetLen)
return packet
}
if (offset > 0) {
compact(offset)
}
return null
}
private fun compact(consumed: Int) {
if (consumed <= 0) {
return
}
if (consumed >= len) {
len = 0
return
}
System.arraycopy(stash, consumed, stash, 0, len - consumed)
len -= consumed
}
}
@@ -0,0 +1,134 @@
package com.droiddeck.launcher.glasses
import android.hardware.usb.UsbConstants
import android.hardware.usb.UsbDevice
import android.hardware.usb.UsbDeviceConnection
import android.hardware.usb.UsbEndpoint
import android.hardware.usb.UsbInterface
import android.hardware.usb.UsbManager
import android.os.SystemClock
import android.util.Log
import java.util.concurrent.atomic.AtomicLong
internal class VitureLink private constructor(
private val connection: UsbDeviceConnection,
private val mcu: UsbInterface,
private val mcuOut: UsbEndpoint,
private val mcuIn: UsbEndpoint?,
private val imu: UsbInterface?,
private val imuIn: UsbEndpoint?,
private val listener: Listener,
) : GlassesLink {
interface Listener {
fun onFrame(frame: VitureProtocol.Frame)
fun onImu(payload: ByteArray, timeNs: Long)
fun onLost()
}
@Volatile private var running = true
private val writeLock = Any()
private var lastSendAt = 0L
private val readers = mutableListOf<Thread>()
private val bad = AtomicLong()
override val hasImu: Boolean get() = imuIn != null
override val badFrames: Long get() = bad.get()
private fun startReaders() {
mcuIn?.let { ep -> readers += reader("viture-mcu", ep) { buf, n -> onPacket(buf, n, false) } }
imuIn?.let { ep ->
readers += reader("viture-imu", ep) { buf, n -> onPacket(buf, n, true) }
}
}
private fun onPacket(buf: ByteArray, size: Int, imuEndpoint: Boolean) {
val frame = VitureProtocol.parse(buf, size) ?: return
if (!frame.crcOk) {
if (bad.incrementAndGet() == 1L) Log.w(TAG, "dropping frames with a bad checksum")
return
}
if (imuEndpoint && frame.magic == VitureProtocol.MAGIC_IMU)
listener.onImu(frame.payload, SystemClock.elapsedRealtimeNanos())
else listener.onFrame(frame)
}
private fun reader(name: String, ep: UsbEndpoint, onPacket: (ByteArray, Int) -> Unit): Thread =
Thread({
val buf = ByteArray(VitureProtocol.PACKET_SIZE)
var failures = 0
while (running) {
val started = SystemClock.uptimeMillis()
val n = connection.bulkTransfer(ep, buf, buf.size, READ_TIMEOUT_MS)
if (!running) break
if (n > 0) {
failures = 0
runCatching { onPacket(buf, n) }.onFailure { Log.w(TAG, "$name packet", it) }
} else if (n < 0 && SystemClock.uptimeMillis() - started < READ_TIMEOUT_MS / 2) {
failures++
SystemClock.sleep(2)
if (failures >= LOST_AFTER) {
Log.w(TAG, "$name: reads keep failing, link lost")
if (running) listener.onLost()
break
}
}
}
}, name).apply { isDaemon = true; priority = Thread.MAX_PRIORITY; start() }
override fun send(msgId: Int, payload: ByteArray): Boolean {
val pkt = VitureProtocol.packet(msgId, payload)
val n = synchronized(writeLock) {
if (!running) return false
val wait = lastSendAt + COMMAND_GAP_MS - SystemClock.uptimeMillis()
if (wait > 0) SystemClock.sleep(wait)
connection.bulkTransfer(mcuOut, pkt, pkt.size, WRITE_TIMEOUT_MS).also { lastSendAt = SystemClock.uptimeMillis() }
}
Log.i(TAG, "send 0x%04X [%s] -> %d".format(msgId, payload.joinToString("") { "%02X".format(it) }, n))
return n == pkt.size
}
override fun close() {
synchronized(writeLock) { running = false }
val self = Thread.currentThread()
readers.forEach { if (it !== self) runCatching { it.join(READ_TIMEOUT_MS * 4L) } }
runCatching { connection.releaseInterface(mcu) }
imu?.let { runCatching { connection.releaseInterface(it) } }
runCatching { connection.close() }
}
companion object {
private const val TAG = "VitureLink"
private const val READ_TIMEOUT_MS = 100
private const val WRITE_TIMEOUT_MS = 250
private const val LOST_AFTER = 50
const val COMMAND_GAP_MS = 60L
fun open(usb: UsbManager, device: UsbDevice, listener: Listener): VitureLink? {
val hid = (0 until device.interfaceCount).map(device::getInterface)
.filter { it.interfaceClass == UsbConstants.USB_CLASS_HID }
.sortedBy { it.id }
for (intf in hid) Log.i(TAG, "iface ${intf.id} eps " + (0 until intf.endpointCount).joinToString {
val ep = intf.getEndpoint(it)
"0x%02X".format(ep.address)
})
val mcu = hid.lastOrNull { it.outEndpoint() != null } ?: return null
val imu = hid.firstOrNull { it.id != mcu.id && it.inEndpoint() != null }
val conn = usb.openDevice(device) ?: return null
if (!conn.claimInterface(mcu, true)) {
conn.close()
return null
}
val imuClaimed = imu != null && conn.claimInterface(imu, true)
val link = VitureLink(conn, mcu, mcu.outEndpoint()!!, mcu.inEndpoint(),
imu.takeIf { imuClaimed }, imu.takeIf { imuClaimed }?.inEndpoint(), listener)
link.startReaders()
return link
}
private fun UsbInterface.outEndpoint(): UsbEndpoint? = (0 until endpointCount).map(::getEndpoint)
.firstOrNull { it.direction == UsbConstants.USB_DIR_OUT && it.type == UsbConstants.USB_ENDPOINT_XFER_INT }
private fun UsbInterface.inEndpoint(): UsbEndpoint? = (0 until endpointCount).map(::getEndpoint)
.firstOrNull { it.direction == UsbConstants.USB_DIR_IN && it.type == UsbConstants.USB_ENDPOINT_XFER_INT }
}
}
@@ -0,0 +1,186 @@
package com.droiddeck.launcher.glasses
import kotlin.math.cos
import kotlin.math.sin
import kotlin.math.sqrt
object VitureProtocol {
const val VENDOR_ID = 0x35CA
const val PACKET_SIZE = 64
private const val HEADER = 0x12
const val MAGIC_COMMAND = 0xFE
const val MAGIC_REPORT = 0xFD
const val MAGIC_IMU = 0xFC
const val MSG_GET_BRIGHTNESS = 0x0005
const val MSG_BRIGHTNESS = 0x0006
const val MSG_GET_DISPLAY = 0x0007
const val MSG_DISPLAY = 0x0008
const val MSG_FILM = 0x000E
const val MSG_IMU = 0x0015
const val MSG_GET_IMU = 0x0017
const val MSG_IMU_RATE = 0x0018
const val MSG_GET_VOLUME = 0x0032
const val MSG_VOLUME = 0x0033
const val EVENT_BRIGHTNESS = 0x0301
const val EVENT_DISPLAY = 0x0302
const val EVENT_VOLUME = 0x0304
const val EVENT_DISPLAY_STEP = 0x030D
const val IMU_RATE_240 = 3
enum class Family { ONE, PRO, LUMA, ULTRA, BEAST, PRO2 }
class Model(override val name: String, val family: Family) : GlassesModel {
override val controllable: Boolean get() = true
val needsSdk: Boolean get() = family == Family.ULTRA || family == Family.BEAST || family == Family.PRO2
val tall: Boolean get() = family == Family.LUMA || family == Family.ULTRA || family == Family.BEAST
override val sbs90: Boolean get() = family == Family.ULTRA
override val brightnessMax: Int get() = 8
override val volumeMax: Int get() = if (family == Family.BEAST) 15 else 8
override val hasSunblock: Boolean get() = family != Family.PRO2
override val diagonalFovDeg: Float get() = when (family) {
Family.ONE -> 43f
Family.PRO -> 46f
Family.PRO2, Family.LUMA, Family.ULTRA -> 50f
Family.BEAST -> 58f
}
override fun mode(hz: Int, sbs: Boolean): Int = when {
tall && sbs -> if (hz >= 90 && sbs90) 0x45 else 0x42
tall -> when { hz >= 120 -> 0x44; hz >= 90 -> 0x43; else -> 0x41 }
sbs -> if (hz >= 90 && sbs90) 0x35 else 0x32
else -> when { hz >= 120 -> 0x34; hz >= 90 -> 0x33; else -> 0x31 }
}
override fun modeForStep(step: Int): Int? = if (tall) null else when (step) {
1 -> 0x31
2 -> 0x33
3 -> 0x34
4 -> 0x32
else -> null
}
}
/** SDK validates the PID before returning this market name. Never guess a new PID. */
fun sdkModel(marketName: String): Model? {
val family = when (marketName) {
"One", "Lite" -> Family.ONE
"Pro" -> Family.PRO
"Pro 2" -> Family.PRO2
"Luma", "Luma Pro", "Luma Cyber" -> Family.LUMA
"Luma Ultra" -> Family.ULTRA
"Beast" -> Family.BEAST
else -> return null
}
return Model("VITURE $marketName", family)
}
fun model(productId: Int, productName: String?): Model? {
val family = when (productId) {
0x1011, 0x1013, 0x1015, 0x1017, 0x101b -> Family.ONE
0x1019, 0x101d -> Family.PRO
0x1121, 0x1131, 0x1141, 0x1151 -> Family.LUMA
0x1101, 0x1104 -> Family.ULTRA
0x1201, 0x1211 -> Family.BEAST
else -> return null
}
val fallback = when (productId) {
0x1015, 0x101b -> "VITURE One Lite"
0x1011, 0x1013, 0x1017 -> "VITURE One"
0x1019 -> "VITURE Pro"
0x101d -> "VITURE Pro XR"
0x1131 -> "VITURE Luma"
0x1121, 0x1141, 0x1151 -> "VITURE Luma Pro"
0x1101, 0x1104 -> "VITURE Luma Ultra"
else -> "VITURE Beast"
}
val name = productName?.trim()?.removeSuffix(" Glasses")?.takeIf { it.startsWith("VITURE", ignoreCase = true) } ?: fallback
return Model(name, family)
}
private val CRC_TABLE = IntArray(256) { i ->
var c = i shl 8
repeat(8) { c = if (c and 0x8000 != 0) (c shl 1) xor 0x1021 else c shl 1 }
c and 0xFFFF
}
fun crc16(data: ByteArray, offset: Int, length: Int): Int {
var crc = 0
for (i in offset until offset + length) {
crc = (CRC_TABLE[((data[i].toInt() and 0xFF) xor (crc ushr 8)) and 0xFF] xor (crc shl 8)) and 0xFFFF
}
return crc
}
fun packet(msgId: Int, payload: ByteArray = ByteArray(0)): ByteArray {
require(payload.size <= PACKET_SIZE - HEADER)
val buf = ByteArray(PACKET_SIZE)
buf[0] = 0xFF.toByte()
buf[1] = MAGIC_COMMAND.toByte()
val len = payload.size + 0x0C
buf[4] = len.toByte()
buf[5] = (len shr 8).toByte()
buf[14] = msgId.toByte()
buf[15] = (msgId shr 8).toByte()
payload.copyInto(buf, HEADER)
val crc = crc16(buf, 4, len + 2)
buf[2] = crc.toByte()
buf[3] = (crc shr 8).toByte()
return buf
}
class Frame(val magic: Int, val msgId: Int, val payload: ByteArray, val crcOk: Boolean)
fun parse(buf: ByteArray, size: Int): Frame? {
if (size < HEADER || size > buf.size || (buf[0].toInt() and 0xFF) != 0xFF) return null
val magic = buf[1].toInt() and 0xFF
if (magic != MAGIC_COMMAND && magic != MAGIC_REPORT && magic != MAGIC_IMU) return null
val len = u16(buf, 4)
if (len < 0x0C || len + 6 > size) return null
val dataLen = len - 0x0C
val crcOk = crc16(buf, 4, len + 2) == u16(buf, 2)
return Frame(magic, u16(buf, 14), buf.copyOfRange(HEADER, HEADER + dataLen), crcOk)
}
private fun u16(buf: ByteArray, at: Int) = (buf[at].toInt() and 0xFF) or ((buf[at + 1].toInt() and 0xFF) shl 8)
fun beFloat(b: ByteArray, at: Int): Float = java.lang.Float.intBitsToFloat(
((b[at].toInt() and 0xFF) shl 24) or ((b[at + 1].toInt() and 0xFF) shl 16) or
((b[at + 2].toInt() and 0xFF) shl 8) or (b[at + 3].toInt() and 0xFF),
)
fun imuQuaternion(payload: ByteArray): FloatArray? {
if (payload.size >= 36) {
val w = beFloat(payload, 20)
val x = beFloat(payload, 24)
val y = beFloat(payload, 28)
val z = beFloat(payload, 32)
val n = sqrt(w * w + x * x + y * y + z * z)
if (n.isFinite() && n > 0.9f && n < 1.1f) return floatArrayOf(x / n, y / n, z / n, w / n)
}
if (payload.size < 12) return null
val roll = beFloat(payload, 0)
val pitch = beFloat(payload, 4)
val yaw = beFloat(payload, 8)
if (!roll.isFinite() || !pitch.isFinite() || !yaw.isFinite()) return null
return eulerZxy(roll, pitch, yaw)
}
fun eulerZxy(rollDeg: Float, pitchDeg: Float, yawDeg: Float): FloatArray {
val r = Math.toRadians(rollDeg.toDouble()) / 2
val p = Math.toRadians(pitchDeg.toDouble()) / 2
val y = Math.toRadians(yawDeg.toDouble()) / 2
val cx = cos(r); val sx = sin(r)
val cy = cos(p); val sy = sin(p)
val cz = cos(y); val sz = sin(y)
return floatArrayOf(
(sx * cy * cz - cx * sy * sz).toFloat(),
(cx * sy * cz + sx * cy * sz).toFloat(),
(cx * cy * sz + sx * sy * cz).toFloat(),
(cx * cy * cz - sx * sy * sz).toFloat(),
)
}
}
@@ -0,0 +1,52 @@
package com.droiddeck.launcher.glasses
import android.util.Log
import kotlin.math.sqrt
/** JNI adapter to VITURE's Android C SDK. The vendor binary is a local build input. */
internal object VitureSdk {
val available: Boolean by lazy {
runCatching {
System.loadLibrary("droiddeck_viture")
nativeAvailable()
}.onFailure { Log.w("VitureSdk", "VITURE driver unavailable", it) }.getOrDefault(false)
}
fun marketName(pid: Int): String? = if (available) nativeMarketName(pid) else null
external fun nativeAvailable(): Boolean
external fun nativeMarketName(pid: Int): String?
external fun nativeOpen(pid: Int, fd: Int, cacheDirectory: String): Long
external fun nativeType(token: Long): Int
external fun nativeTracking(token: Long, enabled: Boolean): Int
external fun nativeCommand(token: Long, command: Int, value: Int): Int
/** Returns CLOCK_BOOTTIME receipt time, zero for no new pose, or a negative SDK error. */
external fun nativePose(token: Long, wxyz: FloatArray): Long
external fun nativeNextState(token: Long, event: IntArray): Int
external fun nativeClose(token: Long)
const val CARINA = 2
const val GET_DISPLAY = 1
const val SET_DISPLAY = 2
const val GET_BRIGHTNESS = 3
const val SET_BRIGHTNESS = 4
const val GET_VOLUME = 5
const val SET_VOLUME = 6
const val SET_FILM = 7
}
internal object VitureSdkPose {
// Pro 2 hardware captures use OpenGL axes despite the generic Gen2 callback's
// NWU documentation. Keep that model-specific exception out of other Gen2 devices.
fun usesOpenGlAxes(carina: Boolean, family: VitureProtocol.Family?) =
carina || family == VitureProtocol.Family.PRO2
/** SDK quaternions are wxyz; normalize their coordinate basis at the boundary. */
fun toGlasses(wxyz: FloatArray, openGlAxes: Boolean): FloatArray? {
if (wxyz.size != 4 || wxyz.any { !it.isFinite() }) return null
val length = sqrt(wxyz.sumOf { it.toDouble() * it }.toFloat())
if (!length.isFinite() || length < 0.5f || length > 1.5f) return null
val q = if (openGlAxes) floatArrayOf(-wxyz[3], -wxyz[1], wxyz[2], wxyz[0])
else floatArrayOf(wxyz[1], wxyz[2], wxyz[3], wxyz[0])
return q.map { it / length }.toFloatArray()
}
}
@@ -0,0 +1,132 @@
package com.droiddeck.launcher.glasses
import android.hardware.usb.UsbDevice
import android.hardware.usb.UsbDeviceConnection
import android.hardware.usb.UsbManager
import android.os.SystemClock
import android.util.Log
import java.io.File
/** SDK owns interface selection (including Ultra bulk endpoints) and all USB I/O. */
internal class VitureSdkLink private constructor(
private val connection: UsbDeviceConnection,
private val token: Long,
private val listener: Listener,
family: VitureProtocol.Family?,
) : GlassesLink {
interface Listener {
fun onValue(msgId: Int, value: Int)
fun onPose(q: FloatArray, timeNs: Long)
fun onLost()
}
private val lock = Any()
@Volatile private var running = true
@Volatile private var tracking = false
override val ready: Boolean get() = running
override val hasImu = true
private val openGlAxes = VitureSdkPose.usesOpenGlAxes(VitureSdk.nativeType(token) == VitureSdk.CARINA, family)
private val worker = Thread({ poll() }, "viture-sdk-pose").apply { isDaemon = true; start() }
override fun send(msgId: Int, payload: ByteArray): Boolean = synchronized(lock) {
if (!running) return false
val value = payload.firstOrNull()?.toInt()?.and(0xff) ?: 0
when (msgId) {
VitureProtocol.MSG_IMU -> {
val wanted = value != 0
if (wanted == tracking) return true
val rc = VitureSdk.nativeTracking(token, wanted)
if (rc == 0 || !wanted) tracking = wanted
return result("tracking", rc)
}
// Frequency is negotiated against SDK capabilities, including Pro 2's pose limit.
VitureProtocol.MSG_IMU_RATE -> return tracking
VitureProtocol.MSG_GET_IMU -> return true
}
val operation = when (msgId) {
VitureProtocol.MSG_GET_DISPLAY -> VitureSdk.GET_DISPLAY
VitureProtocol.MSG_DISPLAY -> VitureSdk.SET_DISPLAY
VitureProtocol.MSG_GET_BRIGHTNESS -> VitureSdk.GET_BRIGHTNESS
VitureProtocol.MSG_BRIGHTNESS -> VitureSdk.SET_BRIGHTNESS
VitureProtocol.MSG_GET_VOLUME -> VitureSdk.GET_VOLUME
VitureProtocol.MSG_VOLUME -> VitureSdk.SET_VOLUME
VitureProtocol.MSG_FILM -> VitureSdk.SET_FILM
else -> return false
}
val rc = VitureSdk.nativeCommand(token, operation, value)
if (rc < 0) return result("command $operation", rc)
if (operation == VitureSdk.GET_DISPLAY || operation == VitureSdk.GET_BRIGHTNESS || operation == VitureSdk.GET_VOLUME)
listener.onValue(msgId, rc)
else if (operation == VitureSdk.SET_DISPLAY) {
val actual = VitureSdk.nativeCommand(token, VitureSdk.GET_DISPLAY, 0)
if (actual >= 0) listener.onValue(VitureProtocol.MSG_GET_DISPLAY, actual)
}
true
}
private fun result(operation: String, rc: Int): Boolean {
if (rc < 0) Log.w("VitureSdkLink", "$operation failed: $rc")
return rc == 0
}
private fun poll() {
val pose = FloatArray(4)
val event = IntArray(2)
var failures = 0
while (running) {
val timestamp = synchronized(lock) {
if (running) repeat(6) {
if (VitureSdk.nativeNextState(token, event) == 1) when (event[0]) {
0 -> listener.onValue(VitureProtocol.MSG_GET_BRIGHTNESS, event[1])
1 -> listener.onValue(VitureProtocol.MSG_GET_VOLUME, event[1])
// A device event can use a different display-mode namespace.
2 -> send(VitureProtocol.MSG_GET_DISPLAY)
3 -> listener.onValue(VitureProtocol.MSG_FILM, event[1])
}
}
if (!running || !tracking) 0L else VitureSdk.nativePose(token, pose)
}
if (timestamp > 0) {
failures = 0
VitureSdkPose.toGlasses(pose, openGlAxes)?.let { listener.onPose(it, timestamp) }
} else if (timestamp == -2L || timestamp == -3L) {
if (++failures >= 10) {
Log.w("VitureSdkLink", "closing after $failures pose USB errors; last SDK result: $timestamp")
listener.onLost()
break
}
}
SystemClock.sleep(if (tracking) 4 else 25)
}
}
override fun close() {
synchronized(lock) {
if (!running) return
running = false
VitureSdk.nativeClose(token)
// Keep Java's descriptor alive until the SDK and its worker threads have stopped.
connection.close()
}
if (Thread.currentThread() !== worker) worker.join(1000)
}
companion object {
fun open(usb: UsbManager, device: UsbDevice, cacheDirectory: File, listener: Listener,
family: VitureProtocol.Family? = null): VitureSdkLink? {
if (!VitureSdk.available) return null
val conn = usb.openDevice(device) ?: return null
val token = runCatching {
cacheDirectory.mkdirs()
VitureSdk.nativeOpen(device.productId, conn.fileDescriptor, cacheDirectory.absolutePath)
}.onFailure { Log.w("VitureSdkLink", "SDK initialization failed", it) }.getOrDefault(0)
if (token <= 0) { conn.close(); return null }
return runCatching { VitureSdkLink(conn, token, listener, family) }.getOrElse {
VitureSdk.nativeClose(token)
conn.close()
Log.w("VitureSdkLink", "SDK worker could not start", it)
null
}
}
}
}
@@ -0,0 +1,88 @@
package com.droiddeck.launcher.glasses
import android.content.Context
import android.net.ConnectivityManager
import android.net.LinkProperties
import android.net.NetworkCapabilities
import java.io.Closeable
import java.io.IOException
import java.io.InputStream
import java.io.OutputStream
import java.net.InetAddress
import java.net.InetSocketAddress
import java.net.Socket
internal interface XrealChannels : Closeable {
val controlInput: InputStream
val controlOutput: OutputStream
val reports: InputStream
}
internal interface XrealConnector : Closeable {
fun connect(): XrealChannels
}
internal class XrealNetworkMissing : IOException("XREAL USB Ethernet network is not available")
/** Bind individual sockets, never the process: Steam must retain its Internet network. */
internal class AndroidXrealConnector(context: Context) : XrealConnector {
private val manager = context.getSystemService(ConnectivityManager::class.java)
private val address = InetAddress.getByName(XrealProtocol.HOST)
private val lock = Any()
private val sockets = mutableSetOf<Socket>()
private var closed = false
companion object {
internal fun hasDirectRoute(properties: LinkProperties, address: InetAddress): Boolean =
properties.routes.any { !it.hasGateway() && it.destination.prefixLength > 0 && it.matches(address) }
}
override fun connect(): XrealChannels {
val network = manager.allNetworks.firstOrNull { network ->
manager.getNetworkCapabilities(network)?.let {
it.hasTransport(NetworkCapabilities.TRANSPORT_ETHERNET) &&
!it.hasTransport(NetworkCapabilities.TRANSPORT_VPN)
} == true && manager.getLinkProperties(network)?.let { hasDirectRoute(it, address) } == true
} ?: throw XrealNetworkMissing()
fun open(port: Int): Socket {
val socket = Socket()
synchronized(lock) {
if (closed) { socket.close(); throw IOException("XREAL connection closed") }
sockets.add(socket)
}
try {
network.bindSocket(socket)
socket.soTimeout = 2000
socket.tcpNoDelay = true
socket.connect(InetSocketAddress(address, port), 1500)
return socket
} catch (e: Exception) {
release(socket)
throw e
}
}
val control = open(XrealProtocol.CONTROL_PORT)
val reportSocket = try { open(XrealProtocol.REPORT_PORT) } catch (e: Exception) {
release(control)
throw e
}
return object : XrealChannels {
override val controlInput = control.getInputStream()
override val controlOutput = control.getOutputStream()
override val reports = reportSocket.getInputStream()
override fun close() { release(control); release(reportSocket) }
}
}
private fun release(socket: Socket) = synchronized(lock) {
runCatching { socket.close() }
sockets.remove(socket)
Unit
}
override fun close() = synchronized(lock) {
closed = true
sockets.forEach { runCatching { it.close() } }
sockets.clear()
}
}
@@ -0,0 +1,201 @@
package com.droiddeck.launcher.glasses
import android.content.Context
import android.util.Log
import java.io.IOException
import java.util.concurrent.ArrayBlockingQueue
import java.util.concurrent.TimeUnit
import java.util.concurrent.atomic.AtomicReference
/** One connection owns two TCP channels. All control requests are serialized off the UI thread. */
internal class XrealLink internal constructor(
private val connector: XrealConnector,
private val listener: Listener,
) : GlassesLink {
interface Listener {
fun onReady()
fun onValue(message: Int, value: Int)
fun onPose(raw: ByteArray, pose: FloatArray, timeNs: Long)
fun onUnavailable(networkMissing: Boolean)
}
private data class Command(val message: Int, val value: Int)
private val commands = ArrayBlockingQueue<Command>(16)
private val connection = AtomicReference<XrealChannels?>()
@Volatile private var running = true
@Volatile private var tracking = false
@Volatile override var ready = false
private set
override val hasImu = true
private val worker = Thread(::run, "xreal-control").apply { isDaemon = true; start() }
companion object {
fun open(context: Context, listener: Listener) = XrealLink(AndroidXrealConnector(context), listener)
}
override fun send(msgId: Int, payload: ByteArray): Boolean {
if (!running || !ready) return false
val value = payload.firstOrNull()?.toInt()?.and(255) ?: -1
if (msgId == VitureProtocol.MSG_IMU) {
if (value !in 0..1) return false
tracking = value == 1
return true
}
if (msgId == VitureProtocol.MSG_GET_IMU) return true
val valid = when (msgId) {
VitureProtocol.MSG_GET_DISPLAY -> true
VitureProtocol.MSG_DISPLAY -> value in 2..5 || value == XrealModel.NATIVE_60HZ
VitureProtocol.MSG_BRIGHTNESS -> value in 0..9
VitureProtocol.MSG_FILM -> value in 0..1
else -> false // No guessed volume, sensor-rate, or VITURE wire commands.
}
return valid && commands.offer(Command(msgId, value))
}
private fun run() {
var lastError: String? = null
while (running) {
var channels: XrealChannels? = null
var reports: Thread? = null
val failure = AtomicReference<Exception?>()
try {
channels = connector.connect()
connection.set(channels)
if (!running) break
val control = XrealControl(channels)
listener.onValue(VitureProtocol.MSG_GET_DISPLAY, control.display())
val current = channels
reports = Thread({
try {
var filter = XrealOrientationFilter()
var wasTracking = false
var lastSensor = System.nanoTime()
while (running && connection.get() === current) {
val frame = XrealProtocol.read(current.reports)
val sample = XrealProtocol.sample(frame)
if (sample != null) {
lastSensor = System.nanoTime()
val enabled = tracking
if (enabled && !wasTracking) filter = XrealOrientationFilter()
wasTracking = enabled
if (enabled) filter.update(sample)?.let {
if (running && ready && tracking && connection.get() === current)
listener.onPose(frame.body, it, System.nanoTime())
}
} else if (System.nanoTime() - lastSensor > 3_000_000_000L) {
throw IOException("No valid XREAL IMU reports")
}
}
} catch (e: Exception) {
failure.set(e)
current.close() // Wake a control read immediately.
}
}, "xreal-reports").apply { isDaemon = true; start() }
ready = true
lastError = null
listener.onReady()
var heartbeat = System.nanoTime()
while (running && failure.get() == null) {
val command = commands.poll(250, TimeUnit.MILLISECONDS)
if (command != null) {
when (command.message) {
VitureProtocol.MSG_GET_DISPLAY ->
listener.onValue(command.message, control.display())
VitureProtocol.MSG_DISPLAY -> {
listener.onValue(VitureProtocol.MSG_GET_DISPLAY, control.changeDisplay(command.value))
}
VitureProtocol.MSG_BRIGHTNESS -> {
control.set(XrealProtocol.SET_BRIGHTNESS, command.value)
listener.onValue(command.message, command.value)
}
VitureProtocol.MSG_FILM -> {
control.set(XrealProtocol.SET_DIMMER, if (command.value == 1) 2 else 0)
listener.onValue(command.message, command.value)
}
}
}
if (System.nanoTime() - heartbeat > 5_000_000_000L) {
listener.onValue(VitureProtocol.MSG_GET_DISPLAY, control.display())
heartbeat = System.nanoTime()
}
}
failure.get()?.let { throw it }
} catch (e: Exception) {
ready = false
if (running && lastError != e.message) {
lastError = e.message
Log.w("XrealLink", "Connection unavailable: ${e.message}")
listener.onUnavailable(e is XrealNetworkMissing)
}
} finally {
ready = false
tracking = false
commands.clear()
connection.compareAndSet(channels, null)
channels?.close()
try { reports?.join(2500) } catch (_: InterruptedException) { }
}
if (running) try { Thread.sleep(2000) } catch (_: InterruptedException) { }
}
}
override fun close() {
running = false
ready = false
tracking = false
commands.clear()
connection.getAndSet(null)?.close()
connector.close() // Includes sockets still inside connect().
worker.interrupt()
}
}
/** Kept independent of Android so the real framing/acknowledgement path is testable. */
internal class XrealControl(private val channels: XrealChannels, private val settleDelayMs: Long = 150) {
private var transaction = 0
private fun exchange(command: Int, value: Int? = null): ByteArray {
transaction = if (transaction == Int.MAX_VALUE) 1 else transaction + 1
channels.controlOutput.write(XrealProtocol.request(command, transaction, value))
channels.controlOutput.flush()
val deadline = System.nanoTime() + 3_000_000_000L
repeat(32) { // Bound unsolicited button/status packets and stale responses.
if (System.nanoTime() > deadline) throw IOException("XREAL control timed out")
XrealProtocol.response(XrealProtocol.read(channels.controlInput), command, transaction)?.let { return it }
}
throw IOException("XREAL control response was not received")
}
fun display(): Int {
val reply = exchange(XrealProtocol.GET_DISPLAY)
return XrealProtocol.numeric(reply)?.takeIf { it in 2..5 || it == XrealModel.NATIVE_60HZ }
?: throw IOException("Unsupported XREAL display response: ${reply.take(32).joinToString("") { "%02x".format(it) }}")
}
fun changeDisplay(mode: Int): Int {
val stereo = mode == 5
set(XrealProtocol.SET_INPUT, if (stereo) 1 else 0)
// On 1S, SET_INPUT itself changes EDID. Sending SET_DISPLAY immediately
// receives error 5001 while that change is pending. GET_DISPLAY returns
// an empty property during the transition; wait for a real mode report.
val current = awaitDisplay { (it == 5) == stereo }
if (current == mode) return current
set(XrealProtocol.SET_DISPLAY, mode)
return awaitDisplay { it == mode }
}
private fun awaitDisplay(matches: (Int) -> Boolean): Int {
val deadline = System.nanoTime() + 8_000_000_000L
repeat(40) {
val reply = exchange(XrealProtocol.GET_DISPLAY)
val value = XrealProtocol.numeric(reply)
if (value != null && matches(value)) return value
if (value == null && !XrealProtocol.acknowledged(reply))
throw IOException("Invalid XREAL display transition response")
if (System.nanoTime() > deadline) throw IOException("XREAL display transition timed out")
Thread.sleep(settleDelayMs)
}
throw IOException("XREAL display transition did not complete")
}
fun set(command: Int, value: Int) {
val reply = exchange(command, value)
if (!XrealProtocol.acknowledged(reply)) throw IOException("XREAL command %04x was not acknowledged: %s".format(
command, reply.take(32).joinToString("") { "%02x".format(it) }))
}
}
@@ -0,0 +1,27 @@
package com.droiddeck.launcher.glasses
/** First One-family target. Recovery and Air USB products must not use this transport. */
object XrealModel : GlassesModel {
const val VENDOR_ID = 0x3318
const val PRODUCT_1S = 0x043e
const val NATIVE_60HZ = 29 // Captured on 1S: 1920 x 1200 at 60 Hz.
override val name = "XREAL 1S"
override val controllable = true
override val brightnessMax = 9
override val volumeMax = 0
override val hasSunblock = true
override val diagonalFovDeg = 52f
override val sbs90 = false
fun matches(vid: Int, pid: Int) = vid == VENDOR_ID && pid == PRODUCT_1S
fun isDisplay(name: String) = name.contains("XREAL", ignoreCase = true)
// One-family EDID configuration values, not VITURE's display mode IDs.
override fun mode(hz: Int, sbs: Boolean) = when {
sbs -> 5
hz >= 120 -> 4
hz >= 90 -> 3
else -> NATIVE_60HZ
}
override fun modeForStep(step: Int): Int? = null
}
@@ -0,0 +1,46 @@
package com.droiddeck.launcher.glasses
import kotlin.math.abs
/** 1S radians/sec and device nanoseconds, mapped using directed hardware captures. */
internal class XrealOrientationFilter {
private var previous = 0L
private var rotation = Quaternion.IDENTITY
private var initialized = false
private var bias = Vec3(0f, 0f, 0f)
fun update(sample: XrealProtocol.Sample): FloatArray? {
if (sample.timeNs <= previous) return null
val dt = if (previous == 0L) 0f else (sample.timeNs - previous) * 1e-9f
previous = sample.timeNs
// 1S reports right/down/forward. Rotate 180 degrees about X into the
// shared right/up/back basis before either gravity alignment or fusion.
// Captures: level gravity -Y; looking left -gyroY; up +gyroX; right tilt +gyroZ.
val accel = Vec3(sample.accel.x, -sample.accel.y, -sample.accel.z)
val gyro = Vec3(sample.gyro.x, -sample.gyro.y, -sample.gyro.z)
val norm = accel.length()
val gravityValid = abs(norm - 9.81f) < 1.5f
if (dt > 0.25f) initialized = false // Never integrate a reconnect/stall as motion.
if (!initialized) {
if (!gravityValid) return null
val up = accel.normalized()
val axis = up.cross(Vec3(0f, 1f, 0f))
rotation = if (up.y < -0.9999f) Quaternion.fromAxisAngle(Vec3(1f, 0f, 0f), Math.PI.toFloat())
else Quaternion(1f + up.y, axis.x, axis.y, axis.z).normalized()
initialized = true
} else if (dt in 0.0001f..0.05f) {
if (gravityValid && gyro.length() < 0.04f) {
val alpha = (dt * 0.5f).coerceAtMost(1f)
bias = bias * (1f - alpha) + gyro * alpha
}
var rate = gyro - bias
if (gravityValid) {
val predictedUp = rotation.conjugate().rotate(Vec3(0f, 1f, 0f))
rate += accel.normalized().cross(predictedUp) * 1.5f
}
rotation = rotation.integrateBodyRate(rate, dt).normalized()
}
// Same right/up/back -> shared glasses basis used by the RayNeo adapter.
return RayNeoOrientationFilter.glassesPose(floatArrayOf(rotation.w, rotation.x, rotation.y, rotation.z))
}
}
@@ -0,0 +1,78 @@
package com.droiddeck.launcher.glasses
import java.io.EOFException
import java.io.IOException
import java.io.InputStream
import java.nio.ByteBuffer
import java.nio.ByteOrder
/** Independent implementation of the One-family wire format; see xreal-investigation.md. */
internal object XrealProtocol {
const val HOST = "169.254.2.1"
const val REPORT_PORT = 52998
const val CONTROL_PORT = 52999
const val REPORT = 0x2836
const val GET_DISPLAY = 0x275e
const val SET_DISPLAY = 0x275f
const val SET_INPUT = 0x2822
const val SET_BRIGHTNESS = 0x271c
const val SET_DIMMER = 0x2727
const val MAX_BODY = 64 * 1024
data class Frame(val command: Int, val body: ByteArray)
data class Sample(val timeNs: Long, val gyro: Vec3, val accel: Vec3)
fun read(input: InputStream): Frame {
val deadline = System.nanoTime() + 3_000_000_000L
val header = ByteArray(6)
input.readFully(header, deadline)
val size = ByteBuffer.wrap(header, 2, 4).int
if (size !in 0..MAX_BODY) throw IOException("Invalid XREAL frame length: $size")
return Frame(((header[0].toInt() and 255) shl 8) or (header[1].toInt() and 255),
ByteArray(size).also { input.readFully(it, deadline) })
}
private fun InputStream.readFully(bytes: ByteArray, deadline: Long) {
var at = 0
while (at < bytes.size) {
if (System.nanoTime() > deadline) throw IOException("XREAL frame timed out")
val count = read(bytes, at, bytes.size - at)
if (count < 0) throw EOFException("XREAL stream closed mid-frame")
if (count == 0) throw IOException("XREAL stream made no progress")
at += count
}
}
fun sample(frame: Frame): Sample? {
if (frame.command != REPORT || frame.body.size != 128) return null
val b = ByteBuffer.wrap(frame.body).order(ByteOrder.LITTLE_ENDIAN)
if (b.getInt(24) != 11) return null // Magnetometer reports are a different type.
val time = b.getLong(8)
val gyro = Vec3(b.getFloat(28), b.getFloat(32), b.getFloat(36))
val accel = Vec3(b.getFloat(40), b.getFloat(44), b.getFloat(48))
if (time <= 0 || !gyro.valid(100f) || !accel.valid(200f)) return null
return Sample(time, gyro, accel)
}
private fun Vec3.valid(limit: Float) = listOf(x, y, z).all { it.isFinite() && kotlin.math.abs(it) <= limit }
fun request(command: Int, transaction: Int, value: Int? = null): ByteArray {
require(transaction > 0)
require(value == null || value in 0..127)
val payload = if (value == null) byteArrayOf(0x18, 0) else byteArrayOf(0x1a, 2, 8, value.toByte())
return ByteBuffer.allocate(10 + payload.size).putShort(command.toShort())
.putInt(4 + payload.size).putInt(transaction or Int.MIN_VALUE).put(payload).array()
}
/** No acknowledgement is inferred from a write; match both command and transaction. */
fun response(frame: Frame, command: Int, transaction: Int): ByteArray? {
if (frame.command != command || frame.body.size < 4 ||
ByteBuffer.wrap(frame.body).int != transaction) return null
return frame.body.copyOfRange(4, frame.body.size)
}
fun acknowledged(payload: ByteArray) = payload.contentEquals(byteArrayOf(0x22, 0))
fun numeric(payload: ByteArray): Int? = if (payload.size == 4 &&
payload[0] == 0x22.toByte() && payload[1] == 2.toByte() && payload[2] == 0x10.toByte() &&
payload[3].toInt() in 0..127) payload[3].toInt() else null
}
@@ -12,13 +12,15 @@ data class Cardboard(
val screenWidthDm: Int = 40,
val screenDistanceDm: Int = 30,
val headTracking: Boolean = true,
val mono: Boolean = false,
) {
val lens: Viewer get() = Viewer.byId(viewer)
fun push(panelWidthMm: Float, panelHeightMm: Float) {
fun push(panelWidthMm: Float, panelHeightMm: Float, fovOverrideDeg: Float = 0f, displayHz: Float = 0f) {
val l = lens
WaylandCompositor.nativeSetCardboard(enabled, source, ipdMm / 1000f, screenDistanceDm / 10f,
screenWidthDm / 10f, l.k1, l.k2, fovDeg(l, panelHeightMm), lensInset(panelWidthMm, l.lensSeparationMm), l.chroma)
screenWidthDm / 10f, l.k1, l.k2, if (fovOverrideDeg > 0f) fovOverrideDeg else fovDeg(l, panelHeightMm),
if (mono) 0f else lensInset(panelWidthMm, l.lensSeparationMm), l.chroma, mono, displayHz)
}
fun encode(): JSONObject = JSONObject()
@@ -29,7 +31,8 @@ data class Cardboard(
val lensSeparationMm: Float, val chroma: Float, val fallbackFovDeg: Float) {
CARDBOARD_V2("cardboard2", 0.34f, 0.55f, 39.3f, 63.9f, 0.006f, 90f),
CARDBOARD_V1("cardboard1", 0.441f, 0.156f, 42f, 60f, 0.004f, 80f),
NONE("none", 0f, 0f, 0f, 0f, 0f, 60f);
NONE("none", 0f, 0f, 0f, 0f, 0f, 60f),
VITURE("viture", 0f, 0f, 0f, 0f, 0f, 23.3f);
companion object {
fun byId(id: String): Viewer = entries.firstOrNull { it.id == id } ?: CARDBOARD_V2
@@ -123,6 +126,10 @@ object HeadPose {
a[r * 3] * b[c] + a[r * 3 + 1] * b[3 + c] + a[r * 3 + 2] * b[6 + c]
}
private val GLASSES_AXES = floatArrayOf(0f, 0f, -1f, -1f, 0f, 0f, 0f, 1f, 0f)
fun fromGlasses(x: Float, y: Float, z: Float, w: Float): FloatArray = multiply(fromQuaternion(x, y, z, w), GLASSES_AXES)
fun transpose(m: FloatArray): FloatArray = floatArrayOf(m[0], m[3], m[6], m[1], m[4], m[7], m[2], m[5], m[8])
fun apply(m: FloatArray, v: FloatArray): FloatArray = floatArrayOf(
@@ -10,19 +10,42 @@ import android.os.HandlerThread
import android.util.Log
import com.droiddeck.launcher.wayland.WaylandCompositor
class HeadTracker(context: Context, private val rotation: () -> Int) : SensorEventListener {
class HeadTracker(context: Context, private val glasses: Boolean = false, private val rotation: () -> Int) : SensorEventListener {
private val sensors = context.getSystemService(Context.SENSOR_SERVICE) as SensorManager
private val sensor: Sensor? = sensors.getDefaultSensor(Sensor.TYPE_GAME_ROTATION_VECTOR)
?: sensors.getDefaultSensor(Sensor.TYPE_ROTATION_VECTOR)
private var thread: HandlerThread? = null
private var started = false
private val glassesLock = Any()
@Volatile private var reference: HeadPose.Reference? = null
@Volatile private var recenterPending = true
@Volatile private var latest: FloatArray? = null
private val glassesSink = com.droiddeck.launcher.glasses.Glasses.PoseSink { x, y, z, w, _ ->
val m = HeadPose.fromGlasses(x, y, z, w)
synchronized(glassesLock) {
if (!started) return@PoseSink
latest = m
apply(m)
}
}
val usesGlasses: Boolean get() = glasses
val available: Boolean get() = sensor != null
val sensorName: String get() = sensor?.let { "${it.name} (${if (it.type == Sensor.TYPE_GAME_ROTATION_VECTOR) "game rotation vector" else "rotation vector"})" } ?: "none"
fun start() {
if (glasses) {
synchronized(glassesLock) {
if (started) return
started = true
recenterPending = true
}
com.droiddeck.launcher.glasses.Glasses.setPoseSink(glassesSink)
WaylandCompositor.nativeLog("cardboard", "head tracking on: VITURE glasses IMU")
return
}
val s = sensor ?: run {
WaylandCompositor.nativeLog("cardboard", "no rotation sensor: the virtual screen stays head-locked")
WaylandCompositor.nativeSetCardboardPose(HeadPose.IDENTITY, false)
@@ -37,6 +60,15 @@ class HeadTracker(context: Context, private val rotation: () -> Int) : SensorEve
}
fun stop() {
if (glasses) {
synchronized(glassesLock) {
if (!started) return
started = false
WaylandCompositor.nativeSetCardboardPose(HeadPose.IDENTITY, false)
}
com.droiddeck.launcher.glasses.Glasses.clearPoseSink(glassesSink)
return
}
val t = thread ?: return
sensors.unregisterListener(this)
t.quitSafely()
@@ -46,7 +78,8 @@ class HeadTracker(context: Context, private val rotation: () -> Int) : SensorEve
fun recenter() {
recenterPending = true
latest?.let { apply(it) }
if (glasses) synchronized(glassesLock) { if (started) latest?.let { apply(it) } }
else latest?.let { apply(it) }
}
override fun onSensorChanged(event: SensorEvent) {
@@ -5,6 +5,8 @@ import com.droiddeck.launcher.agent.AgentGuest
import org.json.JSONObject
object SteamVr {
const val APP_ID = "250820"
enum class Status {
UNAVAILABLE,
NOT_INSTALLED,
@@ -13,6 +15,23 @@ object SteamVr {
RUNNING,
}
internal fun shouldPoll(status: Status?, cardboardEnabled: Boolean, pictureSwitched: Boolean,
glassesControl: Boolean, requested: Boolean): Boolean =
requested || cardboardEnabled || pictureSwitched || glassesControl ||
status == Status.RUNNING || status == Status.INSTALLING
internal fun autoAction(
status: Status, requested: Boolean, loading: Boolean, steamSession: Boolean,
startTried: Boolean, installTried: Boolean,
): String? {
if (!requested || loading || !steamSession || startTried) return null
return when (status) {
Status.READY -> "start"
Status.NOT_INSTALLED -> if (installTried) null else "install"
else -> null
}
}
fun status(context: Context): Status = runCatching {
parse(AgentGuest.call(context, JSONObject().put("kind", "vr").put("action", "status"), STATUS_TIMEOUT_MS))
}.getOrDefault(Status.UNAVAILABLE)
@@ -0,0 +1,86 @@
package com.droiddeck.launcher.gpu
import org.json.JSONObject
import java.io.File
import java.io.RandomAccessFile
import java.nio.file.Files
import java.nio.file.attribute.BasicFileAttributes
/** Only new compositor output is evidence of a fault in the current session. */
internal class SteamVrDeviceLossLog(private val file: File) {
private var identity: Any? = identity()
private var position = file.length()
private var prefix = runCatching {
RandomAccessFile(file, "r").use { input -> ByteArray(minOf(64, input.length()).toInt()).also(input::readFully) }
}.getOrDefault(byteArrayOf())
private var partial = ""
private var handled = false
private fun identity(): Any? = runCatching {
Files.readAttributes(file.toPath(), BasicFileAttributes::class.java).fileKey()
}.getOrNull()
fun poll(): Boolean {
if (handled) return true
if (!file.isFile) return false
return runCatching {
val current = identity()
RandomAccessFile(file, "r").use { input ->
val head = ByteArray(minOf(64, input.length()).toInt()).also(input::readFully)
val replaced = prefix.isNotEmpty() && !head.take(prefix.size).toByteArray().contentEquals(prefix)
if (current != identity || input.length() < position || replaced) {
position = 0
partial = ""
}
identity = current
prefix = head
// Fault loops can produce megabytes per second. Bound work and keep the newest output.
if (input.length() - position > MAX_READ) {
position = input.length() - MAX_READ
partial = ""
}
input.seek(position)
val bytes = ByteArray(minOf(MAX_READ, input.length() - position).toInt())
input.readFully(bytes)
position = input.filePointer
val text = partial + bytes.toString(Charsets.UTF_8)
val end = text.lastIndexOf('\n')
partial = text.substring(end + 1).takeLast(2048)
if (end >= 0 && isDeviceLoss(text.substring(0, end + 1))) handled = true
handled
}
}.getOrDefault(false)
}
companion object {
private const val MAX_READ = 128 * 1024L
internal fun isDeviceLoss(text: String): Boolean =
Regex("\\bVK_ERROR_DEVICE_LOST\\b|\\bvkerror\\s*=\\s*-4\\b", RegexOption.IGNORE_CASE).containsMatchIn(text)
}
}
internal object SteamVrRecoveryPolicy {
const val COOLDOWN_MS = 10 * 60 * 1000L
const val STOP_MS = 15_000L
const val LOGS_MS = 30_000L
private const val POLL_MS = 250L
enum class Shutdown { COMPLETE, LOGS_PENDING, STUCK }
fun canRestartAutomatically(lastAttempt: Long, now: Long): Boolean =
lastAttempt == 0L || (now >= lastAttempt && now - lastAttempt >= COOLDOWN_MS)
fun awaitShutdown(session: () -> JSONObject?, now: () -> Long, sleep: (Long) -> Unit): Shutdown {
fun await(ms: Long, done: (JSONObject) -> Boolean): Boolean {
val deadline = now() + ms
while (true) {
if (session()?.let(done) == true) return true
if (now() >= deadline) return false
sleep(POLL_MS)
}
}
if (!await(STOP_MS) { it.optString("phase") in setOf("IDLE", "FAILED") }) return Shutdown.STUCK
return if (await(LOGS_MS) { !it.optBoolean("artifactsAvailable") || it.optBoolean("artifactsComplete") }) Shutdown.COMPLETE
else Shutdown.LOGS_PENDING
}
}
@@ -0,0 +1,83 @@
package com.droiddeck.launcher.gpu
import android.app.Activity
import android.app.AlertDialog
import android.content.Context
import android.content.Intent
import android.os.Handler
import android.os.Looper
import com.droiddeck.launcher.R
import com.droiddeck.launcher.runtime.LinuxRuntime
import com.droiddeck.launcher.session.SessionEvents
import java.io.File
/** Process-wide cursor survives Activity recreation; a new session ignores older log errors. */
internal object SteamVrHealth {
private var session: String? = null
private var log: SteamVrDeviceLossLog? = null
@Synchronized fun begin(context: Context, sessionId: String?) {
if (sessionId == session && log != null) return
session = sessionId
log = SteamVrDeviceLossLog(File(LinuxRuntime.rootDir(context),
"root/.local/share/Steam/logs/vrcompositor.txt"))
}
@Synchronized fun deviceLost(): Boolean = log?.poll() == true
}
internal class SteamVrRecovery(private val activity: Activity, private val foreground: () -> Boolean) {
private val handler = Handler(Looper.getMainLooper())
private var dialog: AlertDialog? = null
private var handled = false
private var countdown: Runnable? = null
fun detected() {
if (handled || !foreground() || activity.isFinishing || activity.isDestroyed) return
handled = true
SessionEvents.record("steamvr.device_lost")
val prefs = activity.getSharedPreferences(PREFS, Context.MODE_PRIVATE)
val automatic = SteamVrRecoveryPolicy.canRestartAutomatically(prefs.getLong(LAST_ATTEMPT, 0), System.currentTimeMillis())
var seconds = 8
fun message() = if (automatic) activity.getString(R.string.steamvr_recovery_countdown, seconds)
else activity.getString(R.string.steamvr_recovery_repeated)
val prompt = AlertDialog.Builder(activity)
.setTitle(R.string.steamvr_recovery_title)
.setMessage(message())
.setPositiveButton(R.string.steamvr_recovery_restart) { _, _ -> restart() }
.setNegativeButton(R.string.steamvr_recovery_cancel) { _, _ -> }
.create()
prompt.setOnDismissListener { countdown?.let(handler::removeCallbacks); countdown = null; dialog = null }
dialog = prompt
// A touch meant for the game must not silently dismiss the recovery controls.
prompt.setCanceledOnTouchOutside(false)
prompt.show()
if (automatic) {
countdown = object : Runnable {
override fun run() {
if (!foreground() || activity.isFinishing || activity.isDestroyed) { handled = false; prompt.dismiss(); return }
if (--seconds <= 0) { prompt.dismiss(); restart(); return }
prompt.setMessage(message())
handler.postDelayed(this, 1000)
}
}.also { handler.postDelayed(it, 1000) }
}
}
private fun restart() {
if (!foreground()) return
// Commit before crossing processes so a repeated fault cannot create a restart loop.
if (!activity.getSharedPreferences(PREFS, Context.MODE_PRIVATE).edit()
.putLong(LAST_ATTEMPT, System.currentTimeMillis()).commit()) return
SessionEvents.record("steamvr.recovery_requested")
activity.startActivity(Intent(activity, SteamVrRestartActivity::class.java)
.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK))
}
fun close() { countdown?.let(handler::removeCallbacks); countdown = null; dialog?.dismiss() }
companion object {
internal const val PREFS = "steamvr_recovery"
internal const val LAST_ATTEMPT = "lastAttempt"
}
}
@@ -0,0 +1,27 @@
package com.droiddeck.launcher.gpu
import android.util.AtomicFile
import org.json.JSONObject
import java.io.File
internal object SteamVrRecoverySettings {
/** After a graphics crash SteamVR can quarantine our HMD, leaving a server with no VR output. */
fun unblockHeadset(file: File): Boolean {
if (!file.isFile) return false
val settings = JSONObject(file.readText())
val driver = settings.optJSONObject("driver_droiddeck") ?: return false
if (!driver.optBoolean("blocked_by_safe_mode")) return false
// Preserve every other setting, including safe mode and blocks on other drivers.
driver.put("blocked_by_safe_mode", false)
val atomic = AtomicFile(file)
val output = atomic.startWrite()
try {
output.write(settings.toString(3).toByteArray(Charsets.UTF_8))
atomic.finishWrite(output)
} catch (e: Exception) {
atomic.failWrite(output)
throw e
}
return true
}
}
@@ -0,0 +1,72 @@
package com.droiddeck.launcher.gpu
import android.app.Activity
import android.app.ActivityManager
import android.content.Intent
import android.net.Uri
import android.os.Bundle
import android.os.Process
import android.os.SystemClock
import android.view.Gravity
import android.widget.TextView
import com.droiddeck.launcher.R
import com.droiddeck.launcher.SessionActivity
import com.droiddeck.launcher.session.SessionService
import org.json.JSONObject
/** A visible, private process survives destruction of the app's native graphics context. */
class SteamVrRestartActivity : Activity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(TextView(this).apply {
setText(R.string.steamvr_recovery_restarting)
gravity = Gravity.CENTER
textSize = 22f
})
Thread({ recover() }, "steamvr-process-recovery").start()
}
private fun recover() {
try {
val manager = getSystemService(ACTIVITY_SERVICE) as ActivityManager
val main = manager.runningAppProcesses?.firstOrNull {
it.processName == packageName && it.uid == Process.myUid() && it.pid != Process.myPid()
} ?: error("DroidDeck process is unavailable")
val uri = Uri.parse("content://$packageName.agent")
contentResolver.call(uri, "stop", null, null)
val shutdown = SteamVrRecoveryPolicy.awaitShutdown({
runCatching {
contentResolver.call(uri, "state", null, null)?.getString("json")?.let(::JSONObject)?.optJSONObject("session")
}.getOrNull()
}, SystemClock::elapsedRealtime, SystemClock::sleep)
if (shutdown != SteamVrRecoveryPolicy.Shutdown.COMPLETE)
android.util.Log.w("SteamVrRecovery", "Restarting before the session finished stopping: $shutdown")
// Only the verified main process, never a caller-supplied PID or an external app.
Process.killProcess(main.pid)
val exited = SystemClock.elapsedRealtime() + 5000
while (java.io.File("/proc/${main.pid}").exists() && SystemClock.elapsedRealtime() < exited) SystemClock.sleep(50)
check(!java.io.File("/proc/${main.pid}").exists()) { "DroidDeck process did not exit" }
val settings = java.io.File(com.droiddeck.launcher.runtime.LinuxRuntime.rootDir(this),
"root/.local/share/Steam/config/steamvr.vrsettings")
runCatching { SteamVrRecoverySettings.unblockHeadset(settings) }
.onSuccess { if (it) android.util.Log.i("SteamVrRecovery", "Cleared the DroidDeck HMD crash block before restarting SteamVR") }
.onFailure { android.util.Log.w("SteamVrRecovery", "Could not read the SteamVR settings; restarting with them unchanged", it) }
runOnUiThread {
startActivity(Intent(this, SessionActivity::class.java)
.putExtra(SessionService.EXTRA_MODE, SessionService.MODE_STEAM)
.putExtra(SessionActivity.EXTRA_STEAMVR, true)
.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_CLEAR_TASK))
finishAndRemoveTask()
}
} catch (e: Exception) {
android.util.Log.e("SteamVrRecovery", "Could not restart DroidDeck", e)
runOnUiThread {
setContentView(TextView(this).apply {
setText(R.string.steamvr_recovery_failed)
gravity = Gravity.CENTER
textSize = 22f
})
}
}
}
}
@@ -0,0 +1,255 @@
package com.droiddeck.launcher.handtracking
import com.droiddeck.launcher.glasses.Vec3
import kotlin.math.abs
import kotlin.math.atan2
import kotlin.math.hypot
internal class Landmark(val x: Float, val y: Float, val visibility: Float)
internal class WorldPoint(val x: Float, val y: Float, val z: Float)
internal class Intrinsics(val fx: Float, val fy: Float, val cx: Float, val cy: Float, val width: Int, val height: Int)
internal class CameraLevel(up: Vec3) {
val right: Vec3
val up: Vec3
val back: Vec3
val valid: Boolean
init {
val u = up.normalized()
val forward = Vec3(0f, 0f, 1f)
val along = u.x * forward.x + u.y * forward.y + u.z * forward.z
val horizontal = Vec3(forward.x - u.x * along, forward.y - u.y * along, forward.z - u.z * along)
valid = u.length() > 0.5f && horizontal.length() > MIN_HORIZONTAL
this.up = u
back = (horizontal * -1f).normalized()
right = this.up.cross(back)
}
fun level(v: Vec3) = Vec3(dot(v, right), dot(v, up), dot(v, back))
private fun dot(a: Vec3, b: Vec3) = a.x * b.x + a.y * b.y + a.z * b.z
companion object {
private const val MIN_HORIZONTAL = 0.25f
}
}
internal class BodyFrame(
val hands: Array<Vec3?>,
val faceYaw: Float?,
val distance: Float,
val shoulderSpan: Float,
val seen: Boolean,
val estimated: BooleanArray = BooleanArray(2),
val edgeRoom: Float = 1f,
val joints: Array<FloatArray?> = arrayOfNulls(2),
)
internal class HandFix(val palm: Vec3, val x: Float, val y: Float, val extent: Float, val points: FloatArray, val world: List<WorldPoint> = emptyList())
internal object Pnp {
fun translation(count: Int, k: Intrinsics, x: (Int) -> Float, y: (Int) -> Float, world: (Int) -> WorldPoint): Vec3? {
val ata = Array(3) { DoubleArray(3) }
val atb = DoubleArray(3)
for (i in 0 until count) {
val a = ((x(i) - k.cx) / k.fx).toDouble()
val b = ((y(i) - k.cy) / k.fy).toDouble()
val w = world(i)
add(ata, atb, doubleArrayOf(1.0, 0.0, -a), a * w.z - w.x)
add(ata, atb, doubleArrayOf(0.0, 1.0, -b), b * w.z - w.y)
}
val t = solve3(ata, atb) ?: return null
return Vec3(t[0].toFloat(), t[1].toFloat(), t[2].toFloat())
}
private fun add(ata: Array<DoubleArray>, atb: DoubleArray, row: DoubleArray, rhs: Double) {
for (i in 0..2) {
atb[i] += row[i] * rhs
for (j in 0..2) ata[i][j] += row[i] * row[j]
}
}
private fun solve3(a: Array<DoubleArray>, b: DoubleArray): DoubleArray? {
val det = det3(a)
if (abs(det) < 1e-12) return null
return DoubleArray(3) { col -> det3(Array(3) { r -> DoubleArray(3) { c -> if (c == col) b[r] else a[r][c] } }) / det }
}
private fun det3(m: Array<DoubleArray>) = m[0][0] * (m[1][1] * m[2][2] - m[1][2] * m[2][1]) -
m[0][1] * (m[1][0] * m[2][2] - m[1][2] * m[2][0]) + m[0][2] * (m[1][0] * m[2][1] - m[1][1] * m[2][0])
}
internal object BodyPose {
const val NOSE = 0
const val LEFT_EYE = 2
const val RIGHT_EYE = 5
const val LEFT_EAR = 7
const val RIGHT_EAR = 8
const val RIGHT_MOUTH = 10
const val LEFT_SHOULDER = 11
const val RIGHT_SHOULDER = 12
const val LEFT_ELBOW = 13
const val RIGHT_ELBOW = 14
const val LEFT_WRIST = 15
const val RIGHT_WRIST = 16
const val LEFT_PINKY = 17
const val RIGHT_PINKY = 18
const val LEFT_INDEX = 19
const val RIGHT_INDEX = 20
const val LEFT_HIP = 23
const val RIGHT_HIP = 24
private val ANCHORS = intArrayOf(NOSE, LEFT_EYE, RIGHT_EYE, LEFT_EAR, RIGHT_EAR, LEFT_SHOULDER, RIGHT_SHOULDER, LEFT_HIP, RIGHT_HIP)
private val WRISTS = intArrayOf(LEFT_WRIST, RIGHT_WRIST)
private val PINKIES = intArrayOf(LEFT_PINKY, RIGHT_PINKY)
private val INDEXES = intArrayOf(LEFT_INDEX, RIGHT_INDEX)
const val MIN_VISIBILITY = 0.6f
fun solve(
image: List<Landmark>,
world: List<WorldPoint>,
k: Intrinsics,
level: CameraLevel,
fixes: Array<HandFix?>? = null,
held: FloatArray? = null,
): BodyFrame? {
if (image.size <= RIGHT_HIP || world.size <= RIGHT_HIP || !level.valid) return null
val t = translation(image, world, k) ?: return null
fun at(i: Int): Vec3 {
val w = world[i]
val depth = w.z + t.z
val px = image[i].x * k.width
val py = image[i].y * k.height
return Vec3((px - k.cx) / k.fx * depth, (py - k.cy) / k.fy * depth, depth)
}
fun visible(i: Int) = image[i].visibility >= MIN_VISIBILITY && image[i].x in 0f..1f && image[i].y in 0f..1f
val eyesSeen = visible(LEFT_EYE) && visible(RIGHT_EYE)
val shouldersSeen = visible(LEFT_SHOULDER) && visible(RIGHT_SHOULDER)
val hands = arrayOfNulls<Vec3>(2)
val estimated = BooleanArray(2)
val joints = arrayOfNulls<FloatArray>(2)
if (fixes == null) {
val head = if (eyesSeen) (at(LEFT_EYE) + at(RIGHT_EYE)) * 0.5f else null
if (head != null) for (side in 0..1) {
if (!visible(WRISTS[side])) continue
val grip = (at(WRISTS[side]) + at(PINKIES[side]) + at(INDEXES[side])) * (1f / 3f)
hands[side] = level.level(grip - head)
}
} else {
val head = if (eyesSeen) eyes(image, world, k) else null
if (head != null) for (side in 0..1) {
val fix = fixes.getOrNull(side)
if (fix != null) {
hands[side] = level.level(fix.palm - head)
joints[side] = handJoints(fix.world, level)
continue
}
val depth = held?.getOrNull(side)?.takeIf { it.isFinite() && it > 0f } ?: continue
if (!visible(WRISTS[side])) continue
val gx = (image[WRISTS[side]].x + image[PINKIES[side]].x + image[INDEXES[side]].x) / 3f * k.width
val gy = (image[WRISTS[side]].y + image[PINKIES[side]].y + image[INDEXES[side]].y) / 3f * k.height
hands[side] = level.level(Vec3((gx - k.cx) / k.fx * depth, (gy - k.cy) / k.fy * depth, depth) - head)
estimated[side] = true
}
}
val faceYaw = if (visible(NOSE) && visible(LEFT_EAR) && visible(RIGHT_EAR)) {
val n = world[NOSE]
val l = world[LEFT_EAR]
val r = world[RIGHT_EAR]
val f = level.level(Vec3(n.x - (l.x + r.x) / 2, n.y - (l.y + r.y) / 2, n.z - (l.z + r.z) / 2))
if (hypot(f.x, f.z) > 0.02f) atan2(-f.x, -f.z) else null
} else null
val span = if (shouldersSeen) hypot(image[LEFT_SHOULDER].x - image[RIGHT_SHOULDER].x,
(image[LEFT_SHOULDER].y - image[RIGHT_SHOULDER].y) * k.height / k.width) else 0f
var room = 1f
for (i in EDGE_JOINTS) {
if (image[i].visibility < MIN_VISIBILITY) continue
room = minOf(room, image[i].x, 1f - image[i].x, image[i].y, 1f - image[i].y)
}
return BodyFrame(hands, faceYaw, t.z, span, eyesSeen && shouldersSeen, estimated, room, joints)
}
fun handJoints(world: List<WorldPoint>, level: CameraLevel): FloatArray? {
if (world.size < HAND_POINTS) return null
var mx = 0f
var my = 0f
var mz = 0f
for (i in PALM) {
mx += world[i].x
my += world[i].y
mz += world[i].z
}
val n = PALM.size.toFloat()
val out = FloatArray(HAND_POINTS * 3)
for (i in 0 until HAND_POINTS) {
val v = level.level(Vec3(world[i].x - mx / n, world[i].y - my / n, world[i].z - mz / n))
out[i * 3] = v.x
out[i * 3 + 1] = v.y
out[i * 3 + 2] = v.z
}
return out
}
fun eyes(image: List<Landmark>, world: List<WorldPoint>, k: Intrinsics): Vec3? {
val used = HEAD.filter { image[it].visibility >= MIN_VISIBILITY }
if (used.size < MIN_HEAD_POINTS) return null
val t = Pnp.translation(used.size, k, { image[used[it]].x * k.width }, { image[used[it]].y * k.height }, { world[used[it]] })
?: return null
if (!(t.z > MIN_DISTANCE && t.z < MAX_DISTANCE)) return null
val l = world[LEFT_EYE]
val r = world[RIGHT_EYE]
return Vec3(t.x + (l.x + r.x) / 2, t.y + (l.y + r.y) / 2, t.z + (l.z + r.z) / 2)
}
fun handFix(points: FloatArray, world: List<WorldPoint>, k: Intrinsics): HandFix? {
if (world.size < HAND_POINTS || points.size < HAND_POINTS * 2) return null
val t = Pnp.translation(HAND_POINTS, k, { points[it * 2] }, { points[it * 2 + 1] }, { world[it] }) ?: return null
if (!(t.z > MIN_HAND_DISTANCE && t.z < MAX_DISTANCE)) return null
var px = 0f
var py = 0f
var wx = 0f
var wy = 0f
var wz = 0f
for (i in PALM) {
px += points[i * 2]
py += points[i * 2 + 1]
wx += world[i].x
wy += world[i].y
wz += world[i].z
}
val n = PALM.size.toFloat()
var minX = Float.MAX_VALUE
var maxX = -Float.MAX_VALUE
var minY = Float.MAX_VALUE
var maxY = -Float.MAX_VALUE
for (i in 0 until HAND_POINTS) {
minX = minOf(minX, points[i * 2])
maxX = maxOf(maxX, points[i * 2])
minY = minOf(minY, points[i * 2 + 1])
maxY = maxOf(maxY, points[i * 2 + 1])
}
return HandFix(Vec3(t.x + wx / n, t.y + wy / n, t.z + wz / n), px / n, py / n, maxOf(maxX - minX, maxY - minY), points, world)
}
private fun translation(image: List<Landmark>, world: List<WorldPoint>, k: Intrinsics): Vec3? {
val used = ANCHORS.filter { image[it].visibility >= MIN_VISIBILITY }
if (used.size < MIN_ANCHORS) return null
val t = Pnp.translation(used.size, k, { image[used[it]].x * k.width }, { image[used[it]].y * k.height }, { world[used[it]] })
?: return null
if (!(t.z > MIN_DISTANCE && t.z < MAX_DISTANCE)) return null
return t
}
const val HAND_POINTS = 21
private val HEAD = (NOSE..RIGHT_MOUTH).toList()
private val PALM = intArrayOf(0, 5, 9, 13, 17)
private val EDGE_JOINTS = intArrayOf(LEFT_ELBOW, RIGHT_ELBOW, LEFT_WRIST, RIGHT_WRIST)
private const val MIN_ANCHORS = 4
private const val MIN_HEAD_POINTS = 4
private const val MIN_DISTANCE = 0.2f
private const val MIN_HAND_DISTANCE = 0.05f
private const val MAX_DISTANCE = 6f
}
@@ -0,0 +1,323 @@
package com.droiddeck.launcher.handtracking
import android.content.Context
import android.graphics.Bitmap
import android.graphics.Canvas
import android.graphics.Matrix
import android.hardware.Sensor
import android.hardware.SensorEvent
import android.hardware.SensorEventListener
import android.hardware.SensorManager
import android.hardware.camera2.CaptureRequest
import android.os.Build
import android.os.PowerManager
import android.os.SystemClock
import android.util.Log
import android.util.Range
import android.util.Size
import androidx.camera.camera2.interop.Camera2CameraInfo
import androidx.camera.camera2.interop.Camera2Interop
import androidx.camera.camera2.interop.ExperimentalCamera2Interop
import androidx.camera.core.CameraSelector
import androidx.camera.core.ImageAnalysis
import androidx.camera.core.ImageProxy
import androidx.camera.core.resolutionselector.AspectRatioStrategy
import androidx.camera.core.resolutionselector.ResolutionSelector
import androidx.camera.core.resolutionselector.ResolutionStrategy
import androidx.camera.lifecycle.ProcessCameraProvider
import androidx.core.content.ContextCompat
import androidx.lifecycle.Lifecycle
import androidx.lifecycle.LifecycleOwner
import androidx.lifecycle.LifecycleRegistry
import com.droiddeck.launcher.glasses.Vec3
import com.google.mediapipe.framework.image.BitmapImageBuilder
import com.google.mediapipe.tasks.core.BaseOptions
import com.google.mediapipe.tasks.core.Delegate
import com.google.mediapipe.tasks.vision.core.RunningMode
import com.google.mediapipe.tasks.vision.poselandmarker.PoseLandmarker
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import java.io.File
import java.nio.ByteBuffer
import java.nio.ByteOrder
import java.util.concurrent.Executors
import kotlin.math.abs
@androidx.annotation.OptIn(ExperimentalCamera2Interop::class)
internal class HandCamera(
context: Context,
private val output: File?,
private val preview: Boolean,
) : LifecycleOwner {
class View(
val image: Bitmap,
val front: Boolean,
val landmarks: List<Landmark>?,
val placement: Placement,
val advice: Advice,
)
private val context = context.applicationContext
private val registry = LifecycleRegistry(this)
override val lifecycle: Lifecycle get() = registry
private val executor = Executors.newSingleThreadExecutor { Thread(it, "hand-camera") }
private val sensors = this.context.getSystemService(SensorManager::class.java)
private val power = this.context.getSystemService(PowerManager::class.java)
private val state = MutableStateFlow<View?>(null)
val view: StateFlow<View?> = state.asStateFlow()
private val broken = MutableStateFlow(false)
val failed: StateFlow<Boolean> = broken.asStateFlow()
@Volatile private var up: Vec3? = null
@Volatile private var thermal = 0
@Volatile private var stopped = false
@Volatile var bareHands = false
set(value) {
if (field != value) Log.i(TAG, if (value) "no controllers: tracking bare hands" else "controllers in use: placing them only")
field = value
}
private var provider: ProcessCameraProvider? = null
private var analysis: ImageAnalysis? = null
private var landmarker: PoseLandmarker? = null
private var hands: HandModel? = null
private val heldDepth = FloatArray(2) { Float.NaN }
private val heldAt = LongArray(2)
private var shm: HandCameraShm? = null
private var camera: TrackingCamera? = null
private var zoom = 1f
private var lastRun = 0L
private var lastStampMs = 0L
private var skipped = 0
private var recorder: HandCameraRecorder? = null
private var rawFrame: Bitmap? = null
private var turnedFrame: Bitmap? = null
private val gravity = object : SensorEventListener {
override fun onSensorChanged(event: SensorEvent) {
val v = Vec3(event.values[0], event.values[1], event.values[2])
val last = up
up = if (event.sensor.type == Sensor.TYPE_GRAVITY || last == null) v else last + (v - last) * ACCEL_SMOOTHING
}
override fun onAccuracyChanged(sensor: Sensor, accuracy: Int) {}
}
private val heat: Any? = if (Build.VERSION.SDK_INT >= 29) PowerManager.OnThermalStatusChangedListener { thermal = it } else null
init {
registry.currentState = Lifecycle.State.CREATED
}
fun start(side: String?) {
val chosen = TrackingCameras.pick(HandTracking.cameras(context), side)
if (chosen == null) {
broken.value = true
return
}
camera = chosen
zoom = chosen.zoom()
recorder = HandCameraRecorder.start(context, output, preview, chosen)
val sensor = sensors?.getDefaultSensor(Sensor.TYPE_GRAVITY) ?: sensors?.getDefaultSensor(Sensor.TYPE_ACCELEROMETER)
if (sensor != null) sensors?.registerListener(gravity, sensor, SensorManager.SENSOR_DELAY_UI)
if (Build.VERSION.SDK_INT >= 29) {
thermal = power?.currentThermalStatus ?: 0
power?.addThermalStatusListener(ContextCompat.getMainExecutor(context), heat as PowerManager.OnThermalStatusChangedListener)
}
executor.execute { prepare() }
val future = ProcessCameraProvider.getInstance(context)
future.addListener({
if (stopped) return@addListener
runCatching { bind(future.get(), chosen) }.onFailure {
Log.w(TAG, "camera ${chosen.id} could not start", it)
broken.value = true
}
}, ContextCompat.getMainExecutor(context))
}
fun stop() {
if (stopped) return
stopped = true
sensors?.unregisterListener(gravity)
if (Build.VERSION.SDK_INT >= 29) power?.removeThermalStatusListener(heat as PowerManager.OnThermalStatusChangedListener)
analysis?.let {
it.clearAnalyzer()
runCatching { provider?.unbind(it) }
}
registry.currentState = Lifecycle.State.DESTROYED
executor.execute {
runCatching { landmarker?.close() }
landmarker = null
hands?.close()
hands = null
shm?.close()
shm = null
recorder?.close()
recorder = null
}
executor.shutdown()
}
private fun bind(cameras: ProcessCameraProvider, chosen: TrackingCamera) {
val selector = CameraSelector.Builder()
.addCameraFilter { infos -> infos.filter { Camera2CameraInfo.from(it).cameraId == chosen.id } }
.build()
val builder = ImageAnalysis.Builder()
val extender = Camera2Interop.Extender(builder)
chosen.fps?.let { extender.setCaptureRequestOption(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, Range(it.first, it.second)) }
recorder?.let { extender.setSessionCaptureCallback(it.callback) }
val use = builder
.setResolutionSelector(
ResolutionSelector.Builder()
.setAspectRatioStrategy(AspectRatioStrategy.RATIO_4_3_FALLBACK_AUTO_STRATEGY)
.setResolutionStrategy(ResolutionStrategy(Size(ANALYSIS_WIDTH, ANALYSIS_HEIGHT), ResolutionStrategy.FALLBACK_RULE_CLOSEST_HIGHER_THEN_LOWER))
.build(),
)
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
.build()
use.setAnalyzer(executor, ::analyze)
val bound = cameras.bindToLifecycle(this, selector, use)
if (zoom < 1f) bound.cameraControl.setZoomRatio(zoom)
provider = cameras
analysis = use
registry.currentState = Lifecycle.State.RESUMED
Log.i(TAG, "camera ${chosen.id} (${if (chosen.front) "front" else "rear"}, ${chosen.fovDegrees.toInt()}°, zoom $zoom, ${chosen.fps} fps) " +
if (preview) "checking placement" else "tracking hands")
}
private fun prepare() {
if (stopped) return
runCatching {
val bytes = context.assets.open(MODEL).use { it.readBytes() }
val model = ByteBuffer.allocateDirect(bytes.size).order(ByteOrder.nativeOrder()).put(bytes)
model.rewind()
val options = PoseLandmarker.PoseLandmarkerOptions.builder()
.setBaseOptions(BaseOptions.builder().setModelAssetBuffer(model).setDelegate(Delegate.CPU).build())
.setRunningMode(RunningMode.VIDEO)
.setNumPoses(1)
.setMinPoseDetectionConfidence(MIN_CONFIDENCE)
.setMinPosePresenceConfidence(MIN_CONFIDENCE)
.setOutputSegmentationMasks(false)
.build()
landmarker = PoseLandmarker.createFromOptions(context, options)
if (output != null) shm = HandCameraShm.open(output)
}.onFailure {
Log.w(TAG, "pose model could not load", it)
broken.value = true
}
if (landmarker != null) hands = runCatching { HandModel.open(context) }.onFailure { Log.w(TAG, "hand model could not load", it) }.getOrNull()
}
private fun analyze(image: ImageProxy) {
try {
val chosen = camera ?: return
val pose = landmarker ?: return
val arrival = System.nanoTime()
if (stopped || arrival - lastRun < interval()) {
skipped++
return
}
lastRun = arrival
val exposed = exposure(image.imageInfo.timestamp, arrival, chosen)
val stampMs = maxOf(exposed / 1_000_000L, lastStampMs + 1)
lastStampMs = stampMs
val gravity = up
val placement = if (gravity != null) CameraGeometry.placement(chosen.front, gravity) else Placement.UNKNOWN
val turns = if (gravity != null) CameraGeometry.uprightTurns(chosen.front, chosen.orientation, gravity)
else image.imageInfo.rotationDegrees / 90
val frame = upright(raw(image), turns)
val result = pose.detectForVideo(BitmapImageBuilder(frame).build(), stampMs)
val inferred = System.nanoTime()
val found = result.landmarks().firstOrNull()
val marks = found?.map { Landmark(it.x(), it.y(), it.visibility().orElse(0f)) }
val world = result.worldLandmarks().firstOrNull()?.map { WorldPoint(it.x(), it.y(), it.z()) }
val k = chosen.intrinsics(image.width, image.height, zoom, turns)
val model = hands
val fixes = if (marks != null && model != null) model.find(frame, marks, k, stampMs) else null
val handsDone = System.nanoTime()
fixes?.forEachIndexed { side, fix ->
if (fix != null) {
heldDepth[side] = fix.palm.z
heldAt[side] = exposed
}
}
val held = FloatArray(2) { if (exposed - heldAt[it] <= HOLD_NS) heldDepth[it] else Float.NaN }
val body = if (marks != null && world != null && gravity != null) {
BodyPose.solve(marks, world, k, CameraLevel(CameraGeometry.upInCamera(chosen.front, chosen.orientation, turns, gravity)),
if (model != null) fixes ?: arrayOfNulls(2) else null, held)
} else null
shm?.write(exposed, body, bareHands)
if (preview) state.value = View(frame, chosen.front, marks, placement, PlacementAdvice.of(placement, body))
recorder?.frame(HandCameraRecorder.Frame(exposed, image.imageInfo.timestamp, arrival, inferred, System.nanoTime(), skipped,
thermal, turns, placement, gravity, k,
found?.map { floatArrayOf(it.x(), it.y(), it.z(), it.visibility().orElse(0f), it.presence().orElse(0f)) },
world, body, handsDone, fixes), frame)
skipped = 0
} catch (e: Exception) {
Log.w(TAG, "camera frame skipped", e)
} finally {
image.close()
}
}
private fun raw(image: ImageProxy): Bitmap {
val plane = image.planes[0]
if (plane.pixelStride != 4 || plane.rowStride != image.width * 4) return image.toBitmap()
val bitmap = rawFrame?.takeIf { it.width == image.width && it.height == image.height }
?: Bitmap.createBitmap(image.width, image.height, Bitmap.Config.ARGB_8888).also { rawFrame = it }
plane.buffer.rewind()
bitmap.copyPixelsFromBuffer(plane.buffer)
return bitmap
}
private fun upright(raw: Bitmap, turns: Int): Bitmap {
val quarter = Math.floorMod(turns, 4)
if (quarter == 0) return if (preview) raw.copy(Bitmap.Config.ARGB_8888, false) else raw
val width = if (quarter % 2 == 1) raw.height else raw.width
val height = if (quarter % 2 == 1) raw.width else raw.height
val target = if (preview) Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
else turnedFrame?.takeIf { it.width == width && it.height == height }
?: Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888).also { turnedFrame = it }
val matrix = Matrix()
matrix.postRotate(90f * quarter)
when (quarter) {
1 -> matrix.postTranslate(raw.height.toFloat(), 0f)
2 -> matrix.postTranslate(raw.width.toFloat(), raw.height.toFloat())
else -> matrix.postTranslate(0f, raw.width.toFloat())
}
Canvas(target).drawBitmap(raw, matrix, null)
return target
}
private fun exposure(stamp: Long, arrival: Long, chosen: TrackingCamera): Long {
val monotonic = if (chosen.realtime) stamp - (SystemClock.elapsedRealtimeNanos() - System.nanoTime()) else stamp
return if (abs(arrival - monotonic) < MAX_DELAY_NS) monotonic else arrival - TYPICAL_DELAY_NS
}
private fun interval(): Long {
val fps = when {
thermal >= SEVERE -> HOT_FPS
thermal >= MODERATE -> WARM_FPS
else -> FPS
}
return 1_000_000_000L / fps - SLACK_NS
}
companion object {
private const val TAG = "HandCamera"
private const val MODEL = "handtracking/pose_landmarker_lite.task"
private const val ANALYSIS_WIDTH = 960
private const val ANALYSIS_HEIGHT = 720
private const val MIN_CONFIDENCE = 0.5f
private const val FPS = 15
private const val WARM_FPS = 10
private const val HOT_FPS = 6
private const val MODERATE = 2
private const val SEVERE = 3
private const val SLACK_NS = 8_000_000L
private const val MAX_DELAY_NS = 1_000_000_000L
private const val TYPICAL_DELAY_NS = 60_000_000L
private const val ACCEL_SMOOTHING = 0.1f
private const val HOLD_NS = 500_000_000L
}
}
@@ -0,0 +1,203 @@
package com.droiddeck.launcher.handtracking
import android.content.Context
import android.graphics.Bitmap
import android.hardware.camera2.CameraCaptureSession
import android.hardware.camera2.CaptureRequest
import android.hardware.camera2.CaptureResult
import android.hardware.camera2.TotalCaptureResult
import android.util.Log
import com.droiddeck.launcher.glasses.Vec3
import java.io.BufferedWriter
import java.io.File
import java.io.FileOutputStream
import java.text.SimpleDateFormat
import java.util.Date
import java.util.Locale
import java.util.concurrent.Executors
import java.util.concurrent.atomic.AtomicBoolean
import kotlin.math.max
import kotlin.math.roundToInt
internal class HandCameraRecorder private constructor(private val dir: File, private val traceFlag: File?) {
class Capture(val exposureNs: Long, val frameDurationNs: Long, val iso: Int)
class Frame(
val exposureNs: Long,
val sensorNs: Long,
val arrivalNs: Long,
val inferredNs: Long,
val doneNs: Long,
val skipped: Int,
val thermal: Int,
val turns: Int,
val placement: Placement,
val gravity: Vec3?,
val intrinsics: Intrinsics?,
val marks: List<FloatArray>?,
val world: List<WorldPoint>?,
val body: BodyFrame?,
val handsNs: Long = 0L,
val fixes: Array<HandFix?>? = null,
)
private val writer = Executors.newSingleThreadExecutor { Thread(it, "hand-camera-record") }
private val lines: BufferedWriter = File(dir, "frames.jsonl").bufferedWriter()
private val busy = AtomicBoolean(false)
private val captures = object : LinkedHashMap<Long, Capture>() {
override fun removeEldestEntry(eldest: MutableMap.MutableEntry<Long, Capture>?) = size > KEEP_CAPTURES
}
private var count = 0
@Volatile private var closed = false
val callback = object : CameraCaptureSession.CaptureCallback() {
override fun onCaptureCompleted(session: CameraCaptureSession, request: CaptureRequest, result: TotalCaptureResult) {
val stamp = result.get(CaptureResult.SENSOR_TIMESTAMP) ?: return
val capture = Capture(result.get(CaptureResult.SENSOR_EXPOSURE_TIME) ?: 0L,
result.get(CaptureResult.SENSOR_FRAME_DURATION) ?: 0L, result.get(CaptureResult.SENSOR_SENSITIVITY) ?: 0)
synchronized(captures) { captures[stamp] = capture }
}
}
fun frame(frame: Frame, image: Bitmap) {
if (closed || count >= MAX_FRAMES) return
val index = ++count
val capture = synchronized(captures) { captures[frame.sensorNs] }
val small = if (busy.compareAndSet(false, true)) runCatching { shrink(image) }.getOrNull().also { if (it == null) busy.set(false) } else null
val name = if (small != null) "%05d.jpg".format(Locale.ROOT, index) else null
val line = json(index, frame, capture, name)
val queued = runCatching { writer.execute {
if (small != null) {
runCatching { FileOutputStream(File(dir, name!!)).use { small.compress(Bitmap.CompressFormat.JPEG, QUALITY, it) } }
small.recycle()
busy.set(false)
}
runCatching {
lines.write(line)
lines.newLine()
if (index % FLUSH_EVERY == 0) lines.flush()
}
} }
if (queued.isFailure) {
small?.recycle()
busy.set(false)
}
}
fun close() {
if (closed) return
closed = true
traceFlag?.delete()
writer.execute { runCatching { lines.close() } }
writer.shutdown()
Log.i(TAG, "recorded $count frames to $dir")
}
private fun shrink(image: Bitmap): Bitmap {
val scale = SIDE.toFloat() / max(image.width, image.height)
if (scale >= 1f) return image.copy(Bitmap.Config.ARGB_8888, false)
return Bitmap.createScaledBitmap(image, (image.width * scale).roundToInt(), (image.height * scale).roundToInt(), true)
}
private fun json(index: Int, f: Frame, capture: Capture?, jpg: String?): String = buildString {
append("{\"i\":").append(index)
append(",\"exp\":").append(f.exposureNs)
append(",\"sensor\":").append(f.sensorNs)
append(",\"arr\":").append(f.arrivalNs)
append(",\"inf\":").append(f.inferredNs)
append(",\"done\":").append(f.doneNs)
append(",\"hand\":").append(f.handsNs)
append(",\"skip\":").append(f.skipped)
append(",\"heat\":").append(f.thermal)
append(",\"turns\":").append(f.turns)
append(",\"place\":\"").append(f.placement.name).append('"')
append(",\"g\":").vec(f.gravity)
f.intrinsics?.let { append(",\"k\":[").num(it.fx).append(',').num(it.cx).append(',').num(it.cy).append(',').append(it.width).append(',').append(it.height).append(']') }
capture?.let { append(",\"ae\":[").append(it.exposureNs).append(',').append(it.frameDurationNs).append(',').append(it.iso).append(']') }
append(",\"lm\":")
if (f.marks == null) append("null") else {
append('[')
f.marks.forEachIndexed { i, m ->
if (i > 0) append(',')
append('[')
m.forEachIndexed { j, v -> if (j > 0) append(','); num(v) }
append(']')
}
append(']')
}
append(",\"wl\":")
if (f.world == null) append("null") else {
append('[')
f.world.forEachIndexed { i, w ->
if (i > 0) append(',')
append('[').num(w.x).append(',').num(w.y).append(',').num(w.z).append(']')
}
append(']')
}
f.body?.let { b ->
append(",\"hands\":[").vec(b.hands[0]).append(',').vec(b.hands[1]).append(']')
append(",\"face\":"); if (b.faceYaw == null) append("null") else num(b.faceYaw)
append(",\"est\":[").append(b.estimated[0]).append(',').append(b.estimated[1]).append(']')
append(",\"room\":").num(b.edgeRoom)
append(",\"dist\":").num(b.distance)
append(",\"span\":").num(b.shoulderSpan)
append(",\"joints\":[")
b.joints.forEachIndexed { i, j ->
if (i > 0) append(',')
if (j == null) append("null") else {
append('[')
j.forEachIndexed { k, v -> if (k > 0) append(','); num(v) }
append(']')
}
}
append(']')
}
f.fixes?.let { fixes ->
append(",\"fix\":[")
fixes.forEachIndexed { i, fix ->
if (i > 0) append(',')
if (fix == null) append("null") else {
append("{\"palm\":").vec(fix.palm).append(",\"at\":[").num(fix.x).append(',').num(fix.y).append("],\"size\":").num(fix.extent)
append(",\"pts\":[")
fix.points.forEachIndexed { j, v -> if (j > 0) append(','); num(v) }
append("],\"world\":[")
fix.world.forEachIndexed { j, w -> if (j > 0) append(','); num(w.x).append(',').num(w.y).append(',').num(w.z) }
append("]}")
}
}
append(']')
}
append(",\"jpg\":"); if (jpg == null) append("null") else append('"').append(jpg).append('"')
append('}')
}
private fun StringBuilder.num(v: Float): StringBuilder = if (v.isFinite()) append(v) else append("null")
private fun StringBuilder.vec(v: Vec3?): StringBuilder =
if (v == null) append("null") else append('[').num(v.x).append(',').num(v.y).append(',').num(v.z).append(']')
companion object {
private const val TAG = "HandCamera"
const val FLAG = "handcam-record"
const val TRACE_FLAG = "droiddeck-handtrace.on"
private const val MAX_FRAMES = 5400
private const val KEEP_CAPTURES = 32
private const val SIDE = 640
private const val QUALITY = 85
private const val FLUSH_EVERY = 30
fun start(context: Context, output: File?, preview: Boolean, camera: TrackingCamera): HandCameraRecorder? {
if (!File(context.filesDir, FLAG).exists()) return null
return runCatching {
val stamp = SimpleDateFormat("yyyyMMdd-HHmmss", Locale.ROOT).format(Date())
val dir = File(context.cacheDir, "handcam-rec/$stamp-${if (preview) "check" else "track"}").apply { mkdirs() }
File(dir, "camera.json").writeText("{\"id\":\"${camera.id}\",\"front\":${camera.front},\"orientation\":${camera.orientation}," +
"\"focal\":${camera.focalPixels},\"active\":[${camera.activeWidth},${camera.activeHeight}]," +
"\"fps\":${camera.fps?.let { "[${it.first},${it.second}]" } ?: "null"}}")
val flag = output?.parentFile?.let { File(it, TRACE_FLAG) }?.also { it.createNewFile() }
Log.i(TAG, "recording hand camera to $dir")
HandCameraRecorder(dir, flag)
}.onFailure { Log.w(TAG, "hand camera recording unavailable", it) }.getOrNull()
}
}
}
@@ -0,0 +1,103 @@
package com.droiddeck.launcher.handtracking
import android.util.Log
import com.droiddeck.launcher.glasses.Vec3
import com.droiddeck.launcher.input.FakeInputWriter
import java.io.File
import java.io.RandomAccessFile
import java.nio.ByteOrder
import java.nio.MappedByteBuffer
import java.nio.channels.FileChannel
internal class HandCameraShm private constructor(
private val file: RandomAccessFile,
private val map: MappedByteBuffer,
private val fence: () -> Unit,
) {
private var seq = map.getInt(SEQ).let { if (it and 1 != 0) it + 1 else it }
private var frame = 0
init {
begin()
map.putInt(MAGIC_AT, MAGIC)
map.putInt(VERSION_AT, VERSION)
for (at in FLAGS until SIZE step 4) map.putInt(at, 0)
end()
}
fun write(timeNs: Long, body: BodyFrame?, bareHands: Boolean = false) {
begin()
val face = body?.faceYaw
map.putInt(FLAGS, RUNNING or (if (face != null) FACE else 0) or (if (bareHands) BARE_HANDS else 0))
map.putLong(TIME, timeNs)
map.putInt(FRAME, ++frame)
map.putFloat(FACE_YAW, face ?: 0f)
for (side in 0..1) {
val at = HANDS + side * HAND_SIZE
val hand: Vec3? = body?.hands?.getOrNull(side)
val joints = if (hand != null) body?.joints?.getOrNull(side)?.takeIf { it.size == JOINT_FLOATS } else null
val seen = if (body?.estimated?.getOrNull(side) == true) SEEN or ESTIMATED else SEEN
map.putInt(at, if (hand == null) 0 else if (joints != null) seen or JOINTS else seen)
map.putFloat(at + 4, hand?.x ?: 0f)
map.putFloat(at + 8, hand?.y ?: 0f)
map.putFloat(at + 12, hand?.z ?: 0f)
for (i in 0 until JOINT_FLOATS) map.putFloat(at + JOINTS_AT + i * 4, joints?.get(i) ?: 0f)
}
end()
}
fun close() {
begin()
for (at in FLAGS until SIZE step 4) map.putInt(at, 0)
end()
runCatching { file.close() }
}
private fun begin() {
map.putInt(SEQ, ++seq)
fence()
}
private fun end() {
fence()
map.putInt(SEQ, ++seq)
}
companion object {
const val FILE = "droiddeck-handcam"
const val SIZE = 568
private const val MAGIC = 0x4D434844
private const val VERSION = 2
private const val MAGIC_AT = 0
private const val VERSION_AT = 4
private const val SEQ = 8
private const val FLAGS = 12
private const val TIME = 16
private const val FRAME = 24
private const val FACE_YAW = 28
private const val HANDS = 32
private const val HAND_SIZE = 268
private const val JOINTS_AT = 16
private const val JOINT_FLOATS = BodyPose.HAND_POINTS * 3
private const val RUNNING = 1
private const val FACE = 2
private const val BARE_HANDS = 4
private const val SEEN = 1
private const val ESTIMATED = 2
private const val JOINTS = 4
fun open(path: File, fence: () -> Unit = FakeInputWriter::storeFence): HandCameraShm? = runCatching {
path.parentFile?.mkdirs()
val file = RandomAccessFile(path, "rw")
try {
file.setLength(SIZE.toLong())
val map = file.channel.map(FileChannel.MapMode.READ_WRITE, 0, SIZE.toLong())
map.order(ByteOrder.LITTLE_ENDIAN)
HandCameraShm(file, map, fence)
} catch (e: Exception) {
file.close()
throw e
}
}.onFailure { Log.w("HandCamera", "hand camera file unavailable: $path", it) }.getOrNull()
}
}
@@ -0,0 +1,172 @@
package com.droiddeck.launcher.handtracking
import android.content.Context
import android.graphics.Bitmap
import android.graphics.Canvas
import android.graphics.Color
import android.graphics.Matrix
import android.graphics.Paint
import com.google.mediapipe.framework.image.BitmapImageBuilder
import com.google.mediapipe.tasks.components.containers.Landmark as MpLandmark
import com.google.mediapipe.tasks.components.containers.NormalizedLandmark
import com.google.mediapipe.tasks.core.BaseOptions
import com.google.mediapipe.tasks.core.Delegate
import com.google.mediapipe.tasks.vision.core.RunningMode
import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarker
import java.nio.ByteBuffer
import java.nio.ByteOrder
import kotlin.math.hypot
import kotlin.math.max
internal class HandModel private constructor(private val landmarkers: List<HandLandmarker>, private val starter: HandLandmarker) {
private class Track(val x: Float, val y: Float, val size: Float)
private val crops = List(2) { Bitmap.createBitmap(CROP, CROP, Bitmap.Config.ARGB_8888) }
private val startCrop = Bitmap.createBitmap(START_CROP, START_CROP, Bitmap.Config.ARGB_8888)
private val tracks = arrayOfNulls<Track>(2)
private val starts = IntArray(2)
private val paint = Paint(Paint.FILTER_BITMAP_FLAG)
private val matrix = Matrix()
fun find(frame: Bitmap, marks: List<Landmark>, k: Intrinsics, stampMs: Long): Array<HandFix?> {
val fixes = arrayOfNulls<HandFix>(2)
if (marks.size <= BodyPose.RIGHT_INDEX) return fixes
val shoulders = hypot((marks[BodyPose.LEFT_SHOULDER].x - marks[BodyPose.RIGHT_SHOULDER].x) * k.width,
(marks[BodyPose.LEFT_SHOULDER].y - marks[BodyPose.RIGHT_SHOULDER].y) * k.height)
for (side in 0..1) {
val wrist = marks[WRISTS[side]]
val wx = wrist.x * k.width
val wy = wrist.y * k.height
val fix = tracks[side]?.let { follow(frame, it, side, k, stampMs) }?.takeIf { belongs(it, marks, side, shoulders, k) }
?: start(frame, marks, side, wx, wy, shoulders, k)?.takeIf { belongs(it, marks, side, shoulders, k) }
if (fix == null) {
tracks[side] = null
continue
}
fixes[side] = fix
starts[side] = 0
tracks[side] = Track(fix.x, fix.y, max(fix.extent, shoulders * MIN_HAND))
}
val left = fixes[0]
val right = fixes[1]
if (left != null && right != null && hypot(left.x - right.x, left.y - right.y) < max(left.extent, right.extent) * SAME_HAND) {
val drop = if (gap(left, marks, 0, k) <= gap(right, marks, 1, k)) 1 else 0
fixes[drop] = null
tracks[drop] = null
}
return fixes
}
fun close() {
landmarkers.forEach { runCatching { it.close() } }
runCatching { starter.close() }
crops.forEach { it.recycle() }
startCrop.recycle()
}
private fun follow(frame: Bitmap, track: Track, side: Int, k: Intrinsics, stampMs: Long): HandFix? {
val size = max(track.size * CROP_HANDS, MIN_CROP)
val x0 = track.x - size / 2
val y0 = track.y - size / 2
draw(frame, crops[side], x0, y0, size)
val result = landmarkers[side].detectForVideo(BitmapImageBuilder(crops[side]).build(), stampMs)
val found = result.landmarks().firstOrNull() ?: return null
val world = result.worldLandmarks().firstOrNull() ?: return null
return fixOf(found, world, x0, y0, size, k)
}
private fun start(frame: Bitmap, marks: List<Landmark>, side: Int, wx: Float, wy: Float, shoulders: Float, k: Intrinsics): HandFix? {
val wrist = marks[WRISTS[side]]
if (wrist.visibility < MIN_VISIBILITY || shoulders < 1f) return null
if (wrist.x < -EDGE || wrist.x > 1f + EDGE || wrist.y < -EDGE || wrist.y > 1f + EDGE) return null
val wide = starts[side]++ % 2 == 0
val tx = (marks[PINKIES[side]].x + marks[INDEXES[side]].x) / 2 * k.width - wx
val ty = (marks[PINKIES[side]].y + marks[INDEXES[side]].y) / 2 * k.height - wy
val length = hypot(tx, ty)
val reach = if (wide) 0f else shoulders * TOWARD_FINGERS
val cx = if (length > 1e-3f) wx + tx / length * reach else wx
val cy = if (length > 1e-3f) wy + ty / length * reach else wy
val size = max(shoulders * if (wide) WIDE_SHOULDERS else START_SHOULDERS, MIN_CROP)
val x0 = cx - size / 2
val y0 = cy - size / 2
draw(frame, startCrop, x0, y0, size)
val result = starter.detect(BitmapImageBuilder(startCrop).build())
var best: HandFix? = null
var bestGap = Float.MAX_VALUE
result.landmarks().forEachIndexed { i, found ->
val world = result.worldLandmarks().getOrNull(i) ?: return@forEachIndexed
val fix = fixOf(found, world, x0, y0, size, k) ?: return@forEachIndexed
val gap = hypot(fix.points[0] - wx, fix.points[1] - wy)
if (gap < bestGap) {
best = fix
bestGap = gap
}
}
return best
}
private fun draw(frame: Bitmap, crop: Bitmap, x0: Float, y0: Float, size: Float) {
crop.eraseColor(Color.BLACK)
matrix.setTranslate(-x0, -y0)
matrix.postScale(crop.width / size, crop.height / size)
Canvas(crop).drawBitmap(frame, matrix, paint)
}
private fun fixOf(found: List<NormalizedLandmark>, world: List<MpLandmark>, x0: Float, y0: Float, size: Float, k: Intrinsics): HandFix? {
if (found.size < BodyPose.HAND_POINTS || world.size < BodyPose.HAND_POINTS) return null
val points = FloatArray(BodyPose.HAND_POINTS * 2) { i ->
val p = found[i / 2]
if (i % 2 == 0) x0 + p.x() * size else y0 + p.y() * size
}
return BodyPose.handFix(points, world.map { WorldPoint(it.x(), it.y(), it.z()) }, k)
}
private fun belongs(fix: HandFix, marks: List<Landmark>, side: Int, shoulders: Float, k: Intrinsics): Boolean {
val own = marks[WRISTS[side]]
if (hypot(fix.x - own.x * k.width, fix.y - own.y * k.height) > max(fix.extent, shoulders) * MAX_WRIST_GAP) return false
val other = marks[WRISTS[1 - side]]
if (other.visibility < MIN_VISIBILITY || other.x !in 0f..1f || other.y !in 0f..1f) return true
return hypot(fix.points[0] - other.x * k.width, fix.points[1] - other.y * k.height) >=
hypot(fix.points[0] - own.x * k.width, fix.points[1] - own.y * k.height)
}
private fun gap(fix: HandFix, marks: List<Landmark>, side: Int, k: Intrinsics): Float =
hypot(fix.x - marks[WRISTS[side]].x * k.width, fix.y - marks[WRISTS[side]].y * k.height)
companion object {
private const val MODEL = "handtracking/hand_landmarker.task"
private val WRISTS = intArrayOf(BodyPose.LEFT_WRIST, BodyPose.RIGHT_WRIST)
private val PINKIES = intArrayOf(BodyPose.LEFT_PINKY, BodyPose.RIGHT_PINKY)
private val INDEXES = intArrayOf(BodyPose.LEFT_INDEX, BodyPose.RIGHT_INDEX)
private const val CROP = 256
private const val START_CROP = 384
private const val CROP_HANDS = 2.8f
private const val MIN_CROP = 48f
private const val START_SHOULDERS = 2.5f
private const val WIDE_SHOULDERS = 4f
private const val MIN_HAND = 0.25f
private const val TOWARD_FINGERS = 0.35f
private const val MAX_WRIST_GAP = 3f
private const val SAME_HAND = 0.5f
private const val EDGE = 0.05f
private const val MIN_VISIBILITY = 0.5f
private const val MIN_CONFIDENCE = 0.4f
fun open(context: Context): HandModel {
val bytes = context.assets.open(MODEL).use { it.readBytes() }
fun landmarker(mode: RunningMode, hands: Int): HandLandmarker {
val model = ByteBuffer.allocateDirect(bytes.size).order(ByteOrder.nativeOrder()).put(bytes)
model.rewind()
return HandLandmarker.createFromOptions(context, HandLandmarker.HandLandmarkerOptions.builder()
.setBaseOptions(BaseOptions.builder().setModelAssetBuffer(model).setDelegate(Delegate.CPU).build())
.setRunningMode(mode)
.setNumHands(hands)
.setMinHandDetectionConfidence(MIN_CONFIDENCE)
.setMinHandPresenceConfidence(MIN_CONFIDENCE)
.setMinTrackingConfidence(MIN_CONFIDENCE)
.build())
}
return HandModel(List(2) { landmarker(RunningMode.VIDEO, 1) }, landmarker(RunningMode.IMAGE, 2))
}
}
}
@@ -0,0 +1,110 @@
package com.droiddeck.launcher.handtracking
import android.Manifest
import android.content.Context
import android.content.pm.PackageManager
import android.os.Build
import com.droiddeck.launcher.input.ControllerPrefs
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
internal object HandTracking {
data class Settings(
val enabled: Boolean = false,
val asked: Boolean = false,
val side: String? = null,
val permitted: Boolean = false,
val sides: Set<String> = emptySet(),
) {
val usable: Boolean get() = sides.isNotEmpty()
val ready: Boolean get() = enabled && permitted && usable
}
private val state = MutableStateFlow(Settings())
val settings: StateFlow<Settings> = state.asStateFlow()
private val checks = MutableStateFlow(0)
val checkRequests: StateFlow<Int> = checks.asStateFlow()
@Volatile private var cameras: List<TrackingCamera>? = null
val supported: Boolean get() = Build.VERSION.SDK_INT >= 31
fun load(context: Context): Settings {
val prefs = ControllerPrefs.prefs(context)
val found = cameras(context)
val next = Settings(
enabled = prefs.getBoolean(ENABLED, false),
asked = prefs.getBoolean(ASKED, false),
side = prefs.getString(SIDE, null),
permitted = permitted(context),
sides = if (supported) TrackingCameras.sides(found) else emptySet(),
)
state.value = next
return next
}
fun cameras(context: Context): List<TrackingCamera> =
cameras ?: TrackingCameras.list(context.applicationContext).also { cameras = it }
fun camera(context: Context): TrackingCamera? = TrackingCameras.pick(cameras(context), state.value.side)
fun permitted(context: Context): Boolean =
context.checkSelfPermission(Manifest.permission.CAMERA) == PackageManager.PERMISSION_GRANTED
fun refreshPermission(context: Context) {
state.value = state.value.copy(permitted = permitted(context))
}
fun allow(context: Context, side: String?) {
val edit = ControllerPrefs.prefs(context).edit().putBoolean(ASKED, true).putBoolean(ENABLED, true)
if (side != null) edit.putString(SIDE, side)
edit.apply()
load(context)
}
fun decline(context: Context) {
ControllerPrefs.prefs(context).edit().putBoolean(ASKED, true).putBoolean(ENABLED, false).apply()
load(context)
}
fun setEnabled(context: Context, on: Boolean) {
ControllerPrefs.prefs(context).edit().putBoolean(ASKED, true).putBoolean(ENABLED, on).apply()
load(context)
}
fun setSide(context: Context, side: String) {
ControllerPrefs.prefs(context).edit().putString(SIDE, side).apply()
load(context)
}
fun shouldAsk(settings: Settings, headset: Boolean): Boolean = headset && !settings.asked && settings.usable
fun wanted(context: Context): Boolean = supported && load(context).ready
fun requestCheck() {
checks.value++
}
private const val ENABLED = "handCamera"
private const val ASKED = "handCameraAsked"
private const val SIDE = "handCameraSide"
}
internal enum class Advice { STARTING, STAND_UP, UPRIGHT, NOT_SEEN, MOVE_BACK, MOVE_CLOSER, SHOW_HANDS, GOOD }
internal object PlacementAdvice {
const val MIN_SPAN = 0.12f
const val MAX_SPAN = 0.26f
const val MIN_ROOM = 0.05f
fun of(placement: Placement, body: BodyFrame?): Advice = when {
placement == Placement.FACE_UP || placement == Placement.FACE_DOWN -> Advice.STAND_UP
placement == Placement.POINTS_UP || placement == Placement.POINTS_DOWN -> Advice.UPRIGHT
body == null || !body.seen -> Advice.NOT_SEEN
body.shoulderSpan > MAX_SPAN -> Advice.MOVE_BACK
body.shoulderSpan < MIN_SPAN -> Advice.MOVE_CLOSER
body.edgeRoom < MIN_ROOM -> Advice.MOVE_BACK
body.hands.any { it == null } -> Advice.SHOW_HANDS
else -> Advice.GOOD
}
}
@@ -0,0 +1,136 @@
package com.droiddeck.launcher.handtracking
import android.content.Context
import android.hardware.camera2.CameraCharacteristics
import android.hardware.camera2.CameraManager
import android.os.Build
import com.droiddeck.launcher.glasses.Vec3
import kotlin.math.asin
import kotlin.math.atan
import kotlin.math.max
import kotlin.math.min
import kotlin.math.tan
internal class TrackingCamera(
val id: String,
val front: Boolean,
val orientation: Int,
val focalPixels: Float,
val activeWidth: Int,
val activeHeight: Int,
val minZoom: Float,
val realtime: Boolean,
val fps: Pair<Int, Int>? = null,
) {
val fovDegrees: Float get() = Math.toDegrees(2.0 * atan(activeWidth / 2.0 / focalPixels)).toFloat()
fun zoom(): Float {
if (minZoom >= 1f) return 1f
val needed = (tan(Math.toRadians(fovDegrees / 2.0)) / tan(Math.toRadians(MAX_FOV / 2.0))).toFloat()
return min(1f, max(minZoom, needed))
}
fun intrinsics(bufferWidth: Int, bufferHeight: Int, zoom: Float, turns: Int): Intrinsics {
val aspect = bufferWidth.toFloat() / bufferHeight
val region = min(activeWidth.toFloat(), activeHeight * aspect) / zoom
val f = focalPixels * bufferWidth / region
val quarter = turns and 1 == 1
val w = if (quarter) bufferHeight else bufferWidth
val h = if (quarter) bufferWidth else bufferHeight
return Intrinsics(f, f, w / 2f, h / 2f, w, h)
}
companion object {
const val MAX_FOV = 100.0
}
}
internal object TrackingCameras {
const val REAR = "rear"
const val FRONT = "front"
private const val MAX_FPS = 30
fun list(context: Context): List<TrackingCamera> {
val manager = context.getSystemService(CameraManager::class.java) ?: return emptyList()
val ids = runCatching { manager.cameraIdList.toList() }.getOrDefault(emptyList())
return ids.mapNotNull { id -> runCatching { describe(id, manager.getCameraCharacteristics(id)) }.getOrNull() }
}
private fun describe(id: String, c: CameraCharacteristics): TrackingCamera? {
val facing = c.get(CameraCharacteristics.LENS_FACING) ?: return null
if (facing == CameraCharacteristics.LENS_FACING_EXTERNAL) return null
val caps = c.get(CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES) ?: return null
if (CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES_BACKWARD_COMPATIBLE !in caps) return null
val active = c.get(CameraCharacteristics.SENSOR_INFO_ACTIVE_ARRAY_SIZE) ?: return null
val pixels = c.get(CameraCharacteristics.SENSOR_INFO_PIXEL_ARRAY_SIZE) ?: return null
val physical = c.get(CameraCharacteristics.SENSOR_INFO_PHYSICAL_SIZE) ?: return null
val focal = c.get(CameraCharacteristics.LENS_INFO_AVAILABLE_FOCAL_LENGTHS)?.firstOrNull() ?: return null
val calibrated = c.get(CameraCharacteristics.LENS_INTRINSIC_CALIBRATION)?.firstOrNull()?.takeIf { it > 1f }
val focalPixels = calibrated ?: (focal * pixels.width / physical.width)
val minZoom = if (Build.VERSION.SDK_INT >= 30) c.get(CameraCharacteristics.CONTROL_ZOOM_RATIO_RANGE)?.lower ?: 1f else 1f
val realtime = c.get(CameraCharacteristics.SENSOR_INFO_TIMESTAMP_SOURCE) == CameraCharacteristics.SENSOR_INFO_TIMESTAMP_SOURCE_REALTIME
val fps = steadyFps(c.get(CameraCharacteristics.CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES)?.map { it.lower to it.upper }.orEmpty())
return TrackingCamera(id, facing == CameraCharacteristics.LENS_FACING_FRONT, c.get(CameraCharacteristics.SENSOR_ORIENTATION) ?: 0,
focalPixels, active.width(), active.height(), minZoom, realtime, fps)
}
fun steadyFps(ranges: List<Pair<Int, Int>>): Pair<Int, Int>? =
ranges.filter { it.second <= MAX_FPS }.maxWithOrNull(compareBy<Pair<Int, Int>> { it.first }.thenByDescending { it.second })
fun sides(cameras: List<TrackingCamera>): Set<String> = cameras.mapTo(LinkedHashSet()) { if (it.front) FRONT else REAR }
fun pick(cameras: List<TrackingCamera>, side: String?): TrackingCamera? {
val wanted = cameras.filter { side == null || (it.front == (side == FRONT)) }.ifEmpty { cameras }
val preferred = if (side == null) wanted.filter { !it.front }.ifEmpty { wanted } else wanted
return preferred.maxWithOrNull(compareBy<TrackingCamera> { min(widest(it), TrackingCamera.MAX_FOV.toFloat()) }
.thenBy { it.activeWidth.toLong() * it.activeHeight })
}
private fun widest(c: TrackingCamera): Float =
Math.toDegrees(2.0 * atan(tan(Math.toRadians(c.fovDegrees / 2.0)) / c.zoom())).toFloat()
}
internal object CameraGeometry {
fun axes(front: Boolean, orientation: Int, turns: Int): Pair<Vec3, Vec3> {
val t = Math.toRadians(orientation.toDouble())
val c = Math.cos(t).toFloat()
val s = Math.sin(t).toFloat()
var u = if (front) Vec3(-c, -s, 0f) else Vec3(c, -s, 0f)
var v = if (front) Vec3(s, -c, 0f) else Vec3(-s, -c, 0f)
repeat(Math.floorMod(turns, 4)) {
val nu = v * -1f
v = u
u = nu
}
return u to v
}
fun upInCamera(front: Boolean, orientation: Int, turns: Int, up: Vec3): Vec3 {
val (u, v) = axes(front, orientation, turns)
val n = up.normalized()
return Vec3(dot(n, u), dot(n, v), dot(n, u.cross(v)))
}
fun uprightTurns(front: Boolean, orientation: Int, up: Vec3): Int =
(0..3).minBy { turns -> dot(up.normalized(), axes(front, orientation, turns).second) }
fun placement(front: Boolean, up: Vec3): Placement {
val n = up.normalized()
if (n.length() < 0.5f) return Placement.UNKNOWN
if (n.z > FLAT) return Placement.FACE_UP
if (n.z < -FLAT) return Placement.FACE_DOWN
val elevation = Math.toDegrees(asin((if (front) n.z else -n.z).toDouble().coerceIn(-1.0, 1.0)))
return when {
elevation > STEEP -> Placement.POINTS_UP
elevation < -STEEP -> Placement.POINTS_DOWN
else -> Placement.STANDING
}
}
private fun dot(a: Vec3, b: Vec3) = a.x * b.x + a.y * b.y + a.z * b.z
private const val FLAT = 0.9f
private const val STEEP = 45.0
}
internal enum class Placement { UNKNOWN, STANDING, FACE_UP, FACE_DOWN, POINTS_UP, POINTS_DOWN }
@@ -47,6 +47,7 @@ object ControllerPrefs {
val qamButton: Boolean,
val keyboardButton: Boolean,
val selectSteam: Boolean,
val joyConVr: Boolean,
val snap: Boolean,
val customLayout: Boolean,
val mapping: Map<String, String>,
@@ -67,6 +68,7 @@ object ControllerPrefs {
qamButton = p.getBoolean("qamButton", true),
keyboardButton = p.getBoolean("keyboardButton", true),
selectSteam = p.getBoolean("selectSteam", false),
joyConVr = joyConVr(context),
snap = p.getBoolean("snap", true),
customLayout = p.all.keys.any { it.startsWith("layout.") },
mapping = mappable.associate { (id, _) -> id to target(context, id) },
@@ -127,6 +129,12 @@ object ControllerPrefs {
prefs(context).edit().putBoolean("selectSteam", on).apply()
}
fun joyConVr(context: Context): Boolean = prefs(context).getBoolean("joyConVr", true)
fun setJoyConVr(context: Context, on: Boolean) {
prefs(context).edit().putBoolean("joyConVr", on).apply()
}
fun setSnap(context: Context, on: Boolean) {
prefs(context).edit().putBoolean("snap", on).apply()
}
@@ -119,6 +119,10 @@ public class FakeInputWriter {
private static native void nativeStoreFence();
public static void storeFence() {
nativeStoreFence();
}
private final File eventFile;
private final int slot;
private final ByteBuffer buffer = ByteBuffer.allocateDirect(BUFFER_SIZE);
@@ -0,0 +1,143 @@
package com.droiddeck.launcher.input
import android.view.InputDevice
import android.view.KeyEvent
import android.view.MotionEvent
object JoyCon {
const val LEFT = 0
const val RIGHT = 1
const val VR_TRIGGER = 1 shl 0
const val VR_GRIP = 1 shl 1
const val VR_A = 1 shl 2
const val VR_B = 1 shl 3
const val VR_STICK = 1 shl 4
const val VR_SYSTEM = 1 shl 5
const val VR_RUMBLE_SLOT = 0x100
private const val VENDOR = 0x057e
private const val PRODUCT_LEFT = 0x2006
private const val PRODUCT_RIGHT = 0x2007
private const val BTN_SOUTH = 0x130
private const val BTN_EAST = 0x131
private const val BTN_NORTH = 0x133
private const val BTN_WEST = 0x134
private const val BTN_Z = 0x135
private const val BTN_TL = 0x136
private const val BTN_TR = 0x137
private const val BTN_TL2 = 0x138
private const val BTN_TR2 = 0x139
private const val BTN_SELECT = 0x13a
private const val BTN_START = 0x13b
private const val BTN_MODE = 0x13c
private const val BTN_THUMBL = 0x13d
private const val BTN_THUMBR = 0x13e
private const val BTN_DPAD_UP = 0x220
private const val BTN_DPAD_DOWN = 0x221
private const val BTN_DPAD_LEFT = 0x222
private const val BTN_DPAD_RIGHT = 0x223
private val NO_VR_DEVICES = intArrayOf(PadBridge.NO_CONTROLLER, PadBridge.NO_CONTROLLER)
private var vrOwner: Any? = null
@Volatile private var vrDevices = NO_VR_DEVICES
@JvmStatic
fun side(device: InputDevice?): Int = when {
device == null || device.isVirtual || device.vendorId != VENDOR -> -1
device.productId == PRODUCT_LEFT -> LEFT
device.productId == PRODUCT_RIGHT -> RIGHT
else -> -1
}
@JvmStatic
fun stickAxes(side: Int, device: InputDevice, source: Int): IntArray = when {
side == LEFT -> intArrayOf(MotionEvent.AXIS_X, MotionEvent.AXIS_Y)
device.getMotionRange(MotionEvent.AXIS_RX, source) != null -> intArrayOf(MotionEvent.AXIS_RX, MotionEvent.AXIS_RY)
else -> intArrayOf(MotionEvent.AXIS_Z, MotionEvent.AXIS_RZ)
}
@JvmStatic
fun isRail(side: Int, scanCode: Int): Boolean = when (side) {
LEFT -> scanCode == BTN_TR || scanCode == BTN_TR2
RIGHT -> scanCode == BTN_TL || scanCode == BTN_TL2
else -> false
}
internal fun padKey(side: Int, scanCode: Int): Int = when (side) {
LEFT -> when (scanCode) {
BTN_TL -> KeyEvent.KEYCODE_BUTTON_L1
BTN_TL2 -> KeyEvent.KEYCODE_BUTTON_L2
BTN_SELECT -> KeyEvent.KEYCODE_BUTTON_SELECT
BTN_THUMBL -> KeyEvent.KEYCODE_BUTTON_THUMBL
BTN_Z -> KeyEvent.KEYCODE_BUTTON_Z
BTN_DPAD_UP -> KeyEvent.KEYCODE_DPAD_UP
BTN_DPAD_DOWN -> KeyEvent.KEYCODE_DPAD_DOWN
BTN_DPAD_LEFT -> KeyEvent.KEYCODE_DPAD_LEFT
BTN_DPAD_RIGHT -> KeyEvent.KEYCODE_DPAD_RIGHT
else -> KeyEvent.KEYCODE_UNKNOWN
}
RIGHT -> when (scanCode) {
BTN_SOUTH -> KeyEvent.KEYCODE_BUTTON_A
BTN_EAST -> KeyEvent.KEYCODE_BUTTON_B
BTN_NORTH -> KeyEvent.KEYCODE_BUTTON_Y
BTN_WEST -> KeyEvent.KEYCODE_BUTTON_X
BTN_TR -> KeyEvent.KEYCODE_BUTTON_R1
BTN_TR2 -> KeyEvent.KEYCODE_BUTTON_R2
BTN_START -> KeyEvent.KEYCODE_BUTTON_START
BTN_THUMBR -> KeyEvent.KEYCODE_BUTTON_THUMBR
BTN_MODE -> KeyEvent.KEYCODE_BUTTON_MODE
else -> KeyEvent.KEYCODE_UNKNOWN
}
else -> KeyEvent.KEYCODE_UNKNOWN
}
@JvmStatic
fun normalize(event: KeyEvent): KeyEvent {
val side = side(event.device)
if (side < 0) return event
val code = padKey(side, event.scanCode)
if (code == KeyEvent.KEYCODE_UNKNOWN || code == event.keyCode) return event
return KeyEvent(event.downTime, event.eventTime, event.action, code, event.repeatCount, event.metaState,
event.deviceId, event.scanCode, event.flags, event.source)
}
internal fun vrButton(side: Int, scanCode: Int): Int = when (side) {
LEFT -> when (scanCode) {
BTN_TL2 -> VR_TRIGGER
BTN_TL, BTN_TR, BTN_TR2 -> VR_GRIP
BTN_DPAD_DOWN, BTN_DPAD_RIGHT -> VR_A
BTN_DPAD_UP, BTN_DPAD_LEFT, BTN_SELECT -> VR_B
BTN_THUMBL -> VR_STICK
BTN_Z -> VR_SYSTEM
else -> 0
}
RIGHT -> when (scanCode) {
BTN_TR2 -> VR_TRIGGER
BTN_TR, BTN_TL, BTN_TL2 -> VR_GRIP
BTN_SOUTH, BTN_EAST -> VR_A
BTN_NORTH, BTN_WEST, BTN_START -> VR_B
BTN_THUMBR -> VR_STICK
BTN_MODE -> VR_SYSTEM
else -> 0
}
else -> 0
}
@JvmStatic
fun vrRumbleDevice(slot: Int): Int? =
if (slot == VR_RUMBLE_SLOT + LEFT || slot == VR_RUMBLE_SLOT + RIGHT) vrDevices[slot - VR_RUMBLE_SLOT] else null
@Synchronized
internal fun setVrDevices(owner: Any, devices: IntArray) {
vrOwner = owner
vrDevices = devices
}
@Synchronized
internal fun clearVrDevices(owner: Any) {
if (vrOwner !== owner) return
vrOwner = null
vrDevices = NO_VR_DEVICES
}
}
@@ -0,0 +1,213 @@
package com.droiddeck.launcher.input
import android.content.Context
import com.droiddeck.launcher.glasses.Quaternion
import com.droiddeck.launcher.glasses.Vec3
import kotlin.math.abs
import kotlin.math.exp
import kotlin.math.roundToInt
import kotlin.math.sqrt
internal class JoyConAccelCalibration {
var bias = Vec3(0f, 0f, 0f)
private set
var scale = 1f
private set
var ready = false
private set
private val sums = HashMap<Int, DoubleArray>()
fun restore(saved: FloatArray?) {
if (saved == null || saved.size != 4 || saved.any { it.isNaN() || it.isInfinite() }) return
val b = Vec3(saved[0], saved[1], saved[2])
if (!plausible(b, saved[3])) return
bias = b
scale = saved[3]
ready = true
}
fun saved(): FloatArray? = if (ready) floatArrayOf(bias.x, bias.y, bias.z, scale) else null
fun correct(raw: Vec3): Vec3 = if (ready) (raw - bias) * scale else raw
fun addResting(raw: Vec3): Boolean {
val n = raw.length()
if (n < 1e-3f) return false
val key = bin(raw.x / n) * 9 + bin(raw.y / n) * 3 + bin(raw.z / n)
val s = sums.getOrPut(key) { DoubleArray(4) }
s[0] += raw.x.toDouble()
s[1] += raw.y.toDouble()
s[2] += raw.z.toDouble()
s[3] += 1.0
return s[3] % SOLVE_EVERY == 0.0 && solve()
}
private fun solve(): Boolean {
val points = sums.values.filter { it[3] >= MIN_SAMPLES }.map { doubleArrayOf(it[0] / it[3], it[1] / it[3], it[2] / it[3]) }
if (points.size < MIN_BINS || smallestSpread(points) < MIN_SPREAD) return false
val ata = Array(4) { DoubleArray(4) }
val atb = DoubleArray(4)
for (p in points) {
val row = doubleArrayOf(2 * p[0], 2 * p[1], 2 * p[2], 1.0)
val rhs = p[0] * p[0] + p[1] * p[1] + p[2] * p[2]
for (i in 0..3) {
atb[i] += row[i] * rhs
for (j in 0..3) ata[i][j] += row[i] * row[j]
}
}
val x = solve4(ata, atb) ?: return false
val radius2 = x[3] + x[0] * x[0] + x[1] * x[1] + x[2] * x[2]
if (radius2 <= 0) return false
val b = Vec3(x[0].toFloat(), x[1].toFloat(), x[2].toFloat())
val s = (GRAVITY / sqrt(radius2)).toFloat()
if (!plausible(b, s)) return false
bias = b
scale = s
ready = true
return true
}
private fun bin(v: Float) = (v * 1.5f).roundToInt().coerceIn(-1, 1) + 1
companion object {
private const val GRAVITY = 9.80665
private const val MIN_SAMPLES = 20.0
private const val MIN_BINS = 5
private const val MIN_SPREAD = 2.0
private const val SOLVE_EVERY = 25.0
private const val MAX_BIAS = 1.5f
private const val MAX_SCALE_ERROR = 0.1f
private fun plausible(b: Vec3, s: Float) = b.length() <= MAX_BIAS && abs(s - 1f) <= MAX_SCALE_ERROR
private fun smallestSpread(points: List<DoubleArray>): Double {
val mean = DoubleArray(3) { i -> points.sumOf { it[i] } / points.size }
val c = Array(3) { DoubleArray(3) }
for (p in points) for (i in 0..2) for (j in 0..2) c[i][j] += (p[i] - mean[i]) * (p[j] - mean[j])
return sqrt(maxOf(smallestEigenvalue(c), 0.0))
}
private fun smallestEigenvalue(m: Array<DoubleArray>): Double {
val a = Array(3) { m[it].copyOf() }
repeat(50) {
var p = 0
var q = 1
for ((i, j) in listOf(0 to 2, 1 to 2)) if (abs(a[i][j]) > abs(a[p][q])) { p = i; q = j }
if (abs(a[p][q]) < 1e-12) return minOf(a[0][0], a[1][1], a[2][2])
val theta = (a[q][q] - a[p][p]) / (2 * a[p][q])
val t = (if (theta >= 0) 1.0 else -1.0) / (abs(theta) + sqrt(theta * theta + 1))
val c = 1 / sqrt(t * t + 1)
val s = t * c
for (k in 0..2) {
val kp = a[k][p]
val kq = a[k][q]
a[k][p] = c * kp - s * kq
a[k][q] = s * kp + c * kq
}
for (k in 0..2) {
val pk = a[p][k]
val qk = a[q][k]
a[p][k] = c * pk - s * qk
a[q][k] = s * pk + c * qk
}
}
return minOf(a[0][0], a[1][1], a[2][2])
}
private fun solve4(a: Array<DoubleArray>, b: DoubleArray): DoubleArray? {
val m = Array(4) { i -> DoubleArray(5) { j -> if (j < 4) a[i][j] else b[i] } }
for (col in 0..3) {
val pivot = (col..3).maxBy { abs(m[it][col]) }
if (abs(m[pivot][col]) < 1e-9) return null
val swap = m[col]
m[col] = m[pivot]
m[pivot] = swap
for (r in 0..3) {
if (r == col) continue
val f = m[r][col] / m[col][col]
for (k in col..4) m[r][k] -= f * m[col][k]
}
}
return DoubleArray(4) { m[it][4] / m[it][it] }
}
}
}
internal class JoyConMotion {
var velocity = Vec3(0f, 0f, 0f)
private set
var displacement = Vec3(0f, 0f, 0f)
private set
var inertial = Vec3(0f, 0f, 0f)
private set
var resting = false
private set
private var gravity = Float.NaN
private var average = Vec3(0f, 0f, 0f)
private var calmFor = 0f
private var stillFor = 0f
private var movingFor = 0f
fun update(rotation: Quaternion, accel: Vec3, rate: Vec3, calibrated: Boolean, dt: Float) {
if (dt <= 0f) return
if (gravity.isNaN()) {
gravity = accel.length()
average = accel
}
if (calibrated) gravity = GRAVITY
average += (accel - average) * (dt / AVERAGE_SECONDS).coerceAtMost(1f)
val world = rotation.rotate(accel)
val linear = Vec3(world.x, world.y - gravity, world.z)
inertial = inertial * exp(-dt / INERTIAL_SECONDS) + linear * dt
val calm = rate.length() < CALM_RATE && (accel - average).length() < CALM_ACCEL
calmFor = if (calm) calmFor + dt else 0f
val still = rate.length() < CALM_RATE &&
(linear.length() < STILL_ACCEL || calmFor >= if (calibrated) CALIBRATED_CALM_SECONDS else UNCALIBRATED_CALM_SECONDS)
stillFor = if (still) stillFor + dt else 0f
resting = calmFor >= RESTING_SECONDS && rate.length() < RESTING_RATE
if (stillFor >= STILL_SECONDS) {
movingFor = 0f
velocity *= exp(-dt / SETTLE_SECONDS)
if (!calibrated) gravity += (accel.length() - gravity) * (dt / GRAVITY_SECONDS).coerceAtMost(1f)
} else {
movingFor += dt
val leak = if (movingFor < FREE_SECONDS) LEAK_SECONDS else LONG_MOVE_LEAK_SECONDS
velocity = velocity * exp(-dt / leak) + linear * dt
}
displacement += velocity * dt
}
companion object {
private const val GRAVITY = 9.80665f
private const val AVERAGE_SECONDS = 0.1f
private const val CALM_RATE = 0.2f
private const val CALM_ACCEL = 0.25f
private const val STILL_ACCEL = 0.12f
private const val STILL_SECONDS = 0.2f
private const val UNCALIBRATED_CALM_SECONDS = 0.5f
private const val CALIBRATED_CALM_SECONDS = 2f
private const val SETTLE_SECONDS = 0.05f
private const val LEAK_SECONDS = 20f
private const val FREE_SECONDS = 1f
private const val LONG_MOVE_LEAK_SECONDS = 0.2f
private const val GRAVITY_SECONDS = 0.5f
private const val INERTIAL_SECONDS = 30f
private const val RESTING_SECONDS = 0.3f
private const val RESTING_RATE = 0.05f
}
}
internal object JoyConCalibrations {
fun load(context: Context, descriptor: String?): FloatArray? {
if (descriptor.isNullOrEmpty()) return null
val text = ControllerPrefs.prefs(context).getString(KEY + descriptor, null) ?: return null
return text.split(',').mapNotNull { it.toFloatOrNull() }.toFloatArray().takeIf { it.size == 4 }
}
fun save(context: Context, descriptor: String?, values: FloatArray?) {
if (descriptor.isNullOrEmpty() || values == null) return
ControllerPrefs.prefs(context).edit().putString(KEY + descriptor, values.joinToString(",")).apply()
}
private const val KEY = "joyConAccel."
}
@@ -0,0 +1,113 @@
package com.droiddeck.launcher.input
import com.droiddeck.launcher.glasses.Quaternion
import com.droiddeck.launcher.glasses.Vec3
import kotlin.math.abs
internal class JoyConOrientation {
var rotation = Quaternion.IDENTITY
private set
var rate = Vec3(0f, 0f, 0f)
private set
var ready = false
private set
private var bias = Vec3(0f, 0f, 0f)
private var stillFor = 0f
fun update(accelKernel: FloatArray, gyroKernel: FloatArray, dt: Float) = update(axes(accelKernel), axes(gyroKernel), dt)
fun update(accel: Vec3, gyro: Vec3, dt: Float) {
val norm = accel.length()
val gravityValid = abs(norm - GRAVITY) < GRAVITY_TOLERANCE
if (!ready) {
if (!gravityValid) return
val up = accel.normalized()
val axis = up.cross(WORLD_UP)
rotation = if (up.y < -0.9999f) Quaternion.fromAxisAngle(Vec3(1f, 0f, 0f), Math.PI.toFloat())
else Quaternion(1f + up.y, axis.x, axis.y, axis.z).normalized()
rate = Vec3(0f, 0f, 0f)
ready = true
return
}
if (dt <= 0f) return
val still = gravityValid && abs(norm - GRAVITY) < STILL_ACCEL && (gyro - bias).length() < STILL_RATE
stillFor = if (still) stillFor + dt else 0f
if (stillFor >= STILL_SECONDS) {
val alpha = (dt / BIAS_SECONDS).coerceAtMost(1f)
bias = bias * (1f - alpha) + gyro * alpha
}
rate = gyro - bias
var corrected = rate
if (gravityValid && rate.length() < MOVING_RATE) {
val predictedUp = rotation.conjugate().rotate(WORLD_UP)
corrected += accel.normalized().cross(predictedUp) * (GAIN * (1f - abs(norm - GRAVITY) / GRAVITY_TOLERANCE))
}
rotation = rotation.integrateBodyRate(corrected, dt).normalized()
}
companion object {
private val WORLD_UP = Vec3(0f, 1f, 0f)
private const val GRAVITY = 9.80665f
private const val GRAVITY_TOLERANCE = 1.0f
private const val STILL_ACCEL = 0.5f
private const val STILL_RATE = 0.05f
private const val STILL_SECONDS = 0.3f
private const val BIAS_SECONDS = 2f
private const val GAIN = 1.0f
private const val MOVING_RATE = 1.0f
fun axes(v: FloatArray) = Vec3(-v[1], v[2], -v[0])
}
}
internal class JoyConReports {
class Sample(val accel: FloatArray, val gyro: FloatArray, val dt: Float)
private val pending = ArrayList<FloatArray>(4)
private var pendingAt = 0L
private var previousFirst: FloatArray? = null
private var previousLast: FloatArray? = null
private var previousSample: FloatArray? = null
private var previousAt = 0L
private var order = NEWEST_FIRST_START
val size: Int get() = pending.size
fun add(accel: FloatArray, gyro: FloatArray, arrivalNs: Long) {
if (pending.isEmpty()) pendingAt = arrivalNs
pending += floatArrayOf(accel[0], accel[1], accel[2], gyro[0], gyro[1], gyro[2])
}
fun flush(): List<Sample> {
if (pending.isEmpty()) return emptyList()
val first = pending.first()
val last = pending.last()
if (pending.size > 1) {
if (previousFirst?.contentEquals(last) == true) order = (order + 1).coerceAtMost(ORDER_LIMIT)
else if (previousLast?.contentEquals(first) == true) order = (order - 1).coerceAtLeast(-ORDER_LIMIT)
}
val chronological = if (order >= 0) pending.asReversed() else pending
val fresh = ArrayList<FloatArray>(chronological.size)
var before = previousSample
for (sample in chronological) {
if (before?.contentEquals(sample) != true) fresh += sample
before = sample
}
val used = fresh.ifEmpty { listOf(chronological.last()) }
val dt = if (previousAt == 0L) 0f else ((pendingAt - previousAt).coerceAtLeast(0L) * 1e-9f / used.size).coerceAtMost(MAX_STEP_S)
val out = used.map { Sample(it.copyOfRange(0, 3), it.copyOfRange(3, 6), dt) }
previousFirst = first
previousLast = last
previousSample = chronological.last()
previousAt = pendingAt
pending.clear()
return out
}
companion object {
const val SAMPLES_PER_REPORT = 3
private const val NEWEST_FIRST_START = 2
private const val ORDER_LIMIT = 20
private const val MAX_STEP_S = 0.05f
}
}
@@ -0,0 +1,116 @@
package com.droiddeck.launcher.input
import android.util.Log
import com.droiddeck.launcher.glasses.Quaternion
import com.droiddeck.launcher.glasses.Vec3
import java.io.File
import java.io.RandomAccessFile
import java.nio.ByteOrder
import java.nio.MappedByteBuffer
import java.nio.channels.FileChannel
internal class JoyConShm private constructor(
private val file: RandomAccessFile,
private val map: MappedByteBuffer,
private val fence: () -> Unit,
) {
class Hand(
val connected: Boolean,
val charging: Boolean = false,
val buttons: Int = 0,
val rotation: Quaternion = Quaternion.IDENTITY,
val rate: Vec3 = Vec3(0f, 0f, 0f),
val displacement: Vec3 = Vec3(0f, 0f, 0f),
val velocity: Vec3 = Vec3(0f, 0f, 0f),
val inertial: Vec3 = Vec3(0f, 0f, 0f),
val stickX: Float = 0f,
val stickY: Float = 0f,
val battery: Float = -1f,
val recenter: Int = 0,
val timeNs: Long = 0L,
)
private var seq = map.getInt(SEQ).let { if (it and 1 != 0) it + 1 else it }
init {
begin()
map.putInt(MAGIC_AT, MAGIC)
map.putInt(VERSION_AT, VERSION)
for (at in HANDS until SIZE step 4) map.putInt(at, 0)
end()
}
fun write(side: Int, hand: Hand) {
val at = HANDS + side * HAND_SIZE
begin()
map.putInt(at, (if (hand.connected) CONNECTED else 0) or (if (hand.charging) CHARGING else 0))
map.putInt(at + 4, hand.buttons)
map.putFloat(at + 8, hand.rotation.w)
map.putFloat(at + 12, hand.rotation.x)
map.putFloat(at + 16, hand.rotation.y)
map.putFloat(at + 20, hand.rotation.z)
map.putFloat(at + 24, hand.rate.x)
map.putFloat(at + 28, hand.rate.y)
map.putFloat(at + 32, hand.rate.z)
map.putFloat(at + 36, hand.stickX)
map.putFloat(at + 40, hand.stickY)
map.putFloat(at + 44, hand.battery)
map.putInt(at + 48, hand.recenter)
map.putLong(at + 56, hand.timeNs)
map.putFloat(at + 64, hand.displacement.x)
map.putFloat(at + 68, hand.displacement.y)
map.putFloat(at + 72, hand.displacement.z)
map.putFloat(at + 76, hand.velocity.x)
map.putFloat(at + 80, hand.velocity.y)
map.putFloat(at + 84, hand.velocity.z)
map.putFloat(at + 88, hand.inertial.x)
map.putFloat(at + 92, hand.inertial.y)
map.putFloat(at + 96, hand.inertial.z)
map.putInt(at + 100, 0)
end()
}
fun close() {
write(JoyCon.LEFT, Hand(connected = false))
write(JoyCon.RIGHT, Hand(connected = false))
runCatching { file.close() }
}
private fun begin() {
map.putInt(SEQ, ++seq)
fence()
}
private fun end() {
fence()
map.putInt(SEQ, ++seq)
}
companion object {
const val FILE = "droiddeck-joycon"
const val SIZE = 224
private const val MAGIC = 0x4E434A44
private const val VERSION = 3
private const val MAGIC_AT = 0
private const val VERSION_AT = 4
private const val SEQ = 8
private const val HANDS = 16
private const val HAND_SIZE = 104
private const val CONNECTED = 1
private const val CHARGING = 2
fun open(path: File, fence: () -> Unit = FakeInputWriter::storeFence): JoyConShm? = runCatching {
path.parentFile?.mkdirs()
val file = RandomAccessFile(path, "rw")
try {
file.setLength(SIZE.toLong())
val map = file.channel.map(FileChannel.MapMode.READ_WRITE, 0, SIZE.toLong())
map.order(ByteOrder.LITTLE_ENDIAN)
JoyConShm(file, map, fence)
} catch (e: Exception) {
file.close()
throw e
}
}.onFailure { Log.w("JoyConVr", "Joy-Con pose file unavailable: $path", it) }.getOrNull()
}
}
@@ -0,0 +1,363 @@
package com.droiddeck.launcher.input
import android.content.Context
import android.hardware.BatteryState
import android.hardware.Sensor
import android.hardware.SensorEvent
import android.hardware.SensorEventListener
import android.hardware.SensorManager
import android.os.CombinedVibration
import android.os.Handler
import android.os.HandlerThread
import android.os.Looper
import android.os.Process
import android.os.VibrationEffect
import android.util.Log
import android.view.InputDevice
import android.view.KeyEvent
import android.view.MotionEvent
import androidx.annotation.RequiresApi
import java.io.File
import kotlin.math.min
import kotlin.math.sqrt
@RequiresApi(31)
class JoyConVr(
private val context: Context,
private val path: File,
private val onDevicesChanged: (taken: List<Int>, released: List<Int>) -> Unit,
) {
private val thread = HandlerThread("joycon-vr", Process.THREAD_PRIORITY_DISPLAY).apply { start() }
private val worker = Handler(thread.looper)
private val main = Handler(Looper.getMainLooper())
@Volatile private var owned = IntArray(0)
private var wanted = false
private var closed = false
private var retries = 0
private var shm: JoyConShm? = null
private val hands = arrayOfNulls<Hand>(2)
private val retry = Runnable { apply() }
fun setActive(on: Boolean) {
worker.post {
if (closed || wanted == on) return@post
wanted = on
retries = 0
apply()
}
}
fun refresh() {
worker.post {
if (closed) return@post
retries = 0
apply()
}
}
fun owns(deviceId: Int): Boolean = owned.contains(deviceId)
val holding: Boolean get() = owned.isNotEmpty()
fun onKeyEvent(event: KeyEvent): Boolean {
if (!owns(event.deviceId)) return false
if (event.repeatCount > 0) return true
val down = when (event.action) {
KeyEvent.ACTION_DOWN -> true
KeyEvent.ACTION_UP -> false
else -> return true
}
val id = event.deviceId
val scanCode = event.scanCode
worker.post { hand(id)?.key(scanCode, down) }
return true
}
fun onMotionEvent(event: MotionEvent): Boolean {
if (!owns(event.deviceId)) return false
if (!event.isFromSource(InputDevice.SOURCE_JOYSTICK) || event.actionMasked != MotionEvent.ACTION_MOVE) return true
val device = event.device ?: return true
val side = JoyCon.side(device)
if (side < 0) return true
val axes = JoyCon.stickAxes(side, device, event.source)
val x = event.getAxisValue(axes[0])
val y = event.getAxisValue(axes[1])
val id = event.deviceId
worker.post { hand(id)?.stick(x, -y) }
return true
}
fun releaseButtons() {
worker.post { hands.forEach { it?.release() } }
}
fun close() {
worker.post {
closed = true
wanted = false
apply()
shm?.close()
shm = null
}
thread.quitSafely()
}
private fun hand(id: Int): Hand? = hands.firstOrNull { it?.device?.id == id }
private fun apply() {
worker.removeCallbacks(retry)
val found = arrayOfNulls<InputDevice>(2)
var waiting = false
if (wanted && !closed) {
for (id in InputDevice.getDeviceIds()) {
val device = InputDevice.getDevice(id) ?: continue
val side = JoyCon.side(device)
if (side < 0 || found[side] != null) continue
if (hasMotion(device)) found[side] = device else waiting = true
}
}
val before = owned
for (side in 0..1) {
val current = hands[side]
val device = found[side]
if (current != null && current.device.id == device?.id) continue
current?.stop()
hands[side] = null
if (device != null) {
hands[side] = start(side, device)
if (hands[side] == null) waiting = true
}
}
val now = hands.mapNotNull { it?.device?.id }.toIntArray()
owned = now
if (closed) JoyCon.clearVrDevices(this)
else JoyCon.setVrDevices(this, IntArray(2) { hands[it]?.device?.id ?: PadBridge.NO_CONTROLLER })
if (!closed) {
val taken = now.filter { it !in before }
val released = before.filter { it !in now }
if (taken.isNotEmpty() || released.isNotEmpty()) main.post { onDevicesChanged(taken, released) }
}
if (wanted && waiting && retries < MAX_RETRIES) {
retries++
worker.postDelayed(retry, RETRY_MS)
}
}
private fun hasMotion(device: InputDevice): Boolean =
runCatching {
val sensors = device.sensorManager
sensors.getDefaultSensor(Sensor.TYPE_GYROSCOPE) != null && sensors.getDefaultSensor(Sensor.TYPE_ACCELEROMETER) != null
}.getOrDefault(false)
private fun start(side: Int, device: InputDevice): Hand? {
val out = shm ?: JoyConShm.open(path)?.also { shm = it } ?: return null
val sensors = runCatching { device.sensorManager }.getOrNull() ?: return null
val gyro = runCatching { sensors.getDefaultSensor(Sensor.TYPE_GYROSCOPE) }.getOrNull() ?: return null
val accel = runCatching { sensors.getDefaultSensor(Sensor.TYPE_ACCELEROMETER) }.getOrNull() ?: return null
val hand = Hand(side, device, sensors, out)
val listening = runCatching {
sensors.registerListener(hand, accel, SensorManager.SENSOR_DELAY_FASTEST, worker) &&
sensors.registerListener(hand, gyro, SensorManager.SENSOR_DELAY_FASTEST, worker)
}.onFailure { Log.w(TAG, "Joy-Con motion unavailable for \"${device.name}\"", it) }.getOrDefault(false)
if (!listening) {
runCatching { sensors.unregisterListener(hand) }
return null
}
Log.i(TAG, "${SIDES[side]} Joy-Con \"${device.name}\" (id ${device.id}) is a VR controller")
return hand
}
private inner class Hand(
val side: Int,
val device: InputDevice,
private val sensors: SensorManager,
private val out: JoyConShm,
) : SensorEventListener {
private val filter = JoyConOrientation()
private val motion = JoyConMotion()
private val calibration = JoyConAccelCalibration().apply { restore(JoyConCalibrations.load(context, device.descriptor)) }
private val reports = JoyConReports()
private val accel = FloatArray(3)
private var accelStamp = 0L
private var haveAccel = false
private val pressed = HashSet<Int>()
private var stickX = 0f
private var stickY = 0f
private var systemDown = false
private var systemTap = false
private var held = false
private var recenter = 0
private var battery = -1f
private var charging = false
private var batteryAt = 0L
private var sampleAt = 0L
private var live = true
private val hold = Runnable {
if (!systemDown || held) return@Runnable
held = true
recenter++
confirm()
publish()
}
private val tapEnd = Runnable {
systemTap = false
publish()
}
private val flush = Runnable { flushReport() }
override fun onSensorChanged(event: SensorEvent) {
if (!live) return
when (event.sensor.type) {
Sensor.TYPE_ACCELEROMETER -> {
if (event.timestamp == accelStamp) return
accelStamp = event.timestamp
event.values.copyInto(accel, 0, 0, 3)
haveAccel = true
}
Sensor.TYPE_GYROSCOPE -> {
if (!haveAccel) return
reports.add(accel, event.values, System.nanoTime())
worker.removeCallbacks(flush)
if (reports.size >= JoyConReports.SAMPLES_PER_REPORT) flushReport() else worker.postDelayed(flush, FLUSH_MS)
}
}
}
private fun flushReport() {
val samples = reports.flush()
if (samples.isEmpty() || !live) return
for (sample in samples) {
val raw = JoyConOrientation.axes(sample.accel)
val corrected = calibration.correct(raw)
filter.update(corrected, JoyConOrientation.axes(sample.gyro), sample.dt)
if (!filter.ready) continue
motion.update(filter.rotation, corrected, filter.rate, calibration.ready, sample.dt)
if (motion.resting && calibration.addResting(raw)) JoyConCalibrations.save(context, device.descriptor, calibration.saved())
}
val now = System.nanoTime()
sampleAt = now
if (now - batteryAt > BATTERY_NS) refreshBattery(now)
publish()
}
override fun onAccuracyChanged(sensor: Sensor?, accuracy: Int) = Unit
fun key(scanCode: Int, down: Boolean) {
if (!live) return
val bit = JoyCon.vrButton(side, scanCode)
if (bit == 0) return
if (bit == JoyCon.VR_SYSTEM) {
system(down)
return
}
if (down) pressed += scanCode else pressed -= scanCode
publish()
}
fun stick(x: Float, y: Float) {
if (!live) return
val length = sqrt(x * x + y * y)
if (length < STICK_DEADZONE) {
stickX = 0f
stickY = 0f
} else {
val scale = min((length - STICK_DEADZONE) / (1f - STICK_DEADZONE), 1f) / length
stickX = x * scale
stickY = y * scale
}
publish()
}
fun release() {
if (!live) return
pressed.clear()
stickX = 0f
stickY = 0f
systemDown = false
systemTap = false
held = false
worker.removeCallbacks(hold)
worker.removeCallbacks(tapEnd)
publish()
}
fun stop() {
live = false
runCatching { sensors.unregisterListener(this) }
worker.removeCallbacks(hold)
worker.removeCallbacks(tapEnd)
worker.removeCallbacks(flush)
out.write(side, JoyConShm.Hand(connected = false))
Log.i(TAG, "${SIDES[side]} Joy-Con \"${device.name}\" (id ${device.id}) is a game pad again")
}
private fun system(down: Boolean) {
if (down == systemDown) return
systemDown = down
if (down) {
held = false
worker.postDelayed(hold, HOLD_MS)
return
}
worker.removeCallbacks(hold)
if (held) return
systemTap = true
publish()
worker.removeCallbacks(tapEnd)
worker.postDelayed(tapEnd, TAP_MS)
}
private fun buttons(): Int {
var bits = if (systemTap) JoyCon.VR_SYSTEM else 0
for (scanCode in pressed) bits = bits or JoyCon.vrButton(side, scanCode)
return bits
}
private fun publish() {
out.write(side, JoyConShm.Hand(
connected = filter.ready,
charging = charging,
buttons = buttons(),
rotation = filter.rotation,
rate = filter.rate,
displacement = motion.displacement,
velocity = motion.velocity,
inertial = motion.inertial,
stickX = stickX,
stickY = stickY,
battery = battery,
recenter = recenter,
timeNs = sampleAt,
))
}
private fun refreshBattery(now: Long) {
batteryAt = now
val state = runCatching { device.batteryState }.getOrNull() ?: return
battery = if (state.isPresent && !state.capacity.isNaN()) state.capacity else -1f
charging = state.isPresent && state.status == BatteryState.STATUS_CHARGING
}
private fun confirm() {
if (!ControllerPrefs.rumbleEnabled(context)) return
runCatching {
device.vibratorManager.vibrate(CombinedVibration.createParallel(
VibrationEffect.createOneShot(CONFIRM_MS, VibrationEffect.DEFAULT_AMPLITUDE)))
}
}
}
companion object {
const val FILE = JoyConShm.FILE
private const val TAG = "JoyConVr"
private val SIDES = arrayOf("left", "right")
private const val MAX_RETRIES = 6
private const val RETRY_MS = 500L
private const val HOLD_MS = 800L
private const val TAP_MS = 120L
private const val CONFIRM_MS = 80L
private const val BATTERY_NS = 30_000_000_000L
private const val STICK_DEADZONE = 0.12f
private const val FLUSH_MS = 3L
}
}
@@ -92,6 +92,7 @@ class OnScreenControls(
private val groups = controls.map { it.group }.distinct()
private var buttonsOnly = false
val isButtonsOnly: Boolean get() = buttonsOnly
private var quickHidden = false
private var quickPressedBy = -1
private var keyboardPressedBy = -1
@@ -46,6 +46,8 @@ public final class PadBridge {
private final FakeInputWriter writer;
private final PadState state = new PadState();
private final int[] owners = new int[SLOTS];
private final int[] partners = new int[SLOTS];
private volatile java.util.function.IntPredicate excluded = id -> false;
private final FakeInputWriter[] playerWriters = new FakeInputWriter[SLOTS];
private final PadState[] playerStates = new PadState[SLOTS];
private final PadState effectiveState = new PadState();
@@ -72,8 +74,10 @@ public final class PadBridge {
public static final int NO_CONTROLLER = -1;
private static volatile int activeControllerId = NO_CONTROLLER;
private static final AtomicIntegerArray slotControllers = new AtomicIntegerArray(SLOTS);
private static final AtomicIntegerArray slotJoyCons = new AtomicIntegerArray(SLOTS * 2);
static {
for (int slot = 0; slot < SLOTS; slot++) slotControllers.set(slot, NO_CONTROLLER);
for (int i = 0; i < SLOTS * 2; i++) slotJoyCons.set(i, NO_CONTROLLER);
}
public static int activeControllerId() { return activeControllerId; }
@@ -82,10 +86,16 @@ public final class PadBridge {
return slot >= 0 && slot < SLOTS ? slotControllers.get(slot) : activeControllerId;
}
public static int joyConForSlot(int slot, int side) {
return slot >= 0 && slot < SLOTS && (side == JoyCon.LEFT || side == JoyCon.RIGHT)
? slotJoyCons.get(slot * 2 + side) : NO_CONTROLLER;
}
public PadBridge(File fakeInputDir) {
this.fakeInputDir = fakeInputDir;
writer = new FakeInputWriter(fakeInputDir.getAbsolutePath(), 0);
Arrays.fill(owners, NO_CONTROLLER);
Arrays.fill(partners, NO_CONTROLLER);
for (int slot = 1; slot < SLOTS; slot++) playerStates[slot] = new PadState();
}
@@ -93,6 +103,10 @@ public final class PadBridge {
selectSteam = on;
}
public void setExcluded(java.util.function.IntPredicate devices) {
excluded = devices;
}
public synchronized boolean start() {
if (!open) {
open = writer.open();
@@ -106,7 +120,9 @@ public final class PadBridge {
activeControllerId = NO_CONTROLLER;
for (int slot = 0; slot < SLOTS; slot++) {
owners[slot] = NO_CONTROLLER;
partners[slot] = NO_CONTROLLER;
slotControllers.set(slot, NO_CONTROLLER);
clearJoyCons(slot);
if (playerWriters[slot] != null) {
playerWriters[slot].destroy();
playerWriters[slot] = null;
@@ -206,7 +222,7 @@ public final class PadBridge {
public synchronized void syncConnectedControllers() {
for (int deviceId : InputDevice.getDeviceIds()) {
InputDevice device = InputDevice.getDevice(deviceId);
if (isFromController(device)) publish(slotFor(device));
if (isFromController(device) && !excluded.test(deviceId)) publish(slotFor(device));
}
}
@@ -224,8 +240,10 @@ public final class PadBridge {
}
public synchronized boolean onKeyEvent(KeyEvent event) {
if (!isControllerKeyEvent(event)) return false;
if (!isControllerKeyEvent(event) || excluded.test(event.getDeviceId())) return false;
int slot = slotFor(event.getDevice());
int side = JoyCon.side(event.getDevice());
if (side >= 0 && partners[slot] != NO_CONTROLLER && JoyCon.isRail(side, event.getScanCode())) return true;
PadState pad = stateFor(slot);
boolean pressed = event.getAction() == KeyEvent.ACTION_DOWN;
switch (event.getKeyCode()) {
@@ -269,16 +287,32 @@ public final class PadBridge {
}
public synchronized boolean onMotionEvent(MotionEvent event) {
if (!isControllerMotionEvent(event)) return false;
if (!isControllerMotionEvent(event) || excluded.test(event.getDeviceId())) return false;
if (event.getAction() != MotionEvent.ACTION_MOVE) return false;
int slot = slotFor(event.getDevice());
PadState pad = stateFor(slot);
noteDevice(event.getDevice(), slot);
statAxes++;
scheduleStats();
InputDevice device = event.getDevice();
int side = JoyCon.side(device);
if (side >= 0) {
int[] stick = JoyCon.stickAxes(side, device, event.getSource());
float x = axis(event, stick[0]);
float y = axis(event, stick[1]);
if (side == JoyCon.LEFT) {
pad.leftX = x;
pad.leftY = y;
} else {
pad.rightX = x;
pad.rightY = y;
}
if (Math.max(Math.abs(x), Math.abs(y)) > 0.5f) notePlayerInput();
publish(slot);
return true;
}
pad.leftX = axis(event, MotionEvent.AXIS_X);
pad.leftY = axis(event, MotionEvent.AXIS_Y);
InputDevice device = event.getDevice();
boolean wiredXboxLayout = device != null
&& ((device.getVendorId() == 0x20d6 && device.getProductId() == 0x2063)
|| (device.getVendorId() == 0x0e6f && device.getProductId() == 0x02f0))
@@ -358,7 +392,7 @@ public final class PadBridge {
public synchronized void onDeviceRemoved(int deviceId) {
for (int slot = 0; slot < SLOTS; slot++) {
if (owners[slot] == deviceId) free(slot);
if (owners[slot] == deviceId || partners[slot] == deviceId) leave(slot, deviceId);
}
}
@@ -425,12 +459,26 @@ public final class PadBridge {
private int slotFor(InputDevice device) {
int id = device.getId();
if (owners[0] == id) return 0;
if (owners[0] == id || partners[0] == id) return 0;
for (int slot = 1; slot < SLOTS; slot++) {
if (owners[slot] == id) return slot;
if (owners[slot] == id || partners[slot] == id) return slot;
}
for (int slot = 0; slot < SLOTS; slot++) {
if (owners[slot] != NO_CONTROLLER && InputDevice.getDevice(owners[slot]) == null) free(slot);
if (partners[slot] != NO_CONTROLLER && InputDevice.getDevice(partners[slot]) == null) leave(slot, partners[slot]);
if (owners[slot] != NO_CONTROLLER && InputDevice.getDevice(owners[slot]) == null) leave(slot, owners[slot]);
}
int side = JoyCon.side(device);
for (int slot = 0; side >= 0 && slot < SLOTS; slot++) {
if (owners[slot] == NO_CONTROLLER || partners[slot] != NO_CONTROLLER) continue;
InputDevice owner = InputDevice.getDevice(owners[slot]);
if (JoyCon.side(owner) != 1 - side) continue;
partners[slot] = id;
slotJoyCons.set(slot * 2 + side, id);
slotJoyCons.set(slot * 2 + 1 - side, owner.getId());
if (slot == 0) releaseBackPaddles();
stateFor(slot).clear();
Log.i(TAG, "\"" + device.getName() + "\" (id " + id + ") joins \"" + owner.getName() + "\" as player " + (slot + 1));
return slot;
}
for (int slot = 0; slot < SLOTS; slot++) {
if (owners[slot] != NO_CONTROLLER) continue;
@@ -446,7 +494,8 @@ public final class PadBridge {
}
private boolean isPrimary(KeyEvent event) {
return owners[0] == event.getDevice().getId();
int id = event.getDevice().getId();
return owners[0] == id || partners[0] == id;
}
private void releaseBackPaddles() {
@@ -454,9 +503,32 @@ public final class PadBridge {
DeckControls.INSTANCE.setGrip(DeckControls.R4, false);
}
private void leave(int slot, int id) {
if (partners[slot] == NO_CONTROLLER) {
free(slot);
return;
}
Log.i(TAG, "player " + (slot + 1) + " Joy-Con (id " + id + ") left");
if (owners[slot] == id) owners[slot] = partners[slot];
partners[slot] = NO_CONTROLLER;
clearJoyCons(slot);
if (slotControllers.get(slot) == id) slotControllers.set(slot, owners[slot]);
if (activeControllerId == id) activeControllerId = owners[slot];
if (slot == 0) releaseBackPaddles();
stateFor(slot).clear();
publish(slot);
}
private static void clearJoyCons(int slot) {
slotJoyCons.set(slot * 2, NO_CONTROLLER);
slotJoyCons.set(slot * 2 + 1, NO_CONTROLLER);
}
private void free(int slot) {
Log.i(TAG, "player " + (slot + 1) + " (id " + owners[slot] + ") left");
owners[slot] = NO_CONTROLLER;
partners[slot] = NO_CONTROLLER;
clearJoyCons(slot);
slotControllers.set(slot, NO_CONTROLLER);
if (slot == 0) {
releaseBackPaddles();
@@ -7,26 +7,21 @@ import android.content.Context
import android.os.Build
import android.provider.Settings
import android.util.Log
import android.view.Display
object PerfMode {
private const val TAG = "PerfMode"
fun apply(a: Activity, fpsLimit: Int = 0): String {
fun apply(a: Activity, fpsLimit: Int = 0, refreshLimit: Int = 0): String {
val parts = ArrayList<String>(2)
parts += try {
val display = if (Build.VERSION.SDK_INT >= 30) a.display else @Suppress("DEPRECATION") a.windowManager.defaultDisplay
if (display == null || Build.VERSION.SDK_INT < 23) "display mode unchanged" else {
val cur = display.mode
val cap = peakRefreshCap(a)
val sameSize = display.supportedModes
.filter { it.physicalWidth == cur.physicalWidth && it.physicalHeight == cur.physicalHeight }
.filter { it.refreshRate <= cap + 0.5f }
.ifEmpty { display.supportedModes.filter { it.modeId == cur.modeId } }
val best = sameSize.filter { cadenceFits(it.refreshRate, fpsLimit) }.maxByOrNull { it.refreshRate }
?: sameSize.maxByOrNull { it.refreshRate }
if (best != null && best.modeId != cur.modeId &&
(best.refreshRate > cur.refreshRate + 0.5f || !cadenceFits(cur.refreshRate, fpsLimit))) {
val cap = minOf(peakRefreshCap(a), refreshLimit.takeIf { it > 0 }?.toFloat() ?: Float.POSITIVE_INFINITY)
val best = selectMode(display.supportedModes, cur, fpsLimit, cap)
if (best != null && a.window.attributes.preferredDisplayModeId != best.modeId) {
a.window.attributes = a.window.attributes.apply { preferredDisplayModeId = best.modeId }
"display ${cur.refreshRate.toInt()} → ${best.refreshRate.toInt()} Hz (mode ${best.modeId})"
} else "display ${cur.refreshRate.toInt()} Hz"
@@ -59,6 +54,16 @@ object PerfMode {
return if (peak >= 1f) peak else Float.POSITIVE_INFINITY
}
internal fun selectMode(modes: Array<Display.Mode>, current: Display.Mode, fpsLimit: Int, cap: Float): Display.Mode? {
val sameSize = modes.filter {
it.physicalWidth == current.physicalWidth && it.physicalHeight == current.physicalHeight
}
val allowed = sameSize.filter { it.refreshRate <= cap + 0.5f }
return allowed.filter { cadenceFits(it.refreshRate, fpsLimit) }.maxByOrNull { it.refreshRate }
?: allowed.maxByOrNull { it.refreshRate }
?: sameSize.minByOrNull { it.refreshRate }
}
fun cadenceFits(hz: Float, fpsLimit: Int): Boolean {
if (fpsLimit <= 0) return true
val ratio = hz / fpsLimit
@@ -16,6 +16,7 @@ import android.view.InputDevice
import androidx.annotation.RequiresApi
import com.droiddeck.launcher.core.SessionPart
import com.droiddeck.launcher.input.ControllerPrefs
import com.droiddeck.launcher.input.JoyCon
import com.droiddeck.launcher.input.PadBridge
import java.io.DataInputStream
import kotlin.math.max
@@ -26,6 +27,7 @@ class RumbleComponent : SessionPart() {
private val playing = HashMap<Int, Motors>()
private val effects = HashMap<Int, Effect>()
private val controllers = HashMap<Int, Motors?>()
private val joyConPairs = HashMap<Long, Motors>()
private var preferences: SharedPreferences? = null
private var enabled = false
private var motorThread: HandlerThread? = null
@@ -78,6 +80,7 @@ class RumbleComponent : SessionPart() {
motorThread = null
phone = null
controllers.clear()
joyConPairs.clear()
val s = server
server = null
try { s?.close() } catch (e: Exception) { }
@@ -125,10 +128,24 @@ class RumbleComponent : SessionPart() {
}
private fun target(slot: Int): Motors? {
JoyCon.vrRumbleDevice(slot)?.let { hand -> return if (hand == PadBridge.NO_CONTROLLER) null else motors(hand) }
val left = PadBridge.joyConForSlot(slot, JoyCon.LEFT)
val right = PadBridge.joyConForSlot(slot, JoyCon.RIGHT)
if (left != PadBridge.NO_CONTROLLER && right != PadBridge.NO_CONTROLLER) {
val key = (left.toLong() shl 32) or (right.toLong() and 0xFFFFFFFFL)
joyConPairs[key]?.let { return it }
val l = motors(left)
val r = motors(right)
if (l != null && r != null) return JoyConPairMotors(l, r).also { joyConPairs[key] = it }
}
val id = PadBridge.controllerForSlot(slot)
if (id == PadBridge.NO_CONTROLLER) return phone
return motors(id) ?: phone
}
private fun motors(id: Int): Motors? {
if (!controllers.containsKey(id)) controllers[id] = try { controllerMotors(id) } catch (e: Exception) { null }
return controllers[id] ?: phone
return controllers[id]
}
private fun cancelAll() {
@@ -201,6 +218,25 @@ class RumbleComponent : SessionPart() {
}
}
@RequiresApi(31)
private class JoyConMotors(override val name: String, private val manager: VibratorManager) : Motors {
override fun play(strong: Int, weak: Int, ms: Long) =
manager.vibrate(CombinedVibration.createParallel(oneShot(manager.defaultVibrator, max(strong, weak), ms)))
override fun cancel() = manager.cancel()
}
private class JoyConPairMotors(private val left: Motors, private val right: Motors) : Motors {
override val name = "${left.name} and ${right.name}"
override fun play(strong: Int, weak: Int, ms: Long) {
if (strong == 0) left.cancel() else left.play(strong, 0, ms)
if (weak == 0) right.cancel() else right.play(0, weak, ms)
}
override fun cancel() {
left.cancel()
right.cancel()
}
}
private class LegacyPadMotors(override val name: String, private val vibrator: Vibrator) : Motors {
override fun play(strong: Int, weak: Int, ms: Long) = vibrator.vibrate(oneShot(vibrator, max(strong, weak), ms))
override fun cancel() {
@@ -230,7 +266,8 @@ class RumbleComponent : SessionPart() {
if (Build.VERSION.SDK_INT >= 31) {
val manager = device.vibratorManager
val ids = manager.vibratorIds.sortedArray()
return if (ids.isEmpty()) null else PadMotors(name, manager, ids)
if (ids.isEmpty()) return null
return if (JoyCon.side(device) >= 0) JoyConMotors(name, manager) else PadMotors(name, manager, ids)
}
@Suppress("DEPRECATION")
val vibrator = device.vibrator
@@ -31,6 +31,7 @@ object SessionEvents {
SessionState.programArgs = emptyList()
SessionState.steamUi = null
SessionState.steamUrl = null
SessionState.steamVr = false
SessionState.firstFrameSeen = false
SessionState.suspended = false
}
@@ -104,14 +104,19 @@ object SessionFiles {
val msitools = listOf("msiinfo", "cabextract", "7z", "7z.so", "libmsi-1.0.so.0", "libgsf-1.so.114", "libgcab-1.0.so.0")
.map { "usr/local/lib/droiddeck-msitools/$it" } +
listOf("NOTICE", "GPL-2", "GPL-3", "LGPL-2.1", "7zip-License", "7zip-unRarLicense").map { "usr/local/share/licenses/droiddeck-msitools/$it" }
val vr = listOf(
val handModels = listOf("left", "right").flatMap { side ->
listOf("droiddeck_hand_$side.json", "hand.mtl", "hand.png", "hand_open.obj", "hand_fist.obj", "thumb_index_open.obj",
"thumb_index_pinch.obj")
.map { "droiddeck/resources/rendermodels/droiddeck_hand_$side/$it" }
}
val vr = (listOf(
"droiddeck/bin/linuxarm64/driver_droiddeck.so",
"droiddeck/bin/linuxarm64/xr-display-mode",
"droiddeck/driver.vrdrivermanifest",
"droiddeck/resources/settings/default.vrsettings",
"layer/libVkLayer_droiddeck_vr.so",
"layer/VK_LAYER_DROIDDECK_vr.json",
).map { "usr/local/lib/droiddeck-vr/$it" to "usr/local/lib/droiddeck-vr/$it" } +
) + handModels).map { "usr/local/lib/droiddeck-vr/$it" to "usr/local/lib/droiddeck-vr/$it" } +
listOf("usr/local/lib/droiddeck-vr/layer/VK_LAYER_DROIDDECK_vr_implicit.json" to
"usr/share/vulkan/implicit_layer.d/droiddeck_vr.json")
val fexPreloads = listOf("x86_64", "i386").flatMap { arch ->
@@ -1299,6 +1299,7 @@ class SessionService : Service() {
val failureStatus = SessionState.failureStatus ?: status
SessionEvents.fail(code, message, failureStatus)
}
com.droiddeck.launcher.glasses.Glasses.set3dForVr(false)
SessionState.notifyEnded(status)
stopForeground(STOP_FOREGROUND_REMOVE)
stopSelf()
@@ -53,6 +53,9 @@ object SessionState {
var steamUi: String? = null
@Volatile
var steamUrl: String? = null
// A Play SteamVR request can arrive while an existing session is visible.
// Wake the status/launch effect even if no glasses surface has attached yet.
var steamVr by mutableStateOf(false)
var relaunch: android.content.Intent? = null
@JvmStatic var hdr = false
@@ -29,6 +29,7 @@ class ControllerActions(
val onQamButton: (Boolean) -> Unit,
val onKeyboardButton: (Boolean) -> Unit,
val onSelectSteam: (Boolean) -> Unit,
val onJoyConVr: (Boolean) -> Unit,
val onEditLayout: () -> Unit,
val onResetLayout: () -> Unit,
val onMapping: () -> Unit,
@@ -72,6 +73,7 @@ fun ColumnScope.ControllerRows(host: MenuHost, oscMode: String, c: ControllerPre
ToggleRow(host, "controller-qam", stringResource(R.string.ctrl_qam_button), null, c.qamButton, onChange = a.onQamButton)
ToggleRow(host, "controller-keyboard", stringResource(R.string.ctrl_keyboard_button), null, c.keyboardButton, onChange = a.onKeyboardButton)
ToggleRow(host, "controller-select-steam", stringResource(R.string.ctrl_select_steam), stringResource(R.string.ctrl_select_steam_hint), c.selectSteam, onChange = a.onSelectSteam)
ToggleRow(host, "controller-joycon-vr", stringResource(R.string.ctrl_joycon_vr), stringResource(R.string.ctrl_joycon_vr_hint), c.joyConVr, onChange = a.onJoyConVr)
SettingsRow(stringResource(R.string.ctrl_layout), if (c.customLayout) stringResource(R.string.ctrl_layout_custom) else stringResource(R.string.ctrl_layout_default)) {
Row(horizontalArrangement = Arrangement.spacedBy(6.dp)) {
SecondaryButton(stringResource(R.string.ctrl_edit)) { a.onEditLayout() }
@@ -0,0 +1,484 @@
package com.droiddeck.launcher.ui
import android.view.KeyEvent
import androidx.activity.compose.BackHandler
import androidx.compose.animation.AnimatedContent
import androidx.compose.animation.core.Animatable
import androidx.compose.animation.fadeIn
import androidx.compose.animation.fadeOut
import androidx.compose.animation.slideInHorizontally
import androidx.compose.animation.slideOutHorizontally
import androidx.compose.animation.togetherWith
import androidx.compose.foundation.Canvas
import androidx.compose.foundation.LocalIndication
import androidx.compose.foundation.background
import androidx.compose.foundation.clickable
import androidx.compose.foundation.focusGroup
import androidx.compose.foundation.gestures.detectTapGestures
import androidx.compose.foundation.hoverable
import androidx.compose.foundation.interaction.MutableInteractionSource
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.BoxWithConstraintsScope
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.heightIn
import androidx.compose.foundation.layout.offset
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.width
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.foundation.verticalScroll
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.Close
import androidx.compose.material3.Icon
import androidx.compose.material3.MaterialTheme
import androidx.compose.runtime.Composable
import androidx.compose.runtime.CompositionLocalProvider
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberCoroutineScope
import androidx.compose.runtime.rememberUpdatedState
import androidx.compose.runtime.setValue
import androidx.compose.runtime.withFrameNanos
import androidx.compose.ui.Alignment
import androidx.compose.ui.ExperimentalComposeUiApi
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.clip
import androidx.compose.ui.focus.FocusDirection
import androidx.compose.ui.focus.FocusRequester
import androidx.compose.ui.focus.focusProperties
import androidx.compose.ui.focus.focusRequester
import androidx.compose.ui.focus.onFocusChanged
import androidx.compose.ui.geometry.Offset
import androidx.compose.ui.geometry.Rect
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.Path
import androidx.compose.ui.graphics.drawscope.Stroke
import androidx.compose.ui.graphics.graphicsLayer
import androidx.compose.ui.graphics.lerp
import androidx.compose.ui.input.InputMode
import androidx.compose.ui.input.key.onPreviewKeyEvent
import androidx.compose.ui.input.nestedscroll.NestedScrollConnection
import androidx.compose.ui.input.nestedscroll.NestedScrollSource
import androidx.compose.ui.input.nestedscroll.nestedScroll
import androidx.compose.ui.input.pointer.pointerInput
import androidx.compose.ui.layout.boundsInParent
import androidx.compose.ui.layout.onGloballyPositioned
import androidx.compose.ui.layout.onSizeChanged
import androidx.compose.ui.platform.LocalDensity
import androidx.compose.ui.platform.LocalInputModeManager
import androidx.compose.ui.platform.testTag
import androidx.compose.ui.res.stringResource
import androidx.compose.ui.semantics.Role
import androidx.compose.ui.semantics.contentDescription
import androidx.compose.ui.semantics.semantics
import androidx.compose.ui.unit.IntOffset
import androidx.compose.ui.unit.Velocity
import androidx.compose.ui.unit.dp
import androidx.compose.ui.util.lerp
import androidx.compose.ui.zIndex
import com.droiddeck.launcher.R
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Job
import kotlinx.coroutines.coroutineScope
import kotlinx.coroutines.delay
import kotlinx.coroutines.launch
import kotlin.math.roundToInt
private const val DROP_PIECES = 12
internal class DropSheetState(private val scope: CoroutineScope) {
var open by mutableStateOf(false)
var shown by mutableStateOf(false)
var lipBack by mutableStateOf(false)
var tab by mutableStateOf(0)
var travel by mutableStateOf(1)
var settledByDrag = false
var engaged = false
private set
val pull = Animatable(0f)
val lift = Animatable(0f)
val ride = Animatable(0f)
val sink = Animatable(0f)
val hot = Animatable(1f)
val lipShow = Animatable(0f)
val lipGone = Animatable(0f)
val pieces = List(DROP_PIECES) { Animatable(0f) }
var travelPx = 1f
private var from = 0f
var progress = 0f
private set
private var dragJob: Job? = null
fun cancelDrag() {
dragJob?.cancel()
dragJob = null
}
fun reset() {
cancelDrag()
engaged = false
settledByDrag = false
open = false
shown = false
lipBack = false
scope.launch {
listOf(pull, lift, ride, sink, lipGone).forEach { it.snapTo(0f) }
pieces.forEach { it.snapTo(0f) }
hot.snapTo(1f)
}
}
fun beginDrag() {
cancelDrag()
settledByDrag = false
engaged = true
from = if (open) 1f else 0f
progress = from
if (!open) shown = true
}
fun drag(dy: Float) = follow((progress + dy / travelPx).coerceIn(0f, 1.04f))
fun follow(p: Float) {
progress = p
scope.launch {
lift.snapTo(p); ride.snapTo(p); sink.snapTo(p.coerceIn(0f, 1f)); hot.snapTo(0f)
lipGone.snapTo(1f)
pieces.forEach { it.snapTo(p.coerceIn(0f, 1f)) }
}
}
fun settle(v: Float, flick: Float) {
engaged = false
val opens = when {
v > flick -> true
v < -flick -> false
from > 0.5f -> progress > 0.7f
else -> progress > 0.3f
}
settledByDrag = true
cancelDrag()
dragJob = scope.launch {
val to = if (opens) 1f else 0f
coroutineScope {
launch { lift.animateTo(to, Motion.sp(if (opens) 0.7f else 1f, 380f)) }
launch { ride.animateTo(to, Motion.sp(if (opens) 0.7f else 0.55f, if (opens) 380f else 300f)) }
launch { sink.animateTo(to, Motion.tw(220)) }
pieces.forEachIndexed { i, piece -> launch { if (opens) delay(Motion.ms(i * 35).toLong()); piece.animateTo(to, Motion.tw(if (opens) 200 else 90)) } }
}
if (!opens) { shown = false; lipGone.snapTo(0f); hot.snapTo(1f); lipBack = false }
}
if (open != opens) open = opens else settledByDrag = false
}
}
@Composable
internal fun rememberDropSheetState(): DropSheetState {
val scope = rememberCoroutineScope()
return remember { DropSheetState(scope) }
}
@OptIn(ExperimentalComposeUiApi::class, androidx.compose.animation.ExperimentalAnimationApi::class)
@Composable
internal fun BoxWithConstraintsScope.DropSheet(
state: DropSheetState,
tabs: List<Pair<String, String>>,
covered: () -> Float = { 0f },
button: @Composable (enabled: Boolean) -> Unit,
rows: @Composable (tab: String) -> Unit,
) {
val density = LocalDensity.current
val colors = MaterialTheme.colorScheme
val pal = LocalPalette.current
val inputMode = LocalInputModeManager.current
val narrow = LocalNarrowPane.current
val chipFocus = remember(tabs.size) { List(tabs.size) { FocusRequester() } }
val buttonFocus = remember { FocusRequester() }
val leftForButton = remember { booleanArrayOf(false) }
var rest by remember { mutableStateOf<Rect?>(null) }
val openState = rememberUpdatedState(state.open)
if (state.tab > tabs.lastIndex) state.tab = 0
fun focusButton() { if (inputMode.inputMode == InputMode.Keyboard) runCatching { buttonFocus.requestFocus() } }
fun close() {
if (!state.open) return
state.open = false
focusButton()
}
val lifecycle = androidx.compose.ui.platform.LocalLifecycleOwner.current.lifecycle
DisposableEffect(lifecycle) {
val obs = androidx.lifecycle.LifecycleEventObserver { _, e ->
if (e == androidx.lifecycle.Lifecycle.Event.ON_STOP && (state.open || state.shown || state.engaged)) state.reset()
}
lifecycle.addObserver(obs)
onDispose { lifecycle.removeObserver(obs) }
}
LaunchedEffect(state.open) {
if (state.settledByDrag) {
state.settledByDrag = false
if (state.open) {
if (inputMode.inputMode == InputMode.Keyboard) { withFrameNanos { }; runCatching { chipFocus[state.tab].requestFocus() } }
} else focusButton()
return@LaunchedEffect
}
if (state.engaged) return@LaunchedEffect
state.cancelDrag()
if (state.open) {
state.lipBack = false
state.shown = true
coroutineScope {
launch { state.pull.animateTo(24f, Motion.sp(0.7f, 380f)) }
launch { state.sink.animateTo(1f, Motion.tw(260)) }
launch {
delay(Motion.ms(120).toLong())
launch { state.lipGone.animateTo(1f, Motion.tw(140)) }
launch { state.pull.animateTo(0f, Motion.sp(1f, 500f)) }
launch { state.lift.animateTo(1f, Motion.sp(0.7f, 380f)) }
launch { state.ride.animateTo(1f, Motion.sp(0.7f, 380f)) }
launch { state.hot.snapTo(1f); delay(Motion.ms(100).toLong()); state.hot.animateTo(0f, Motion.tw(200)) }
}
state.pieces.forEachIndexed { i, p -> launch { delay(Motion.ms(120 + 260 + i * 55).toLong()); p.animateTo(1f, Motion.tw(220)) } }
if (inputMode.inputMode == InputMode.Keyboard) launch {
delay(Motion.ms(300).toLong())
withFrameNanos { }
runCatching { chipFocus[state.tab].requestFocus() }
}
}
} else if (state.shown) {
coroutineScope {
state.pieces.forEach { launch { it.animateTo(0f, Motion.tw(90)) } }
launch { delay(Motion.ms(60).toLong()); state.lift.animateTo(0f, Motion.sp(1f, 380f)) }
launch { delay(Motion.ms(100).toLong()); state.ride.animateTo(0f, Motion.sp(0.55f, 300f)) }
launch { state.sink.animateTo(0f, Motion.tw(300)) }
launch { delay(Motion.ms(420).toLong()); state.lipGone.snapTo(0f); state.lipBack = true }
}
state.shown = false
state.lipBack = false
state.lipGone.snapTo(0f)
state.hot.snapTo(1f)
} else {
listOf(state.pull, state.sink, state.lift, state.ride, state.lipGone).forEach { it.snapTo(0f) }
state.pieces.forEach { it.snapTo(0f) }
state.hot.snapTo(1f)
}
}
BackHandler(enabled = state.open) { close() }
if (state.ride.isRunning || (state.shown && !state.open)) VeilRing()
val lipWanted = !state.open && (!state.shown || state.lipBack) && covered() < 0.5f
LaunchedEffect(lipWanted) {
if (lipWanted) {
if (!state.lipBack) delay(Motion.ms(160).toLong())
state.lipShow.animateTo(1f, Motion.tw(220))
} else {
delay(Motion.ms(140).toLong())
state.lipShow.animateTo(0f, Motion.tw(220))
}
}
val flick = with(density) { 600.dp.toPx() }
val rowsPush = remember(state) {
object : NestedScrollConnection {
override fun onPreScroll(available: Offset, source: NestedScrollSource): Offset {
if (!state.engaged || available.y <= 0f) return Offset.Zero
val before = state.progress
state.follow((before + available.y / state.travelPx).coerceAtMost(1f))
return Offset(0f, (state.progress - before) * state.travelPx)
}
override fun onPostScroll(consumed: Offset, available: Offset, source: NestedScrollSource): Offset {
if (available.y >= 0f || source != NestedScrollSource.Drag || !openState.value) return Offset.Zero
if (!state.engaged) state.beginDrag()
state.follow((state.progress + available.y / state.travelPx).coerceIn(0f, 1f))
return available
}
override suspend fun onPreFling(available: Velocity): Velocity {
if (!state.engaged) return Velocity.Zero
state.settle(available.y, flick)
return available
}
}
}
val w = constraints.maxWidth.toFloat()
val h = constraints.maxHeight.toFloat()
val inset = with(density) { 12.dp.toPx() }
val edge = with(density) { 3.dp.toPx() }
val buttonH = rest?.height ?: with(density) { 56.dp.toPx() }
val openBottom = h - sheetTop(h, buttonH, with(density) { 16.dp.toPx() }, edge)
val lower = rest?.let { (openBottom - edge - buttonH / 2f) - it.top } ?: 0f
state.travelPx = openBottom.coerceAtLeast(1f)
if (state.shown || state.open) Box(
Modifier.fillMaxSize()
.graphicsLayer { alpha = state.sink.value }.background(Color.Black.copy(alpha = 0.42f))
.focusProperties { canFocus = false }
.clickable(interactionSource = remember { MutableInteractionSource() }, indication = null, enabled = state.open) { close() },
)
if (state.shown) {
val bottom = lerp(0f, openBottom, state.lift.value)
val sheetLeft = inset
val sheetRight = w - inset
val corner = with(density) { 18.dp.toPx() }
val rim = with(density) { 1.4.dp.toPx() }
val path = remember { Path() }
val rimPath = remember { Path() }
Canvas(Modifier.fillMaxSize()) {
val t = -corner
rimPath.reset()
rimPath.moveTo(sheetLeft, t)
rimPath.lineTo(sheetLeft, bottom - corner)
rimPath.arcTo(Rect(sheetLeft, bottom - 2 * corner, sheetLeft + 2 * corner, bottom), 180f, -90f, false)
rimPath.lineTo(sheetRight - corner, bottom)
rimPath.arcTo(Rect(sheetRight - 2 * corner, bottom - 2 * corner, sheetRight, bottom), 90f, -90f, false)
rimPath.lineTo(sheetRight, t)
path.reset()
path.addPath(rimPath)
path.close()
drawPath(path, lerp(colors.surface, pal.signal, 0.22f * state.hot.value))
drawPath(rimPath, pal.signal.copy(alpha = lerp(0.55f, 0.9f, state.hot.value)), style = Stroke(rim))
}
val pill = rest?.translate(0f, lower)
val sheetW = sheetRight - sheetLeft
val chipsH = with(density) { 44.dp.toPx() }
val closeW = with(density) { 64.dp.toPx() }
val chipsBottom = with(density) {
if (narrow && pill != null) pill.top - 10.dp.toPx() else openBottom - 14.dp.toPx()
}
val chipsTop = chipsBottom - chipsH
val chipsLeft = with(density) {
if (narrow || pill == null) 16.dp.toPx() else pill.right - sheetLeft + 20.dp.toPx()
}
val chipsRight = sheetW - closeW
val rowsBottom = chipsTop - with(density) { 8.dp.toPx() }
Box(
Modifier
.offset { IntOffset(sheetLeft.roundToInt(), (bottom - openBottom).roundToInt()) }
.size(with(density) { sheetW.toDp() }, with(density) { openBottom.toDp() })
.pointerInput(Unit) { detectTapGestures { } }
.controllerBack { close() }
.bumpers(
onPrevious = { if (state.tab > 0) { state.travel = -1; state.tab--; runCatching { chipFocus[state.tab].requestFocus() } } },
onNext = { if (state.tab < tabs.lastIndex) { state.travel = 1; state.tab++; runCatching { chipFocus[state.tab].requestFocus() } } },
)
.focusProperties {
exit = { dir ->
if (!openState.value) FocusRequester.Default
else if (dir == FocusDirection.Left) { leftForButton[0] = true; buttonFocus }
else FocusRequester.Cancel
}
}
.focusGroup(),
) {
Box(
Modifier
.nestedScroll(rowsPush)
.fillMaxWidth()
.height(with(density) { rowsBottom.coerceAtLeast(0f).toDp() }),
) {
AnimatedContent(
targetState = state.tab.coerceIn(0, tabs.lastIndex.coerceAtLeast(0)),
transitionSpec = {
val d = state.travel
val shift = with(density) { 24.dp.roundToPx() }
(slideInHorizontally(Motion.tw(240)) { shift * d } + fadeIn(Motion.tw(240)))
.togetherWith(slideOutHorizontally(Motion.tw(240)) { -shift * d } + fadeOut(Motion.tw(240)))
},
label = "dropTab",
) { t ->
val leaving = transition.targetState == androidx.compose.animation.EnterExitState.PostExit
val leavingNow = rememberUpdatedState(leaving)
CompositionLocalProvider(LocalSheetBeat provides remember { SheetBeat(state.pieces) }) {
Column(
Modifier.fillMaxSize()
.focusProperties { enter = { if (leavingNow.value) FocusRequester.Cancel else FocusRequester.Default } }
.focusGroup()
.verticalScroll(rememberScrollState())
.padding(start = 16.dp, end = 16.dp, top = 8.dp, bottom = 8.dp),
) { tabs.getOrNull(t)?.let { rows(it.first) } }
}
}
}
Box(
Modifier
.offset { IntOffset(chipsLeft.roundToInt(), chipsTop.roundToInt()) }
.zIndex(1f)
.width(with(density) { (chipsRight - chipsLeft).coerceAtLeast(0f).toDp() })
.graphicsLayer { val v = state.pieces[0].value; alpha = v; translationY = (1f - v) * -6.dp.toPx() }
.onPreviewKeyEvent { e ->
val k = e.nativeKeyEvent
if (k.keyCode == KeyEvent.KEYCODE_DPAD_DOWN) { if (k.action == KeyEvent.ACTION_DOWN) close(); true } else false
},
) {
SheetChips(tabs.map { it.second }, state.tab, chipFocus, width = LocalSheetChipWidth.current) { i -> state.travel = if (i > state.tab) 1 else -1; state.tab = i }
}
val closeSource = remember { MutableInteractionSource() }
val closeHot = rememberHot(closeSource)
Box(
contentAlignment = Alignment.Center,
modifier = Modifier
.offset { IntOffset((sheetW - 52.dp.toPx()).roundToInt(), chipsTop.roundToInt()) }
.zIndex(1f).size(44.dp).testTag("drop-close")
.clip(CircleShape).glideBorder(closeHot, CircleShape, pal.signal)
.hoverable(closeSource)
.clickable(interactionSource = closeSource, indication = LocalIndication.current, enabled = state.open, role = Role.Button) { close() }
.controllerConfirm(enabled = state.open) { close() },
) { Icon(Icons.Filled.Close, stringResource(R.string.common_close), tint = colors.onSurfaceVariant, modifier = Modifier.size(20.dp)) }
}
}
val enabled = covered() < 0.5f
val sideInset = if (narrow) 20.dp else 44.dp
val play = LocalPlayRect.current
Box(
Modifier
.offset { IntOffset(play?.left?.roundToInt() ?: sideInset.roundToPx(), sideInset.roundToPx()) }
.then(if (play != null) Modifier.width(with(density) { play.width.toDp() }) else Modifier)
.onGloballyPositioned { rest = it.boundsInParent() }
.graphicsLayer { translationY = lower * state.ride.value; alpha = 1f - covered() }
.zIndex(2f)
.focusRequester(buttonFocus)
.focusProperties { canFocus = enabled }
.onFocusChanged {
if (it.hasFocus && leftForButton[0]) { leftForButton[0] = false; if (openState.value) close() }
}
.onPreviewKeyEvent { e ->
val k = e.nativeKeyEvent
if (k.keyCode != KeyEvent.KEYCODE_DPAD_UP && !(state.open && k.keyCode == KeyEvent.KEYCODE_DPAD_RIGHT)) false
else {
if (k.action == KeyEvent.ACTION_DOWN && k.repeatCount == 0) {
if (state.open) runCatching { chipFocus[state.tab].requestFocus() } else if (enabled) state.open = true
}
true
}
},
) { button(enabled) }
val r0 = rest
if (r0 != null && (state.lipShow.value > 0.01f || state.pull.value > 0.5f)) {
var handleSize by remember { mutableStateOf(androidx.compose.ui.unit.IntSize.Zero) }
val lipH = 32.dp + state.pull.value.dp
val drop = { if (!state.open && enabled) state.open = true }
val settingsLabel = stringResource(R.string.drawer_page_settings)
Box(
contentAlignment = Alignment.BottomCenter,
modifier = Modifier
.offset { IntOffset((r0.center.x - handleSize.width / 2f).roundToInt(), (r0.top + lower * state.ride.value - handleSize.height + (1f - state.lipShow.value) * 8.dp.toPx()).roundToInt()) }
.onSizeChanged { handleSize = it }
.graphicsLayer { alpha = state.lipShow.value * (1f - state.lipGone.value) * (1f - covered()) }
.zIndex(2f)
.heightIn(min = 44.dp).testTag("drop-open")
.semantics { contentDescription = settingsLabel }
.focusProperties { canFocus = false }
.clickable(interactionSource = remember { MutableInteractionSource() }, indication = null, role = Role.Button, onClick = drop),
) {
Handle(lipH, above = true)
}
}
}
@@ -151,6 +151,7 @@ class FrontEndState(
class FrontEndActions(
val onPlay: () -> Unit,
val onPlaySteamVr: () -> Unit = {},
val onPlayDesktopUi: () -> Unit,
val onSteamUrl: (String) -> Unit = {},
val onSteamGame: (Library.SteamGame) -> Unit,
@@ -206,7 +206,7 @@ internal fun SetupPanel(
),
) {
PageHeader(stringResource(R.string.setup_title)) {
Box(Modifier.weight(1f)) { SheetChips(tabs, tab, chipFocus, pick) }
Box(Modifier.weight(1f)) { SheetChips(tabs, tab, chipFocus, onPick = pick) }
}
Column(modifier = Modifier.weight(1f).fillMaxWidth().verticalScroll(rememberScrollState()).padding(bottom = 24.dp)) {
when (tab) {
@@ -22,6 +22,7 @@ import androidx.compose.foundation.interaction.collectIsHoveredAsState
import androidx.compose.foundation.interaction.collectIsPressedAsState
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.BoxWithConstraints
import androidx.compose.foundation.layout.ExperimentalLayoutApi
import androidx.compose.foundation.layout.FlowRow
import androidx.compose.foundation.layout.Row
@@ -38,6 +39,7 @@ import androidx.compose.material.icons.filled.ChevronLeft
import androidx.compose.ui.graphics.vector.ImageVector
import androidx.compose.material3.CircularProgressIndicator
import androidx.compose.material3.Icon
import androidx.compose.material3.LocalTextStyle
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
@@ -57,6 +59,8 @@ import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.graphicsLayer
import androidx.compose.ui.semantics.Role
import androidx.compose.ui.semantics.contentDescription
import androidx.compose.ui.text.TextStyle
import androidx.compose.ui.text.rememberTextMeasurer
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.dp
@@ -148,6 +152,7 @@ internal fun PrimaryButton(
modifier: Modifier = Modifier,
large: Boolean = false,
icon: ImageVector? = null,
fill: Boolean = false,
onClick: () -> Unit,
) {
val colors = MaterialTheme.colorScheme
@@ -165,7 +170,8 @@ internal fun PrimaryButton(
val lift by animateFloatAsState(if (hot) 14f else 6f, Motion.tw(300), label = "btnLift")
val placed = remember { arrayOfNulls<androidx.compose.ui.layout.LayoutCoordinates>(1) }
Row(
verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = if (fill) Arrangement.spacedBy(8.dp, Alignment.CenterHorizontally) else Arrangement.spacedBy(8.dp),
modifier = modifier.then(track)
.onGloballyPositioned { placed[0] = it }
.heightIn(min = if (compact) 44.dp else 48.dp)
@@ -181,18 +187,34 @@ internal fun PrimaryButton(
onClick()
}
.padding(
start = if (large) 20.dp else if (compact) 10.dp else if (icon != null) 14.dp else 18.dp,
end = if (large) 26.dp else if (compact) 10.dp else 18.dp,
start = if (fill) 12.dp else if (large) 20.dp else if (compact) 10.dp else if (icon != null) 14.dp else 18.dp,
end = if (fill) 12.dp else if (large) 26.dp else if (compact) 10.dp else 18.dp,
top = if (large) 16.dp else if (compact) 7.dp else 11.dp,
bottom = if (large) 16.dp else if (compact) 7.dp else 11.dp,
),
) {
val fg = if (enabled) colors.onPrimary else colors.onSurfaceVariant
if (icon != null) Icon(icon, contentDescription = null, tint = fg, modifier = Modifier.size(if (large) 20.dp else 17.dp))
Text(
text, fontSize = if (large) 17.sp else if (compact) 12.sp else 15.sp,
fontWeight = if (large) FontWeight.Bold else FontWeight.SemiBold, letterSpacing = 0.5.sp, color = fg, maxLines = 1,
)
val style = LocalTextStyle.current.merge(TextStyle(
fontSize = if (large) 17.sp else if (compact) 12.sp else 15.sp,
fontWeight = if (large) FontWeight.Bold else FontWeight.SemiBold, letterSpacing = 0.5.sp, color = fg,
))
if (fill) FittedLabel(text, style, Modifier.weight(1f, fill = false))
else Text(text, style = style, maxLines = 1)
}
}
@Composable
private fun FittedLabel(text: String, style: TextStyle, modifier: Modifier = Modifier) {
BoxWithConstraints(modifier) {
val measurer = rememberTextMeasurer()
val max = constraints.maxWidth
val size = remember(text, style, max) {
var size = style.fontSize
while (size.value > 10f && measurer.measure(text, style.copy(fontSize = size), maxLines = 1, softWrap = false).size.width > max) size = (size.value - 0.5f).sp
size
}
Text(text, style = style.copy(fontSize = size), maxLines = 1, softWrap = false)
}
}
@@ -0,0 +1,64 @@
package com.droiddeck.launcher.ui
import androidx.compose.foundation.layout.Column
import androidx.compose.material3.AlertDialog
import androidx.compose.material3.LinearProgressIndicator
import androidx.compose.material3.Text
import androidx.compose.material3.TextButton
import androidx.compose.runtime.*
import androidx.compose.ui.res.stringResource
import com.droiddeck.launcher.R
import com.droiddeck.launcher.glasses.CalibrationPhase
import com.droiddeck.launcher.glasses.GlassesCalibrationState
@Composable
internal fun GlassesCalibrationControls(
state: GlassesCalibrationState,
onStart: () -> Unit,
onCancel: () -> Unit,
onReset: () -> Unit,
onDone: () -> Unit,
) {
var shown by remember { mutableStateOf(false) }
var reset by remember { mutableStateOf(false) }
val cancel by rememberUpdatedState(onCancel)
DisposableEffect(Unit) { onDispose { cancel() } }
ActionRow(stringResource(R.string.glasses_calibration),
stringResource(if (state.saved) R.string.glasses_calibration_saved_hint else R.string.glasses_calibration_hint),
stringResource(R.string.glasses_calibration_open), { onCancel(); shown = true })
if (state.saved && !state.running) ActionRow(stringResource(R.string.glasses_calibration_reset), null,
stringResource(R.string.glasses_calibration_reset_button), { reset = true })
if (reset) AlertDialog(
onDismissRequest = { reset = false },
title = { Text(stringResource(R.string.glasses_calibration_reset)) },
text = { Text(stringResource(R.string.glasses_calibration_reset_hint)) },
confirmButton = { TextButton(onClick = { onReset(); reset = false }) { Text(stringResource(R.string.glasses_calibration_reset_button)) } },
dismissButton = { TextButton(onClick = { reset = false }) { Text(stringResource(R.string.common_cancel)) } },
)
if (shown) AlertDialog(
onDismissRequest = { onCancel(); shown = false },
title = { Text(stringResource(R.string.glasses_calibration)) },
text = { Column {
Text(stringResource(when (state.phase) {
CalibrationPhase.IDLE -> R.string.glasses_calibration_instructions
CalibrationPhase.SETTLING -> R.string.glasses_calibration_settling
CalibrationPhase.COLLECTING -> R.string.glasses_calibration_collecting
CalibrationPhase.MOVING -> R.string.glasses_calibration_moving
CalibrationPhase.TURN_OVER -> R.string.glasses_calibration_turn_over
CalibrationPhase.SAVED -> R.string.glasses_calibration_complete
CalibrationPhase.FAILED -> R.string.glasses_calibration_failed
}))
if (state.running) {
LinearProgressIndicator(progress = { state.progress / 100f })
Text("${state.progress}%")
}
} },
confirmButton = {
if (!state.running) TextButton(onClick = {
if (state.phase == CalibrationPhase.SAVED) { onDone(); onCancel(); shown = false }
else onStart()
}) { Text(stringResource(if (state.phase == CalibrationPhase.SAVED) R.string.glasses_calibration_done else R.string.glasses_calibration_start)) }
},
dismissButton = { TextButton(onClick = { onCancel(); shown = false }) { Text(stringResource(R.string.common_cancel)) } },
)
}
@@ -0,0 +1,268 @@
package com.droiddeck.launcher.ui
import androidx.compose.foundation.background
import androidx.compose.foundation.border
import androidx.compose.foundation.clickable
import androidx.compose.foundation.interaction.MutableInteractionSource
import androidx.compose.foundation.interaction.collectIsPressedAsState
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.WindowInsets
import androidx.compose.foundation.layout.displayCutout
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.offset
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.windowInsetsPadding
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material3.Icon
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.remember
import androidx.compose.ui.Alignment
import androidx.compose.ui.draw.clip
import androidx.compose.ui.platform.testTag
import androidx.compose.ui.semantics.Role
import androidx.compose.ui.unit.IntOffset
import androidx.compose.ui.unit.sp
import androidx.compose.runtime.getValue
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.SolidColor
import androidx.compose.ui.graphics.StrokeCap
import androidx.compose.ui.graphics.StrokeJoin
import androidx.compose.ui.graphics.vector.ImageVector
import androidx.compose.ui.graphics.vector.path
import androidx.compose.ui.res.stringResource
import androidx.compose.ui.unit.dp
import androidx.compose.runtime.collectAsState
import com.droiddeck.launcher.R
import com.droiddeck.launcher.glasses.Glasses
import com.droiddeck.launcher.glasses.GlassesRendering
import com.droiddeck.launcher.glasses.GlassesSettings
import com.droiddeck.launcher.glasses.GlassesState
import com.droiddeck.launcher.handtracking.HandTracking
import java.util.Locale
internal val GlassesIcon: ImageVector by lazy {
ImageVector.Builder("Glasses", 24.dp, 24.dp, 24f, 24f).apply {
path(fill = null, stroke = SolidColor(Color.Black), strokeLineWidth = 1.8f, strokeLineCap = StrokeCap.Round, strokeLineJoin = StrokeJoin.Round) {
moveTo(2.5f, 10f); lineTo(21.5f, 10f)
moveTo(3.2f, 10f); lineTo(3.8f, 15.2f); quadTo(4f, 17f, 6f, 17f); lineTo(8.6f, 17f); quadTo(10.2f, 17f, 10.6f, 15.4f); lineTo(11.2f, 12.6f)
quadTo(12f, 11.6f, 12.8f, 12.6f); lineTo(13.4f, 15.4f); quadTo(13.8f, 17f, 15.4f, 17f); lineTo(18f, 17f); quadTo(20f, 17f, 20.2f, 15.2f); lineTo(20.8f, 10f)
moveTo(2.5f, 10f); lineTo(1.8f, 7.6f)
moveTo(21.5f, 10f); lineTo(22.2f, 7.6f)
}
}.build()
}
@Composable
internal fun rememberGlasses(): Pair<GlassesState, GlassesSettings> {
val g by Glasses.state.collectAsState()
val s by Glasses.settings.collectAsState()
return g to s
}
internal object GlassesTab {
const val DISPLAY = "display"
const val HARDWARE = "hardware"
const val TRACKING = "tracking"
}
@Composable
internal fun glassesTabs(g: GlassesState): List<Pair<String, String>> = buildList {
add(GlassesTab.DISPLAY to stringResource(R.string.glasses_display))
if (g.canControl) {
if (g.hardwareControls) add(GlassesTab.HARDWARE to stringResource(R.string.glasses_hardware))
add(GlassesTab.TRACKING to stringResource(R.string.glasses_tracking))
}
}
@Composable
internal fun GlassesGroups(
host: MenuHost,
g: GlassesState,
s: GlassesSettings,
track: (String) -> Modifier = { Modifier },
onRecenter: (() -> Unit)? = null,
inSession: Boolean = true,
outputActive: Boolean = inSession,
steamVrRunning: Boolean = false,
tab: String? = null,
steamVr: @Composable () -> Unit = {},
) {
val display = tab == null || tab == GlassesTab.DISPLAY
val picture = GlassesRendering.picture(s, g.sbs, steamVrRunning)
val usesGlasses = GlassesRendering.usesGlasses(g, outputActive, steamVrRunning)
val virtualScreen = usesGlasses && picture == GlassesRendering.Picture.VIRTUAL_SCREEN
val directView = usesGlasses && picture == GlassesRendering.Picture.STEREO
if (display) SettingsGroup(g.model) {
if (g.experimental) SettingsRow(stringResource(R.string.glasses_control), stringResource(R.string.glasses_xreal_experimental)) {}
val status = if (g.hasDisplay) stringResource(R.string.glasses_status, g.width, g.height, kotlin.math.round(g.hz).toInt(),
stringResource(if (g.sbs) R.string.glasses_mode_3d else R.string.glasses_mode_2d)) else null
when {
g.noSignal -> SettingsRow(stringResource(R.string.glasses_output), stringResource(R.string.glasses_no_signal)) {}
!g.controllable -> {
if (status != null) SettingsRow(stringResource(R.string.glasses_output), status) {}
SettingsRow(stringResource(R.string.glasses_control), g.controlError ?: stringResource(R.string.glasses_no_control)) {}
}
g.permission == Glasses.Permission.ASKING ->
SettingsRow(stringResource(R.string.glasses_control), stringResource(R.string.glasses_asking)) {}
!g.connected -> ActionRow(stringResource(R.string.glasses_control),
g.controlError ?: stringResource(if (g.permission == Glasses.Permission.DENIED) R.string.glasses_denied else R.string.glasses_allow_hint),
stringResource(R.string.glasses_allow), { Glasses.requestControl() })
status != null -> SettingsRow(stringResource(R.string.glasses_output), status) {}
}
ToggleRow(host, "glasses-game", stringResource(R.string.glasses_game_on), stringResource(R.string.glasses_game_on_hint),
s.gameOnGlasses, chipModifier = track("glasses-game")) { Glasses.setGameOnGlasses(it) }
}
if (display && (g.canControl || g.hasDisplay)) SettingsGroup(stringResource(R.string.glasses_display)) {
if (g.canControl) {
if (!g.sbs) ChoiceRow(host, "glasses-hz", stringResource(R.string.glasses_refresh), null,
listOf(60, 90, 120).map { it to "$it Hz" }, s.refreshHz, chipModifier = track("glasses-hz")) { Glasses.setRefreshHz(it) }
if (g.sbsSupport == GlassesState.SBS_FAILS && !g.sbs) {
ActionRow(stringResource(R.string.glasses_3d), stringResource(R.string.glasses_3d_failed),
stringResource(R.string.stores_dl_retry), { Glasses.set3d(true) })
} else {
if (inSession || g.sbs) ToggleRow(host, "glasses-3d", stringResource(R.string.glasses_3d), stringResource(R.string.glasses_3d_hint),
g.sbs, chipModifier = track("glasses-3d")) { Glasses.set3d(it) }
if (g.sbs) {
if (g.sbs90) ChoiceRow(host, "glasses-3d-hz", stringResource(R.string.glasses_3d_refresh), null,
listOf(60, 90).map { it to "$it Hz" }, s.sbsHz, chipModifier = track("glasses-3d-hz")) { Glasses.setSbsHz(it) }
else SettingsRow(stringResource(R.string.glasses_3d_refresh), "60 Hz") {}
}
}
}
ToggleRow(host, "glasses-auto3d", stringResource(R.string.glasses_auto3d), stringResource(R.string.glasses_auto3d_hint),
s.auto3d, chipModifier = track("glasses-auto3d")) { Glasses.setAuto3d(it) }
ChoiceRow(host, "glasses-picture", stringResource(R.string.glasses_picture), stringResource(R.string.glasses_picture_hint),
listOf(
GlassesSettings.PICTURE_AUTO to stringResource(R.string.glasses_picture_auto),
GlassesSettings.PICTURE_SCREEN to stringResource(R.string.glasses_picture_screen),
GlassesSettings.PICTURE_SBS to stringResource(R.string.glasses_picture_sbs),
), s.picture3d, chipModifier = track("glasses-picture"), hintLines = 2) { Glasses.setPicture3d(it) }
when {
!inSession -> SettingsRow(stringResource(R.string.glasses_picture_screen), stringResource(R.string.glasses_screen_next_session)) {}
!usesGlasses -> SettingsRow(stringResource(R.string.glasses_picture_screen), stringResource(R.string.glasses_screen_no_output)) {}
picture == GlassesRendering.Picture.FLAT -> ActionRow(
stringResource(R.string.glasses_picture_screen), stringResource(R.string.glasses_screen_flat_hint),
stringResource(R.string.glasses_screen_use), { Glasses.setPicture3d(GlassesSettings.PICTURE_SCREEN) })
directView -> SettingsRow(stringResource(R.string.glasses_picture_screen),
stringResource(if (steamVrRunning) R.string.glasses_screen_direct_hint else R.string.glasses_screen_sbs_hint)) {}
}
if (virtualScreen || (!inSession && s.picture3d != GlassesSettings.PICTURE_SBS)) {
SliderRow(stringResource(R.string.glasses_screen_size), null, s.screenWidthDm, 5..60, step = 1,
format = { String.format(Locale.US, "%.1f m", it / 10f) }, modifier = track("glasses-width")) { Glasses.setScreenWidthDm(it) }
SliderRow(stringResource(R.string.glasses_screen_distance), null, s.screenDistanceDm, 5..100, step = 1,
format = { String.format(Locale.US, "%.1f m", it / 10f) }, modifier = track("glasses-distance")) { Glasses.setScreenDistanceDm(it) }
}
steamVr()
if ((directView && steamVrRunning) || !inSession) SliderRow(stringResource(R.string.glasses_vr_fov), stringResource(R.string.glasses_vr_fov_hint), s.vrFovDeg,
GlassesSettings.VR_FOV_MIN..GlassesSettings.VR_FOV_MAX, step = 1,
format = { "$it°" }, modifier = track("glasses-vr-fov")) { Glasses.setVrFovDeg(it) }
}
if ((tab == null || tab == GlassesTab.HARDWARE) && g.canControl && g.hardwareControls) SettingsGroup(stringResource(R.string.glasses_hardware)) {
if (g.brightnessMax > 0) SliderRow(stringResource(R.string.glasses_brightness), null, g.brightness.coerceAtLeast(0), 0..g.brightnessMax, step = 1,
modifier = track("glasses-brightness")) { Glasses.setBrightness(it) }
if (g.volumeMax > 0) SliderRow(stringResource(R.string.glasses_volume), null, g.volume.coerceAtLeast(0), 0..g.volumeMax, step = 1,
modifier = track("glasses-volume")) { Glasses.setVolume(it) }
if (g.hasSunblock) ToggleRow(host, "glasses-sunblock", stringResource(R.string.glasses_sunblock), stringResource(R.string.glasses_sunblock_hint),
g.sunblock, chipModifier = track("glasses-sunblock")) { Glasses.setSunblock(it) }
}
if ((tab == null || tab == GlassesTab.TRACKING) && g.canControl) SettingsGroup(stringResource(R.string.glasses_tracking)) {
val trackingHint = when {
inSession && !usesGlasses -> R.string.glasses_screen_no_output
inSession && picture == GlassesRendering.Picture.FLAT -> R.string.glasses_tracking_flat_hint
inSession && directView && !steamVrRunning -> R.string.glasses_tracking_sbs_hint
else -> R.string.glasses_head_tracking_hint
}
ToggleRow(host, "glasses-tracking", stringResource(R.string.glasses_head_tracking), stringResource(trackingHint),
s.headTracking, chipModifier = track("glasses-tracking")) { Glasses.setHeadTracking(it) }
if (onRecenter != null && s.headTracking && (virtualScreen || (directView && steamVrRunning))) ActionRow(stringResource(R.string.glasses_recenter), null,
stringResource(R.string.cardboard_recenter_button), onRecenter)
val calibration by Glasses.calibration.collectAsState()
if (calibration.supported) GlassesCalibrationControls(calibration,
onStart = Glasses::startCalibration, onCancel = Glasses::cancelCalibration,
onReset = Glasses::resetCalibration, onDone = { onRecenter?.invoke() })
}
val context = androidx.compose.ui.platform.LocalContext.current
val hands by HandTracking.settings.collectAsState()
androidx.compose.runtime.LaunchedEffect(Unit) { HandTracking.load(context) }
if ((tab == null || tab == GlassesTab.TRACKING) && g.canControl && hands.usable) SettingsGroup(stringResource(R.string.hand_camera_group)) {
HandCameraRows(host, track, inSession)
}
}
private fun padButtonColors(tint: Int, opacity: Int, pressed: Boolean): Pair<Color, Color> {
val base = Color(tint)
val alpha = opacity.coerceIn(0, 100) / 100f
fun light(a: Int, f: Float) = Color(base.red + (1 - base.red) * f, base.green + (1 - base.green) * f, base.blue + (1 - base.blue) * f, a * alpha / 255f)
return Color(base.red * 0.12f, base.green * 0.12f, base.blue * 0.12f, 190 / 255f) to (if (pressed) light(230, 0.7f) else light(150, 0.25f))
}
@Composable
internal fun GlassesCornerButton(slot: Int, tint: Int, opacity: Int, onClick: () -> Unit) {
val density = androidx.compose.ui.platform.LocalDensity.current
androidx.compose.foundation.layout.BoxWithConstraints(
Modifier.fillMaxSize().windowInsetsPadding(WindowInsets.displayCutout),
) {
val x = if (slot < 0) maxWidth / 2 + 56.dp else 34.dp + 56.dp * slot
val bottom = if (slot < 0) 44.dp else 34.dp
val source = remember { MutableInteractionSource() }
val pressed by source.collectIsPressedAsState()
val (fill, line) = padButtonColors(tint, opacity, pressed)
Box(
contentAlignment = Alignment.Center,
modifier = Modifier
.offset { with(density) { IntOffset((x - 18.dp).roundToPx(), (maxHeight - bottom - 18.dp).roundToPx()) } }
.size(36.dp)
.clip(CircleShape)
.background(fill)
.border(1.5.dp, line, CircleShape)
.clickable(interactionSource = source, indication = null, role = Role.Button, onClick = onClick)
.testTag("glasses-button"),
) {
Icon(GlassesIcon, stringResource(R.string.glasses_open), tint = line, modifier = Modifier.size(22.dp))
}
}
}
@Composable
internal fun GlassesRecenterButton(tint: Int, opacity: Int, onClick: () -> Unit) {
val source = remember { MutableInteractionSource() }
val pressed by source.collectIsPressedAsState()
val (fill, line) = padButtonColors(tint, opacity, pressed)
val shape = RoundedCornerShape(50)
Box(
Modifier.fillMaxSize().windowInsetsPadding(WindowInsets.displayCutout).padding(top = 12.dp),
contentAlignment = Alignment.TopCenter,
) {
Box(
contentAlignment = Alignment.Center,
modifier = Modifier
.height(36.dp)
.clip(shape)
.background(fill)
.border(1.5.dp, line, shape)
.clickable(interactionSource = source, indication = null, role = Role.Button, onClick = onClick)
.padding(horizontal = 18.dp)
.testTag("glasses-recenter"),
) {
Text(stringResource(R.string.glasses_recenter_button), color = line, fontSize = 14.sp)
}
}
}
@Composable
internal fun GlassesPlayingHint(model: String) {
Box(Modifier.fillMaxSize(), contentAlignment = Alignment.Center) {
Column(horizontalAlignment = Alignment.CenterHorizontally) {
Icon(GlassesIcon, null, tint = Color.White.copy(alpha = 0.35f), modifier = Modifier.size(56.dp))
Spacer(Modifier.height(10.dp))
Text(stringResource(R.string.glasses_playing, model), color = Color.White.copy(alpha = 0.45f), fontSize = 15.sp)
}
}
}
@@ -0,0 +1,345 @@
package com.droiddeck.launcher.ui
import android.Manifest
import android.content.Intent
import android.net.Uri
import android.provider.Settings
import android.speech.tts.TextToSpeech
import androidx.activity.compose.rememberLauncherForActivityResult
import androidx.activity.result.contract.ActivityResultContracts
import androidx.compose.foundation.Canvas
import androidx.compose.foundation.Image
import androidx.compose.foundation.background
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.aspectRatio
import androidx.compose.foundation.layout.fillMaxHeight
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.heightIn
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.selection.selectable
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.foundation.verticalScroll
import androidx.compose.material3.AlertDialog
import androidx.compose.material3.CircularProgressIndicator
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.RadioButton
import androidx.compose.material3.RadioButtonDefaults
import androidx.compose.material3.Text
import androidx.compose.material3.TextButton
import androidx.compose.runtime.Composable
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberUpdatedState
import androidx.compose.runtime.saveable.rememberSaveable
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.clip
import androidx.compose.ui.geometry.Offset
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.StrokeCap
import androidx.compose.ui.graphics.asImageBitmap
import androidx.compose.ui.graphics.drawscope.DrawScope
import androidx.compose.ui.graphics.graphicsLayer
import androidx.compose.ui.layout.ContentScale
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.platform.LocalLifecycleOwner
import androidx.compose.ui.platform.testTag
import androidx.compose.ui.res.stringResource
import androidx.compose.ui.semantics.Role
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
import androidx.compose.ui.window.Dialog
import androidx.compose.ui.window.DialogProperties
import androidx.lifecycle.Lifecycle
import androidx.lifecycle.LifecycleEventObserver
import com.droiddeck.launcher.R
import com.droiddeck.launcher.handtracking.Advice
import com.droiddeck.launcher.handtracking.BodyPose
import com.droiddeck.launcher.handtracking.HandCamera
import com.droiddeck.launcher.handtracking.HandTracking
import com.droiddeck.launcher.handtracking.Landmark
import com.droiddeck.launcher.handtracking.TrackingCameras
import kotlinx.coroutines.delay
import java.util.Locale
@Composable
internal fun HandCameraHost(check: Boolean?, onAskPermission: () -> Unit, onPlay: () -> Unit, onClose: () -> Unit) {
if (!HandTracking.supported) return
val context = LocalContext.current
val (glasses, _) = rememberGlasses()
val settings by HandTracking.settings.collectAsState()
LaunchedEffect(Unit) { HandTracking.load(context) }
var prompted by rememberSaveable { mutableStateOf(false) }
if (!prompted && HandTracking.shouldAsk(settings, glasses.present && glasses.canControl)) {
HandCameraPrompt(settings.sides,
onAllow = { side ->
prompted = true
HandTracking.allow(context, side)
if (!HandTracking.permitted(context)) onAskPermission()
},
onNotNow = {
prompted = true
HandTracking.decline(context)
})
}
val requests by HandTracking.checkRequests.collectAsState()
var handled by remember { mutableStateOf(requests) }
val fromSettings = requests != handled
if (check != null || fromSettings) {
PlacementCheck(settings, play = check == true && !fromSettings,
onPlay = { handled = requests; onPlay() },
onClose = { handled = requests; onClose() })
}
}
@Composable
internal fun HandCameraPrompt(sides: Set<String>, onAllow: (String?) -> Unit, onNotNow: () -> Unit) {
val colors = MaterialTheme.colorScheme
val pal = LocalPalette.current
val choose = sides.size > 1
var side by remember { mutableStateOf(TrackingCameras.REAR) }
AlertDialog(
onDismissRequest = onNotNow,
modifier = Modifier.testTag("hand-camera-prompt"),
title = { Text(stringResource(R.string.hand_camera_prompt_title)) },
text = {
Column(Modifier.verticalScroll(rememberScrollState()), verticalArrangement = Arrangement.spacedBy(10.dp)) {
Text(stringResource(R.string.hand_camera_prompt_body), fontSize = 14.sp)
if (choose) listOf(
Triple(TrackingCameras.REAR, R.string.hand_camera_rear, R.string.hand_camera_rear_hint),
Triple(TrackingCameras.FRONT, R.string.hand_camera_front, R.string.hand_camera_front_hint),
).forEach { (id, label, hint) ->
Row(
verticalAlignment = Alignment.CenterVertically,
modifier = Modifier.fillMaxWidth().clip(RoundedCornerShape(8.dp))
.selectable(side == id, role = Role.RadioButton) { side = id }
.testTag("hand-camera-$id")
.padding(vertical = 4.dp),
) {
RadioButton(side == id, onClick = null,
colors = RadioButtonDefaults.colors(selectedColor = pal.signal, unselectedColor = colors.onSurfaceVariant))
Column(Modifier.padding(start = 10.dp)) {
Text(stringResource(label), fontSize = 15.sp, color = colors.onBackground)
Text(stringResource(hint), fontSize = 13.sp, color = colors.onSurfaceVariant)
}
}
}
}
},
confirmButton = {
TextButton(onClick = { onAllow(if (choose) side else null) }, modifier = Modifier.testTag("hand-camera-allow")) {
Text(stringResource(R.string.hand_camera_allow))
}
},
dismissButton = {
TextButton(onClick = onNotNow, modifier = Modifier.testTag("hand-camera-not-now")) { Text(stringResource(R.string.hand_camera_not_now)) }
},
)
}
@Composable
internal fun HandCameraRows(host: MenuHost, track: (String) -> Modifier, inSession: Boolean) {
if (!HandTracking.supported) return
val context = LocalContext.current
val lifecycle = LocalLifecycleOwner.current.lifecycle
DisposableEffect(lifecycle) {
HandTracking.load(context)
val watch = LifecycleEventObserver { _, event -> if (event == Lifecycle.Event.ON_RESUME) HandTracking.refreshPermission(context) }
lifecycle.addObserver(watch)
onDispose { lifecycle.removeObserver(watch) }
}
val s by HandTracking.settings.collectAsState()
if (!s.usable) return
var denied by remember { mutableStateOf(false) }
val ask = rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
HandTracking.refreshPermission(context)
denied = !granted
}
ToggleRow(host, "hand-camera", stringResource(R.string.hand_camera_setting), stringResource(R.string.hand_camera_setting_hint),
s.enabled, chipModifier = track("hand-camera")) { on ->
HandTracking.setEnabled(context, on)
if (on && !s.permitted) ask.launch(Manifest.permission.CAMERA)
}
if (!s.enabled) return
if (s.sides.size > 1) ChoiceRow(host, "hand-camera-side", stringResource(R.string.hand_camera_choice), null,
listOf(TrackingCameras.REAR to stringResource(R.string.hand_camera_rear), TrackingCameras.FRONT to stringResource(R.string.hand_camera_front)),
s.side ?: TrackingCameras.REAR, chipModifier = track("hand-camera-side")) { HandTracking.setSide(context, it) }
when {
!s.permitted && denied -> ActionRow(stringResource(R.string.hand_camera_access), stringResource(R.string.hand_camera_access_denied),
stringResource(R.string.hand_camera_open_settings), {
context.startActivity(Intent(Settings.ACTION_APPLICATION_DETAILS_SETTINGS, Uri.fromParts("package", context.packageName, null))
.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK))
})
!s.permitted -> ActionRow(stringResource(R.string.hand_camera_access), stringResource(R.string.hand_camera_access_hint),
stringResource(R.string.hand_camera_allow_button), { ask.launch(Manifest.permission.CAMERA) })
!inSession -> ActionRow(stringResource(R.string.hand_camera_placement), stringResource(R.string.hand_camera_placement_hint),
stringResource(R.string.hand_camera_check), { HandTracking.requestCheck() })
}
}
@Composable
private fun PlacementCheck(settings: HandTracking.Settings, play: Boolean, onPlay: () -> Unit, onClose: () -> Unit) {
val context = LocalContext.current
val side = settings.side
val camera = remember(side) { HandCamera(context, null, preview = true) }
DisposableEffect(camera) {
camera.start(side)
onDispose { camera.stop() }
}
val view by camera.view.collectAsState()
val failed by camera.failed.collectAsState()
val front = view?.front ?: (TrackingCameras.pick(HandTracking.cameras(context), side)?.front == true)
val advice = when {
failed -> null
else -> view?.advice ?: Advice.STARTING
}
val adviceText = stringResource(advice?.let(::adviceLabel) ?: R.string.hand_camera_failed)
val start by rememberUpdatedState(onPlay)
if (play) LaunchedEffect(advice == Advice.GOOD) {
if (advice == Advice.GOOD) {
delay(GOOD_HOLD_MS)
start()
}
}
if (!front) SpokenAdvice(advice, adviceText)
Dialog(onDismissRequest = onClose, properties = DialogProperties(usePlatformDefaultWidth = false, decorFitsSystemWindows = false)) {
PlacementCheckContent(view, front, advice, adviceText, play, switchable = settings.sides.size > 1,
onSwitch = { HandTracking.setSide(context, if (front) TrackingCameras.REAR else TrackingCameras.FRONT) },
onPlay = onPlay, onClose = onClose)
}
}
@Composable
internal fun PlacementCheckContent(
view: HandCamera.View?,
front: Boolean,
advice: Advice?,
adviceText: String,
play: Boolean,
switchable: Boolean,
onSwitch: () -> Unit,
onPlay: () -> Unit,
onClose: () -> Unit,
) {
val colors = MaterialTheme.colorScheme
val pal = LocalPalette.current
Row(
horizontalArrangement = Arrangement.spacedBy(24.dp),
modifier = Modifier.fillMaxSize().background(pal.background).padding(24.dp).testTag("hand-camera-check"),
) {
Box(Modifier.weight(1.3f).fillMaxHeight(), contentAlignment = Alignment.Center) { CameraPreview(view) }
Column(Modifier.weight(1f).fillMaxHeight().verticalScroll(rememberScrollState()), verticalArrangement = Arrangement.spacedBy(14.dp)) {
Text(stringResource(R.string.hand_camera_check_title), fontSize = 22.sp, fontWeight = FontWeight.Bold, color = colors.onBackground)
Text(stringResource(if (front) R.string.hand_camera_how_front else R.string.hand_camera_how_rear), fontSize = 15.sp, color = colors.onSurfaceVariant)
Text(adviceText, fontSize = 19.sp, fontWeight = FontWeight.SemiBold,
color = if (advice == Advice.GOOD) pal.good else colors.onBackground, modifier = Modifier.testTag("hand-camera-advice"))
Spacer(Modifier.heightIn(min = 8.dp))
Row(horizontalArrangement = Arrangement.spacedBy(10.dp)) {
if (switchable) SecondaryButton(stringResource(R.string.hand_camera_switch), compact = true, onClick = onSwitch)
if (play) {
SecondaryButton(stringResource(R.string.hand_camera_skip), compact = true, modifier = Modifier.testTag("hand-camera-skip"), onClick = onPlay)
SecondaryButton(stringResource(R.string.common_cancel), compact = true, onClick = onClose)
} else {
SecondaryButton(stringResource(R.string.hand_camera_done), compact = true, modifier = Modifier.testTag("hand-camera-done"), onClick = onClose)
}
}
}
}
}
@Composable
private fun CameraPreview(view: HandCamera.View?) {
val pal = LocalPalette.current
val shape = RoundedCornerShape(16.dp)
if (view == null) {
Box(Modifier.fillMaxWidth().aspectRatio(4f / 3f).clip(shape).background(pal.surface), contentAlignment = Alignment.Center) {
CircularProgressIndicator(color = pal.signal, modifier = Modifier.size(36.dp))
}
return
}
val image = remember(view) { view.image.asImageBitmap() }
Box(Modifier.aspectRatio(view.image.width.toFloat() / view.image.height).clip(shape)) {
Image(image, null, contentScale = ContentScale.FillBounds,
modifier = Modifier.fillMaxSize().graphicsLayer { if (view.front) scaleX = -1f })
Canvas(Modifier.fillMaxSize()) { drawSkeleton(view.landmarks, view.front, pal.signal, pal.good) }
}
}
private fun DrawScope.drawSkeleton(marks: List<Landmark>?, mirrored: Boolean, line: Color, hand: Color) {
if (marks == null || marks.size <= BodyPose.RIGHT_HIP) return
fun at(i: Int) = Offset((if (mirrored) 1f - marks[i].x else marks[i].x) * size.width, marks[i].y * size.height)
fun seen(i: Int) = marks[i].visibility >= BodyPose.MIN_VISIBILITY
val width = size.minDimension / 120f
for ((a, b) in BONES) if (seen(a) && seen(b)) drawLine(line, at(a), at(b), strokeWidth = width, cap = StrokeCap.Round)
for (i in JOINTS) if (seen(i)) drawCircle(line, width * 1.2f, at(i))
for (wrist in listOf(BodyPose.LEFT_WRIST, BodyPose.RIGHT_WRIST)) {
if (!seen(wrist)) continue
val grip = (at(wrist) + at(wrist + 2) + at(wrist + 4)) / 3f
drawCircle(hand.copy(alpha = 0.35f), width * 5f, grip)
drawCircle(hand, width * 2.2f, grip)
}
}
@Composable
private fun SpokenAdvice(advice: Advice?, text: String) {
val context = LocalContext.current
var ready by remember { mutableStateOf(false) }
val speech = remember {
var engine: TextToSpeech? = null
engine = TextToSpeech(context.applicationContext) { status ->
if (status == TextToSpeech.SUCCESS) {
engine?.language = Locale.getDefault()
ready = true
}
}
engine
}
DisposableEffect(speech) { onDispose { speech.shutdown() } }
LaunchedEffect(advice, ready) {
if (!ready || advice == null || advice == Advice.STARTING) return@LaunchedEffect
delay(SPEAK_AFTER_MS)
while (true) {
speech.speak(text, TextToSpeech.QUEUE_FLUSH, null, "hand-camera")
if (advice == Advice.GOOD) break
delay(SPEAK_AGAIN_MS)
}
}
}
private fun adviceLabel(advice: Advice): Int = when (advice) {
Advice.STARTING -> R.string.hand_camera_starting
Advice.STAND_UP -> R.string.hand_camera_stand_up
Advice.UPRIGHT -> R.string.hand_camera_upright
Advice.NOT_SEEN -> R.string.hand_camera_not_seen
Advice.MOVE_BACK -> R.string.hand_camera_move_back
Advice.MOVE_CLOSER -> R.string.hand_camera_move_closer
Advice.SHOW_HANDS -> R.string.hand_camera_show_hands
Advice.GOOD -> R.string.hand_camera_good
}
private val BONES = listOf(
BodyPose.LEFT_SHOULDER to BodyPose.RIGHT_SHOULDER,
BodyPose.LEFT_SHOULDER to BodyPose.LEFT_ELBOW, BodyPose.LEFT_ELBOW to BodyPose.LEFT_WRIST,
BodyPose.RIGHT_SHOULDER to BodyPose.RIGHT_ELBOW, BodyPose.RIGHT_ELBOW to BodyPose.RIGHT_WRIST,
BodyPose.LEFT_SHOULDER to BodyPose.LEFT_HIP, BodyPose.RIGHT_SHOULDER to BodyPose.RIGHT_HIP, BodyPose.LEFT_HIP to BodyPose.RIGHT_HIP,
BodyPose.NOSE to BodyPose.LEFT_EYE, BodyPose.NOSE to BodyPose.RIGHT_EYE,
BodyPose.LEFT_EYE to BodyPose.LEFT_EAR, BodyPose.RIGHT_EYE to BodyPose.RIGHT_EAR,
)
private val JOINTS = listOf(BodyPose.NOSE, BodyPose.LEFT_SHOULDER, BodyPose.RIGHT_SHOULDER, BodyPose.LEFT_ELBOW, BodyPose.RIGHT_ELBOW,
BodyPose.LEFT_HIP, BodyPose.RIGHT_HIP)
private const val GOOD_HOLD_MS = 1200L
private const val SPEAK_AFTER_MS = 1200L
private const val SPEAK_AGAIN_MS = 6000L
@@ -23,6 +23,7 @@ import androidx.compose.foundation.interaction.collectIsHoveredAsState
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.BoxWithConstraints
import androidx.compose.foundation.layout.BoxWithConstraintsScope
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
@@ -42,6 +43,7 @@ import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.foundation.verticalScroll
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.Close
import androidx.compose.material.icons.filled.KeyboardArrowDown
import androidx.compose.material.icons.filled.KeyboardArrowUp
import androidx.compose.ui.platform.testTag
import kotlinx.coroutines.Job
@@ -52,6 +54,7 @@ import androidx.compose.runtime.Composable
import androidx.compose.runtime.CompositionLocalProvider
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.compositionLocalOf
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateMapOf
import androidx.compose.runtime.mutableStateOf
@@ -81,6 +84,7 @@ import androidx.compose.ui.geometry.RoundRect
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.Path
import androidx.compose.ui.graphics.drawscope.Stroke
import androidx.compose.ui.graphics.drawscope.scale
import androidx.compose.ui.graphics.graphicsLayer
import androidx.compose.ui.graphics.lerp
import androidx.compose.ui.input.InputMode
@@ -102,6 +106,7 @@ import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.Dp
import androidx.compose.ui.unit.IntOffset
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.isSpecified
import androidx.compose.ui.unit.sp
import androidx.compose.ui.util.lerp
import com.droiddeck.launcher.R
@@ -138,6 +143,15 @@ internal class SheetBeat(private val pieces: List<Animatable<Float, AnimationVec
internal val LocalSheetBeat = staticCompositionLocalOf<SheetBeat?> { null }
internal val LocalSheetChipWidth = compositionLocalOf { Dp.Unspecified }
internal val LocalPlayRect = compositionLocalOf<Rect?> { null }
private val ChipGap = 6.dp
internal fun sheetChipWidth(row: Dp, count: Int): Dp =
if (count > 0 && row / count >= 96.dp) (row - ChipGap * (count - 1)) / count else 0.dp
@Composable
internal fun Modifier.sheetPiece(): Modifier {
val beat = LocalSheetBeat.current ?: return this
@@ -173,6 +187,8 @@ internal fun LiftHero(
onOpenChange: (Boolean) -> Unit = {},
start: Dp = 44.dp,
bottom: Dp = 44.dp,
drop: DropSheetState? = null,
overlay: @Composable BoxWithConstraintsScope.(lifted: () -> Float) -> Unit = {},
) {
val density = LocalDensity.current
val inputMode = LocalInputModeManager.current
@@ -306,7 +322,7 @@ internal fun LiftHero(
val lipFocus = remember { FocusRequester() }
val settingsLabel = stringResource(R.string.drawer_page_settings)
var lipFocused by remember { mutableStateOf(false) }
val lipWanted = !open && (!shown || lipBack) && (playFocused || playHovered || touchOnly || lipFocused)
val lipWanted = !open && (!shown || lipBack) && (playFocused || playHovered || touchOnly || lipFocused) && drop?.let { it.open || it.shown } != true
LaunchedEffect(lipWanted) {
if (lipWanted) {
if (!lipBack) delay(Motion.ms(160).toLong())
@@ -385,18 +401,35 @@ internal fun LiftHero(
}
BoxWithConstraints(
modifier.fillMaxSize().onGloballyPositioned { heroAt = it.boundsInRoot().topLeft }
.pointerInput(Unit) {
.pointerInput(drop) {
val tracker = androidx.compose.ui.input.pointer.util.VelocityTracker()
var decided = false
val dropFlick = 600.dp.toPx()
detectVerticalDragGestures(
onDragStart = { tracker.resetTracking(); decided = false; drag.engaged = false },
onDragEnd = { if (drag.engaged) settleDrag(tracker.calculateVelocity().y) },
onDragCancel = { if (drag.engaged) settleDrag(0f) },
onDragEnd = {
if (drag.engaged) settleDrag(tracker.calculateVelocity().y)
else if (drop?.engaged == true) drop.settle(tracker.calculateVelocity().y, dropFlick)
},
onDragCancel = {
if (drag.engaged) settleDrag(0f)
else if (drop?.engaged == true) drop.settle(0f, dropFlick)
},
) { change, amount ->
if (launching) return@detectVerticalDragGestures
if (!decided) {
decided = true
if (if (open) amount > 0f else amount < 0f) beginDrag()
when {
drop != null && (drop.open || drop.shown) -> if (drop.open && amount < 0f) drop.beginDrag()
open || shown || amount < 0f -> if (if (open) amount > 0f else amount < 0f) beginDrag()
drop != null && amount > 0f -> drop.beginDrag()
}
}
if (drop?.engaged == true) {
change.consume()
tracker.addPosition(change.uptimeMillis, change.position)
drop.drag(amount)
return@detectVerticalDragGestures
}
if (!drag.engaged) return@detectVerticalDragGestures
change.consume()
@@ -413,9 +446,15 @@ internal fun LiftHero(
val openTop = sheetTop(h, buttonH, with(density) { 16.dp.toPx() }, with(density) { 3.dp.toPx() })
val raise = restRect?.let { (openTop + with(density) { 3.dp.toPx() } - buttonH / 2f) - it.top } ?: 0f
travelPx[0] = (h - openTop).coerceAtLeast(1f)
val pill = restRect?.translate(0f, raise)
val chipsX = with(density) {
if (narrow || pill == null) 16.dp.toPx() else pill.right + 20.dp.toPx() - inset
}
val chipWidth = with(density) { sheetChipWidth((w - 2 * inset - chipsX - 64.dp.toPx()).coerceAtLeast(0f).toDp(), tabs.size) }
val foot = h
Box(Modifier.fillMaxSize().clipToBounds().graphicsLayer { val k = 1f - 0.05f * sink.value; scaleX = k; scaleY = k }) { background() }
val wallSink = { maxOf(sink.value, drop?.sink?.value ?: 0f) }
Box(Modifier.fillMaxSize().clipToBounds().graphicsLayer { val k = 1f - 0.05f * wallSink(); scaleX = k; scaleY = k }) { background() }
scrim { sink.value }
if (shown || open) Box(
Modifier.fillMaxSize()
@@ -432,7 +471,6 @@ internal fun LiftHero(
val rimPath = remember { Path() }
val corner = with(density) { SheetCorner.toPx() }
val rim = with(density) { 1.4.dp.toPx() }
val pill = restRect?.translate(0f, raise)
Canvas(Modifier.fillMaxSize()) {
val f = fold.value
val ft = foldTop.value
@@ -459,9 +497,6 @@ internal fun LiftHero(
drawPath(rimPath, rimColor, style = Stroke(rim))
}
}
val chipsX = with(density) {
if (narrow || pill == null) 16.dp.toPx() else pill.right + 20.dp.toPx() - sheetLeft
}
val chipsY = with(density) { if (narrow && pill != null) (pill.bottom - openTop) + 10.dp.toPx() else ChipsTop.toPx() }
val rowsY = with(density) { maxOf(RowsTop.toPx(), chipsY + 52.dp.toPx()) }
val sheetW = sheetRight - sheetLeft
@@ -537,13 +572,13 @@ internal fun LiftHero(
}
Column(Modifier.align(Alignment.BottomStart).padding(start = start, bottom = bottom, end = 20.dp)) {
Box(Modifier.graphicsLayer { alpha = 1f - sink.value }) { words() }
Box(Modifier.graphicsLayer { alpha = 1f - wallSink() }) { words() }
Spacer(Modifier.height(18.dp))
Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(10.dp)) {
Box(
Modifier
.onGloballyPositioned { rest = it.boundsInRoot() }
.graphicsLayer { translationY = raise * ride.value }
.graphicsLayer { translationY = raise * ride.value; alpha = 1f - (drop?.sink?.value ?: 0f) }
.onFocusChanged {
if (it.hasFocus && leftForPlay[0]) { leftForPlay[0] = false; if (openState.value && !launchingState.value) close() }
playFocused = it.hasFocus
@@ -570,7 +605,7 @@ internal fun LiftHero(
val besideShown = !(open || shown)
Row(
verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(10.dp),
modifier = Modifier.graphicsLayer { alpha = 1f - sink.value }
modifier = Modifier.graphicsLayer { alpha = 1f - wallSink() }
.focusProperties { canFocus = besideShown },
) { if (besideShown || sink.value < 0.99f) beside() }
}
@@ -585,7 +620,7 @@ internal fun LiftHero(
Modifier
.offset { IntOffset((r0.center.x - handleW / 2f).roundToInt(), (r0.bottom + raise * ride.value + (lipShow.value - 1f) * 8.dp.toPx()).roundToInt()) }
.onSizeChanged { handleW = it.width.toFloat() }
.graphicsLayer { alpha = lipShow.value * (1f - lipGone.value) }
.graphicsLayer { alpha = lipShow.value * (1f - lipGone.value) * (1f - (drop?.sink?.value ?: 0f)) }
.heightIn(min = 44.dp).testTag("sheet-open")
.semantics { contentDescription = settingsLabel }
.then(
@@ -599,14 +634,15 @@ internal fun LiftHero(
}
}
}
CompositionLocalProvider(LocalSheetChipWidth provides chipWidth, LocalPlayRect provides restRect) { overlay { sink.value } }
}
}
@Composable
private fun Handle(height: Dp) {
internal fun Handle(height: Dp, above: Boolean = false) {
val colors = MaterialTheme.colorScheme
val pal = LocalPalette.current
val shape = RoundedCornerShape(bottomStart = 12.dp, bottomEnd = 12.dp)
val shape = if (above) RoundedCornerShape(topStart = 12.dp, topEnd = 12.dp) else RoundedCornerShape(bottomStart = 12.dp, bottomEnd = 12.dp)
val still = Motion.scale == 0f
val bob = androidx.compose.animation.core.rememberInfiniteTransition(label = "handle")
val lift by bob.animateFloat(
@@ -625,18 +661,18 @@ private fun Handle(height: Dp) {
Row(
verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(4.dp),
modifier = Modifier.height(height).clip(shape).background(colors.surface)
.drawBorderSidesAndFoot(pal.line, 12.dp)
.drawBorderSidesAndFoot(pal.line, 12.dp, flip = above)
.padding(start = 14.dp, end = 10.dp),
) {
Text(stringResource(R.string.drawer_page_settings), fontSize = 14.sp, fontWeight = FontWeight.SemiBold, color = colors.onSurfaceVariant, maxLines = 1)
Icon(
Icons.Filled.KeyboardArrowUp, null, tint = colors.onSurfaceVariant,
modifier = Modifier.size(16.dp).graphicsLayer { translationY = if (still) 0f else -3.dp.toPx() * lift },
if (above) Icons.Filled.KeyboardArrowDown else Icons.Filled.KeyboardArrowUp, null, tint = colors.onSurfaceVariant,
modifier = Modifier.size(16.dp).graphicsLayer { translationY = if (still) 0f else (if (above) 3.dp else -3.dp).toPx() * lift },
)
}
}
private fun Modifier.drawBorderSidesAndFoot(color: Color, corner: Dp): Modifier =
private fun Modifier.drawBorderSidesAndFoot(color: Color, corner: Dp, flip: Boolean = false): Modifier =
this.drawWithContent {
drawContent()
val c = corner.toPx()
@@ -649,30 +685,40 @@ private fun Modifier.drawBorderSidesAndFoot(color: Color, corner: Dp): Modifier
arcTo(Rect(size.width - 2 * c, size.height - 2 * c + sw / 2, size.width - sw / 2, size.height - sw / 2), 90f, -90f, false)
lineTo(size.width - sw / 2, 0f)
}
drawPath(p, color, style = Stroke(sw))
if (flip) scale(1f, -1f) { drawPath(p, color, style = Stroke(sw)) } else drawPath(p, color, style = Stroke(sw))
}
@OptIn(ExperimentalComposeUiApi::class)
@Composable
internal fun SheetChips(labels: List<String>, selected: Int, focus: List<FocusRequester>? = null, onPick: (Int) -> Unit) {
internal fun SheetChips(
labels: List<String>,
selected: Int,
focus: List<FocusRequester>? = null,
width: Dp = Dp.Unspecified,
onPick: (Int) -> Unit,
) {
val pal = LocalPalette.current
val bounds = remember { mutableStateMapOf<Int, Rect>() }
val tint = rememberUpdatedState(pal.signal)
val chips = focus ?: remember(labels.size) { List(labels.size) { FocusRequester() } }
var within by remember { mutableStateOf(false) }
LaunchedEffect(selected) { if (within) runCatching { chips[selected].requestFocus() } }
BoxWithConstraints(Modifier.fillMaxWidth().onFocusChanged { within = it.hasFocus }) {
val equal = maxWidth / labels.size.coerceAtLeast(1) >= 96.dp
BoxWithConstraints(
Modifier.fillMaxWidth().onFocusChanged { within = it.hasFocus },
contentAlignment = Alignment.CenterStart,
) {
val equal = !width.isSpecified && maxWidth / labels.size.coerceAtLeast(1) >= 96.dp
val least = if (width.isSpecified) maxOf(width, 64.dp) else 64.dp
val scroll = rememberScrollState()
Box(if (equal) Modifier.fillMaxWidth() else Modifier.edgeFade(scroll).horizontalScroll(scroll).padding(3.dp)) {
Row(
horizontalArrangement = Arrangement.spacedBy(6.dp),
horizontalArrangement = Arrangement.spacedBy(ChipGap),
modifier = (if (equal) Modifier.fillMaxWidth() else Modifier)
.focusProperties { enter = { chips.getOrNull(selected) ?: FocusRequester.Default } }
.focusGroup(),
) {
labels.forEachIndexed { i, label ->
val m = (if (equal) Modifier.weight(1f) else Modifier.widthIn(min = 64.dp))
val m = (if (equal) Modifier.weight(1f) else Modifier.widthIn(min = least))
.onGloballyPositioned { bounds[i] = it.boundsInParent() }
.focusRequester(chips[i])
SheetChip(label, i == selected, m.onFocusChanged { if (it.isFocused && i != selected) onPick(i) }) { onPick(i) }
@@ -2,6 +2,7 @@ package com.droiddeck.launcher.ui
internal enum class SessionDrawerPage(val firstControl: String) {
CONTROLLER("touch"),
GLASSES("glasses-game"),
SECOND_SCREEN("second-screen:none"),
DISPLAY("hud"),
EFFECTS("look"),
@@ -10,8 +11,10 @@ internal enum class SessionDrawerPage(val firstControl: String) {
SESSION("suspend"),
}
internal fun sessionDrawerPages(hasSecondScreen: Boolean): List<SessionDrawerPage> =
SessionDrawerPage.entries.filter { hasSecondScreen || it != SessionDrawerPage.SECOND_SCREEN }
internal fun sessionDrawerPages(hasSecondScreen: Boolean, hasGlasses: Boolean = false): List<SessionDrawerPage> =
SessionDrawerPage.entries.filter {
(hasSecondScreen || it != SessionDrawerPage.SECOND_SCREEN) && (hasGlasses || it != SessionDrawerPage.GLASSES)
}
internal fun List<SessionDrawerPage>.step(current: SessionDrawerPage, direction: Int): SessionDrawerPage {
val index = indexOf(current).coerceAtLeast(0)
@@ -149,6 +149,7 @@ import java.util.Locale
private val SessionDrawerPage.title: Int get() = when (this) {
SessionDrawerPage.CONTROLLER -> R.string.drawer_page_controller
SessionDrawerPage.GLASSES -> R.string.drawer_page_glasses
SessionDrawerPage.SECOND_SCREEN -> R.string.drawer_second_screen
SessionDrawerPage.DISPLAY -> R.string.drawer_page_display
SessionDrawerPage.VR -> R.string.vr_title
@@ -159,6 +160,7 @@ private val SessionDrawerPage.title: Int get() = when (this) {
private val SessionDrawerPage.icon get() = when (this) {
SessionDrawerPage.CONTROLLER -> Icons.Outlined.SportsEsports
SessionDrawerPage.GLASSES -> GlassesIcon
SessionDrawerPage.SECOND_SCREEN -> Icons.Outlined.ScreenShare
SessionDrawerPage.DISPLAY -> Icons.Outlined.DesktopWindows
SessionDrawerPage.VR -> VrHeadsetIcon
@@ -211,6 +213,8 @@ class DrawerActions(
val cardboard: Cardboard = Cardboard.OFF,
val cardboardSensor: Boolean = false,
val steamVr: SteamVr.Status? = null,
val glassesPresent: Boolean = false,
val glassesOutput: Boolean = false,
val immersiveSteamVr: Boolean = false,
val textureAnisotropy: Int? = 0,
val textureLodBias: String? = TextureFiltering.LOD_BIAS_OFF,
@@ -247,6 +251,7 @@ class DrawerActions(
val onEffects: (ScreenEffects) -> Unit = {},
val onCardboard: (Cardboard) -> Unit = {},
val onCardboardRecenter: () -> Unit = {},
val onGlassesRecenter: () -> Unit = {},
val onSteamVr: (String) -> Unit = {},
val onTextureAnisotropy: (Int?) -> Unit = {},
val onTextureLodBias: (String?) -> Unit = {},
@@ -295,7 +300,7 @@ class DrawerActions(
@OptIn(ExperimentalComposeUiApi::class)
@Composable
internal fun SessionDrawer(open: Boolean, requestedPage: SessionDrawerPage, controllerActive: Boolean, onPageChange: (SessionDrawerPage) -> Unit, a: DrawerActions) {
val pages = sessionDrawerPages(a.steam && a.secondScreenDisplays.isNotEmpty())
val pages = sessionDrawerPages(a.steam && a.secondScreenDisplays.isNotEmpty(), a.glassesPresent)
val page = requestedPage.takeIf { it in pages } ?: SessionDrawerPage.CONTROLLER
LaunchedEffect(requestedPage, pages) { if (page != requestedPage) onPageChange(page) }
val colors = MaterialTheme.colorScheme
@@ -465,7 +470,7 @@ internal fun SessionDrawer(open: Boolean, requestedPage: SessionDrawerPage, cont
}
SessionDrawerPage.VR -> {
SteamVrGroup(a) { key -> focus.track(page, key) }
CardboardGroup(host, a) { key -> focus.track(page, key) }
if (!a.glassesOutput) CardboardGroup(host, a) { key -> focus.track(page, key) }
}
SessionDrawerPage.CONTROLLER -> {
SettingsGroup(stringResource(R.string.drawer_page_controller)) {
@@ -532,6 +537,12 @@ internal fun SessionDrawer(open: Boolean, requestedPage: SessionDrawerPage, cont
SessionDrawerPage.SECOND_SCREEN -> {
SecondScreenGroup(host, a) { key -> focus.track(page, key) }
}
SessionDrawerPage.GLASSES -> {
val (g, s) = rememberGlasses()
GlassesGroups(host, g, s, { key -> focus.track(page, key) }, onRecenter = a.onGlassesRecenter,
outputActive = a.glassesOutput, steamVrRunning = a.steamVr == SteamVr.Status.RUNNING,
steamVr = { SteamVrRow(a) })
}
SessionDrawerPage.GAMES -> {
if (a.steam) SelectedGameProfileGroup(host, a) { key -> focus.track(page, key) }
Text(stringResource(R.string.fex_next_launch), fontSize = 12.sp,
@@ -954,6 +965,19 @@ private fun SteamVrGroup(a: DrawerActions, track: (String) -> Modifier) {
}
}
@Composable
private fun SteamVrRow(a: DrawerActions) {
val vrLabel = stringResource(R.string.cardboard_steamvr)
when (a.steamVr) {
SteamVr.Status.NOT_INSTALLED -> ActionRow(vrLabel, null, stringResource(R.string.cardboard_steamvr_install), { a.onSteamVr("install") })
SteamVr.Status.INSTALLING -> SettingsRow(vrLabel, stringResource(R.string.cardboard_steamvr_installing)) {}
SteamVr.Status.READY -> ActionRow(vrLabel, null, stringResource(R.string.cardboard_steamvr_start), { a.onSteamVr("start") })
SteamVr.Status.RUNNING -> ActionRow(vrLabel, null, stringResource(R.string.cardboard_steamvr_stop), { a.onSteamVr("stop") })
SteamVr.Status.UNAVAILABLE -> SettingsRow(vrLabel, stringResource(R.string.cardboard_steamvr_unavailable)) {}
null -> {}
}
}
@Composable
private fun SecondScreenGroup(host: MenuHost, a: DrawerActions, track: (String) -> Modifier) {
val colors = MaterialTheme.colorScheme
@@ -1,6 +1,7 @@
package com.droiddeck.launcher.ui
import androidx.compose.ui.platform.testTag
import androidx.compose.foundation.layout.fillMaxWidth
import com.droiddeck.launcher.R
import androidx.compose.ui.res.stringResource
import androidx.compose.animation.core.FastOutSlowInEasing
@@ -104,8 +105,24 @@ internal fun SteamHome(s: FrontEndState, a: FrontEndActions, modifier: Modifier)
}
},
)
val (glasses, glassesSettings) = rememberGlasses()
val headset = glasses.present && glasses.canControl
val drop = rememberDropSheetState()
LaunchedEffect(headset) { if (!headset && (drop.open || drop.shown)) drop.reset() }
LiftHero(
key = "steam", tabs = tabs,
drop = drop.takeIf { headset },
overlay = { lifted ->
if (headset) DropSheet(
drop, glassesTabs(glasses), covered = lifted,
button = { enabled ->
PrimaryButton(stringResource(R.string.games_play_steamvr), enabled = !s.busy && glasses.usable, large = true, icon = Icons.Filled.PlayArrow, fill = true,
modifier = Modifier.fillMaxWidth().testTag("play-steamvr")) { if (enabled) { drop.open = false; a.onPlaySteamVr() } }
},
) { tab ->
GlassesGroups(rememberMenuHost(), glasses, glassesSettings, inSession = false, tab = tab)
}
},
onOpen = a.onSteamSettings, onPrimary = a.onPlay,
modifier = modifier.clipToBounds(),
start = if (narrow) 20.dp else 44.dp, bottom = if (narrow) 20.dp else 44.dp,
@@ -41,7 +41,6 @@ object CompositorHost {
resumeVsync()
return false
}
WaylandCompositor.nativeSetOutputRefreshRate(refreshHz)
com.droiddeck.launcher.gpu.TpUbwcHint.applyToProcess()
WaylandCompositor.nativeStartWithSurface(
surface, xdgRuntimeDir, driverPath, libraryName, nativeLibDir,
@@ -55,6 +54,13 @@ object CompositorHost {
private fun pace(refreshHz: Float, fpsLimit: Int) {
WaylandCompositor.nativeSetFpsLimit(fpsLimit)
WaylandCompositor.nativeSetLayerFrameRate(if (fpsLimit > 0) fpsLimit.toFloat() else refreshHz)
WaylandCompositor.nativeSetOutputRefreshRate(refreshHz)
SessionState.refreshHz = refreshHz
}
@Synchronized
fun updatePacing(refreshHz: Float, fpsLimit: Int) {
if (started) pace(refreshHz, fpsLimit)
}
@Synchronized
@@ -20,8 +20,8 @@ class HdrProbe(
)
object HdrSupport {
fun probe(context: Context): HdrProbe {
val display: Display? = if (Build.VERSION.SDK_INT >= 30) {
fun probe(context: Context, output: Display? = null): HdrProbe {
val display: Display? = output ?: if (Build.VERSION.SDK_INT >= 30) {
try { context.display } catch (e: Exception) { null }
} else {
@Suppress("DEPRECATION")
@@ -273,7 +273,7 @@ public final class WaylandCompositor {
public static native void nativeSetCardboard(boolean enabled, int source, float ipdM, float screenDistM,
float screenWidthM, float k1, float k2, float fovDeg,
float lensInset, float chroma);
float lensInset, float chroma, boolean mono, float displayHz);
public static native void nativeSetCardboardPose(float[] rotation, boolean live);