diff --git a/README.md b/README.md index 5ead27303..8a8a20746 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ On Android 14 and newer, before Steam launches, you must turn on **Disable child ## Smart glasses -Use **Setup → Smart glasses** for app-wide virtual-screen settings across DroidDeck menus, launchers, desktop sessions, and games. Supported VITURE, RayNeo, and XREAL connections provide 3DoF; Luma Ultra adds optional experimental 6DoF and stereo-assisted hand/Joy-Con tracking. Model support and SDK requirements differ: see the [smart-glasses support matrix](docs/smart-glasses-support.md) for supported devices, controls, setup, and limitations. +Use **Setup → Smart glasses** for app-wide virtual-screen settings across DroidDeck menus, launchers, desktop sessions, and games. Supported VITURE, RayNeo, XREAL, and Rokid connections provide 3DoF; Luma Ultra adds optional experimental 6DoF and stereo-assisted hand/Joy-Con tracking. Model support and SDK requirements differ: see the [smart-glasses support matrix](docs/smart-glasses-support.md) for supported devices, controls, setup, and limitations. ## Community @@ -49,7 +49,7 @@ Compatibility and performance vary by device; hardware validation is limited. De GPL-3.0. Runtime, shim, input, and controller work build on WinNative and Bannerlator (maxjivi05). Epic Games and GOG support build on [Phobos’s](https://github.com/phobos665) adaptation of [Legendary](https://github.com/legendary-gl/legendary) and [gogdl](https://github.com/Heroic-Games-Launcher/heroic-gogdl), respectively. The registry entries a GOG game's install script (`goggame-.script`) sets are read the way [Heroic Games Launcher](https://github.com/Heroic-Games-Launcher/HeroicGamesLauncher)'s GOG setup (GPL-3.0) reads them. LSFG frame generation is from the work of Camille LaVey and the [Eden](https://eden-emu.dev) emulator project, following [lsfg-vk](https://github.com/PancakeTAS/lsfg-vk), ported to WinNative and DroidDeck by [@maxjivi05](https://github.com/maxjivi05); it needs your own copy of [Lossless Scaling](https://store.steampowered.com/app/993090/) and ships none of its shaders. x86 AppImages, and any whose own runtime cannot unpack them, are unpacked with [uruntime](https://github.com/VHSgunzo/uruntime) by VHSgunzo (MIT), shipped unmodified with its licence, which also starts the arm64 emulator AppImages whose own runtime cannot run its extractor on Android. The Steam client's audio goes through [@The412Banner](https://github.com/The412Banner)'s [DirectAudio](https://github.com/The412Banner/directaudio) PulseAudio sink (LGPL-2.1-or-later), built from `tools/directaudio/sink` with its request size kept to its ring so clients are not asked to queue 100 ms ahead. mmap ring-buffer controller input was developed by ribbit384 ([WinNative #345](https://github.com/WinNative-Emu/WinNative/pull/345)). Why the x86-64 Steam client refused the arm64 client's sign-in was checked by reading Steam's saved token with the `ConnectCache` format from [deadlock-api-ingest](https://github.com/deadlock-api/deadlock-api-ingest). x86 programs that link OpenSSL 3 under FEX use the copy in the [Steam Linux Runtime 4.0](https://gitlab.steamos.cloud/steamrt/steamrt) platform. Hooks written with `WriteProcessMemory` into code [FEX](https://github.com/FEX-Emu/FEX) has translated go through `/proc//mem` again when the kernel refuses `process_vm_writev`, as [Wine](https://gitlab.winehq.org/wine/wine)'s `server/ptrace.c` wrote them before Wine 11. x86 Linux games under FEX get the same `/etc/asound.conf` that [pressure-vessel](https://gitlab.steamos.cloud/steamrt/steam-runtime-tools) writes into the Steam Linux Runtime, so ALSA plays through PulseAudio. Games render with [Mesa](https://gitlab.freedesktop.org/mesa/mesa) Turnip's own per-render-pass choice between tile memory and system memory, while the Steam client keeps `TU_DEBUG=sysmem`; [FEX](https://github.com/FEX-Emu/FEX)'s code disk cache was checked in its source and is already on through Proton. See [LICENSE](LICENSE). Steam and Proton belong to Valve Corporation; this project is not affiliated with Valve. -XR glasses support builds on WinNative's VITURE glasses work. VITURE protocol details were checked against [XRLinuxDriver](https://github.com/wheaney/XRLinuxDriver) by [@wheaney](https://github.com/wheaney), including the VITURE protocol header it ships, [ar-drivers-rs](https://github.com/badicsalex/ar-drivers-rs) by [@badicsalex](https://github.com/badicsalex) and the [go-macos/iokit VITURE package](https://pkg.go.dev/github.com/go-macos/iokit/viture); the optional VITURE SDK belongs to VITURE and is not included. RayNeo support adapts [@informalTechCode](https://github.com/informalTechCode)'s [RayNeo-Air-Series-3DoF-Scaffold](https://github.com/informalTechCode/RayNeo-Air-Series-3DoF-Scaffold) (Apache-2.0) and references [@verncat](https://github.com/verncat)'s [RayNeo-Air-3S-Pro-OpenVR](https://github.com/verncat/RayNeo-Air-3S-Pro-OpenVR) (MIT) and [Monado](https://gitlab.freedesktop.org/monado/monado) (BSL-1.0); see [docs/third-party/rayneo](docs/third-party/rayneo/README.md). XREAL 1S support references [XRLinuxDriver](https://github.com/wheaney/XRLinuxDriver) and [xrealOneDeviceKit](https://github.com/wheaney/xrealOneDeviceKit) by @wheaney, [xreal_one_driver](https://github.com/rohitsangwan01/xreal_one_driver) by [@rohitsangwan01](https://github.com/rohitsangwan01) (MIT) and [xr-tools](https://github.com/0xcaff/xr-tools) by [@0xcaff](https://github.com/0xcaff). The REDMAGIC glasses-display findings draw on [@mekhontsev](https://github.com/mekhontsev)'s [MagicDesk investigation](https://github.com/mekhontsev/magicdesk/blob/main/docs/xr-resolution-fix.md) and [@Coding-BR](https://github.com/Coding-BR)'s [decompiled Nubia kernel](https://github.com/Coding-BR/android_kernel_nubia_sm8850_qwjujube). VITURE, RayNeo, XREAL and REDMAGIC are trademarks of their owners; this project is not affiliated with them. +XR glasses support builds on WinNative's VITURE glasses work. VITURE protocol details were checked against [XRLinuxDriver](https://github.com/wheaney/XRLinuxDriver) by [@wheaney](https://github.com/wheaney), including the VITURE protocol header it ships, [ar-drivers-rs](https://github.com/badicsalex/ar-drivers-rs) by [@badicsalex](https://github.com/badicsalex) and the [go-macos/iokit VITURE package](https://pkg.go.dev/github.com/go-macos/iokit/viture); the optional VITURE SDK belongs to VITURE and is not included. RayNeo support adapts [@informalTechCode](https://github.com/informalTechCode)'s [RayNeo-Air-Series-3DoF-Scaffold](https://github.com/informalTechCode/RayNeo-Air-Series-3DoF-Scaffold) (Apache-2.0) and references [@verncat](https://github.com/verncat)'s [RayNeo-Air-3S-Pro-OpenVR](https://github.com/verncat/RayNeo-Air-3S-Pro-OpenVR) (MIT) and [Monado](https://gitlab.freedesktop.org/monado/monado) (BSL-1.0); see [docs/third-party/rayneo](docs/third-party/rayneo/README.md). XREAL 1S support references [XRLinuxDriver](https://github.com/wheaney/XRLinuxDriver) and [xrealOneDeviceKit](https://github.com/wheaney/xrealOneDeviceKit) by @wheaney, [xreal_one_driver](https://github.com/rohitsangwan01/xreal_one_driver) by [@rohitsangwan01](https://github.com/rohitsangwan01) (MIT) and [xr-tools](https://github.com/0xcaff/xr-tools) by [@0xcaff](https://github.com/0xcaff). Rokid support follows the Rokid protocol notes in [ar-drivers-rs](https://github.com/badicsalex/ar-drivers-rs) and @badicsalex's [Void Computing write-ups](https://voidcomputing.hu/blog/worse-better-prettier/), and [Monado](https://gitlab.freedesktop.org/monado/monado)'s Rokid driver. The REDMAGIC glasses-display findings draw on [@mekhontsev](https://github.com/mekhontsev)'s [MagicDesk investigation](https://github.com/mekhontsev/magicdesk/blob/main/docs/xr-resolution-fix.md) and [@Coding-BR](https://github.com/Coding-BR)'s [decompiled Nubia kernel](https://github.com/Coding-BR/android_kernel_nubia_sm8850_qwjujube). VITURE, RayNeo, XREAL, Rokid and REDMAGIC are trademarks of their owners; this project is not affiliated with them. Joy-Con VR controllers port the arm model of Google's [Google VR SDK for Unity](https://github.com/googlevr/gvr-unity-sdk) (Apache-2.0), adapt the hand skeleton of Valve's [OpenVR](https://github.com/ValveSoftware/openvr) `handskeletonsimulation` sample (BSD-3-Clause), which follows Moshi Turner's hand simulation in [Monado](https://gitlab.freedesktop.org/monado/monado), and use the Index controller offsets from Monado's `vive_poses.c` (BSL-1.0). Joy-Con behaviour was researched in Linux's [hid-nintendo](https://github.com/torvalds/linux/blob/master/drivers/hid/hid-nintendo.c) driver by Daniel J. Ogorchock, [@dekuNukem](https://github.com/dekuNukem)'s [Nintendo_Switch_Reverse_Engineering](https://github.com/dekuNukem/Nintendo_Switch_Reverse_Engineering), [SDL](https://github.com/libsdl-org/SDL) and the Android Open Source Project, and checked against [ALVR](https://github.com/alvr-org/ALVR), [OpenComposite](https://gitlab.com/znixian/OpenOVR), [PSMoveSteamVRBridge](https://github.com/HipsterSloth/PSMoveSteamVRBridge), [@F8LEFT](https://github.com/F8LEFT)'s [JoyConVr](https://github.com/F8LEFT/JoyConVr), [@krnelpan1c](https://github.com/krnelpan1c)'s [Daydream-Controller-SteamVR](https://github.com/krnelpan1c/Daydream-Controller-SteamVR), [@yoyobuae](https://github.com/yoyobuae)'s [joyconvr](https://github.com/yoyobuae/joyconvr), [Moonlight](https://github.com/moonlight-stream/moonlight-android) and [Dolphin](https://github.com/dolphin-emu/dolphin); see [docs/third-party/joycon](docs/third-party/joycon/README.md). Nintendo Switch and Joy-Con are trademarks of Nintendo; this project is not affiliated with Nintendo. diff --git a/app/src/main/java/com/droiddeck/launcher/glasses/Glasses.kt b/app/src/main/java/com/droiddeck/launcher/glasses/Glasses.kt index 89d5fcbde..947ed5fd2 100644 --- a/app/src/main/java/com/droiddeck/launcher/glasses/Glasses.kt +++ b/app/src/main/java/com/droiddeck/launcher/glasses/Glasses.kt @@ -50,6 +50,7 @@ data class GlassesState( val noSignal: Boolean = false, val sbsSupport: Int = SBS_UNKNOWN, val sbs90: Boolean = false, + val rates2d: List = listOf(60, 90, 120), val experimental: Boolean = false, val supportsScreen3dof: Boolean = false, val supports30Hz: Boolean = false, @@ -613,12 +614,13 @@ object Glasses { switching3d = modeState.pendingStereo, brightnessMax = m?.brightnessMax ?: 8, volumeMax = m?.volumeMax ?: 8, sunblock = if (m === XrealModel) it.sunblock else _settings.value.sunblock ?: it.sunblock, - hardwareControls = device?.vendorId == VitureProtocol.VENDOR_ID || m === XrealModel, + hardwareControls = device?.vendorId == VitureProtocol.VENDOR_ID || m === XrealModel || m is RokidModel, hasSunblock = m?.hasSunblock == true, diagonalFovDeg = m?.diagonalFovDeg ?: 46f, tracking = it.tracking && link != null, noSignal = noSignal, sbs90 = m?.sbs90 == true, + rates2d = m?.rates2d ?: listOf(60, 90, 120), experimental = m === XrealModel, supportsScreen3dof = link?.hasImu == true, supports30Hz = display?.let(::supports30Hz) == true, @@ -634,7 +636,8 @@ object Glasses { scheduleModeCheck() } - internal fun isGlassesVendor(vid: Int) = vid == VitureProtocol.VENDOR_ID || RayNeoModel.isVendor(vid) || vid == XrealModel.VENDOR_ID + internal fun isGlassesVendor(vid: Int) = vid == VitureProtocol.VENDOR_ID || RayNeoModel.isVendor(vid) || + vid == XrealModel.VENDOR_ID || vid == RokidModel.VENDOR_ID internal fun supports30Hz(display: Display): Boolean { val current = display.mode @@ -654,11 +657,13 @@ object Glasses { ?: VitureProtocol.model(device.productId, name) } else if (XrealModel.matches(device.vendorId, device.productId)) XrealModel - else RayNeoModel.model(device.vendorId, device.productId, name) + else RokidModel.model(device.vendorId, device.productId, name) + ?: RayNeoModel.model(device.vendorId, device.productId, name) } internal fun isGlassesDisplayName(name: String) = - name.contains("VITURE", ignoreCase = true) || RayNeoModel.isDisplay(name) || XrealModel.isDisplay(name) + name.contains("VITURE", ignoreCase = true) || RayNeoModel.isDisplay(name) || XrealModel.isDisplay(name) || + RokidModel.isDisplay(name) private fun findDisplay(usbPresent: Boolean): Display? { return selectGlassesDisplay(displays.getDisplays(DisplayManager.DISPLAY_CATEGORY_PRESENTATION).toList(), usbPresent) @@ -723,6 +728,8 @@ object Glasses { } else VitureLink.open(usb, device, linkListener(generation)) } else if (model === XrealModel) { XrealLink.open(app, xrealListener(generation)) + } else if (model is RokidModel) { + RokidLink.open(usb, device, rokidListener(generation)) } else { val baseKey = calibrationKey!! val gtUsb = RayNeoProtocol.isGt(device.vendorId, device.productId) @@ -790,6 +797,33 @@ object Glasses { } } + private fun rokidListener(generation: Long) = object : RokidLink.Listener { + override fun onReady() = post { + if (generation != linkGeneration) return@post + publish { it.copy(connected = true, controlError = null) } + updateImu() + applyMode(force = false) + } + override fun onValue(message: Int, value: Int) = post { + if (generation != linkGeneration) return@post + when (message) { + VitureProtocol.MSG_GET_DISPLAY -> glassesMode = value + VitureProtocol.MSG_BRIGHTNESS -> publish { it.copy(brightness = value) } + } + } + override fun onPose(raw: ByteArray, pose: FloatArray, timeNs: Long) { + if (generation != linkGeneration) return + imuPackets++ + probe?.invoke(raw, pose) + if (simulating.get() == null) deliver(pose, timeNs) + } + override fun onLost() = post { + if (generation != linkGeneration) return@post + closeLink() + refresh() + } + } + private fun sdkListener(generation: Long) = object : VitureSdkLink.Listener { override fun onSpatial(pose: FloatArray, timeNs: Long, valid: Boolean) { if (generation == linkGeneration) SpatialTracking.accept(pose, timeNs, generation, valid) diff --git a/app/src/main/java/com/droiddeck/launcher/glasses/GlassesHardware.kt b/app/src/main/java/com/droiddeck/launcher/glasses/GlassesHardware.kt index 0f6c28833..7362ebf44 100644 --- a/app/src/main/java/com/droiddeck/launcher/glasses/GlassesHardware.kt +++ b/app/src/main/java/com/droiddeck/launcher/glasses/GlassesHardware.kt @@ -8,6 +8,7 @@ interface GlassesModel { val volumeMax: Int val diagonalFovDeg: Float val sbs90: Boolean + val rates2d: List get() = listOf(60, 90, 120) val hasSunblock: Boolean get() = true val resolutions: List get() = emptyList() fun mode(hz: Int, sbs: Boolean, resolution: GlassesResolution?): Int = mode(hz, sbs) diff --git a/app/src/main/java/com/droiddeck/launcher/glasses/GlassesRendering.kt b/app/src/main/java/com/droiddeck/launcher/glasses/GlassesRendering.kt index 4cd7f0a6d..f8d6f4c5f 100644 --- a/app/src/main/java/com/droiddeck/launcher/glasses/GlassesRendering.kt +++ b/app/src/main/java/com/droiddeck/launcher/glasses/GlassesRendering.kt @@ -12,8 +12,8 @@ internal object GlassesRendering { fun refreshRates(settings: GlassesSettings, state: GlassesState): List = buildList { if (native30HzAvailable(settings, state)) add(30) add(60) - if (!state.sbs || state.sbs90) add(90) - if (!state.sbs) add(120) + if (state.sbs) { if (state.sbs90) add(90) } + else addAll(state.rates2d.filter { it > 60 }) } fun outputHz(settings: GlassesSettings, state: GlassesState): Int { @@ -21,6 +21,7 @@ internal object GlassesRendering { return when { requested == 30 -> if (native30HzAvailable(settings, state)) 30 else 60 state.sbs && !state.sbs90 -> 60 + !state.sbs && requested !in state.rates2d -> state.rates2d.filter { it <= requested }.maxOrNull() ?: 60 else -> requested } } diff --git a/app/src/main/java/com/droiddeck/launcher/glasses/RokidLink.kt b/app/src/main/java/com/droiddeck/launcher/glasses/RokidLink.kt new file mode 100644 index 000000000..dbf10a6e6 --- /dev/null +++ b/app/src/main/java/com/droiddeck/launcher/glasses/RokidLink.kt @@ -0,0 +1,148 @@ +package com.droiddeck.launcher.glasses + +import android.hardware.usb.UsbConstants +import android.hardware.usb.UsbDevice +import android.hardware.usb.UsbDeviceConnection +import android.hardware.usb.UsbEndpoint +import android.hardware.usb.UsbInterface +import android.hardware.usb.UsbManager +import android.os.SystemClock +import android.util.Log + +/** Controls are vendor control transfers; the HID interrupt endpoint streams IMU reports unprompted. */ +internal class RokidLink private constructor( + private val connection: UsbDeviceConnection, + private val usbInterface: UsbInterface, + private val endpoint: UsbEndpoint, + private val listener: Listener, +) : GlassesLink { + interface Listener { + fun onReady() + fun onValue(message: Int, value: Int) + fun onPose(raw: ByteArray, pose: FloatArray, timeNs: Long) + fun onLost() + } + + @Volatile private var running = true + @Volatile override var ready = false + private set + @Volatile private var tracking = false + private val controlLock = Any() + private lateinit var worker: Thread + override val hasImu = true + + override fun send(msgId: Int, payload: ByteArray): Boolean { + if (!running) return false + val value = payload.firstOrNull()?.toInt()?.and(0xff) ?: -1 + return when (msgId) { + VitureProtocol.MSG_IMU -> (value in 0..1).also { if (it) tracking = value == 1 } + VitureProtocol.MSG_GET_DISPLAY -> display()?.let { listener.onValue(msgId, it) } != null + VitureProtocol.MSG_DISPLAY -> value in RokidModel.MODES && + control(RokidModel.REQUEST_SET_DISPLAY, value, RokidModel.INDEX_DISPLAY) && + true.also { listener.onValue(VitureProtocol.MSG_GET_DISPLAY, value) } + VitureProtocol.MSG_GET_BRIGHTNESS -> read(RokidModel.REQUEST_GET_BRIGHTNESS, RokidModel.INDEX_BRIGHTNESS) + ?.let { listener.onValue(VitureProtocol.MSG_BRIGHTNESS, RokidModel.brightnessStep(it)) } != null + VitureProtocol.MSG_BRIGHTNESS -> value in 0..RokidModel.BRIGHTNESS_STEPS && + control(RokidModel.REQUEST_SET_BRIGHTNESS, RokidModel.brightnessRaw(value), RokidModel.INDEX_BRIGHTNESS) + else -> false // Volume is the glasses' USB audio device; there is no film or IMU-rate control. + } + } + + private fun display(): Int? = read(RokidModel.REQUEST_GET_DISPLAY, RokidModel.INDEX_DISPLAY) + + private fun read(request: Int, index: Int): Int? = synchronized(controlLock) { + if (!running) return null + val buffer = ByteArray(64) + val count = connection.controlTransfer(VENDOR_IN, request, 0, index, buffer, buffer.size, TIMEOUT_MS) + // Display mode is reported in the second byte, brightness in the first. + val offset = if (request == RokidModel.REQUEST_GET_DISPLAY) 1 else 0 + if (count > offset) buffer[offset].toInt() and 0xff else null + } + + private fun control(request: Int, value: Int, index: Int): Boolean = synchronized(controlLock) { + running && connection.controlTransfer(VENDOR_OUT, request, value, index, byteArrayOf(1), 1, TIMEOUT_MS) >= 0 + } + + private fun run() { + val buffer = ByteArray(maxOf(endpoint.maxPacketSize, 64)) + val parser = RokidProtocol.Parser() + var fusion = ImuFusion() + var wasTracking = false + var brightness = -1 + var lastSensor = SystemClock.elapsedRealtime() + try { + ready = true + listener.onReady() + while (running) { + val count = connection.bulkTransfer(endpoint, buffer, buffer.size, 250) + val now = SystemClock.elapsedRealtime() + val sample = if (count > 0) parser.parse(buffer, count) else null + if (sample == null) { + if (now - lastSensor > 3000) error("Rokid IMU stream stalled") + continue + } + lastSensor = now + sample.brightness?.takeIf { it != brightness }?.let { + brightness = it + listener.onValue(VitureProtocol.MSG_BRIGHTNESS, RokidModel.brightnessStep(it)) + } + val enabled = tracking + if (enabled && !wasTracking) fusion = ImuFusion() + wasTracking = enabled + if (!enabled) continue + // Max captures: already right/up/back; level gravity +Y, turn left +gyroY, look up +gyroX, roll right -gyroZ. + fusion.update(sample.timeNs, sample.accel, sample.gyro)?.let { + if (running && tracking) listener.onPose(buffer.copyOf(count), it, SystemClock.elapsedRealtimeNanos()) + } + } + } catch (e: Exception) { + if (running) Log.w(TAG, "Rokid stream failed", e) + } finally { + ready = false + if (running) listener.onLost() + } + } + + override fun close() { + synchronized(controlLock) { + if (!running) return + running = false + } + if (Thread.currentThread() !== worker) worker.join(500) + runCatching { connection.releaseInterface(usbInterface) } + connection.close() + } + + companion object { + private const val TAG = "RokidLink" + private const val TIMEOUT_MS = 250 + private const val VENDOR_IN = UsbConstants.USB_DIR_IN or UsbConstants.USB_TYPE_VENDOR + private const val VENDOR_OUT = UsbConstants.USB_DIR_OUT or UsbConstants.USB_TYPE_VENDOR + + internal fun sensorEndpoint(device: UsbDevice): Pair? { + for (i in 0 until device.interfaceCount) { + val iface = device.getInterface(i) + if (iface.interfaceClass != UsbConstants.USB_CLASS_HID) continue + for (e in 0 until iface.endpointCount) { + val endpoint = iface.getEndpoint(e) + if (endpoint.direction == UsbConstants.USB_DIR_IN && + endpoint.type == UsbConstants.USB_ENDPOINT_XFER_INT) return iface to endpoint + } + } + return null + } + + fun open(usb: UsbManager, device: UsbDevice, listener: Listener): RokidLink? { + val (iface, endpoint) = sensorEndpoint(device) ?: return null + val conn = usb.openDevice(device) ?: return null + if (!conn.claimInterface(iface, true)) { + conn.close() + return null + } + return RokidLink(conn, iface, endpoint, listener).also { link -> + link.worker = Thread({ link.run() }, "rokid-imu").apply { isDaemon = true } + link.worker.start() + } + } + } +} diff --git a/app/src/main/java/com/droiddeck/launcher/glasses/RokidModel.kt b/app/src/main/java/com/droiddeck/launcher/glasses/RokidModel.kt new file mode 100644 index 000000000..a1ad6e83a --- /dev/null +++ b/app/src/main/java/com/droiddeck/launcher/glasses/RokidModel.kt @@ -0,0 +1,104 @@ +package com.droiddeck.launcher.glasses + +import java.nio.ByteBuffer +import java.nio.ByteOrder + +/** Rokid Air/Max share one USB identity; the product string distinguishes them. */ +internal class RokidModel private constructor(override val name: String, private val max: Boolean) : GlassesModel { + override val controllable = true + override val brightnessMax = BRIGHTNESS_STEPS + override val volumeMax = 0 + override val hasSunblock = false + override val diagonalFovDeg = if (max) 50f else 43f + override val sbs90 = max + override val rates2d = if (max) listOf(60, 120) else listOf(60) + override val resolutions = if (max) listOf(GlassesResolution(1920, 1080), GlassesResolution(1920, 1200)) + else listOf(GlassesResolution(1920, 1080)) + + override fun mode(hz: Int, sbs: Boolean): Int = mode(hz, sbs, null) + + override fun mode(hz: Int, sbs: Boolean, resolution: GlassesResolution?): Int = when { + !max -> if (sbs) MODE_SBS_1080_60 else MODE_2D_60 + sbs && resolution?.height == 1080 -> MODE_SBS_1080_60 + sbs -> if (hz >= 90) MODE_SBS_1200_90 else MODE_SBS_1200_60 + hz >= 120 -> MODE_2D_120 + else -> MODE_2D_60 + } + + override fun modeForStep(step: Int): Int? = null + + override fun equals(other: Any?) = other is RokidModel && other.name == name && other.max == max + override fun hashCode() = name.hashCode() + + companion object { + const val VENDOR_ID = 0x04d2 + const val PRODUCT_ID = 0x162f + const val BRIGHTNESS_STEPS = 10 + + // Vendor control requests, device recipient. + const val REQUEST_SET_DISPLAY = 0x01 + const val REQUEST_GET_DISPLAY = 0x81 + const val REQUEST_SET_BRIGHTNESS = 0x02 + const val REQUEST_GET_BRIGHTNESS = 0x82 + const val INDEX_DISPLAY = 0x01 + const val INDEX_BRIGHTNESS = 0x02 + + const val MODE_2D_60 = 0 + const val MODE_SBS_1080_60 = 1 + const val MODE_2D_120 = 3 + const val MODE_SBS_1200_90 = 4 + const val MODE_SBS_1200_60 = 5 + val MODES = setOf(MODE_2D_60, MODE_SBS_1080_60, MODE_2D_120, MODE_SBS_1200_90, MODE_SBS_1200_60) + + // 0x04d2 is a shared vendor ID, so the Rokid product string is required too. + fun model(vid: Int, pid: Int, name: String?): GlassesModel? { + if (vid != VENDOR_ID || pid != PRODUCT_ID) return null + val product = name?.trim()?.takeIf { it.startsWith("Rokid", ignoreCase = true) } ?: return null + return RokidModel(product, product.contains("Max", ignoreCase = true)) + } + + fun isDisplay(name: String) = name.contains("Rokid", ignoreCase = true) + + /** Device brightness is 0..100; the shared slider uses ten steps. */ + fun brightnessStep(raw: Int): Int = ((raw.coerceIn(0, 100) + 5) / 10).coerceIn(0, BRIGHTNESS_STEPS) + fun brightnessRaw(step: Int): Int = step.coerceIn(0, BRIGHTNESS_STEPS) * 10 + } +} + +internal object RokidProtocol { + data class Sample(val timeNs: Long, val accel: Vec3, val gyro: Vec3, val brightness: Int?) + + private const val COMBINED = 17 + private const val SEPARATE = 4 + private const val SENSOR_ACCEL = 1 + private const val SENSOR_GYRO = 2 + + /** Max sends one combined report; Air sends accel and gyro reports that share a timestamp. */ + class Parser { + private var accel: Vec3? = null + private var accelTime = -1L + private var gyro: Vec3? = null + private var gyroTime = -2L + + fun parse(packet: ByteArray, length: Int = packet.size): Sample? { + if (length < 64) return null + val b = ByteBuffer.wrap(packet, 0, length).order(ByteOrder.LITTLE_ENDIAN) + return when (packet[0].toInt() and 0xff) { + COMBINED -> Sample(b.getLong(1), vec(b, 9), vec(b, 21), packet[59].toInt() and 0xff) + SEPARATE -> { + val time = b.getLong(9) * 1000 // Air reports microseconds; Max reports nanoseconds. + when (packet[1].toInt() and 0xff) { + SENSOR_ACCEL -> { accel = vec(b, 21); accelTime = time } + SENSOR_GYRO -> { gyro = vec(b, 21); gyroTime = time } + else -> return null + } + if (accelTime != gyroTime) return null + Sample(time, accel!!, gyro!!, null).also { accelTime = -1L; gyroTime = -2L } + } + else -> null + } + } + + private fun vec(b: ByteBuffer, offset: Int) = Vec3(b.getFloat(offset), b.getFloat(offset + 4), b.getFloat(offset + 8)) + } +} diff --git a/app/src/main/java/com/droiddeck/launcher/glasses/XrealOrientationFilter.kt b/app/src/main/java/com/droiddeck/launcher/glasses/XrealOrientationFilter.kt index f6ac28f69..e8d34f44a 100644 --- a/app/src/main/java/com/droiddeck/launcher/glasses/XrealOrientationFilter.kt +++ b/app/src/main/java/com/droiddeck/launcher/glasses/XrealOrientationFilter.kt @@ -4,20 +4,27 @@ import kotlin.math.abs /** 1S radians/sec and device nanoseconds, mapped using directed hardware captures. */ internal class XrealOrientationFilter { + private val fusion = ImuFusion() + + // 1S reports right/down/forward. Rotate 180 degrees about X into the + // shared right/up/back basis before either gravity alignment or fusion. + // Captures: level gravity -Y; looking left -gyroY; up +gyroX; right tilt +gyroZ. + fun update(sample: XrealProtocol.Sample): FloatArray? = fusion.update(sample.timeNs, + Vec3(sample.accel.x, -sample.accel.y, -sample.accel.z), + Vec3(sample.gyro.x, -sample.gyro.y, -sample.gyro.z)) +} + +/** Gyro integration with gravity correction in the shared right/up/back basis, rad/s and device ns. */ +internal class ImuFusion { private var previous = 0L private var rotation = Quaternion.IDENTITY private var initialized = false private var bias = Vec3(0f, 0f, 0f) - fun update(sample: XrealProtocol.Sample): FloatArray? { - if (sample.timeNs <= previous) return null - val dt = if (previous == 0L) 0f else (sample.timeNs - previous) * 1e-9f - previous = sample.timeNs - // 1S reports right/down/forward. Rotate 180 degrees about X into the - // shared right/up/back basis before either gravity alignment or fusion. - // Captures: level gravity -Y; looking left -gyroY; up +gyroX; right tilt +gyroZ. - val accel = Vec3(sample.accel.x, -sample.accel.y, -sample.accel.z) - val gyro = Vec3(sample.gyro.x, -sample.gyro.y, -sample.gyro.z) + fun update(timeNs: Long, accel: Vec3, gyro: Vec3): FloatArray? { + if (timeNs <= previous) return null + val dt = if (previous == 0L) 0f else (timeNs - previous) * 1e-9f + previous = timeNs val norm = accel.length() val gravityValid = abs(norm - 9.81f) < 1.5f if (dt > 0.25f) initialized = false // Never integrate a reconnect/stall as motion. diff --git a/app/src/test/java/com/droiddeck/launcher/glasses/GlassesRenderingTest.kt b/app/src/test/java/com/droiddeck/launcher/glasses/GlassesRenderingTest.kt index 401d24701..51d9aa7b7 100644 --- a/app/src/test/java/com/droiddeck/launcher/glasses/GlassesRenderingTest.kt +++ b/app/src/test/java/com/droiddeck/launcher/glasses/GlassesRenderingTest.kt @@ -5,6 +5,14 @@ import org.junit.Assert.* import org.junit.Test class GlassesRenderingTest { + @Test fun modelTwoDRatesLimitChoicesAndSavedRatesFallBackToAnOfferedOne() { + val rokid = GlassesState(displayId = 2, width = 1920, height = 1080, hz = 60f, rates2d = listOf(60, 120)) + assertEquals(listOf(60, 120), GlassesRendering.refreshRates(GlassesSettings(), rokid)) + assertEquals(60, GlassesRendering.outputHz(GlassesSettings(refreshHz = 90), rokid)) + assertEquals(120, GlassesRendering.outputHz(GlassesSettings(refreshHz = 120), rokid)) + assertEquals(listOf(60, 90, 120), GlassesRendering.refreshRates(GlassesSettings(), rokid.copy(rates2d = listOf(60, 90, 120)))) + } + private val ultra = GlassesState( model = "VITURE Luma Ultra", connected = true, controllable = true, displayId = 2, width = 1920, height = 1200, diff --git a/app/src/test/java/com/droiddeck/launcher/glasses/GlassesUsbTest.kt b/app/src/test/java/com/droiddeck/launcher/glasses/GlassesUsbTest.kt index d4ee21f79..ae274cece 100644 --- a/app/src/test/java/com/droiddeck/launcher/glasses/GlassesUsbTest.kt +++ b/app/src/test/java/com/droiddeck/launcher/glasses/GlassesUsbTest.kt @@ -209,6 +209,14 @@ class GlassesUsbTest { assertEquals(null, Glasses.deviceModel(device(0x043e, 0x1234, "XREAL 1S"))) } + @Test fun rokidMaxRoutesThroughItsOwnModelAndSharedVendorIdNeedsTheName() { + val max = Glasses.deviceModel(device(0x162f, 0x04d2, "Rokid Max")) + assertTrue(max is RokidModel) + assertEquals("Rokid Max", max?.name) + assertEquals(null, Glasses.deviceModel(device(0x162f, 0x04d2, "Generic HID"))) + assertTrue(Glasses.isGlassesDisplayName("Rokid Max")) + } + private fun device(pid: Int, vid: Int = VitureProtocol.VENDOR_ID, name: String = "VITURE device"): UsbDevice = ReflectionHelpers.callConstructor( UsbDevice::class.java, from(String::class.java, "/dev/bus/usb/$pid"), diff --git a/app/src/test/java/com/droiddeck/launcher/glasses/RokidProtocolTest.kt b/app/src/test/java/com/droiddeck/launcher/glasses/RokidProtocolTest.kt new file mode 100644 index 000000000..c2cf38e33 --- /dev/null +++ b/app/src/test/java/com/droiddeck/launcher/glasses/RokidProtocolTest.kt @@ -0,0 +1,94 @@ +package com.droiddeck.launcher.glasses + +import com.droiddeck.launcher.gpu.HeadPose +import org.junit.Assert.* +import org.junit.Test +import java.nio.ByteBuffer +import java.nio.ByteOrder + +class RokidProtocolTest { + private fun hex(s: String) = s.chunked(2).map { it.toInt(16).toByte() }.toByteArray() + private fun matrix(q: FloatArray) = HeadPose.fromGlasses(q[0], q[1], q[2], q[3]) + + @Test fun identityRequiresRokidProductString() { + assertEquals("Rokid Max", RokidModel.model(0x04d2, 0x162f, "Rokid Max")?.name) + assertEquals("Rokid Air", RokidModel.model(0x04d2, 0x162f, "Rokid Air")?.name) + assertNull(RokidModel.model(0x04d2, 0x162f, "USB Keyboard")) + assertNull(RokidModel.model(0x04d2, 0x162f, null)) + assertNull(RokidModel.model(0x04d2, 0x1234, "Rokid Max")) + assertTrue(RokidModel.isDisplay("Rokid Max")) + } + + @Test fun maxModesFollowRefreshAndStereoResolution() { + val max = RokidModel.model(0x04d2, 0x162f, "Rokid Max")!! + assertEquals(RokidModel.MODE_2D_60, max.mode(60, false, null)) + assertEquals(RokidModel.MODE_2D_120, max.mode(120, false, null)) + assertEquals(RokidModel.MODE_2D_60, max.mode(90, false, null)) + assertEquals(listOf(60, 120), max.rates2d) + assertEquals(RokidModel.MODE_SBS_1200_60, max.mode(60, true, null)) + assertEquals(RokidModel.MODE_SBS_1200_90, max.mode(90, true, null)) + assertEquals(RokidModel.MODE_SBS_1080_60, max.mode(90, true, GlassesResolution(1920, 1080))) + assertTrue(max.sbs90) + assertFalse(max.hasSunblock) + } + + @Test fun airStaysOnSixtyHertzModes() { + val air = RokidModel.model(0x04d2, 0x162f, "Rokid Air")!! + assertEquals(RokidModel.MODE_2D_60, air.mode(120, false, null)) + assertEquals(RokidModel.MODE_SBS_1080_60, air.mode(90, true, null)) + assertFalse(air.sbs90) + assertEquals(listOf(60), air.rates2d) + } + + @Test fun brightnessStepsRoundTripTheDevicePercent() { + assertEquals(8, RokidModel.brightnessStep(80)) + assertEquals(10, RokidModel.brightnessStep(100)) + assertEquals(3, RokidModel.brightnessStep(32)) + assertEquals(0, RokidModel.brightnessStep(0)) + assertEquals(30, RokidModel.brightnessRaw(3)) + assertEquals(100, RokidModel.brightnessRaw(12)) + } + + @Test fun capturedMaxCombinedReportParsesAccelGyroAndBrightness() { + val packet = hex("11d09b3e3700000000bb9eb03f9147f0c06e86c340306a43bc9c227b3ad3e6aebb000055c29a99abc1cdccfac10000000000000000000000000000503c000000") + val sample = RokidProtocol.Parser().parse(packet)!! + assertEquals(0x373e9bd0L, sample.timeNs) + assertEquals(1.380f, sample.accel.x, 0.001f) + assertEquals(-7.509f, sample.accel.y, 0.001f) + assertEquals(6.110f, sample.accel.z, 0.001f) + assertEquals(-0.0119f, sample.gyro.x, 0.0001f) + assertEquals(80, sample.brightness) + assertNull(RokidProtocol.Parser().parse(packet.copyOf(32))) + } + + @Test fun airSeparateReportsPairOnSharedTimestamp() { + fun report(type: Int, time: Long, v: Vec3) = ByteBuffer.allocate(64).order(ByteOrder.LITTLE_ENDIAN).apply { + put(0, 4); put(1, type.toByte()); putLong(9, time) + putFloat(21, v.x); putFloat(25, v.y); putFloat(29, v.z) + }.array() + val parser = RokidProtocol.Parser() + assertNull(parser.parse(report(1, 500, Vec3(0f, 9.81f, 0f)))) + val sample = parser.parse(report(2, 500, Vec3(0.1f, 0f, 0f)))!! + assertEquals(500_000L, sample.timeNs) + assertEquals(9.81f, sample.accel.y, 0f) + assertEquals(0.1f, sample.gyro.x, 0f) + assertNull(sample.brightness) + assertNull(parser.parse(report(2, 600, Vec3(0f, 0f, 0f)))) + } + + @Test fun capturedRestingGravityStartsUpright() { + val pose = ImuFusion().update(1_000_000, Vec3(0.01f, 9.58f, -2.20f), Vec3(0f, 0f, 0f))!! + assertTrue(kotlin.math.abs(pose[3]) > 0.97f) + } + + @Test fun capturedLeftTurnSignLooksLeft() { + val fusion = ImuFusion() + val first = matrix(fusion.update(1_000_000, Vec3(0f, 9.81f, 0f), Vec3(0f, 0f, 0f))!!) + var last = FloatArray(4) + repeat(1000) { + last = fusion.update((it + 2) * 1_000_000L, Vec3(0f, 9.81f, 0f), Vec3(0f, (Math.PI / 6).toFloat(), 0f))!! + } + val view = HeadPose.relative(HeadPose.recenter(first), matrix(last)) + assertArrayEquals(floatArrayOf(-0.5f, 0f, -0.8660254f), HeadPose.apply(view, floatArrayOf(0f, 0f, -1f)), 0.01f) + } +} diff --git a/docs/smart-glasses-support.md b/docs/smart-glasses-support.md index dd70fbdaf..416b0c3f4 100644 --- a/docs/smart-glasses-support.md +++ b/docs/smart-glasses-support.md @@ -23,9 +23,10 @@ This describes the integrations implemented in DroidDeck as of 2026-10-17. A rec | Recognized RayNeo / TCL Air-series devices | Integrated RayNeo USB transport | Yes, with orientation calibration controls | No | No | | RayNeo GT-series devices matching the supported USB identity, including GT Max identification | Integrated GT USB/IMU path | Yes, with orientation calibration controls | No | No | | **XREAL 1S** | **Experimental**; exact USB identity plus the device's USB-Ethernet control/IMU connection | Yes, when that connection is ready | No | No | +| Rokid Max; Rokid Air under the same USB identity | Integrated Rokid USB transport (vendor control requests and HID IMU reports) | Yes | No | No | | Other XREAL models; unrecognized glasses; display adapters and docks | Display-only where Android exposes a usable display; automatic glasses routing also requires display recognition | No supported app tracking connection | No | No | -RayNeo's Air-series recognition uses its known USB identity or a RayNeo/TCL Air name under the accepted vendor IDs; it is not a per-model certification list. The GT vendor path requires the specific supported product ID. XREAL support currently targets **1S only**: One, One Pro, Air-family devices, and recovery-mode products are not enabled by the 1S transport. VITURE Pro 2 is identified through SDK validation rather than a guessed product ID. A dock does not supply the connected glasses' sensors by itself. +RayNeo's Air-series recognition uses its known USB identity or a RayNeo/TCL Air name under the accepted vendor IDs; it is not a per-model certification list. The GT vendor path requires the specific supported product ID. Rokid recognition requires USB 04d2:162f plus a Rokid product name, since that vendor ID is shared with unrelated devices; it was verified on a Rokid Max, and the Air follows the same published protocol untested. XREAL support currently targets **1S only**: One, One Pro, Air-family devices, and recovery-mode products are not enabled by the 1S transport. VITURE Pro 2 is identified through SDK validation rather than a guessed product ID. A dock does not supply the connected glasses' sensors by itself. The final column refers specifically to Luma Ultra's stereo tracking cameras. DroidDeck's separate phone-camera and physical-controller features are independent of this matrix. @@ -42,6 +43,8 @@ All display modes depend on the phone's video output, the cable/adapter, and And | VITURE Pro 2 | 1080p per eye; 2D and SBS | Brightness and volume; no electrochromic-film control | | RayNeo Air/GT | 2D/SBS switching; resolution and refresh follow available Android modes | Orientation calibration; no app brightness, volume, or dimming commands | | XREAL 1S | Supported 1S display-mode commands, including native 1200p/60 Hz and SBS | Brightness and dimming toggle; no app volume command | +| Rokid Max | 1080p at 60 or 120 Hz in 2D; SBS at 1080p/60, 1200p/60, or 1200p/90 | Brightness; volume follows Android's USB audio volume; no dimming film | +| Rokid Air | 1080p/60 in 2D and SBS | Brightness; volume follows Android's USB audio volume | DroidDeck saves separate **2D resolution**, **3D resolution**, and **VR resolution** preferences. Unsupported saved choices fall back to the connected model's automatic behavior. Native **30 Hz** is offered only when Android advertises a matching mode and glasses presentation output is enabled. Higher refresh selections are constrained by the active 2D/SBS layout and model; a firmware-rejected switch is not treated as successful until the physical display confirms it. @@ -78,4 +81,4 @@ If only a flat picture is available, check video output first, then the control - [Agent diagnostics and capture commands](agent-control.md) - [RayNeo attribution and protocol references](third-party/rayneo/README.md) -The capability definitions are maintained in [VITURE models](../app/src/main/java/com/droiddeck/launcher/glasses/VitureProtocol.kt), [RayNeo models](../app/src/main/java/com/droiddeck/launcher/glasses/RayNeoModel.kt), [XREAL 1S](../app/src/main/java/com/droiddeck/launcher/glasses/XrealModel.kt), and [connection capability routing](../app/src/main/java/com/droiddeck/launcher/glasses/Glasses.kt). Update this matrix when those integrations change. +The capability definitions are maintained in [VITURE models](../app/src/main/java/com/droiddeck/launcher/glasses/VitureProtocol.kt), [RayNeo models](../app/src/main/java/com/droiddeck/launcher/glasses/RayNeoModel.kt), [XREAL 1S](../app/src/main/java/com/droiddeck/launcher/glasses/XrealModel.kt), [Rokid](../app/src/main/java/com/droiddeck/launcher/glasses/RokidModel.kt), and [connection capability routing](../app/src/main/java/com/droiddeck/launcher/glasses/Glasses.kt). Update this matrix when those integrations change.