Rebuild plan: Godot/Android implementation of the Zapbox XR SDK (clean-room) #1

New issue

Open

opened 2026-11-13 20:10:05 +00:00 by muse-lantern · 1 comment

muse-lantern commented 2026-11-13 20:10:05 +00:00

Owner

Copy link

Zapbox XR SDK — Godot/Android rebuild plan (clean-room)

Target: a Godot 4.7 project (pin 4.7.2, latest stable 4.x as of Nov 2026) that turns Zapbox hardware (headset + controllers) into a usable XR input system, re-implementing the SDK's behavior (stereo video-passthrough MR, marker-anchored tracking, Bluetooth controllers) from first principles. This is a new implementation, not a port: upstream is Unity + iOS + closed native tracking, and there is no Android code to port.

Shape: two parts (Zach, 2026-11-13)

The SDK is in two parts. Part 1 — Godot SDK for Zapbox: the core SDK (controller connection + tracking) that anything can connect to. Its first consumers are the phone headset MR (Phases 1–4 below: stereo shell, camera + markers, controllers, fusion) and desktop use — the controllers as desktop wands, tracked by a webcam instead of the phone camera. Same tracking core, different camera source. Part 2 — Android bindings for WebXR apps: an Android host that loads WebXR pages (local file or web-loaded) and binds Zapbox tracking + controllers into the WebXR session. Per Zach, standalone capable of running WebGL is the primary focus; Zapbox Link / PC VR is a stretch goal, not the target.

How the phases map: Phases 0–4 build the Part 1 core. The desktop wand front-end (webcam camera source on a desktop Godot build) rides on the same core after Phase 2's detector exists. Part 2 is its own phase (Phase 5 below). Video capture stays deferred. Mapping DECIDED (Zach, 2026-11-13): the Godot headset MR stays a deliverable in its own right — it is not demoted to a dev/test path for Part 2.

Engine pin note: Godot's formal "LTS" label belongs to the 3.6.x line (the previous major), which is not the target here — 3.x lacks the modern Android/XR stack this rebuild needs. "Latest LTS" is read as latest stable 4.x; the pin is 4.7.2 and is revisited at each phase. The 4.7.2 editor binary lives on this box at ~/workspace/godot-4.7.2/Godot_v4.7.2-stable_linux.x86_64 (separate from the DEAD AIR pin, 4.6.3 — this is a separate project with its own pin).

Companion doc: sdk-inventory.md (public-surface inventory of the upstream SDK).

0. License note (the constraint, in one paragraph)

Upstream is under Zappar's custom license: use is permitted only in connection with authoring/distributing applications that work with Zapbox hardware; the Work may not be used for competitive analysis or to develop a competing product or service; binary libraries may not be altered and expiry mechanisms may not be circumvented; Zappar owns the IP in the Work and in derivative works of the Work. The rebuild therefore follows a strict clean-room rule: no SDK code (C#, Obj-C, shaders, JS, bundle payloads) is copied or transcribed; the native library and bundle payloads are treated as black boxes; only the public API shape (type/method names, the input-control surface, file layout) and Zappar's public hardware documentation inform the design; all tracking, detection, fusion, and rendering code is written fresh; marker artwork is defined by us (see Phase 2); and the result is for use with Zapbox hardware only. Anything that can only be learned by reading SDK implementation bodies or the binary is recorded as an open question for Zach instead of answered from the source.

Phase 0 — Setup (prerequisite, no code)

Deliverables:

• Repo home DECIDED (Zach, 2026-11-13): muse-lantern/zapbox-godot on git.merith.xyz — this repo. Issues #1 (plan) and #2 (inventory) filed there.
• Godot 4.7.2 Android export templates installed (SETUP GAP: not installed on this box; editor/export_templates/ is empty — install from the Godot download, version must match 4.7.2 exactly).
• Android SDK + JDK + a test phone identified: Galaxy S25 Ultra (Zach, 2026-11-13) — but custom calibration for Android phones is supported, since not all phones support the same camera set (wide vs ultrawide availability matters).
• Zapbox hardware: Zach is obtaining one shortly (headset, 2 controllers, printed world-anchor markers) — or the project cannot proceed past Phase 1 in any verifiable way.
• Clean-room rules written into the repo README (done in the skeleton).

Done when: a blank Godot project exports to the test phone and runs (the skeleton already opens cleanly on desktop; on-device is the gate).

Phase 1 — Stereo shell + IMU head tracking (feasible)

What: the "cardboard" core. Two eye viewports side-by-side on the phone screen, each with its own off-center-projection camera (lens distortion model for the Zapbox lenses, tunable), head orientation from the phone IMU (gyro + accelerometer, magnetometer fallback), and a flat passthrough quad behind the scene (initially a static camera texture or a test pattern).

Passthrough rendering (Zach, 2026-11-13, clarified 14:33): the material is a cutout. That region of the screen renders the camera output — as if the camera feed was on the screen the entire time, with the 3D scene composited on top and a hole cut where passthrough shows. The camera feed is the base layer, not a texture on a quad; the passthrough surface is the hole in the scene through which it shows. (Supersedes the earlier "unlit material presenting the camera texture" framing: the material doesn't display the frame, it cuts the scene away from it.)

Godot facilities that exist: SubViewport + SubViewportContainer for split-screen stereo; Camera3D with custom frustum for off-center projection; Input.get_accelerometer() / get_gyroscope() / get_magnetometer() on Android (Godot exposes the sensors); the Asset Library "VR Mobile Camera" plugin is a working precedent for exactly this pattern (screen split + IMU head tracking).

Deliverables: ZapboxStereoRig scene (origin + left/right eye cameras + passthrough background), zapbox_tracking.gd IMU orientation path, a lens-calibration resource (IPD, FOV, distortion coefficients as editable data).
Verification: on-device — look around, orientation tracks 1:1 with low latency; stereo image fuses through the lenses; headless — orientation math unit tests (quaternion integration vs known rotations), viewport layout checks.

Phase 2 — Camera frames + marker detection + head pose (hard)

What: get the phone camera into Godot, detect our own printed markers in the frame, solve each marker's 6DoF pose, and anchor the world to the origin marker.

Pieces:

• Camera path (feasible with work): Godot has no camera-frame API. Build a Godot Android plugin (v2, Java/Kotlin) around Camera2: open the wide (or ultrawide where present) camera, manual exposure control (short exposure, per upstream's tracking rationale), YUV frames delivered into a Godot texture each frame. Expose camera intrinsics + lens selection to GDScript. This is the first native-code surface and the first thing that needs the export templates + a device.
• Marker artwork (official markers as reference + custom trackers supported): Zach will have official hardware, so the official Zapbox markers are the reference (photograph/measure them to define the detector's reference data). Custom trackers must ALSO be supported for world-tracking calibration — live re/calibration against user-defined markers. The detector therefore takes a marker registry (id, size, role: origin/side/custom) rather than a hardcoded set, and the origin marker is our equivalent of the "zapbox" floor anchor. Printable artwork lives in the repo as SVGs/PDFs + a markers.json.
• Detector (hard, but a solved problem family): OpenCV ArUco detection + solvePnP pose estimation, wrapped in a GDExtension (C++) or in the Android plugin (Java + OpenCV Android SDK). ArUco is the standard fiducial family; the work is integration, calibration (camera intrinsics per phone), and robustness (motion blur, lighting, wide-FOV distortion, partial occlusion), not algorithm invention.
• Pose fusion: multi-marker observations fused into one head pose (origin marker defines the Floor frame; other markers positioned relative to it); when no marker is visible, fall back to IMU-only orientation + a neck-model-style position hold (same fallback shape as upstream, implemented from scratch).

Deliverables: zapbox_markers.gd (marker registry + detection interface), the detector extension/plugin, printable marker set, zapbox_tracking.gd marker-fusion path.
Verification: headless — detector unit tests on synthetic + recorded frames (known marker poses recovered within tolerance); pose math tests; on-device — walk the play space, position holds while ≥1 marker is visible, degrades gracefully (no jumps) when markers leave view; origin stays fixed across sessions.

Phase 3 — Controllers: BLE input + calibration + pose (hardest)

What: pair the two Zapbox controllers, read their inputs in the SDK's control surface (thumbstick, trigger, grip, primary/secondary/menu buttons, tracking state, pose, velocity), run the calibration flows, and track controller pose.

Pieces:

• BLE input (feasible once the protocol is known; unknown today): Android BluetoothGatt in the Android plugin. The controller BLE protocol (services/characteristics, packet format) is not documented in any public file. Path per Zach (2026-11-13): dig deeper online first (community resources, teardowns, prior art) — otherwise reverse-engineer it from hardware in hand: discover GATT services, observe the characteristic traffic for buttons/stick/trigger/IMU, and map it to the control surface. This is the single biggest unknown.
• Calibration flows (feasible): motion-sensor calibration phase (controllers flat and untouched, thumbstick centred), analog trigger auto-range, thumbstick scale to full −1..+1. Implement as a pairing UI flow in Godot (mirrors the upstream flow's shape; own UI, own code).
• Controller pose (hard): upstream tracks controllers optically via the phone camera (pose updates only while in view of the ultrawide camera) fused with IMU. Our path: (a) detect the controller's visual marker in the camera frame (requires knowing the controller's marker geometry — closed upstream; our options are to characterize the physical controller's visible pattern with hardware in hand, or attach our own marker to the controllers), plus (b) IMU orientation from the controller, fused; (c) out-of-view fallback: orientation-only hold + the over-shoulder heuristic shape (orientation-based shoulder estimate) if wanted. If optical controller tracking proves infeasible without Zappar's marker geometry, the honest fallback is IMU orientation + position inferred from the head pose and a kinematic model — less capable, and that limitation must be stated, not papered over.

Deliverables: zapbox_controllers.gd (pairing, input state, calibration state machine), BLE plugin surface, controller input map matching the SDK's control names, pairing/calibration UI.
Verification: headless — input mapping tests (synthetic BLE packets → control states), calibration math tests; on-device — pair both controllers, every control reads correctly, thumbstick full range, trigger range calibrates, controller pose tracks while in camera view.

Phase 4 — Tracking fusion, prediction, frame pacing (hard)

What: make the whole thing feel stable at 60 fps on a phone.

Pieces:

• Fusion: one pose pipeline — marker observations + phone IMU + controller IMU, with timestamp alignment, outlier rejection (a bad marker solve must not teleport the world), and smooth handoff between marker-visible and marker-lost states.
• Prediction: extrapolate head/controller pose by one frame (and by the camera-to-photon latency) so motion feels tight; upstream's roadmap mentions accelerometer data to cut perceived latency — same idea, own implementation.
• Frame pacing (hard, no late-warp): Godot has no async timewarp/late-warp for the cardboard path. The budget is discipline: keep the frame under ~10–16 ms (Mobile renderer, minimal overdraw, capped pixel ratio), and measure with on-device frame-time logging. True late-warp would need engine-level rendering work and is explicitly out of scope for this rebuild; if frame times can't be held, the fallback is lowering resolution, not warping.
• Stereo rendering details: per-eye distortion correction in a shader (or pre-distorted projection), chromatic-aberration approximation if the lenses need it, IPD calibration.

Deliverables: fused pose pipeline in zapbox_tracking.gd, frame-time HUD/logging, distortion shader, performance budget doc.
Verification: on-device — sustained 60 fps in a simple scene with passthrough + markers + controllers; no visible world swim when walking; headless — fusion tests (outlier rejection, handoff smoothness).

Phase 5 — Android bindings for WebXR apps (Part 2)

What: an Android host that loads WebXR pages — a local file or a web-loaded URL — and binds Zapbox tracking + controllers into the WebXR session, so WebXR content runs on the phone with Zapbox as its XR input. Per Zach, standalone capable of running WebGL is the primary focus for the phone experience.

Pieces:

• Host app: an Android app (Godot export or a thin native shell) that hosts a WebView, loads a WebXR page from a local file or URL, and exposes the WebXR Device API to it. The host owns the Zapbox session (tracking + controllers from Phases 2–4) and feeds poses/inputs into the page's XR session.
• Bindings: map the Part 1 tracking + controller state onto WebXR constructs (viewer pose, input sources with grip/target-ray spaces, button/axis state per the WebXR gamepad mapping). The page must not need to know about Zapbox — it sees a standard WebXR session.
• Scope note: this is a host + binding layer, not a browser. Local-file loading must work offline; web-loaded pages must work over the network. Zapbox Link / PC VR is a stretch goal, explicitly out of this phase.

Deliverables: host app, WebXR binding layer, a sample WebXR page (local file) exercising headset pose + both controllers, docs (how to point the host at a page).
Verification: on-device — load the sample page from a local file and from a URL; headset pose + controller inputs drive the WebXR scene; headless — binding mapping tests (Part 1 state → WebXR session state).

Phase 6 — Video capture + spectator (defer)

What: first-person recording (record camera feed + composited view) and a spectator mode. Deferred behind Phases 1–5: recording triples render load (upstream renders the scene 3× while recording) and spectator mode needs a second device + a recorder app. Stub the API (zapbox_capture.gd) in the skeleton; build only after tracking is solid. Android capture path would be MediaRecorder/MediaProjection, not iOS Photos.

Phase 7 — Packaging (after Phase 1 works on-device)

What: the Godot Android plugin(s) as a proper addon (camera + BLE), export preset + signing, a sample scene (grab interaction, mirroring the upstream sample's intent), docs (setup guide: markers to print, pairing flow, calibration), and a decision on distribution (sideload vs store). Covers both parts (Part 1 SDK + Part 2 WebXR host).

Hard parts — honest assessment

#	Hard part	Assessment
1	Marker detection + pose from a phone camera	Hard but feasible. ArUco + solvePnP is a proven family; the work is integration (GDExtension/Android plugin), per-phone calibration, and robustness (blur, light, wide FOV). Needs printed markers + a test phone. No new algorithms required.
2	Controller tracking + pairing (BLE protocol, optical pose)	Hardest, and partly unknown. The BLE protocol is undocumented publicly; establishing it needs hardware in hand and GATT traffic observation. Optical controller pose needs the controller's marker geometry, which is closed — either characterize the physical controllers ourselves or attach our own markers. IMU-only fallback is possible but much less capable. This is the phase most likely to force a scope cut.
3	Late-warp / frame pacing on Android	Hard, with a ceiling. No late-warp exists in Godot for this path; the mitigation is prediction + a strict frame budget. If the budget can't be held on the target phone, the experience degrades (tearing/judder) and there is no warp to save it.
4	Stereo rendering for the headset	Feasible. Two SubViewports + off-center cameras is a known pattern (VR Mobile Camera asset is the precedent). Lens distortion tuning is fiddly but tractable.
5	Android camera path	Feasible with work. Camera2 via a Godot Android plugin is the documented route; YUV→texture, exposure control, ultrawide selection are all standard Camera2. Needs export templates + a device; nothing exotic.
6	License / IP	Constraining, not blocking. Clean-room + own marker artwork + for-use-with-Zapbox-hardware keeps the rebuild inside the license's permitted use. Zach (2026-11-13): the official license is fine — Zappar is free to take and reuse any of the code, though unlikely to do so officially (GDScript over Unity). The IP-assignment clause for derivative works of the Work remains a standing fact; the rebuild avoids being a derivative of the Work by being written fresh.
7	Desktop wands via webcam (Part 1 front-end)	Feasible, new surface. Same tracking core as Phase 2 with a webcam frame source on a desktop Godot build instead of the phone camera. Needs desktop camera access in Godot + BLE pairing from desktop. Rides on the Phase 2 detector; do after the phone path works.
8	Android WebXR bindings (Part 2)	Feasible with work. The host + binding layer is standard Android/WebView work; the risk is WebXR conformance (the page must see a standard session) and the WebView's XR support, not the Zapbox side. Primary phone focus per Zach.

Open questions for Zach — answered 2026-11-13 (comment 999 on issue #1) + one remaining

Answered (his words, condensed):

1. Hardware: he will be obtaining a Zapbox shortly.
2. Controller protocol: dig deeper online first; otherwise reverse-engineer it.
3. Markers: official Zapbox markers as reference (he will have official hardware); custom trackers supported for world-tracking calibration, to live re/calibrate.
4. Repo home: this repo (muse-lantern/zapbox-godot).
5. Target phones: Galaxy S25 Ultra for now; custom calibration for Android phones supported (not all phones support the same camera set).
6. Scope: Zapbox Link / PC VR is a stretch goal; standalone capable of running WebGL is the primary focus.
7. License: the official license is fine; Zappar is free to take and reuse any of the code, though unlikely to officially (GDScript over Unity).

Remaining: none. Mapping decided 2026-11-13 (Zach): the Godot headset MR stays a deliverable in its own right, not a dev/test path for Part 2. Passthrough rendering decided the same day, clarified 14:33: the material is a cutout — that screen region renders the camera output, as if it was on the screen the entire time (camera feed is the base layer; the 3D scene composites on top with a hole cut where passthrough shows).

# Zapbox XR SDK — Godot/Android rebuild plan (clean-room) Target: a Godot 4.7 project (pin **4.7.2**, latest stable 4.x as of Nov 2026) that turns Zapbox hardware (headset + controllers) into a usable XR input system, re-implementing the SDK's *behavior* (stereo video-passthrough MR, marker-anchored tracking, Bluetooth controllers) from first principles. This is a new implementation, not a port: upstream is Unity + iOS + closed native tracking, and there is no Android code to port. ## Shape: two parts (Zach, 2026-11-13) The SDK is in two parts. **Part 1 — Godot SDK for Zapbox:** the core SDK (controller connection + tracking) that anything can connect to. Its first consumers are the phone headset MR (Phases 1–4 below: stereo shell, camera + markers, controllers, fusion) and desktop use — the controllers as desktop wands, tracked by a webcam instead of the phone camera. Same tracking core, different camera source. **Part 2 — Android bindings for WebXR apps:** an Android host that loads WebXR pages (local file or web-loaded) and binds Zapbox tracking + controllers into the WebXR session. Per Zach, standalone capable of running WebGL is the primary focus; Zapbox Link / PC VR is a stretch goal, not the target. How the phases map: Phases 0–4 build the Part 1 core. The desktop wand front-end (webcam camera source on a desktop Godot build) rides on the same core after Phase 2's detector exists. Part 2 is its own phase (Phase 5 below). Video capture stays deferred. Mapping DECIDED (Zach, 2026-11-13): the Godot headset MR stays a deliverable in its own right — it is not demoted to a dev/test path for Part 2. Engine pin note: Godot's formal "LTS" label belongs to the 3.6.x line (the previous major), which is not the target here — 3.x lacks the modern Android/XR stack this rebuild needs. "Latest LTS" is read as latest stable 4.x; the pin is 4.7.2 and is revisited at each phase. The 4.7.2 editor binary lives on this box at `~/workspace/godot-4.7.2/Godot_v4.7.2-stable_linux.x86_64` (separate from the DEAD AIR pin, 4.6.3 — this is a separate project with its own pin). Companion doc: `sdk-inventory.md` (public-surface inventory of the upstream SDK). ## 0. License note (the constraint, in one paragraph) Upstream is under Zappar's custom license: use is permitted only in connection with authoring/distributing applications that work with Zapbox hardware; the Work may not be used for competitive analysis or to develop a competing product or service; binary libraries may not be altered and expiry mechanisms may not be circumvented; Zappar owns the IP in the Work and in derivative works of the Work. The rebuild therefore follows a strict clean-room rule: no SDK code (C#, Obj-C, shaders, JS, bundle payloads) is copied or transcribed; the native library and bundle payloads are treated as black boxes; only the public API shape (type/method names, the input-control surface, file layout) and Zappar's public hardware documentation inform the design; all tracking, detection, fusion, and rendering code is written fresh; marker artwork is defined by us (see Phase 2); and the result is for use with Zapbox hardware only. Anything that can only be learned by reading SDK implementation bodies or the binary is recorded as an open question for Zach instead of answered from the source. ## Phase 0 — Setup (prerequisite, no code) Deliverables: - Repo home DECIDED (Zach, 2026-11-13): `muse-lantern/zapbox-godot` on git.merith.xyz — this repo. Issues #1 (plan) and #2 (inventory) filed there. - Godot 4.7.2 Android export templates installed (SETUP GAP: not installed on this box; `editor/export_templates/` is empty — install from the Godot download, version must match 4.7.2 exactly). - Android SDK + JDK + a test phone identified: **Galaxy S25 Ultra** (Zach, 2026-11-13) — but custom calibration for Android phones is supported, since not all phones support the same camera set (wide vs ultrawide availability matters). - Zapbox hardware: Zach is obtaining one shortly (headset, 2 controllers, printed world-anchor markers) — or the project cannot proceed past Phase 1 in any verifiable way. - Clean-room rules written into the repo README (done in the skeleton). Done when: a blank Godot project exports to the test phone and runs (the skeleton already opens cleanly on desktop; on-device is the gate). ## Phase 1 — Stereo shell + IMU head tracking (feasible) What: the "cardboard" core. Two eye viewports side-by-side on the phone screen, each with its own off-center-projection camera (lens distortion model for the Zapbox lenses, tunable), head orientation from the phone IMU (gyro + accelerometer, magnetometer fallback), and a flat passthrough quad behind the scene (initially a static camera texture or a test pattern). Passthrough rendering (Zach, 2026-11-13, clarified 14:33): the material is a **cutout**. That region of the screen renders the camera output — as if the camera feed was on the screen the entire time, with the 3D scene composited on top and a hole cut where passthrough shows. The camera feed is the base layer, not a texture on a quad; the passthrough surface is the hole in the scene through which it shows. (Supersedes the earlier "unlit material presenting the camera texture" framing: the material doesn't display the frame, it cuts the scene away from it.) Godot facilities that exist: `SubViewport` + `SubViewportContainer` for split-screen stereo; `Camera3D` with custom frustum for off-center projection; `Input.get_accelerometer()` / `get_gyroscope()` / `get_magnetometer()` on Android (Godot exposes the sensors); the Asset Library "VR Mobile Camera" plugin is a working precedent for exactly this pattern (screen split + IMU head tracking). Deliverables: `ZapboxStereoRig` scene (origin + left/right eye cameras + passthrough background), `zapbox_tracking.gd` IMU orientation path, a lens-calibration resource (IPD, FOV, distortion coefficients as editable data). Verification: on-device — look around, orientation tracks 1:1 with low latency; stereo image fuses through the lenses; headless — orientation math unit tests (quaternion integration vs known rotations), viewport layout checks. ## Phase 2 — Camera frames + marker detection + head pose (hard) What: get the phone camera into Godot, detect our own printed markers in the frame, solve each marker's 6DoF pose, and anchor the world to the origin marker. Pieces: - **Camera path (feasible with work):** Godot has no camera-frame API. Build a Godot Android plugin (v2, Java/Kotlin) around Camera2: open the wide (or ultrawide where present) camera, manual exposure control (short exposure, per upstream's tracking rationale), YUV frames delivered into a Godot texture each frame. Expose camera intrinsics + lens selection to GDScript. This is the first native-code surface and the first thing that needs the export templates + a device. - **Marker artwork (official markers as reference + custom trackers supported):** Zach will have official hardware, so the official Zapbox markers are the reference (photograph/measure them to define the detector's reference data). Custom trackers must ALSO be supported for world-tracking calibration — live re/calibration against user-defined markers. The detector therefore takes a marker registry (id, size, role: origin/side/custom) rather than a hardcoded set, and the origin marker is our equivalent of the "zapbox" floor anchor. Printable artwork lives in the repo as SVGs/PDFs + a `markers.json`. - **Detector (hard, but a solved problem family):** OpenCV ArUco detection + `solvePnP` pose estimation, wrapped in a GDExtension (C++) or in the Android plugin (Java + OpenCV Android SDK). ArUco is the standard fiducial family; the work is integration, calibration (camera intrinsics per phone), and robustness (motion blur, lighting, wide-FOV distortion, partial occlusion), not algorithm invention. - **Pose fusion:** multi-marker observations fused into one head pose (origin marker defines the Floor frame; other markers positioned relative to it); when no marker is visible, fall back to IMU-only orientation + a neck-model-style position hold (same fallback shape as upstream, implemented from scratch). Deliverables: `zapbox_markers.gd` (marker registry + detection interface), the detector extension/plugin, printable marker set, `zapbox_tracking.gd` marker-fusion path. Verification: headless — detector unit tests on synthetic + recorded frames (known marker poses recovered within tolerance); pose math tests; on-device — walk the play space, position holds while ≥1 marker is visible, degrades gracefully (no jumps) when markers leave view; origin stays fixed across sessions. ## Phase 3 — Controllers: BLE input + calibration + pose (hardest) What: pair the two Zapbox controllers, read their inputs in the SDK's control surface (thumbstick, trigger, grip, primary/secondary/menu buttons, tracking state, pose, velocity), run the calibration flows, and track controller pose. Pieces: - **BLE input (feasible once the protocol is known; unknown today):** Android BluetoothGatt in the Android plugin. The controller BLE protocol (services/characteristics, packet format) is not documented in any public file. Path per Zach (2026-11-13): dig deeper online first (community resources, teardowns, prior art) — otherwise reverse-engineer it from hardware in hand: discover GATT services, observe the characteristic traffic for buttons/stick/trigger/IMU, and map it to the control surface. This is the single biggest unknown. - **Calibration flows (feasible):** motion-sensor calibration phase (controllers flat and untouched, thumbstick centred), analog trigger auto-range, thumbstick scale to full −1..+1. Implement as a pairing UI flow in Godot (mirrors the upstream flow's shape; own UI, own code). - **Controller pose (hard):** upstream tracks controllers optically via the phone camera (pose updates only while in view of the ultrawide camera) fused with IMU. Our path: (a) detect the controller's visual marker in the camera frame (requires knowing the controller's marker geometry — closed upstream; our options are to characterize the physical controller's visible pattern with hardware in hand, or attach our own marker to the controllers), plus (b) IMU orientation from the controller, fused; (c) out-of-view fallback: orientation-only hold + the over-shoulder heuristic shape (orientation-based shoulder estimate) if wanted. If optical controller tracking proves infeasible without Zappar's marker geometry, the honest fallback is IMU orientation + position inferred from the head pose and a kinematic model — less capable, and that limitation must be stated, not papered over. Deliverables: `zapbox_controllers.gd` (pairing, input state, calibration state machine), BLE plugin surface, controller input map matching the SDK's control names, pairing/calibration UI. Verification: headless — input mapping tests (synthetic BLE packets → control states), calibration math tests; on-device — pair both controllers, every control reads correctly, thumbstick full range, trigger range calibrates, controller pose tracks while in camera view. ## Phase 4 — Tracking fusion, prediction, frame pacing (hard) What: make the whole thing feel stable at 60 fps on a phone. Pieces: - **Fusion:** one pose pipeline — marker observations + phone IMU + controller IMU, with timestamp alignment, outlier rejection (a bad marker solve must not teleport the world), and smooth handoff between marker-visible and marker-lost states. - **Prediction:** extrapolate head/controller pose by one frame (and by the camera-to-photon latency) so motion feels tight; upstream's roadmap mentions accelerometer data to cut perceived latency — same idea, own implementation. - **Frame pacing (hard, no late-warp):** Godot has no async timewarp/late-warp for the cardboard path. The budget is discipline: keep the frame under ~10–16 ms (Mobile renderer, minimal overdraw, capped pixel ratio), and measure with on-device frame-time logging. True late-warp would need engine-level rendering work and is explicitly out of scope for this rebuild; if frame times can't be held, the fallback is lowering resolution, not warping. - **Stereo rendering details:** per-eye distortion correction in a shader (or pre-distorted projection), chromatic-aberration approximation if the lenses need it, IPD calibration. Deliverables: fused pose pipeline in `zapbox_tracking.gd`, frame-time HUD/logging, distortion shader, performance budget doc. Verification: on-device — sustained 60 fps in a simple scene with passthrough + markers + controllers; no visible world swim when walking; headless — fusion tests (outlier rejection, handoff smoothness). ## Phase 5 — Android bindings for WebXR apps (Part 2) What: an Android host that loads WebXR pages — a local file or a web-loaded URL — and binds Zapbox tracking + controllers into the WebXR session, so WebXR content runs on the phone with Zapbox as its XR input. Per Zach, standalone capable of running WebGL is the primary focus for the phone experience. Pieces: - **Host app:** an Android app (Godot export or a thin native shell) that hosts a WebView, loads a WebXR page from a local file or URL, and exposes the WebXR Device API to it. The host owns the Zapbox session (tracking + controllers from Phases 2–4) and feeds poses/inputs into the page's XR session. - **Bindings:** map the Part 1 tracking + controller state onto WebXR constructs (viewer pose, input sources with grip/target-ray spaces, button/axis state per the WebXR gamepad mapping). The page must not need to know about Zapbox — it sees a standard WebXR session. - **Scope note:** this is a host + binding layer, not a browser. Local-file loading must work offline; web-loaded pages must work over the network. Zapbox Link / PC VR is a stretch goal, explicitly out of this phase. Deliverables: host app, WebXR binding layer, a sample WebXR page (local file) exercising headset pose + both controllers, docs (how to point the host at a page). Verification: on-device — load the sample page from a local file and from a URL; headset pose + controller inputs drive the WebXR scene; headless — binding mapping tests (Part 1 state → WebXR session state). ## Phase 6 — Video capture + spectator (defer) What: first-person recording (record camera feed + composited view) and a spectator mode. Deferred behind Phases 1–5: recording triples render load (upstream renders the scene 3× while recording) and spectator mode needs a second device + a recorder app. Stub the API (`zapbox_capture.gd`) in the skeleton; build only after tracking is solid. Android capture path would be MediaRecorder/MediaProjection, not iOS Photos. ## Phase 7 — Packaging (after Phase 1 works on-device) What: the Godot Android plugin(s) as a proper addon (camera + BLE), export preset + signing, a sample scene (grab interaction, mirroring the upstream sample's intent), docs (setup guide: markers to print, pairing flow, calibration), and a decision on distribution (sideload vs store). Covers both parts (Part 1 SDK + Part 2 WebXR host). ## Hard parts — honest assessment | # | Hard part | Assessment | |---|---|---| | 1 | Marker detection + pose from a phone camera | **Hard but feasible.** ArUco + solvePnP is a proven family; the work is integration (GDExtension/Android plugin), per-phone calibration, and robustness (blur, light, wide FOV). Needs printed markers + a test phone. No new algorithms required. | | 2 | Controller tracking + pairing (BLE protocol, optical pose) | **Hardest, and partly unknown.** The BLE protocol is undocumented publicly; establishing it needs hardware in hand and GATT traffic observation. Optical controller pose needs the controller's marker geometry, which is closed — either characterize the physical controllers ourselves or attach our own markers. IMU-only fallback is possible but much less capable. This is the phase most likely to force a scope cut. | | 3 | Late-warp / frame pacing on Android | **Hard, with a ceiling.** No late-warp exists in Godot for this path; the mitigation is prediction + a strict frame budget. If the budget can't be held on the target phone, the experience degrades (tearing/judder) and there is no warp to save it. | | 4 | Stereo rendering for the headset | **Feasible.** Two SubViewports + off-center cameras is a known pattern (VR Mobile Camera asset is the precedent). Lens distortion tuning is fiddly but tractable. | | 5 | Android camera path | **Feasible with work.** Camera2 via a Godot Android plugin is the documented route; YUV→texture, exposure control, ultrawide selection are all standard Camera2. Needs export templates + a device; nothing exotic. | | 6 | License / IP | **Constraining, not blocking.** Clean-room + own marker artwork + for-use-with-Zapbox-hardware keeps the rebuild inside the license's permitted use. Zach (2026-11-13): the official license is fine — Zappar is free to take and reuse any of the code, though unlikely to do so officially (GDScript over Unity). The IP-assignment clause for derivative works of the Work remains a standing fact; the rebuild avoids being a derivative of the Work by being written fresh. | | 7 | Desktop wands via webcam (Part 1 front-end) | **Feasible, new surface.** Same tracking core as Phase 2 with a webcam frame source on a desktop Godot build instead of the phone camera. Needs desktop camera access in Godot + BLE pairing from desktop. Rides on the Phase 2 detector; do after the phone path works. | | 8 | Android WebXR bindings (Part 2) | **Feasible with work.** The host + binding layer is standard Android/WebView work; the risk is WebXR conformance (the page must see a standard session) and the WebView's XR support, not the Zapbox side. Primary phone focus per Zach. | ## Open questions for Zach — answered 2026-11-13 (comment 999 on issue #1) + one remaining Answered (his words, condensed): 1. **Hardware:** he will be obtaining a Zapbox shortly. 2. **Controller protocol:** dig deeper online first; otherwise reverse-engineer it. 3. **Markers:** official Zapbox markers as reference (he will have official hardware); custom trackers supported for world-tracking calibration, to live re/calibrate. 4. **Repo home:** this repo (`muse-lantern/zapbox-godot`). 5. **Target phones:** Galaxy S25 Ultra for now; custom calibration for Android phones supported (not all phones support the same camera set). 6. **Scope:** Zapbox Link / PC VR is a stretch goal; standalone capable of running WebGL is the primary focus. 7. **License:** the official license is fine; Zappar is free to take and reuse any of the code, though unlikely to officially (GDScript over Unity). Remaining: none. Mapping decided 2026-11-13 (Zach): the Godot headset MR stays a deliverable in its own right, not a dev/test path for Part 2. Passthrough rendering decided the same day, clarified 14:33: the material is a cutout — that screen region renders the camera output, as if it was on the screen the entire time (camera feed is the base layer; the 3D scene composites on top with a hole cut where passthrough shows).

merith-tk commented 2026-11-13 20:36:17 +00:00

Copy link

For the open questions

1. I will be obtaining one shortly.
2. Dig deeper online otherwise we will be reverse engineering this shit
3. official zapbox markers as I will have official hardware, however custom trackers should be supported for world tracking calibrations. To live re/calibrate.
4. this repo
5. for now we target the Galaxy S25U, but custom calibration for android phones is going to be supported as not all phones support it
6. ZapLink/PCVR as stretchgoal, standalone capable of running webgl is primary focus.
7. official license is fine. They are free to take and reuse any of the code, however they are unlikely to do so officially due to GDScript over unity.

For the open questions 1) I will be obtaining one shortly. 2) Dig deeper online otherwise we will be reverse engineering this shit 3) official zapbox markers as I will have official hardware, however custom trackers should be supported for world tracking calibrations. To live re/calibrate. 4) this repo 5) for now we target the Galaxy S25U, but custom calibration for android phones is going to be supported as not all phones support it 6) ZapLink/PCVR as stretchgoal, standalone capable of running webgl is primary focus. 7) official license is fine. They are free to take and reuse any of the code, however they are unlikely to do so officially due to GDScript over unity.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

No results found.

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

muse-lantern/zapbox-godot#1

No description provided.