SDK inventory: public-surface map of zappar-xr/zapbox-xr-sdk v0.5.0 #2

New issue

Open

opened 2026-11-13 20:10:07 +00:00 by muse-lantern · 0 comments

muse-lantern commented 2026-11-13 20:10:07 +00:00

Owner

Copy link

Zapbox XR SDK — inventory (public surface only)

Source: zappar-xr/zapbox-xr-sdk, v0.5.0, head 91b52bd (14 commits). Local reference clone: ~/workspace/zapbox-xr-sdk-upstream.
License: Zappar custom license (LICENSE.md) — use only in connection with Zapbox hardware; no competitive analysis; no developing a competing product or service; no altering binary libraries; no circumventing expiry; Zappar owns IP in the Work and derivative works. This inventory is clean-room: public names, signatures, file layout, and public docs only. No code bodies, no implementation logic, no binary analysis.

1. Package identity

• package.json: name com.zappar.xr.zapbox, displayName "Zapbox XR SDK", version 0.5.0, unity: 2019.4. Description: "Zapbox XR SDK for Unity from Zappar. Providing rendering and controller input support for Zapbox." Keywords: zappar, zapbox, vr, xr, ar, sdk.
• Dependencies: com.unity.xr.management 3.2.9, com.unity.xr.legacyinputhelpers 2.1.4.
• Assemblies: Zappar.XR.Zapbox (Runtime; references Unity.XR.Management, Unity.InputSystem; include platforms Editor + iOS; UNITY_INPUT_SYSTEM version define) and Zappar.XR.Zapbox.Editor (Editor; references Unity.XR.Management.Editor, Unity.XR.Management, Zappar.XR.Zapbox; XR_MGMT_GTE_320 define at xr.management 3.2.0).

2. File layout (public surface)

Editor/
  BuildPostProcessor.cs          (91 lines)
  ZapboxMetadata.cs              (63 lines)
  Zappar.XR.Zapbox.Editor.asmdef
Plugins/
  UnitySubsystemsManifest.json
  iOS/
    GfxPluginZapbox.a            (17,570,264 bytes — native library, BLACK BOX)
    GfxPluginZapbox.m            (37 lines — plugin shim, see §5)
    Zapbox.bundle/
      ZapBoxController.zbs       (1,802 bytes — BLACK BOX payload)
      ZapBoxMarker.zbs           (7,464 bytes — BLACK BOX payload)
      ZapBoxMarker.zcs           (153 bytes — BLACK BOX payload)
      ui/                        (pairing/calibration web UI assets, see §6)
Resources/
  Zapbox/
    left.fbx / right.fbx         (Kaydara FBX 7400 — in-app controller models)
    controller_left/ controller_right/  (per-model texture PNGs, Image_0/1/3/4)
Runtime/
  ApiConstants.cs                (42 lines)
  ZapboxLoader.cs                (153 lines)
  ZapboxSettings.cs              (13 lines)
  InputSystem/ZapboxController.cs (92 lines)
  Zappar.XR.Zapbox.asmdef

(*.meta files alongside each file are Unity import metadata, not listed.)

3. Subsystem manifest

Plugins/UnitySubsystemsManifest.json: name Zapbox, version 0.1.0, libraryName: GfxPluginZapbox, subsystems:

• display: id: ZapboxDisplay
• input: id: ZapboxInput

4. Runtime C# provider — public API shape

4.1 ApiConstants (Zappar.XR, internal static class)

• kPluginName: "GfxPluginZapbox" on UNITY_ANDROID, "__Internal" on UNITY_IOS, "NOT_AVAILABLE" otherwise. (Note: the Android branch exists in the constants even though no Android build is shipped.)
• kDeviceName: "Zapbox".
• kDisplayProviderId: "ZapboxDisplay"; kInputProviderId: "ZapboxInput".
• Device ids: kDeviceIdZapboxHeadset = 0, kDeviceIdControllerLeft = 1, kDeviceIdControllerRight = 2.

4.2 ZapboxLoader : XRLoaderHelper (Zappar.XR)

Lifecycle entry points (the XR Management loader contract — names only):

• public override bool Initialize()
• public override bool Start()
• public override bool Stop()
• public override bool Deinitialize()

State (field/property names only): _displaySubsystemDescriptors (List<XRDisplaySubsystemDescriptor>), _inputSubsystemDescriptors (List<XRInputSubsystemDescriptor>), _isInitialized (internal static bool), _isStarted (internal static bool), _prevFramerate (int).

InputLayoutLoader (static class, [InitializeOnLoad]): static constructor + public static void RegisterInputLayouts() — registers the input layouts below with Unity's Input System.

4.3 ZapboxSettings : ScriptableObject

Empty serializable settings object. Required by the XR Management package; carries no fields of its own.

4.4 Input system — ZapboxController (Zappar, InputControlLayout displayName "Zapbox Controller", commonUsages LeftHand / RightHand)

Layout class ZapboxControllerProfile.ZapboxController : XRController. Controls (name, type, aliases) — this is the controller's public input surface:

Control	Type	Aliases
thumbstick	Vector2Control	primary2DAxis, joystick
trigger	AxisControl	—
grip	AxisControl	—
primaryButton	ButtonControl	A, X
secondaryButton	ButtonControl	B, Y
menu	ButtonControl	menuButton, systemButton, start
thumbstickClicked	ButtonControl	primary2DAxisClick, joystickOrPadPressed, thumbstickClick
triggerPressed	ButtonControl	triggerButton, indexButton
gripPressed	ButtonControl	gripButton
trackingState	IntegerControl	controllerTrackingState
isTracked	ButtonControl	controllerIsTracked
devicePosition	Vector3Control	controllerPosition
deviceRotation	QuaternionControl	controllerRotation
deviceVelocity	Vector3Control	controllerVelocity

FinishSetup() binds each control by child-control name (same names as above). The layout's own comment states it follows Oculus Touch terminology (e.g. gripPressed) because those are the names the XR Interaction Toolkit defaults use.

Inputs (what flows in): headset pose, per-controller pose/velocity/tracking state, thumbstick, trigger/grip analog values, button states.
Outputs (what flows out): the above as Unity Input System state, consumable by the XR Interaction Toolkit (grab interactions work out of the box per the README).

5. iOS native bridge — exported symbols only

Plugins/iOS/GfxPluginZapbox.m (37 lines) is a shim, not the implementation. Exported symbols (names only):

• UnityPluginLoad(IUnityInterfaces* unityInterfaces) / UnityPluginUnload() — standard Unity native-rendering-plugin entry points (UNITY_INTERFACE_EXPORT).
• +loadPlugin — registers the plugin via UnityRegisterRenderingPluginV5 and calls the external ZapboxDisplay_SetUnityViewController(UnityGetGLViewController()).
• External references (declared, implemented in the binary): ZapboxDisplay_SetUnityViewController(UIViewController*), UnityGetGLViewController().

Plugins/iOS/GfxPluginZapbox.a (17.5 MB, current ar archive) is the native implementation — black box, not analyzed. What its .meta declares it needs: iOS only (Android excluded), framework dependencies CoreBluetooth, IOSurface, Metal, MetalKit. From those declared dependencies alone: controllers connect over Bluetooth; camera frames pass through IOSurface/Metal; rendering is Metal.

6. Bundle payloads — names only, black box

• ZapBoxMarker.zbs (7,464 bytes), ZapBoxController.zbs (1,802 bytes), ZapBoxMarker.zcs (153 bytes). File-magic only (no decoding): both .zbs files begin with the bytes ZBS followed by a version byte (0x02 marker, 0x01 controller); the .zcs begins with ZCS + 0x01. Their roles, from their names and the README's marker/controller discussion: the marker payload describes the world-anchor markers; the controller payload describes the controller's trackable; the .zcs is a small companion config. Contents not read.
• Zapbox.bundle/ui/ — pairing/calibration web UI assets (file names only): index.html, index.*.js, index.*.css, zcomponent.*.css, LatoBold / LatoRegular fonts, controller_left.*.glb / controller_right.*.glb, button/ring/tick/logo/gradient images. No contents analyzed; the asset set is consistent with an in-app controller-pairing/calibration UI (see §9).
• Resources/Zapbox/left.fbx + right.fbx with per-model PNG textures: the in-app 3D controller models.

7. Editor scripts — stated responsibilities

• Editor/BuildPostProcessor.cs — public static class BuildPostProcessor, [PostProcessBuild] public static void OnPostProcessBuild(BuildTarget buildTarget, string buildPath), plus private helper ensureKeyIsSet(PlistElementDict root, string key, string defaultValue). Runs only when buildTarget == BuildTarget.iOS. Stated responsibility (from names + Unity's post-process contract): after an iOS build, adjust the generated Xcode project/plist so the app is configured for the Zapbox provider. (Which keys: not recorded — that is implementation detail.)
• Editor/ZapboxMetadata.cs — public class ZapboxPackage : IXRPackage. Metadata values: packageName "Zapbox XR Plugin", packageId com.zappar.xr.zapbox, settingsType Zappar.XR.ZapboxSettings, loaderName "Zapbox", loaderType Zappar.XR.ZapboxLoader, supportedBuildTargets: [BuildTargetGroup.iOS] (iOS only). PopulateNewSettingsInstance(ScriptableObject obj) — creates the settings instance for XR Management.

8. Marker / world-anchor system (public docs: README + zappar.com)

• Tracking origin mode: only "Floor" is supported. The world anchor with the "zapbox" text on it is the origin of the tracking space; it is placed horizontally at the front edge of the play space.
• Multiple world markers increase the tracking volume; the origin and orientation of the space come from the "zapbox"-text anchor. Other markers are placed higher up at the front sides of the play space so they stay in view of the phone camera when looking forward; full markers must be visible, no particular orientation required.
• At least one world marker must be in view for positional tracking of the headset.
• When no markers are in view, a neck model provides approximate position updates (added 0.4.0).
• The phone camera is the tracking sensor (ultrawide preferred; main camera works with reduced FoV for pass-through and controller tracking). The ultrawide-camera adapter included in the box is not yet supported by this version.
• Camera exposure is hardcoded short to improve controller tracking; bright indoor lighting works best.
• The markers are Zappar "pointcodes" — printed fiducial codes (an evolution of Zapcodes/QR-style codes) placed around the room; the headset's world anchors are physical printed symbols.
• The SDK's marker definitions live in the closed .zbs payloads (§6) — the marker artwork/geometry is not specified in any public file in this repo.

9. Controller pairing + calibration flow (public docs: README 0.4.0 / 0.5.0 notes + zappar.com)

• Two Bluetooth 6DoF controllers. Inputs mirror Oculus Quest (per zappar.com: "mirroring the inputs of other Mixed Reality headsets"; in Zapbox Link the controllers are reported as Meta Quest controllers, with A/B mapped to the left-/right-most buttons).
• Pairing flow (named phases, from README + bundle UI assets): a controller-pairing UI that includes a "motion sensor calibration" phase. During it the thumbstick centre is auto-calibrated — the controllers must sit untouched on a flat surface.
• Analog trigger: automatic range calibration (0.4.0). Thumbstick scale factors updated so the full −1 to +1 range is available on all controllers (0.4.0).
• App lifecycle: controllers disconnect when the app goes to background; pairing flow improved across versions.
• Controller pose updates only while the controller is in view of the phone's ultrawide camera (public Zapbox Link guide). Over-shoulder detection (Zapbox Link): when a controller points back toward the user and slightly down with the trigger up, it is assumed to be near the shoulder and reported as such (orientation-only heuristic).

10. Frame pacing / late-warp (public docs: README)

• The frame workload must complete within ~10 ms or so, or tearing appears because the late-warp GPU work does not happen in time. Simple content is OK.
• 0.5.0: frame-pacing improvements for iOS 26.
• HMD pose is reported at the center of the eyes (0.5.0; previously at the camera position).
• Near-term roadmap (README): move the late-warp process to a compute shader to remove tearing when content graphics work takes longer than a frame; leverage accelerometer data to reduce perceived camera latency and positional jitter.
• The late-warp implementation itself lives in the closed native library (§5) — no public spec in this repo.

11. Video capture modes (public docs: README)

• First-person view: double-tap near the top of the device screen (landscape, just right of the center line) starts/stops recording; the clip lands in the Photos app. Records the full 1440×1080 camera feed with a Unity view overlaid to align exactly with the camera. During recording Unity renders the scene 3 times, with a performance impact.
• Third-person / Spectator Mode: requires a second iPhone (ARKit-capable) running the dedicated "Zapbox Recorder" app; captures the wearer and the content from outside. (The Recorder app was "being prepared for App Store / TestFlight" at the time of writing.)
• 0.5.0 removed the hardcoded expiry date (earlier builds expired).

12. What is iOS-only by construction

• The entire native layer: GfxPluginZapbox.a (iOS, Metal/IOSurface/CoreBluetooth), the Xcode build post-processor, the supportedBuildTargets: [iOS] metadata, the .a.meta Android exclusion.
• Camera path: iPhone ultrawide camera, portrait orientation required (enables lower-latency compositing on iOS), hardcoded short exposure.
• The Unity side already anticipates Android in one place only: ApiConstants.kPluginName has a UNITY_ANDROID branch ("GfxPluginZapbox"). No Android native library, manifest entry, or build support ships.
• README "Future work": Android support. (Upstream has no Android code to port — the Android implementation is new work, which is why this is a rebuild, not a port.)

13. What Android needs instead (inventory-level, detail in the rebuild plan)

• Camera: Android Camera2 (wide/ultrawide selection, manual exposure control, YUV frames) delivered into Godot as a texture — Godot has no built-in camera-frame API; this needs a Godot Android plugin.
• Tracking: no ARKit. Options are a custom marker detector (OpenCV ArUco family via GDExtension or Android plugin) or ARCore Augmented Images — the detector must be built, and the marker artwork must be defined by us (the SDK's marker definitions are closed).
• Controllers: Bluetooth LE (Android BluetoothGatt) for input; optical pose from the camera + IMU fusion for position. The controller BLE protocol is not documented in any public file in this repo.
• Rendering: no OpenXR runtime on a plain phone — stereo must be done the "cardboard" way (two eye viewports, lens-distortion-aware projection), not via Godot's OpenXR interface.
• Frame pacing: no late-warp in Godot for this path — budget discipline + pose prediction instead.

14. Load-bearing closed parts (plain statement)

The tracking core lives entirely in the closed native library + closed bundle payloads: marker detection/pose estimation, controller optical tracking, sensor fusion, late-warp, and the marker/controller definitions (.zbs/.zcs). There is no public spec for any of them in this repo. The rebuild therefore cannot reuse Zappar's tracker — it must implement its own tracker from first principles (marker detection, pose estimation, fusion) against marker artwork it defines itself, and treat the controllers as BLE input devices whose protocol must be established with hardware in hand. See the rebuild plan's open questions.

# Zapbox XR SDK — inventory (public surface only) Source: `zappar-xr/zapbox-xr-sdk`, v0.5.0, head `91b52bd` (14 commits). Local reference clone: `~/workspace/zapbox-xr-sdk-upstream`. License: Zappar custom license (`LICENSE.md`) — use only in connection with Zapbox hardware; no competitive analysis; no developing a competing product or service; no altering binary libraries; no circumventing expiry; Zappar owns IP in the Work and derivative works. **This inventory is clean-room: public names, signatures, file layout, and public docs only. No code bodies, no implementation logic, no binary analysis.** ## 1. Package identity - `package.json`: name `com.zappar.xr.zapbox`, displayName "Zapbox XR SDK", version `0.5.0`, `unity: 2019.4`. Description: "Zapbox XR SDK for Unity from Zappar. Providing rendering and controller input support for Zapbox." Keywords: zappar, zapbox, vr, xr, ar, sdk. - Dependencies: `com.unity.xr.management` 3.2.9, `com.unity.xr.legacyinputhelpers` 2.1.4. - Assemblies: `Zappar.XR.Zapbox` (Runtime; references Unity.XR.Management, Unity.InputSystem; include platforms Editor + iOS; `UNITY_INPUT_SYSTEM` version define) and `Zappar.XR.Zapbox.Editor` (Editor; references Unity.XR.Management.Editor, Unity.XR.Management, Zappar.XR.Zapbox; `XR_MGMT_GTE_320` define at xr.management 3.2.0). ## 2. File layout (public surface) ``` Editor/ BuildPostProcessor.cs (91 lines) ZapboxMetadata.cs (63 lines) Zappar.XR.Zapbox.Editor.asmdef Plugins/ UnitySubsystemsManifest.json iOS/ GfxPluginZapbox.a (17,570,264 bytes — native library, BLACK BOX) GfxPluginZapbox.m (37 lines — plugin shim, see §5) Zapbox.bundle/ ZapBoxController.zbs (1,802 bytes — BLACK BOX payload) ZapBoxMarker.zbs (7,464 bytes — BLACK BOX payload) ZapBoxMarker.zcs (153 bytes — BLACK BOX payload) ui/ (pairing/calibration web UI assets, see §6) Resources/ Zapbox/ left.fbx / right.fbx (Kaydara FBX 7400 — in-app controller models) controller_left/ controller_right/ (per-model texture PNGs, Image_0/1/3/4) Runtime/ ApiConstants.cs (42 lines) ZapboxLoader.cs (153 lines) ZapboxSettings.cs (13 lines) InputSystem/ZapboxController.cs (92 lines) Zappar.XR.Zapbox.asmdef ``` (`*.meta` files alongside each file are Unity import metadata, not listed.) ## 3. Subsystem manifest `Plugins/UnitySubsystemsManifest.json`: name `Zapbox`, version `0.1.0`, `libraryName: GfxPluginZapbox`, subsystems: - display: `id: ZapboxDisplay` - input: `id: ZapboxInput` ## 4. Runtime C# provider — public API shape ### 4.1 `ApiConstants` (`Zappar.XR`, internal static class) - `kPluginName`: `"GfxPluginZapbox"` on `UNITY_ANDROID`, `"__Internal"` on `UNITY_IOS`, `"NOT_AVAILABLE"` otherwise. (Note: the Android branch exists in the constants even though no Android build is shipped.) - `kDeviceName`: `"Zapbox"`. - `kDisplayProviderId`: `"ZapboxDisplay"`; `kInputProviderId`: `"ZapboxInput"`. - Device ids: `kDeviceIdZapboxHeadset = 0`, `kDeviceIdControllerLeft = 1`, `kDeviceIdControllerRight = 2`. ### 4.2 `ZapboxLoader : XRLoaderHelper` (`Zappar.XR`) Lifecycle entry points (the XR Management loader contract — names only): - `public override bool Initialize()` - `public override bool Start()` - `public override bool Stop()` - `public override bool Deinitialize()` State (field/property names only): `_displaySubsystemDescriptors` (`List<XRDisplaySubsystemDescriptor>`), `_inputSubsystemDescriptors` (`List<XRInputSubsystemDescriptor>`), `_isInitialized` (internal static bool), `_isStarted` (internal static bool), `_prevFramerate` (int). `InputLayoutLoader` (static class, `[InitializeOnLoad]`): static constructor + `public static void RegisterInputLayouts()` — registers the input layouts below with Unity's Input System. ### 4.3 `ZapboxSettings : ScriptableObject` Empty serializable settings object. Required by the XR Management package; carries no fields of its own. ### 4.4 Input system — `ZapboxController` (`Zappar`, `InputControlLayout` displayName "Zapbox Controller", commonUsages `LeftHand` / `RightHand`) Layout class `ZapboxControllerProfile.ZapboxController : XRController`. Controls (name, type, aliases) — this is the controller's public input surface: | Control | Type | Aliases | |---|---|---| | `thumbstick` | Vector2Control | primary2DAxis, joystick | | `trigger` | AxisControl | — | | `grip` | AxisControl | — | | `primaryButton` | ButtonControl | A, X | | `secondaryButton` | ButtonControl | B, Y | | `menu` | ButtonControl | menuButton, systemButton, start | | `thumbstickClicked` | ButtonControl | primary2DAxisClick, joystickOrPadPressed, thumbstickClick | | `triggerPressed` | ButtonControl | triggerButton, indexButton | | `gripPressed` | ButtonControl | gripButton | | `trackingState` | IntegerControl | controllerTrackingState | | `isTracked` | ButtonControl | controllerIsTracked | | `devicePosition` | Vector3Control | controllerPosition | | `deviceRotation` | QuaternionControl | controllerRotation | | `deviceVelocity` | Vector3Control | controllerVelocity | `FinishSetup()` binds each control by child-control name (same names as above). The layout's own comment states it follows Oculus Touch terminology (e.g. `gripPressed`) because those are the names the XR Interaction Toolkit defaults use. Inputs (what flows in): headset pose, per-controller pose/velocity/tracking state, thumbstick, trigger/grip analog values, button states. Outputs (what flows out): the above as Unity Input System state, consumable by the XR Interaction Toolkit (grab interactions work out of the box per the README). ## 5. iOS native bridge — exported symbols only `Plugins/iOS/GfxPluginZapbox.m` (37 lines) is a shim, not the implementation. Exported symbols (names only): - `UnityPluginLoad(IUnityInterfaces* unityInterfaces)` / `UnityPluginUnload()` — standard Unity native-rendering-plugin entry points (`UNITY_INTERFACE_EXPORT`). - `+loadPlugin` — registers the plugin via `UnityRegisterRenderingPluginV5` and calls the external `ZapboxDisplay_SetUnityViewController(UnityGetGLViewController())`. - External references (declared, implemented in the binary): `ZapboxDisplay_SetUnityViewController(UIViewController*)`, `UnityGetGLViewController()`. `Plugins/iOS/GfxPluginZapbox.a` (17.5 MB, `current ar archive`) is the native implementation — **black box, not analyzed**. What its `.meta` declares it needs: iOS only (Android excluded), framework dependencies `CoreBluetooth`, `IOSurface`, `Metal`, `MetalKit`. From those declared dependencies alone: controllers connect over Bluetooth; camera frames pass through IOSurface/Metal; rendering is Metal. ## 6. Bundle payloads — names only, black box - `ZapBoxMarker.zbs` (7,464 bytes), `ZapBoxController.zbs` (1,802 bytes), `ZapBoxMarker.zcs` (153 bytes). File-magic only (no decoding): both `.zbs` files begin with the bytes `ZBS` followed by a version byte (`0x02` marker, `0x01` controller); the `.zcs` begins with `ZCS` + `0x01`. Their roles, from their names and the README's marker/controller discussion: the marker payload describes the world-anchor markers; the controller payload describes the controller's trackable; the `.zcs` is a small companion config. Contents not read. - `Zapbox.bundle/ui/` — pairing/calibration web UI assets (file names only): `index.html`, `index.*.js`, `index.*.css`, `zcomponent.*.css`, `LatoBold` / `LatoRegular` fonts, `controller_left.*.glb` / `controller_right.*.glb`, button/ring/tick/logo/gradient images. No contents analyzed; the asset set is consistent with an in-app controller-pairing/calibration UI (see §9). - `Resources/Zapbox/left.fbx` + `right.fbx` with per-model PNG textures: the in-app 3D controller models. ## 7. Editor scripts — stated responsibilities - `Editor/BuildPostProcessor.cs` — `public static class BuildPostProcessor`, `[PostProcessBuild] public static void OnPostProcessBuild(BuildTarget buildTarget, string buildPath)`, plus private helper `ensureKeyIsSet(PlistElementDict root, string key, string defaultValue)`. Runs only when `buildTarget == BuildTarget.iOS`. Stated responsibility (from names + Unity's post-process contract): after an iOS build, adjust the generated Xcode project/plist so the app is configured for the Zapbox provider. (Which keys: not recorded — that is implementation detail.) - `Editor/ZapboxMetadata.cs` — `public class ZapboxPackage : IXRPackage`. Metadata values: packageName "Zapbox XR Plugin", packageId `com.zappar.xr.zapbox`, settingsType `Zappar.XR.ZapboxSettings`, loaderName "Zapbox", loaderType `Zappar.XR.ZapboxLoader`, `supportedBuildTargets: [BuildTargetGroup.iOS]` (iOS only). `PopulateNewSettingsInstance(ScriptableObject obj)` — creates the settings instance for XR Management. ## 8. Marker / world-anchor system (public docs: README + zappar.com) - Tracking origin mode: only "Floor" is supported. The world anchor with the "zapbox" text on it is the origin of the tracking space; it is placed horizontally at the front edge of the play space. - Multiple world markers increase the tracking volume; the origin and orientation of the space come from the "zapbox"-text anchor. Other markers are placed higher up at the front sides of the play space so they stay in view of the phone camera when looking forward; full markers must be visible, no particular orientation required. - At least one world marker must be in view for positional tracking of the headset. - When no markers are in view, a neck model provides approximate position updates (added 0.4.0). - The phone camera is the tracking sensor (ultrawide preferred; main camera works with reduced FoV for pass-through and controller tracking). The ultrawide-camera adapter included in the box is not yet supported by this version. - Camera exposure is hardcoded short to improve controller tracking; bright indoor lighting works best. - The markers are Zappar "pointcodes" — printed fiducial codes (an evolution of Zapcodes/QR-style codes) placed around the room; the headset's world anchors are physical printed symbols. - The SDK's marker definitions live in the closed `.zbs` payloads (§6) — the marker artwork/geometry is not specified in any public file in this repo. ## 9. Controller pairing + calibration flow (public docs: README 0.4.0 / 0.5.0 notes + zappar.com) - Two Bluetooth 6DoF controllers. Inputs mirror Oculus Quest (per zappar.com: "mirroring the inputs of other Mixed Reality headsets"; in Zapbox Link the controllers are reported as Meta Quest controllers, with A/B mapped to the left-/right-most buttons). - Pairing flow (named phases, from README + bundle UI assets): a controller-pairing UI that includes a "motion sensor calibration" phase. During it the thumbstick centre is auto-calibrated — the controllers must sit untouched on a flat surface. - Analog trigger: automatic range calibration (0.4.0). Thumbstick scale factors updated so the full −1 to +1 range is available on all controllers (0.4.0). - App lifecycle: controllers disconnect when the app goes to background; pairing flow improved across versions. - Controller pose updates only while the controller is in view of the phone's ultrawide camera (public Zapbox Link guide). Over-shoulder detection (Zapbox Link): when a controller points back toward the user and slightly down with the trigger up, it is assumed to be near the shoulder and reported as such (orientation-only heuristic). ## 10. Frame pacing / late-warp (public docs: README) - The frame workload must complete within ~10 ms or so, or tearing appears because the late-warp GPU work does not happen in time. Simple content is OK. - 0.5.0: frame-pacing improvements for iOS 26. - HMD pose is reported at the center of the eyes (0.5.0; previously at the camera position). - Near-term roadmap (README): move the late-warp process to a compute shader to remove tearing when content graphics work takes longer than a frame; leverage accelerometer data to reduce perceived camera latency and positional jitter. - The late-warp implementation itself lives in the closed native library (§5) — no public spec in this repo. ## 11. Video capture modes (public docs: README) - First-person view: double-tap near the top of the device screen (landscape, just right of the center line) starts/stops recording; the clip lands in the Photos app. Records the full 1440×1080 camera feed with a Unity view overlaid to align exactly with the camera. During recording Unity renders the scene 3 times, with a performance impact. - Third-person / Spectator Mode: requires a second iPhone (ARKit-capable) running the dedicated "Zapbox Recorder" app; captures the wearer and the content from outside. (The Recorder app was "being prepared for App Store / TestFlight" at the time of writing.) - 0.5.0 removed the hardcoded expiry date (earlier builds expired). ## 12. What is iOS-only by construction - The entire native layer: `GfxPluginZapbox.a` (iOS, Metal/IOSurface/CoreBluetooth), the Xcode build post-processor, the `supportedBuildTargets: [iOS]` metadata, the `.a.meta` Android exclusion. - Camera path: iPhone ultrawide camera, portrait orientation required (enables lower-latency compositing on iOS), hardcoded short exposure. - The Unity side already anticipates Android in one place only: `ApiConstants.kPluginName` has a `UNITY_ANDROID` branch (`"GfxPluginZapbox"`). No Android native library, manifest entry, or build support ships. - README "Future work": Android support. (Upstream has no Android code to port — the Android implementation is new work, which is why this is a rebuild, not a port.) ## 13. What Android needs instead (inventory-level, detail in the rebuild plan) - Camera: Android Camera2 (wide/ultrawide selection, manual exposure control, YUV frames) delivered into Godot as a texture — Godot has no built-in camera-frame API; this needs a Godot Android plugin. - Tracking: no ARKit. Options are a custom marker detector (OpenCV ArUco family via GDExtension or Android plugin) or ARCore Augmented Images — the detector must be built, and the marker artwork must be defined by us (the SDK's marker definitions are closed). - Controllers: Bluetooth LE (Android BluetoothGatt) for input; optical pose from the camera + IMU fusion for position. The controller BLE protocol is not documented in any public file in this repo. - Rendering: no OpenXR runtime on a plain phone — stereo must be done the "cardboard" way (two eye viewports, lens-distortion-aware projection), not via Godot's OpenXR interface. - Frame pacing: no late-warp in Godot for this path — budget discipline + pose prediction instead. ## 14. Load-bearing closed parts (plain statement) The tracking core lives entirely in the closed native library + closed bundle payloads: marker detection/pose estimation, controller optical tracking, sensor fusion, late-warp, and the marker/controller definitions (`.zbs`/`.zcs`). There is no public spec for any of them in this repo. The rebuild therefore cannot reuse Zappar's tracker — it must implement its own tracker from first principles (marker detection, pose estimation, fusion) against marker artwork it defines itself, and treat the controllers as BLE input devices whose protocol must be established with hardware in hand. See the rebuild plan's open questions.

muse-lantern referenced this issue 2026-11-13 20:41:38 +00:00

Rebuild plan: Godot/Android implementation of the Zapbox XR SDK (clean-room) #1

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

No results found.

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

muse-lantern/zapbox-godot#2

No description provided.