feat(Touchscreen): edge-swipe gesture detection for handheld#571
feat(Touchscreen): edge-swipe gesture detection for handheld#571honjow wants to merge 8 commits into
Conversation
Add Touchscreen as a new top-level Capability variant with Touch and Gesture sub-types. Gesture covers single-finger edge-swipe directions (SwipeLeft/Right/Up/Down) with optional zone qualifiers (Top/Bottom). Update InputValue translation, evdev/dbus event enums, and capability map config to handle the new types.
… replay Implement single-finger edge-swipe gesture detection in the touchscreen source device: - Detect swipes starting from left/right/top/bottom screen edges - Optional exclusive grab mode: suppresses raw touch events while a gesture is being evaluated, replaying them as a tap if no gesture is confirmed - Orientation config to rotate coordinates for rotated displays - TouchscreenConfig extended with grab and orientation fields
…le mapping - Pass Touchscreen capability events through translate_event so they can be mapped to gamepad buttons via device profiles - Bypass intercept-mode checks for touchscreen events (gesture handling is always active) - Add Touchscreen capability to the output capability lists of DualSense, SteamDeckUhid and UnifiedGamepad target devices
Map SwipeRight:Bottom to Guide and SwipeLeft:Bottom to QuickAccess in the default profile, matching hhd-style Steam/QAM gestures. Update composite_device_v1.json to add grab field to TouchscreenConfig. Update device_profile_v1.json to include Touchscreen gesture capability definitions.
Button press+release pairs arriving within 80ms now have their release delayed at the TargetDriver layer, so all target devices benefit automatically. Previously this logic only existed in steam_deck. Add ScheduledNativeEvent::event() accessor to support filtering scheduled events during ClearState. Also clear pending button events when ClearState is called to avoid ghost releases after intercept mode resets device state.
…NEXPLAYER X1 GPD Win5: enable touchscreen source with grab mode. ONEXPLAYER X1 Mini: enable touchscreen source with grab mode and left orientation (display is rotated 90 degrees).
honjow
changed the title
ShadowApex
left a comment
ShadowApex
left a comment
There was a problem hiding this comment.
This feature is definitely something we would like to include, but we may need to consider exactly how we want to implement this.
| "grab": { | ||
| "description": "If true, the touchscreen will be exclusively grabbed so raw touch events are not passed to other consumers. Defaults to false.", | ||
| "type": "boolean" | ||
| }, |
There was a problem hiding this comment.
This is not necessary as this is already managed using the passthrough config option.
| /// If true, the device will be grabbed exclusively. Touch events will be | ||
| /// fully managed by InputPlumber; a virtual touchscreen target device will | ||
| /// be created and edge-swipe gestures will suppress raw touch delivery. | ||
| /// Defaults to false (pass-through mode). | ||
| #[serde(skip_serializing_if = "Option::is_none")] | ||
| pub grab: Option<bool>, |
There was a problem hiding this comment.
Also here, this is already implemented by passthrough.
| /// True if this composite device has a grabbed touchscreen source. | ||
| /// When true, the touchscreen target will always be kept in the target list. | ||
| has_touchscreen_source: bool, |
There was a problem hiding this comment.
We should not always keep a touchscreen target in the target list. This should be controllable by the user. There may be cases where the user does not want to emulate a touchscreen (e.g. by converting the touchscreen inputs into touchpad inputs).
| // If the composite device has a grabbed touchscreen source, always keep | ||
| // touchscreen in the target list regardless of what the caller requested. | ||
| if self.has_touchscreen_source { | ||
| let touchscreen_id: TargetDeviceTypeId = TOUCHSCREEN_TARGET_ID.try_into().unwrap(); | ||
| if !device_types.contains(&touchscreen_id) { | ||
| log::debug!("[{dbus_path}] Auto-adding touchscreen target (grabbed touchscreen source detected)"); | ||
| device_types.push(touchscreen_id); | ||
| } | ||
| } |
There was a problem hiding this comment.
Also here, we should not assume to always emulate a virtual touchscreen. Users may not want to use their touchscreen as a touchscreen.
| /// Lifecycle of a single-finger edge-swipe gesture within one touch sequence | ||
| #[derive(Debug, Default)] | ||
| enum GesturePhase { | ||
| /// No touch in progress. | ||
| #[default] | ||
| Idle, | ||
| /// Touch started; watching for a gesture. | ||
| /// In grab mode, `suppressing` indicates whether slot-0 events are being | ||
| /// held back because the touch started in an edge zone. | ||
| Tracking { suppressing: bool }, | ||
| /// A gesture was recognized; suppress remaining touch events until lift. | ||
| Triggered, | ||
| /// A second finger arrived; gesture detection disabled for this touch. | ||
| Invalidated, | ||
| } |
There was a problem hiding this comment.
I'm not sure if we actually want gesture detection to take place in the source device implementation. We may want to instead handle this in the composite device.
@pastaq thoughts?
There was a problem hiding this comment.
I agree, but I'm having difficulty imagining how that would work exactly. They don't cleanly map to native events, and we don't want to lose the rest of the touchscreen.
Perhaps we should allow for a config file or dbus command to assign a capability_map to specific regions by start/end for x and y by % of the screen height/width, then the driver can translate those regions to a specific native event. This would be more flexible than just using gestures as it would allow for a fully virtual gamepad or touchpad on the screen itself. An overlay program could then assign those regions to match the location for its glyphs over the top of a game.
3 participants
Summary
Adds single-finger edge-swipe gesture detection to the touchscreen source device, allowing touchscreen edges to be mapped to gamepad buttons via device profiles — enabling gestures like swiping from the left edge to open the Steam menu, or from the right edge to open the Quick Access Menu, without requiring changes to the compositor.
Tested and verified on GPD Win5 and ONEXPLAYER X1 Mini.
How gesture detection works
Touch coordinates are normalized to
[0.0, 1.0]. A gesture is recognized when the touch starts within 3% of a screen edge, travels inward by at least 12%, and completes within 400 ms.Gesture names describe the direction of finger movement (e.g. swiping inward from the left edge moves rightward →
SwipeRight). Each direction carries aTop/Bottomzone qualifier split at 33% along the perpendicular axis.Default profile mappings:
SwipeRight:BottomGuideSwipeLeft:BottomQuickAccessTwo operating modes: passthrough vs. grab
Passthrough (
grab: false, default): Raw touch events are forwarded normally alongside any gesture events. Suitable for desktop use; avoid in game compositors as gestures will also register as taps.Grab (
grab: true): The touchscreen is exclusively captured viaEVIOCGRAB. Touches starting within 1% of an edge are held back until the gesture outcome is known — confirmed gestures suppress the touch, unconfirmed ones replay it as a normal tap. Multi-finger touch immediately cancels gesture detection and passes all events through. IfEVIOCGRABfails, the source falls back to passthrough with a warning.Grab mode is recommended for devices running a game compositor such as Gamescope.
Why MIN_FRAME_TIME was moved to TargetDriver
Gesture detection emits press and release nearly simultaneously. Without a minimum press duration, both events fall within a single HID poll interval and the compositor sees a report where the button was never pressed.
steam_deckalready handled this with an 80 ms minimum, but only for itself. This PR moves the enforcement intoTargetDriverso every target benefits automatically, and removes the now-redundantsteam_deckimplementation.