2013‐10‐16 Zero Latency UI: Why We Care, How We’re Doing It 1 2013‐10‐16 Sarah Williams Paul Dietz Maxim Mazeev Robert Levy Mike Cronin Hrvoje Benko Katie White Albert Ng Daniel Wigdor, Sarah Williams, Michael Cronin, Robert Levy, Katie White, Maxim Mazeev, and Hrvoje Benko. 2009. Ripples: utilizing percontact visualizations to improve user interaction with touch displays. In Proceedings of the 22nd annual ACM symposium on User interface software and technology (UIST ’09). ACM, New York, NY, USA, 3-12. Albert Ng, Julian Lepinski, Daniel Wigdor, Steven Sanders, and Paul Dietz. 2012. Designing for low-latency direct-touch input. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST ’12). ACM, New York, NY, USA, 453-464. Ricardo Jota, Albert Ng, Paul Dietz, and Daniel Wigdor. 2013. How Fast is Fast Enough? A Study of the Effects of Latency in Direct-Touch Pointing Tasks. In Proceedings of the 2013 ACM annual conference on Human Factors in Computing Systems (CHI ’13). ACM, New York, NY, USA, (in press). 2 2013‐10‐16 High Performance Input High Performance Input Touch & Feedback High Performance Input High Performance Input Touch & Feedback Touch & Feedback Perception of Latency Perception of Latency Performance under Latency High Performance Input High Performance Input Touch & Feedback Touch & Feedback Perception of Latency Perception of Latency Performance under Latency Performance under Latency Architecting Performance Architecting Performance 3 2013‐10‐16 HELP! Raw Input Error Ambiguity: 6 Causes 4 2013‐10‐16 Activation Action Activation Action Accidental Activation Accidental Activation Disabled Content Disabled Content 5 2013‐10‐16 Fat Fingers & Selection Fat Fingers & Selection Manipulation Constraints Manipulation Constraints 6 2013‐10‐16 Over-Capture Over-Capture Error Ambiguity Resolution 7 2013‐10‐16 Capture Capture Fat Fingers Fat Fingers Fat Fingers Over-Capture 8 2013‐10‐16 Over-Capture Tabletop Debris Over-Capture Tabletop Debris Over-Capture Buttons Over-Capture Buttons Constraint Visualization Constraint Visualization: Resize 9 2013‐10‐16 Constraint Visualization: Resize Constraint Visualization: Scroll Constraint Visualization: Scroll Constraint Visualization: Orthogonal Constraint Visualization: Orthogonal 10 2013‐10‐16 Study: Fat Fingers Errors 15 With Feedback Ripples Without Feedback no‐Ripples 10 5 0 33px 43px 53px Study: Fat Fingers 2 8 3 Study: Preference 11 2013‐10‐16 I like to see what is happening, the visuals help me figure out what's going on I know where I am touching technically and I’m reassured I know where I am touching technically and I’m reassured it lets you know it was getting the signal I know where I am touching technically and I’m reassured the sensors caught my actions better [the] table was faster it lets you know it was getting the signal 12 2013‐10‐16 [the] table was faster This one was more responsive Filmed at 80 fps Avoiding Lag Filmed at 80 fps Avoiding Lag 13 2013‐10‐16 14 2013‐10‐16 15 2013‐10‐16 High Performance Input Touch & Feedback Perception of Latency Performance under Latency Architecting Performance 16 2013‐10‐16 Footage at 1/8 speed Footage at 1/8 speed 17 2013‐10‐16 Sources of Direct-Touch Latency Low Latency Touch Demo CPU/GPU Input Sensor Device Driver Operating System ~5-17 ms Application ~25-100 ms Graphics Display Custom Input Sensor ~17 ms 1 ms FPGA Processing < 0.1 ms High Speed Display < 0.1 ms Total Latency: 1 ms Lag Distance @ 1 m/s: 0.1 cm 2 3 Touch perception 4 1 Perception response Distance to surface Distance to surface Perception response Time Time 18 2013‐10‐16 Perception response Distance to surface Distance to surface Perception response Perception response Perception response Distance to surface Time Distance to surface Time first touch Time feedback Time Low Latency touch device 19 2013‐10‐16 First Experiment High-speed processing Just-noticeable difference (JND) 1 ms - 120 ms 75% accuracy rate Low Latency touch device 20 participants First Experiment Latency 120ms Results 0ms Trial Results Latency (in ms) Latency (in ms) Results 20 2013‐10‐16 Latency (in ms) Results Dragging 24 ms Condition A Audience Test 60 ms Condition B Audience Test 42 ms 21 2013‐10‐16 Condition A Condition B Condition A Audience Test 26 ms Condition B Audience Test 10 ms 22 2013‐10‐16 Condition A Condition B Condition A Audience Test 5 ms Condition B Audience Test 2 ms 23 2013‐10‐16 Condition A Condition B Mean JND by Participant Mean JND by Participant Mean JND by Participant High Performance Input Touch & Feedback Perception of Latency Performance under Latency 2 ms Architecting Performance 24 2013‐10‐16 Fitts’ Law W How does latency affect directtouch pointing tasks? D Fitts’ Law Fitts’ Law MacKenzie, I. S., and Ware, C. (1993). Lag as a determinant of human performance in interactive systems. ACM CHI 1993. Target width Target distance + 2 cm + 3 cm 4 cm 5 cm 3.5 cm 8.5 cm Cursor Targets 15 cm 25 2013‐10‐16 Latency applied + + + 1 ms 10 ms + 25 ms 50 ms First experiment x 3 Target widths 3 Target distances 4 Levels of Latency 8 Repetitions 45 Participants Results = 12,960 total trials Does latency affect performance? Mean time (in seconds) How fast is fast enough? Latency (in ms) 26 2013‐10‐16 Index of Difficulty Mean time (in seconds) Mean time (in seconds) How fast is fast enough? Index of difficulty (in bits) Latency (in ms) Throughput Throughput (in bits/s) Mean time (in seconds) Index of Difficulty Index of difficulty (in bits) Latency (in ms) Throughput (in bits/s) Throughput Does latency interact with target size? Latency (in ms) 27 2013‐10‐16 Target size Target size + + 3 cm 2 cm Cursor 4 cm 5 cm Targets 3 cm 2 cm Cursor 4 cm 5 cm Targets Mean time (in seconds) Target distance Does latency interact with target distance? Distance (in pixels) Target distance Target distance Mean time (in seconds) + Cursor + Distance (in pixels) 28 2013‐10‐16 Target distance How fast is fast enough? + + How fast is fast enough? Performance: ~10ms How fast is fast enough? Performance: ~10ms Perception (tap): 24ms How fast is fast enough? Performance: ~10ms Perception (tap): 24ms Perception (drag): 2ms 29 2013‐10‐16 Kindle Fire Microsoft Surface Nexus 7 Kindle Fire iPhone 5 Nexus 10 Microsoft Surface Nexus 7 Kindle Fire 142ms Nexus 7 Kindle Fire 113ms 142ms 97ms 113ms 142ms Nexus 10 Microsoft Surface Nexus 7 Kindle Fire 96ms 97ms 113ms 142ms 90ms 96ms 97ms 113ms 142ms Nexus 4 iPhone 5 Nexus 10 Microsoft Surface Nexus 7 Kindle Fire 88ms 90ms 96ms 97ms 113ms 142ms 30 2013‐10‐16 Samsung Galaxy Tab Nexus 4 iPhone 5 Nexus 10 Microsoft Surface Nexus 7 Kindle Fire iPad Mini iPad 2 Samsung Galaxy Tab Nexus 4 iPhone 5 Nexus 10 Microsoft Surface Nexus 7 Kindle Fire 80ms 88ms 90ms 96ms 97ms 113ms 142ms iPad 2 Samsung Galaxy Tab Nexus 4 iPhone 5 Nexus 10 Microsoft Surface Nexus 7 Kindle Fire 77ms 80ms 88ms 90ms 96ms 97ms 113ms 142ms 73ms 77ms 80ms 88ms 90ms 96ms 97ms 113ms 142ms iPad 3 iPad Mini iPad 2 Samsung Galaxy Tab Nexus 4 iPhone 5 Nexus 10 Microsoft Surface Nexus 7 Kindle Fire 71ms 73ms 77ms 80ms 88ms 90ms 96ms 97ms 113ms 142ms High Performance Input Touch & Feedback Perception of Latency Performance under Latency Architecting Performance 31 2013‐10‐16 Low Latency Touch Demo Custom Input Sensor High Speed Display FPGA Processing 1 ms Low-Latency Touch Toolkit < 0.1 ms Touch Sensor < 0.1 ms R1 R1 R1 R1 R2 R2 R2 R2 R3 R3 R3 R3 R4 R4 R4 R4 R5 R5 R5 R5 R6 R6 R6 R6 R7 R7 R7 R7 C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 32 2013‐10‐16 R1 R1 R2 R2 R3 R3 R4 R4 R5 R5 R6 R6 R7 R7 C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 Low-Latency Touch Toolkit C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 Low-Latency Touch Toolkit LowLatency Subsystem Touch Sensor Frames Touch Processor Touch Sensor Frames Touch Processor High-Latency Subsystem Low-Latency Touch Toolkit Low-Latency Touch Toolkit LowLatency Subsystem Touch Sensor Frames Touch Processor State Info. High-Latency Subsystem LowLatency Subsystem Touch Sensor Frames Touch Processor State Info. Display High-Latency Subsystem 33 2013‐10‐16 Low-Latency Toolkit: Vision Insert Steven Animatics Low-Latency Toolkit: Vision Insert Steven Animatics Low-Latency Toolkit: Vision Insert Steven Animatics Low-Latency Toolkit: Vision Insert Steven Animatics Low-Latency Toolkit: Vision Insert Steven Animatics Low-Latency Toolkit: Vision Insert Steven Animatics 34 2013‐10‐16 High Performance Input Challenges: Synchronization Touch & Feedback Perception of Latency Performance under Latency Architecting Performance Distance to surface Perception response Distance to surface Distance to surface Perception response 2 ms Time Time Perception response Take away message 24 ms We observed increases in performance, for direct-touch pointing tasks, by reducing latency down to 10ms. 2 ms Time 35 2013‐10‐16 Acknowledgments Thank you! We would like to thank Steven Sanders and Kim Wright for their input on the document and study. www.cs.toronto.edu/~jotacosta jotacosta@dgp.toronto.edu Ricardo Jota 2013 2013 2014 36