20 10 13 Eye Tracking System and Mental Load Estimation Authors – Yu Wang and Hoe Kin Malcolm Wong Background and Motivation Supervisor – Dr. Julien Epps 3.Outputs and visualisations Real time video sequence playback with: Introduction Centre trajectory Pupil diameter plots and blink counts The relationships between eye activity and mental load are of great interest in the cross fields of DSP, Machine Learning and biotechnology. For instance, if a pair of smart glasses can identify whether its user’s mind is heavily loaded or not based on their eye behaviors, it can then help them reschedule their agendas and delay all notifications until the user’s mental load level returns to low to avoid causing any interruption of an undertaken task. 3 Results Current frame histograms for blink detection Aims and Objectives To apply a top down design to build low cost accurate and robust eye-tracking system by: •Selecting and modifying a low-cost web-camera •Evaluating and applying selected algorithms •Adding improvements to current algorithms And then to achieve mental load estimation by: •Designing appropriate cognitive and perceptual load tasks •Collecting data from potential users •Using proper statistical models to investigate relationships between pupillary responses and mental loads Methods A top-down approach system design Outputs and visualizations 4. Mental load estimation Detailed classifications of the impact of cognitive load (C.L.) or perceptual load (P.L.) only on eye activities have been done (yellow region), but the green region which takes into account both P.L. and C.L. has not been explored extensively in previous papers. (L = LOW; H = HIGH) Mental task design and data collection •Environment: MATLAB’s GUIDE •Participants: 12 Uni students (10 male 2 female) •Task duration: approx. 45 min •Devices required: laptop, mouse and web-camera. •Outputs & feedbacks: Integrated In MATLAB and stored in CSV files 1 Grid Search Low Level Working Memory P.L. 6r Head-mounted web-camera modification 2. Processing •Blob identification and edge detection: Improved Haar-like feature, Sobel+GS blur, Canny edge. Processing step two: noise removal Close •Parzen Window Classification K-fold leave one out cross validation: One test data, 11 training data. Four big classification groups: LL, HL, LH and HH. Each has three mean value data. A total of 12 repetitions applied for each eye feature of each participant. Flowchart of the algorithm Processing step three: blink detection and ellipse fitting Step two : statistical model analysis Parzen Window PDF (H = 3) applied to mean pupil diameter under LL (blue), HL (red), LH (green), HH (yellow) mental status of participant 3. (above diagram) Evaluation and Discussion Algorithm Speed, Robustness and Accuracy Conclusion UNSW 4 Strong differences are found between LL and HH mental load levels by looking into the saccade features (p-value 0.12). Some differences are found between a HH and a HL mental load levels (p-value 0.2116). Blink detection state machine A detected blink frame and its histogram 32 shapes in grid Hypothesis: no differences are found between pupil features and different levels of combined mental load. Non-blink pupil tracking and its histogram A detected half-occlusion frame and its histogram 6 shapes in grid Mental load estimation step one: mental task design 2 •Better noise removal for corneal reflection and eyelashes: Cascade Morphological operation •Ellipse fitting: RANSAC algorithm •Improved blink detection: K-mean clustering method and early return 3 shapes •Friedman test (Differences) •30 FPS low-cost USB webcamera for recording. •Infrared LEDs added for distinguishing pupil region •Visible light filter to minimise impact of illumination variation •Attached to frames via flexible metal arm Processing step one: edge detection 2 shapes Statistical model analysis 1. Head-mounted Webcamera Processing step one: blob identification High Level Section(s) for speed evaluation (2.3GHz Processor) Time per Frame Integral image 6.03ms Pupil preprocessing (Non-linear) 4.51ms Ellipse fitting 2.50ms Blink detection 0.49ms Haar-like responses 0.40ms Robust against eyelashes, glasses and corneal reflections Accuracy Chen’s Our With extreme Test Data algorithm algorithm half occlusion Core speed (sum of above) 13.93ms Plotting and others 3.32ms Blink detection 99.6% Total = Core + Plots 17.25ms rate 99.97% 99.87% The price, speed, robustness and accuracy of the eye tracking system are improved. The classifications do show some relationships between eye activities and both perceptual and cognitive loads invoked in one task but it is not always consistent. Further investigation is needed for rigorous proof. ENGINEERING @ UNSW