1. technology and computing
2. internet technology
3. isps

Integrating the Microsoft Kinect With Simulink:
Real-Time Object Tracking Example
Joshua Fabian
Tyler Young
James C. Peyton Jones
Garrett M. Clayton
(Villanova University ,PA,USA)
• Introduction
• VU-Kinect Block
• Using The VU-Kinect Block
•
•
•
Depth Camera Calibration
Extracting Position From RGB Camera Image
Depth/RGB Camera Registration
• Simulink Model
• Experimental Results
• Conclusion
2
• Introduction
• VU-Kinect Block
• Using The VU-Kinect Block
•
•
•
Depth Camera Calibration
Extracting Position From RGB Camera Image
Depth/RGB Camera Registration
• Simulink Model
• Experimental Results
• Conclusion
3
1. To develop a “VU-Kinect” block to be incorporated seamlessly
within a higher level, Simulink-based, image-processing and realtime control strategy.
2. To address implementation issues associated with the Kinect,
such as sensor calibration.
3. To show the utility of both the VU-Kinect block and the Kinect itself
through a simple 3-D object tracking example.
4
• Introduction the Microsoft Kinect, has considerable potential in
autonomous system applications.
• To date, the majority of Kinect applications have been coded in C
• But the use of higher-level control and image processing design
languages are now commonplace both in academia and industry.
• These tools allow even inexperienced users to simulate their
designs, then implement them on target hardware (using automatic
code generation), and finally to tune system parameters while the
code is actually running in real time.
5
• In particular, MATLAB and Simulink provide a widely used
environment for designing, simulating, and implementing control
and image-processing algorithms.
• Simulink, developed by MathWorks, is a data flow graphical
programming language tool for modeling, simulating and analyzing
multi-domain dynamic systems.
6
Simulink model of a wind turbine
7
• Libfreenect API
• Libfreenect is mainly a driver which exposes the Kinect device's
features: - depth stream - IR stream - color(RGB) stream - motor
control - LED control – accelerometer.
• Does not provide any advanced processing features like scene
segmentation, skeleton tracking, etc.
8
• Introduction
• VU-Kinect Block
• Using The VU-Kinect Block
•
•
•
Depth Camera Calibration
Extracting Position From RGB Camera Image
Depth/RGB Camera Registration
• Simulink Model
• Experimental Results
• Conclusion
9
• VU-Kinect Block(Villanova University Real-Time Kinect)
• A application which streams parallel camera and depth images from
the Kinect into the user’s Simulink model.
• The (VU-Kinect) block provides a high-level interface to the Kinect
hardware for Simulink users, as well as the low-level back-end code
necessary to interface to the libfreenect API
Simulink
Model
VU-Kinect
Block
libfreenect
API
Kinect
10
Tune Parameters
+ View/Log Results in Real Time
11
• Introduction
• VU-Kinect Block
• Using The VU-Kinect Block
•
•
•
Depth Camera Calibration
Extracting Position From RGB Camera Image
Depth/RGB Camera Registration
• Simulink Model
• Experimental Results
• Conclusion
12
• Details of the calibration of the Kinect sensor, the experimental
setup, and experimental results are presented.
• Calibration:
• 1) Depth Camera Calibration
• 2) Extracting Position From RGB Camera Image
• 3) Depth/RGB Camera Registration
13
Top of view
Depth calibration curve relating Kinect
depth output to distance in centimeters.
14
• The pendulum will be tracked in Kinect coordinates, so the pixel
coordinates in the RGB image need to be converted to this
coordinate system.
Projective Coordinate System
Real World Coordinate System
15
(𝑥1 , 𝑦1 )𝑖𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑢𝑝𝑝𝑒𝑟 𝑙𝑒𝑓𝑡 𝑐𝑜𝑟𝑛𝑒𝑟
(𝑥, , 𝑦 )𝑖𝑠 𝑡ℎ𝑒 𝑐𝑒𝑛𝑡𝑒𝑟 𝑜𝑓 𝑡ℎ𝑒 𝑖𝑚𝑎𝑔𝑒
H and W are the height and width of the image in pixels
16
• The Kinect device uses separate cameras for the RGB and depth
videos, causing misalignment between the two images.
(a) unregistered
(b) registered images.
17
• Therefore, the two images must be geometrically transformed
(or registered) into the same coordinate system
0.93
0.0069
• Using affine transformation matrix 𝐻 = −0.0069 0.93
40
151
• The form of this transformation matrix is chosen to conform with
the Apply Geometric Transformation block in Simulink
• 𝑥 = ℎ11𝑥 + ℎ21𝑦 + ℎ31
𝑦 = ℎ12 𝑥 + ℎ22 𝑦 + ℎ32
where the pixel at location (x, y) in the original image appears
at the location (𝑥, 𝑦) in the transformed image.
18
• Introduction
• VU-Kinect Block
• Using The VU-Kinect Block
•
•
•
Depth Camera Calibration
Extracting Position From RGB Camera Image
Depth/RGB Camera Registration
• Simulink Model
• Experimental Results
• Conclusion
19
• Simulink model
is capable of the complicated task of tracking a
pendulum in 3 dimensions.( Without the need to develop C code.)
20
21
• Introduction
• VU-Kinect Block
• Using The VU-Kinect Block
•
•
•
Depth Camera Calibration
Extracting Position From RGB Camera Image
Depth/RGB Camera Registration
• Simulink Model
• Experimental Results
• Conclusion
22
• Case 1: Approximate 2-D motion in a plane parallel to the Kinect
Y
X
Kinect
Z
23
• Case 2: Approximate 2-D motion in a plane perpendicular to the
Kinect
Y
Z
Kinect
X
24
• Case 3: 3-D elliptical motion.
Y
Z
Kinect
X
25
Experimental position plots measured using the Kinect.
26
• Approximate 2-D motion in a plane parallel to the Kinect
27
• Approximate 2-D motion in a plane perpendicular to the Kinect
28
• 3-D elliptical motion.
29
• Oscillation Period 𝑇 = 2𝜋
𝑙
𝑔
𝑙: 𝑡ℎ𝑒 𝑙𝑒𝑛𝑔𝑡ℎ 𝑜𝑓 , 𝑠𝑡𝑟𝑖𝑛𝑔 𝑔: 9.81𝑚/𝑠 2
30
• Introduction
• VU-Kinect Block
• Using The VU-Kinect Block
•
•
•
Depth Camera Calibration
Extracting Position From RGB Camera Image
Depth/RGB Camera Registration
• Simulink Model
• Experimental Results
• Conclusion
31
• The VU-Kinect block was used to track the 3-D motion of a
pendulum with great success.
• The Kinect in conjunction with Simulink’s real-time workshop
toolbox will enable the easy programming of remote targets
platforms. (e.g.mobile robots).
32