Kinect

Kinect As A Human Interface Device And A Controller

Kinect

Page 1

Kinect
Smt Shantaben Haribhai Gajera Charitable Trust, Surat. LAXMI INSTITUTE OF TECHNOLOGY, SARIGAM CERTIFICATE

This is to certify that the project entitled Kinect is a bonafied report of the work carried out by Mr. Jay N. Patel under the guidance and supervision for the award of the degree of Bachelor of Computer Science & Engineering/Information Technology at Laxmi Institute of Technology -Sarigam , Gujarat.

To the best of my knowledge and belief, this work embodies the work of candidate himself/herself, has duly been completed, fulfills the requirement of the ordinance relating to the Bachelor degree of the university and is up to the standard in respect of content, presentation and language for being referred to the examiner.

Guide Sign

Jury Sign

HOD Sign

Kinect

Page 2

Kinect

Acknowledgment
Firstly I would like to thank our guide Mr. Mihir Mishra for his valuable and deep involvement during the creation of the project. I extend my deep sense of gratitude towards him who has engineered the cause of its existing status. Further, I thank him for his utmost co-operation regarding the freedom and the atmosphere he gave us during the development of the project.

I am also extremely grateful to Prof. Vikas Chaturvedi (H.O.D of Computer Department) for providing all facility and help for smooth progress of project work.

Finally, we thank each other for the valuable co-operation I had even in the hard times while powering this project.

Jay N. Patel B.E 3rd year (Computer) Laxmi Institute of Technology.

Kinect

Page 3

Kinect

INDEX
Abstract 1. Introduction.....6 1.1 1.2 Introduction to Kinect......6 Objective to Study.......6

2. Company Profile.8 2.1 2.2 2.3 Introduction to Company8 History.9 Achivements...10

3. Theortical Background.12 3.1 Fundamental Concepts..12

4. Technological Background..14 4.1 Research Conceptual Clarification14

5. Pros And Cons..21 6. Conclusion....23 7. References24

Kinect

Page 4

Kinect
Abstract
My project evaluates the Kinect game sensor input/ output and its suitability to perform as part of a human interface for a spacecraft application. The primary objective is to evaluate, understand, and communicate the Kinect systems ability to sense and track fine (human) position and motion. The project will analyze the performance characteristics and capabilities of this game system hardware and its applicability for gross and fine motion tracking. The software development kit for the Kinect was also investigated and some experimentation has begun to understand its development environment. To better understand the software development of the Kinect game sensor, research in hacking communities has brought a better understanding of the potential for a wide range of personal computer (PC) application development. The project also entails the disassembly of the Kinect game sensor. This analysis would involve disassembling a sensor, photographing it, and identifying components and describing its operation.

Thi s thesis will discuss how a s ystem like this can be implemented. It will d i s c u s s all the di erent technologies available that could help. It will also o er an indepth look of di erent approaches and methodologies used when solving problems such as gesture r e c o g n i t i o n a n d b o d y m e a s u r e m e n t s . T h e s y s t e m s g o a l i s t o p r o v i d e t h e u s e r w i t h t h e whole personal trainer experience, at home. That experience is mainly split into modules,the feedback and the timely inspection. This thesis will explain thoroughly how each of them (and other modules) where implemented. This project aims to increase the number of individuals playing sports and working out on a daily basis, which will in turn bene t them with better health.

Kinect

Page 5

Kinect
1
1.1

: Introduction
Introduction to Kinect

The Microsoft Kinect game sensor is a revolutionary device that is changing the way electronic consumers interact with their Xbox 360 console and PCs. This piece of gaming technology has an abundance of capabilities that enables it to track human position, gestures, and recognize speech for voice commands. Because the software development kit was not available at the time of the Kinects release in November, 2010, independent developers all over the world have hacked USB interface to allow repurposing of sensor capabilities, controls and 3D environmental mapping, resultin in many amazing interactive PC applications. The Microsoft Kinect is processed by Primesenses PS1080 System-on-a-chip SoC, processor. The Kinect is equipped with an infrared I.R. transmitter, a color image complementary metal oxide semiconductor CMOS camera, and a depth image CMOS camera. This trio of tracking technology allows the Kinect to track up to four people.

1.2

Objective To Study Kinect for Windows Software Development Kit (Beta Version)
The Kinect software development kit beta, SDK beta, became available to download in mid June, 2011. The SDK runs on the Visual Basic development environment by Microsoft. The programming language used for practice and implementation for this project was C++. The SDK is a toolkit for application developers. It accesses the Kinect device connected to computers running on Windows 7 operating system. The Kinect SDK includes drivers, APIs for sensor

Kinect

Page 6

Kinect
video streaming and human motion tracking, installation documents, and resource materials. It offers capabilities to developers to build applications with C++, C# or Visual Basic by using Visual Studio 2010. The Kinect for Windows SDK beta includes: 1. Raw Sensor Streams to access raw data from the depth and color camera sensors, and also the microphone array. 2. Skeletal Tracking to track one or two people moving within the range of the Kinects tracking capabilities. 3. Advanced Audio Capabilities to allow developers access to the sophisticated noise suppression and echo cancellation, beam forming microphones to identify the current sound source and integration with Windows 7 speech recognition API. 4. Sample Code Documentation which includes more than 100 pages of technical information. 5. Easy Installation which is a complete installation of 100 megabytes. Developers can install the SDK in minutes with a Kinect device unit. The kinect SDK also includes a sample game called Shape Game Demo which is a simple application that includes the applications source code.

Kinect for Gaming Controller

The Microsoft Kinect game sensor is a revolutionary device that is changing the way electronic consumers interact with their Xbox 360 console and PCs. This piece of gaming technology has an abundance of capabilities that enables it to track human position, gestures, and recognize speech for voice commands.
Kinect Page 7

Kinect

2
2.1

:Company Profile
Introduction to Company

Microsoft Corporation (NASDAQ: MSFT) is an American multinational corporation headquartered in Redmond, Washington that develops, manufactures, licenses and supports a wide range of products and services related to computing. The company was founded by Bill Gates and Paul Allen on April 4, 1975. Microsoft is the world's largest software maker measured by revenues.It is also one of the world's most valuable companies.Microsoft was established to develop and sell BASIC interpreters for the Altair 8800. It rose to dominate the personal computer operating system market with MS-DOS in the mid-1980s, followed by the Microsoft Windows line of operating systems. The company's 1986 initial public offering, and subsequent rise in its share price, created an estimated three billionaires and 12,000 millionaires from Microsoft employees. Since the 1990s, it has increasingly diversified from the operating system market and has made a number of corporate acquisitions. In May 2011, Microsoft acquired Skype Technologies for $8.5 billion in its largest acquisition to date.As of 2012, Microsoft is market dominant in both the PC operating system and office suite markets (the latter with Microsoft Office). The company also produces a wide range of other software for desktops and servers, and is active in areas including internet search (with Bing), the video game industry (with the Xbox and Xbox 360consoles), the digital services market (through MSN), and mobile phones (via the Windows Phone OS). In June 2012, Microsoft announced that it would be entering the PC vendor market for the first time, with the launch of the Microsoft Surface tablet computer.In the 1990s, critics began to contend that Microsoft used monopolistic business practices and anti-competitive strategies including refusal to deal and tying, put unreasonable restrictions in the use of its software, and used

Kinect

Page 8

Kinect
misrepresentative marketing tactics; both the U.S. Department of Justice and European Commissionfound the company in violation of antitrust laws.

2.2

History
The depth sensing technology behind Kinect was invented in 2005 by Zeev Zalevsky, Alexander Shpunt, Aviad Maizels and Javier Garcia.Kinect itself was first announced on June 1, 2009 at E3 2009 under the code name "Project Natal".Following in Microsoft's tradition of using cities as code names,"Project Natal" was named after the Brazilian city of Natal as a tribute to the country by Brazilian-born Microsoft director Alex Kipman, who incubated the project.The name Natal was also chosen because the word natal means "of or relating to birth", reflecting Microsoft's view of the project as "the birth of the next generation of home entertainment".Three demos were shown to showcase Kinect when it was revealed at Microsoft's E3 2009 Media Briefing: Ricochet, Paint Party and Milo & Kate.A demo based on Burnout Paradise was also shown outside of Microsoft's media briefing.The skeletal mapping technology shown at E3 2009 was capable of simultaneously tracking four people,with a feature extraction of 48 skeletal points on a human body at 30 Hz.It was rumored that the launch of Project Natal would be accompanied with the release of a new Xbox 360 console (as either a new retail configuration,a significant design revision and/or a modest hardware upgrade). Microsoft dismissed the reports in public and repeatedly emphasized that Project Natal would be fully compatible with all Xbox 360 consoles. Microsoft indicated that the company considers it to be a significant initiative, as fundamental to the Xbox brand as Xbox Live,and with a launch akin to that of a new Xbox console platform.Kinect was even referred to as a "new Xbox" by Microsoft CEO Steve Ballmer at a speech for the Executives' Club of Chicago.When asked if the introduction will extend the time before the next-generation console platform is launched (historically about 5 years between platforms),Microsoft corporate vice president Shane Kim reaffirmed that the company believes that the life cycle of the Xbox 360 will last through 2015 (10 years).During Kinect's development, project team members experimentally adapted numerous games to Kinect-based control schemes to help evaluate usability. Among these games were Beautiful Katamari and Space Invaders Extreme, which were demonstrated at the Tokyo Game Show in September 2009.According to creative director Kudo Tsunoda, adding Kinect-based control to pre-existing games would involve significant code alterations, making it unlikely for Kinect features to be added through software updates. Although the sensor unit was originally planned to contain a microprocessor that would perform operations such as the system's skeletal mapping, it was revealed in January 2010 that the sensor would no longer feature a dedicated processor. Instead, processing would be handled by one of the processor cores of the Xbox 360's Xenon CPU.According to Alex Kipman, the Kinect system consumes about 10-15% of the Xbox 360's computing resources.However, in November, Alex Kipman made a statement that "the new motion control tech now only uses a single-digit percentage of the Xbox 360's processing power, down from the previously stated 10 to 15 percent."A number of observers commented that the computational load required for Kinect

Kinect

Page 9

Kinect
makes the addition of Kinect functionality to pre-existing games through software updates even less likely, with concepts specific to Kinect more likely to be the focus for developers using the platform.On March 25, Microsoft sent out a save the date flier for an event called the "World Premiere 'Project Natal' for the Xbox 360 Experience" at E3 2010. The event took place on the evening of Sunday, June 13, 2010 at the Galen Center and featured a performance by Cirque du Soleil. It was announced that the system would officially be called Kinect, a portmanteau of the words "kinetic" and "connect", which describe key aspects of the initiative.Microsoft also announced that the North American launch date for Kinect will be November 4, 2010.Despite previous statements dismissing speculation of a new Xbox 360 to accompany the launch of the new control system, Microsoft announced at E3 2010 that it was introducing a redesigned Xbox 360, complete with a connector port ready for Kinect. In addition, on July 20, 2010, Microsoft announced a Kinect bundle with a redesigned Xbox 360, to be available with the Kinect launch.On June 16, 2011, Microsoft announced its official release of its SDK for noncommercial use.On July 21, 2011, Microsoft announced that the first ever white Kinect sensor would be available as part of the "Xbox 360 Limited Edition Kinect Star Wars Bundle", which also includes custom a Star Wars-themed console and controller, and copies of Kinect Adventures and Star Wars Kinect.Previously, all Kinect sensors had been glossy black.On October 31, 2011, Microsoft announced launching of the commercial version of Kinect for Windows program with release of the SDK to companies. David Dennis, Product Manager at Microsoft, said, "There are hundreds of organizations we are working with to help them determine what's possible with the tech".On February 1, 2012, Microsoft released the commercial version of Kinect for Windows SDK and told that more than 300 companies from over 25 countries are working on Kinect-ready apps.

2.3

Achivements
Launch Microsoft had an advertising budget of US$500 million for the launch of Kinect, a larger sum than the investment at launch of the Xbox console.The marketing campaign You Are the Controller, aiming to reach new audiences, included advertisements on Kellogg's cereal boxes and Pepsi bottles, commercials during shows such as Dancing with the Stars and Glee as well as print ads in various magazines such as People and InStyle.On October 19, Microsoft advertised Kinect on The Oprah Winfrey Show by giving free Xbox 360 consoles and Kinect sensors to the people in the audience.Two weeks later, Kinect bundles with Xbox 360 consoles were also given away to the audience of Late Night with Jimmy Fallon.On October 23, Microsoft held a prelaunch party for Kinect in Beverly Hills. The party was hosted by Ashley Tisdale and was attended by soccer star David Beckham and his three sons, Cruz, Brooklyn, and Romeo. Guests were treated to sessions with Dance Central and Kinect Adventures, followed by Tisdale having a Kinect voice chat with Nick Cannon.Between November 1 and November 28, Burger King gave away a free Kinect bundle "every 15 minutes".A

Kinect

Page 10

Kinect
major event was organized on November 3 in Times Square, where singer NeYo performed with hundreds of dancers in anticipation of Kinect's midnight launch.During the festivities, Microsoft gave away T-shirts and Kinect games. Sales 18 million units of the Kinect sensor had been shipped by January 2012.Having sold 8 million units in its first 60 days on the market, Kinect has claimed the Guinness World Record of being the "fastest selling consumer electronics device".According to Wedbush analyst Michael Pachter, Kinect bundles accounted for about half of all Xbox 360 console sales in December 2010 and for more than two-thirds in February 2011.More than 750,000 Kinect units were sold during the week of Black Friday 2011. Award o machine learning work on human motion capture within Kinect won the 2011 MacRobert Award for engineering innovation. o Kinect Won T3's "Gadget of the Year" award for 2011. It also won the "Gaming Gadget of the Year" prize. o 'Microsoft Kinect for Windows Software Development Kit' was ranked second in "The 10 Most Innovative Tech Products of 2011" at Popular Mechanics Breakthrough Awards ceremony in New York City.

Kinect

Page 11

Kinect

3.
3.1

:Theortical Background
Fundamental Concepts

Figure shows a diagram of how the PS1080 SoC functions. The chip includes two optional input capabilities: color (RGB) image and audio while the I.R. light and depth image CMOS are mandatory. Microsoft states that the infrared camera resolution is 320x480 pixels. The lens are relatively large, and have autofocus function. Primesenses PS1080 SoC is the brains behind the Kinect. The PS1080 SoC is a multi-sense system that synchronizes live depth image, color image and audio streaming. The Kinect contains an I.R. transmitter, a color image CMOS, and a depth image CMOS. The infrared is emitted in a matrix of dots that reflect off objects in a given area. The sensor has an angular field of view of 57 horizontally and 43 vertically, while the motorized pivot is capable of tilting the sensor up to 27 up or down. The time that it takes each
Kinect Page 12

Kinect
dot to return to the depth image CMOS is then translated into a depth image internally digitized depth image map. The folks over at Microsoft Research have come up with a new augmented reality concept called the Beamatron. Combining a projector and a Kinect sensor on a pan tilt moving head, it allows people to combine the physical with the virtual world. This means virtual cars can be driven in a room, bumping into real obstacles and speeding over ramps allowing for some interesting interactions between graphics and the real world. Technology like this could be used to bring notifications and other graphics to the attention of a user by automatically placing it within the users view. Imagine reading a newspaper and then seeing a notification for an incoming call or new email on the top corner of the page. That would make life pretty interesting. Check out a demo of the Beamatron in the video above.

Kinect

Page 13

Kinect

4
4.1

:Technologcal Background
Research conceptual clarification

Figure 1. Primesense Depth Image CMOS. The I.R. invisible light is emitted and tracked by the depth image CMOS. The PS1080 SoC then generates a depth image.

Kinect

Page 14

Kinect

Figure 2. Shape Game Demo. The Kinect SDK beta includes this as a sample application. Kinect Sensor Teardown Disassembling the Kinect sensor was a fairly easy process. The teardown of the Kinect sensor is nothing new to technology enthusiasts. Many websites offer suggestions, pointers and even offer the disassembling process for you. The dismantling process took a total of ten phases, each with additional dismantling processes for the minor things. The duration for the Kinect disassembling process took a weeks time while at the same time photographing the process and documenting the findings. The main resource for the disassembling process was a website by ifixit. This website serves as a free repair manual that anybody with technical knowledge can edit. The ifixit website offers repair manuals for just about every piece of technology from tablets to smart phones. At the time of the Kinect disassembling process, ifixit hosted a teardown webinar. The engineering behind the Kinect is amazing; everything from the placement of the microphone array to the integrated circuits makes the Kinect a highly sophisticated device thats available for $150 versus earlier systems with similar capabilities, priced around $280K.

Kinect

Page 15

Kinect
Figure 3. Locations of the various components. Depicted are the locations of the components that make up the Kinect game sensor. The Kinect game sensor is a glossy black device that has integrated in it an I.R. transmitter, 3D Depth Sensors, (RGB) Camera, a multi-array microphone, and a motorized tilt base as shown in Figure 3. The teardown was a hands on procedure to evaluat the physical structure of the Kinect and to learn about the Kinects components such as integrated circuit IC boards. Each phase is a disassembly of each component of the Kinect from the outside casing to the structural support frame.

Figure 4. Removal of the Motorized base Cover. The case that houses the motor can then be removed also. The first step was to disassemble the motorized base. Within step one, a rubber pad had to be removed to get to four T6 Torx screws, one in each corner as indicated by the red circles in Figure 4.

Kinect

Page 16

Kinect

Figure 5. Inside the Motorized Base. The DC power supply had to be disconnected. The second step, once the T6 Torx screws to the motor casing were removed, was disconnecting the direct current D.C. power supply cable. The D.C. power cable was the only thing that was holding the motor assembly to the base at this point. The plastic gears in the motorized base are fragile and can easily be warped or stripped if forced into position. This could be considered a design flaw.

Kinect

Page 17

Kinect
Figure 6. Removal of the Grills. Depicted are the grilles removed and also the locations of the six T10 Torx security screws indicated by the red circles. Step four essentially was the removal of the large I.C. board that carries the Primesense PS1080 SoC. The microphone array, labled in blue in Figure 7., was the first to be disconnected. The heat sink, labled in green in Figure 7, had to be heated with a heat gun to loosen the adhesive holding it on to the board. Then the seven T10 Torx screws, labeled in red, could be removed.

Figure 7. Removal of the microphone array, heat sink, seven T10 Torx security screws and microphone array. Each component is color coded in figure.

Kinect

Page 18

Kinect

Figure 8. Photo : Step five was to disconnect the depth image CMOS and the color image CMOS cameras and the I.R. transmitter ribbon from the mid size I.C. board. Refer to Figure 9. for color code information

Kinect

Page 19

Kinect
Figure 9. The camera ribbon connectors as well as the I.R. transmitter are disconnected. Depicted are the depth image CMOS ribbon in the blue box, color image ribbon in the red box, and the I.R. transmitter ribbon in the green box. Step five was to disconnect the depth image CMOS and the color image CMOS cameras and the I.R. transmitter ribbon from the mid size I.C. board. Refer to Figure 9. for color code information .

Kinect

Page 20

Kinect

Figure 10. Xbox 360 cables can now be removed. The red arrow indicates the path to which the cable is to be removed. Step six depicts Figure 10. which shows the bottom view of the Kinect. Once all the components were removed from the frame, the cable that connects the Kinect to the universal serial bus USB device can now be slipped out through the bottom center.

Figure 11. Subassemblies are bagged and tagged. The kinect took a total of ten phases to dismantle. Steps seven through ten is a simple teardown of the Kinect game sensors frame. Everything from the IR emitter to the wiring had been removed during the last three phases of the teardown. The Kinect took a total of 10 phases to dismantle that are not all described. As depicted in Figure 11, the subassemblies are bagged and tagged and ready for the next co-op or intern to reassemble.
Kinect Page 21

Kinect
5 :Pros & Cons
Advantages
1. Facilitate Research-Supported Learning Active learning increases academic performance. For many classroom teachers, the most challenging question has been how. There are so many constraints and parameters that prevent active learning from becoming a consistent classroom experience. Its my belief that Kinect can overcome the parameters isolating active learning from classrooms. Furthermore, The Horizon Report reveals that gesture-based learning is an upcoming trend that will be integrated in the field of education over the next few years. Conclusively, gesture-based learning facilitates active learning; Kinect is a consumer-friendly tool providing the gesture-based experience. 2. Seamlessly Integrate Technology The irony of technology integration is that its most effective when it disappears, meaning that users are so engaged in the content that reflecting on the actual technology is minimal. For instance, the Wii offers a great active experience, but users are required to be constantly aware of the device and follow (at times) non-intuitive rules. Kinect is seamless; as long as the software is wellprogrammed, user awareness of the device is rare. 3. Embrace Cultural Diversity Combining a virtual lobby with full facial and body tracking, users can globally connect with others from around the globe. Students can control their avatars in augmented learning environments with other students from anywhere in the world.. Since the device is cheap, this excludes no population. For instance, Microsoft is piloting Kinect in Africa, a sure sign this device can reach learners from all demographics and socioeconomic statuses. 4. Establish Content Relevancy It would be ideal if every classroom could have an authentic lab to further explore content, but it would also be nave to think this is feasible. Virtual labs are great, especially in schools that may not be able to provide real labs. However, augmented labs that integrate gestures and Kinect will bridge the gap between virtual labs and true-to-life labs. 5. Explore New Environments One moment, you may be exploring the universe; the next moment, exploring the human body. All of this can be done within a standard classroom. Pundits will argue that this is far-fetched. I understand their angle and arguments; after all, this new technology has had little classroom exposure. But when we evaluate learning research, instructional needs, and the relatively inexpensive price for Kinect, it
Page 22

Kinect

Kinect
becomes further evident that this technology will be a 21st century tool to facilitate instruction.

Disadvantages
It limits the number of simultaneous players, so far its been used to track 4 people, but only 2 could be active. The device is very fast on tracking but it does require a minimum distance before it can track. The rays realeesd from kinect may be harmful and may cause side effects like skin infections and other health problems. Due to kinect children will addict to them and will not come back from the gaming. People wil stop playing football and all other games outdors.

Kinect

Page 23

Kinect

:Conclusion
In future, a lot of things can be done to better enhance t h i s p r o j e c t i n t o a m o r e established system and hopefully a platform for coaches and players collaborating andsharing training routines. To reach this level a social layer needs to developed, a layer that will allow players to communicate with other friends or coaches. This communication can help raise the level of competition between players and help novice players get in contact with coaches to help them out with their specic moves. The social layer brings another i m p o r t a n t f e a t u r e t o t h e t a b l e , t h e u s e r prole. Each user should have some kind of a prole saved with their calibration le, this will save the user the hassle of c a l i b r a t i o n each and every time the game starts. The system already implements basic functionality required to build this social layer. Also, some cosmetic changes in the user interface can really help this project advance more, for an example a new 3D avatar that resembles areal human coach. All of this, combined with more work on the current features to raise their accuracies and iron out the bu ys and limitations, would d e n i t e l y p r o d u c e a w e l l established game that could help people all around the globe. However, as much as the system has potential, it currently su ers a handful of limi t a t i o n s t h a t m i g h t l i m i t o r d e g r a d e t h e r a t e o f t h e p r o j e c t s g r o w t h i n t h e f u t u r e . A l l the limitation described thoroughly in chapter3can be grouped into two categories, the PC driver and frequent disk access. The PC driver is the open-source driver provided by PrimeSense, the driver provides API for most common languages ;the API c a n b e u s e d to retrieve skeletal information and depth information among a lot of other things. The problem with that driver is the low accuracy when its tracking player(s). The accuracy i s c o n s i d e r e d l o w w h e n i t s c o m p a r e d t o t h e c o m m e r c i a l a p p l i c a t i o n b u i l t f o r t h e X b o x 360, its apparent that Microsoft uses another techniques and algorithms than the ones used in the open-source driver. The other major problem is the frequent use of the disk.The need for disk access this frequent, makes the system su er a major FPS drop. Which in turn causes the game to slow down to an unplayable level.

Kinect

Page 24

Kinect
7 :References

http://www.scribd.com/doc/60688637/7/Advantages-of-Kinect http://en.wikipedia.org/wiki/Kinect http://in.answers.yahoo.com/question/index?qid=20110519074121AAQD6o7

Kinect

Page 25