Not all hackers try to steal credit card information and personal data. Many of them, in fact, are having a positive impact. They are developing open-source code for video-game interfaces such as Microsoft’s Kinect. The resulting applications are often innovative ideas the folks in Redmond, Wash., never imagined.
The Kinect is designed to replace a game controller that normally plugs into the Xbox 360. It lets users play virtual games such as volleyball with their onscreen avatar just by moving their bodies. The device’s combination of cameras, sensors, and software let it detect movement, depth, and the shape and position of the human body.
But Kinect hackers have taken the device far beyond its gaming roots. Programmers such as computer scientist Oliver Kreylos from the University of Calif., Davis, have no interest in playing gesture-based video games. Instead of plugging a Kinect into an Xbox, he hooks it to a computer. The Kinect’s “eyes” are actually a pair of cameras – a depth camera and a color camera -- each of which uses a complimentary metal-oxide semiconductor (CMOS) image sensor. Kreylos converts the Kinect into a 3D camera by combining the depth and color image streams and projecting them back out into 3D space. Data are reconstructed in such a way that real 3D objects inside the camera field-of-view are recreated virtually, at their proper sizes. The result is what Kreylos calls a 3D holographic image.
Kreylos says he had been looking for inexpensive cameras like those in the Kinect for use in his work in virtual reality for scientific data visualization. He contacted PrimeSense, Tel-Aviv, Israel, which developed the original technology that later became Kinect. “Unfortunately, this happened shortly after PrimeSense had entered an exclusive agreement with Microsoft,” he says. “So although I could not purchase the cameras back then, I have known exactly what the Kinect would be capable of for the last two years or so.”
The Kinect’s depth camera is paired with a near-infrared laser projector, which sprinkles the room and its occupants with a pattern of dots. The camera sees this pattern and based on differences between the preset and the observed pattern calculates the distance to each visible dot. From this data, it produces a "depth image," which contains a distance value for each pixel. This method is a variation of the well-known "structured light" approach in which the 3D structure of a scene is found based on the distortion of a projected pattern. Color is added to the 3D model via RGB input from the CMOS image sensor.
In hacking the Kinect, Kreylos wrote all his own code in C++, except for the sequence of control commands that goes to the Kinect to initialize its cameras and start recording. “I had no access to these commands because I do not own an Xbox console,” he says. “So I used the sequence extracted by Hector Martin, the Spanish engineering student who won the Kinect hacking prize for being the first to create an open-source driver.”
Says Kreylos, “The term ‘holographic’ is actually technically incorrect. But I felt I had to use it to distinguish the true 3D video coming from the Kinect from pseudo-3D stereoscopic video such as what's shown in ‘3D’ movies such as Avatar, or captured by available ‘3D cameras.’ The difference is that pseudo-3D video can only be viewed from the point-of-view of the camera that originally recorded it, just like regular 2D video. True 3D video, on the other hand, can be viewed from any arbitrary viewpoint, even after it has been captured. It has this property in common with real holograms, hence I chose the moniker as a short-hand.”
Kreylos explains that, colloquially speaking, Jim Cameron could not go into the editing room after shooting a live-action scene in Avatar, and say, "OK that's great, now let's see this scene from the point-of-view of that blue guy on the right, says Kreylos. “Although this is possible with true 3D video, a true 3D video camera is still a camera and it cannot see through or around solid objects. This means users would need multiple Kinects to pull off the full effect. For example, because a 3D camera can only create a 3D model of what it sees, a 3D holographic model of me would only shows one side of my body. We call this half-representation a ‘facade.’ Using multiple cameras would allow combining respective facades into complete 3D objects, an ongoing area of my research.”
Users can save models, either as snapshots or animations, and edit them freely, says Kreylos. “That’s because the models are just a bunch of triangles in 3D space, exactly the same format used by typical 3D modeling and animation software,” he says. “CAD software is somewhat different, but the models could be converted to a format for editing. In fact, it’s likely that Kinect, or similar 3D cameras, will soon be used as low-cost 3D scanners for engineers and hobbyists. There is still a good amount of software to be developed, but it should happen.”
As for other potentially creative uses of the Kinect, Kreylos says, “We are only seeing the tip of the iceberg right now. There are already tech demos of things including hands-free Web browsing interfaces, virtual keyboards, and autonomous aerial vehicles. We are sure to be surprised what future developments will bring.”
Resources:
Kreylos projects and code, http://idav.ucdavis.edu/~okreylos/ResDev
Authored by: Leslie Gordon, mailto:[email protected]