Vision Sensor Helps Make Robot a Guitar Hero

Engineering student Peter Nikrin got a bit upset when his friend bested him at Guitar Hero, a video game where players score points by correctly hitting keys on a plastic guitar as “notes” to popular rock tunes scroll by on a TV. Rather than buckle down and practice so he could beat his friend’s score, Nikrin decided to build a robot that could.

The robot depends to a large degree on a vision sensor able to quickly detect the notes as they scroll by. Nikrin’s first attempt used five fiber-optic photoelectric sensors, one for each key/note color. But the sensors could not detect the notes, which were just moving spots of color on a video screen. It turned out the fiber-optic sensors didn’t match up well with video; they worked best at detecting light reflecting from a solid object.

A PresencePlus P4 Omni sensor from Banner Engineering, Minneapolis, worked better. His school had purchased one as a start-up educational kit. Nikrin also used a right-angle lens from Banner so that the sensor would fit inside the robot’s head. It took some programming effort before the robot could read signals from the monochromatic sensor to detect and identify notes by position rather than color. A bit more programming went into recognizing the white circle surrounding a note that indicates the right instant to play that note.

It takes the robot about 9 msec to recognize that it’s time to play a note. That’s when it sends a signal to one of five solenoids that activate the robot’s five fingers.

The robot, named Roxanne, averages 98% accuracy on the game’s medium setting. “The higher the difficulty level, the less accurate the robot is, but this is mostly a mechanical issue,” says Nikrin. “I spent so much time on programming that I didn’t have time to focus on the solenoids, so the robot can’t press buttons fast enough to beat the most difficult levels. But the vision system itself hits the notes correctly 100% of the time at all difficulty levels.”