The lights dim and music fills the air as Taiwan’s Kun Shan University chamber orchestra begins to play.

Dancers take the stage. The audience is treated to a show of performance art that combines the grace and style of dance with the smooth sound of song.

It doesn’t stop there. Thanks to five URI students and Professor Jean-Yves Hervé from the Department of Computer Science and Statistics, the action is captured in real-time, 3-D animation to create an interactive multimedia aspect to the show for the audience.

This will be the scene next February in Taiwan when the Kun Shan chamber orchestra reshapes the concert experience. This past summer, URI and Kun Shan teams collaborated in Taiwan for 11 weeks. The project was funded by a National Science Foundation Research Experiences for Undergraduates grant.

The URI students—who are part of URI’s 3D Group for Interactive Visualization—are a blend of art, computer science, and engineering majors. They worked with professors and students from Kun Shan’s College of Creative Media to produce interactive multimedia for performance art.

Motion capture is the process through which video sequences, typically of a human body in motion, are analyzed to reconstruct the body’s joint movements. These recorded movements are then used to produce realistic character actions in video games or computer-animated graphics for movies.

URI’s portion of the project was three-pronged: motion capture; analysis of motion capture data (with respect to the kinetics of human motion); and visualization.

“We want to show how exciting the world of computer science can be, especially when it is combined with other disciplines,” Hervé said. “There is so much more to computer science than sitting at a desk.”

Junior Stacie Waleyko is a computer science/applied math major who became interested in 3D graphics after completing several internships at the Naval Undersea Warfare Center in Newport. She and Charles Morace, also a computer science/applied math major, focused on creating 3D models and animating them with motion capture data.

“My role required me to be both technical and artistic,” Waleyko said. “Not only do I spend time creating models and animations in Maya and Unity 3D, but I spend just as much time recreating those same models by programming them and applying the complex mathematical equations that calculate human motion. I love the work because it is the perfect balance between computer science and art.

“Getting out of our usual lab at URI and coming here to work with the Taiwanese students has really opened my eyes to the many ways that we can apply our computer science research. For me, it is so exciting to get out and finally learn some of the software designed for artists—like Flash and Maya—and then come up with ways to use them in expressing our research. As a result of our experiences here, we have evolved into an extremely versatile group of computer scientists and engineers.”

Using real-time image processing and motion capture data, Waleyko and Morace are able to use multiple cameras to analyze two-dimensional images captured at each frame and compute a person’s body configuration.

“Working with the Kun Shan students has been exciting and inspiring,” Morace said. “I have appreciated their work using multimedia because, like mathematics, it is a universal language of expression. In their work they focus on evoking emotion and telling a story with their designs, which has inspired me to do the same with our research project.”

Nidal Fakhouri ’11, a computer engineering major, became technically proficient using cameras to follow objects moving around and relaying information about those objects to other people.

“I have reaffirmed the importance of finding a crossover between the technical and creative, and between the scientist and the artist,” Fakhouri said. “Opportunities like these allow you the freedom to learn a lot and to teach a lot, in a manner that you would not ordinarily do.”

Ronald Duarte ’11, a computer engineering major, worked on message passing interfaces and system architecture for the project.

“It has been an amazing experience,” said Duarte, a native of the Dominican Republic who served in the U.S. Army’s 101st Airborne Division before coming to URI. “The students and the professors at Kun Shan University are very friendly and open to our ideas and have many great ideas themselves.

“By working with students who aren’t in computer engineering or computer science, we can be exposed to a different set of skills that we might not encounter in the areas of computer engineering and computer science. In addition, after working with students at Kun Shan University, I have become more experienced about working in other cultures,” said Duarte.

The five students on the trip: Duarte, Fakhouri, Morace, Waleyko and Joel Barruos, a computer science major, all received grants from URI’s Research Office to be part of the project.

“The best part of this trip was the environment we got to work in,” Waleyko said. “I have never had so much fun with a project or worked in a place saturated with this much creativity. Every day we come to work with these incredible artists who have the best ideas and applications for the research we are doing. The days fly by because I get to spend every day doing something I love with other people who feel the same way.”

By Shane Donaldson ’99
Photos Courtesy of Jean-Yves Hervé