SYNTHETIC HUMANS©

Virtual reality company Hyperreal Digital Inc. collaborates with Drexel’s Animation Capture & Effects Lab on the creation of synthetic human actors used in metaverse advertising by brands like PepsiCo.

IMAGE ABOVE COURTESY OF HYPERREAL DIGITAL INC.

One weekend in late October, a highly confidential mission was under way inside Drexel’s Animation Capture & Effects Lab in West Philadelphia. The group of students present were all sworn to secrecy, as was Assistant Professor of Digital Media Nick Jushchyshyn, who oversees the lab.

Such is the protocol when Jushchyshyn opens the lab to industry partners. On this particular Saturday last year, he and his students were collaborating with Hyperreal Digital Inc., a New York-based metaverse entertainment company known as a pioneer in the creation of hyper-realistic digital humans for films, commercials and video games. From time to time, technology companies ask Hyperreal to test cutting-edge equipment, and when they do, CEO Remington Scott brings them to Drexel.

“When we do these tests, it’s super secretive,” says Scott. “Every time I go in there, no one outside of the lab can know what we’re working on.”

The work on this day involved bringing the late rap star Notorious B.I.G. back into living color.

Scott and his team had previously shot footage of a virtual concert performed by a hyper-realistic avatar of Notorious B.I.G., commissioned by a unit of Meta to mark what would have been Biggie’s 50th birthday.

But as Scott was later processing the footage, watching the avatar’s performance, he realized something was off. “We weren’t looking at it through a virtual camera controlled by a human,” Scott observed. “It was missing this breathing, moving human element behind the camera.”

So, Scott made a trip to Drexel, and Jushchyshyn and his team of students went to work. As Scott held a virtual camera, he was able to reframe the rap star with new shots and angles. The virtual, simulated concert was released on Dec. 16, 2022.

Hyperreal turns to Drexel’s ACE Lab — and its companion, the Immersive Research Lab (IRL), also run by Jushchyshyn — because the labs have what Scott needs: A fully equipped professional motion-capture studio with equipment seldom found outside the entertainment industry.

The facilities at Drexel are unique and rare for the East Coast, let alone for a university.

“When Nick showed me the facilities at Drexel, I couldn’t get over the standard he was setting and the technology available in his lab,” Scott says. “I was blown away. It’s world class and the students are up and running and ready to be engaged. Having those talented minds that are learning, asking the right questions, thinking something through and playing around with the technology, is extremely liberating.”

Built for the future

The ACE Lab and IRL in the Antoinette Westphal College of Media Arts & Design focus on the study and research of immersive media such as virtual reality (VR), augmented reality (AR), immersive projection, virtual creation and production.

The quality of the labs’ equipment underscores Drexel’s commitment to training students in the field of study.

Students have access to dozens of AR headsets from a range of manufacturers; numerous 360° cameras systems including monoscopic, stereoscopic and volumetric; and motion-capture systems from Vicon and OptiTrack, two of the leading brands. While regular Macs and PCs are part of the equipment, so are HP-Z VR Backpack computers that users wear to move around virtual spaces. There are three different screening rooms including motion capture and green screen studios.

Jushchyshyn envisions eventually having an LED wall in the studio to display real-time computer graphics — a forest, a mountain, a desert — at a pace of 60 frames a second, which produces clear images and can be used as backgrounds when filming. “It’s a wish list item,” he smiles.

LED walls cost between $500,000 to $1 million and, though the funds have yet to be secured, Jushchyshyn has prepared the lab for the technology by having additional electrical wiring installed earlier this year to support it.

“We’re always thinking about cutting-edge technology even though we don’t always have every piece of it here,” he says.

It was this mindset that positioned Drexel to be a leader in VR and AR production. In September 2012, the University, with Jushchyshyn’s guidance, established a studio with a green screen wall, motion-capture system and other equipment to do virtual production in television and film.

The studio included technology that allowed computer graphics and line movement to occur simultaneously, which was uncommon. Most production facilities stationed them in separate spaces rather than a single room. “This was on the vanguard then and now,” Jushchyshyn says.

The following year, in 2013, several standout graduate students who would go on to successful careers in the field urged Westphal College to bolster the facility further. “We were building these facilities to facilitate their research,” Jushchyshyn recalls. “It was also a strategic interest for us as a department and as a college.”

Melissa Cell, BS digital media ’12, was one the standouts. She’s now technical supervisor at the Los Angeles-based digital production company Digital Domain and known for “X-Men: Days of Future Past,” “Spider-Man: Homecoming,” “Thor: Ragnarok,” among others. Another, Glenn Winters (MS digital media ’13), is now manager of virtual production at Rockstar Games in New York, the maker of “Grand Theft Auto,” “L.A. Noire” and “Red Dead Redemption.” Third was Girish Balakrishnan (BS digital media ’12, MS digital media ’13), who for his master’s thesis proved that you could collaborate on making a film with someone in a different location, a novel idea at the time. He is now director of virtual production at Netflix.

“Girish was one of the first students to go through this studio; that was 10 years ago, and we have never let up,” Jushchyshyn says.

What has helped keep Drexel ahead of other universities is a combination of the University’s network of alumni working in the entertainment industry on the West Coast, its educational philosophy to integrate industry as a partner, and Jushchyshyn’s personal involvement in feature films.

Raised in Upper Darby, Jushchyshyn (BS ’93) received his undergraduate degree in commerce and engineering from Drexel and an MFA from the Academy of Art University in San Francisco. His film career didn’t start until later in his life, though as a child, Jushchyshyn’s interest in visual effects was sparked after seeing a television special documenting how “Star Wars” was made.

“It was an amazing, seemingly magical world created with blood, sweat and tears, a pile of junk and a lot of creativity,” he recalls.

But a career in that magical world seemed elusive, so upon graduation, Jushchyshyn went to work for what is now Teledyne Princeton Instruments, a maker of digital camera technology. That experience exposed him to the inner workings of the devices and ended up serving as a building block to his work in filmmaking. He began making software for the cameras and, when desktop computers came on the scene, explored ways to make production more efficient. When not at work, Jushchyshyn tinkered with creating computer graphics by self-studying on a Commodore 64, an 8-bit home computer popular in the 1980s.

Around the same time, the film industry was transitioning to new technology and the skills Jushchyshyn developed at work and through his hobbies proved valuable. He made a move into feature films and spent more than two decades building a portfolio of credits that include being on the team that won an Academy Award in 2008 for the best visual effects for “The Curious Case of Benjamin Button.” His other credits include “The Girl with the Dragon Tattoo,” “I Am Number Four” and “The Road.”

Jushchyshyn joined Drexel in January 2012 part-time and moved into a full-time position that September. Three years later, he convinced the department that Drexel and a group of students should attend Siggraph, an annual industry conference in Los Angeles focused on computer graphics and interactive techniques.

During that trip, the group toured studios and visited Balakrishnan, who was then working at a virtual and digital production company, to learn about the field. The tours enlightened Jushchyshyn on the direction of the industry and where Drexel needed to go. He concluded that no school was teaching animation, virtual reality and augmented reality and convinced Drexel to launch a degree program in 2018 focused on digital media and virtual production.

“The program was unique — in the world,” Jushchyshyn says. “That’s what makes it special; it doesn’t exist anywhere else.”

While other schools have since adopted similar programs, Drexel’s remains distinct. Other schools typically incorporate these disciplines as a small part of a more generalized program, or into other degree programs such as game design and animation, which tend to be focused mainly on the needs of the entertainment industry.

Drexel distinguishes itself by having a dedicated degree program specifically focused on preparing students to be well versed in the most critical production tools for metaverse media creation, Jushchyshyn says. That means teaching them about motion capture, real-world virtualization and visualization, real-time 3D computer graphics and human interaction, VR, AR and other immersive technologies. Students then apply these tools across a spectrum of industries including entertainment, medicine, education, industrial design, travel, hospitality, merchandising and branding, among others.

Since 2016, Drexel has had a booth at Siggraph staffed by students. It has led to job opportunities for them. It also keeps Drexel’s capabilities in the spotlight.

Regina Mae “Redg” Libunao, a fourth-year student, took a course on 3D programming for animation and was hooked. She’s majoring in animated visual effects and participated in the Motion Capture Club at the ACE Lab. Aside from motion-capture work, Libunao has been gaining experience on character rigging, which is a bridge from model to animation. A model of a human or animal needs to be “rigged” with virtual bones before it can be animated.

“It’s a lot of fun,” she says. “Motion-capture systems are used by actual companies who do this type of work. It’s not accessible to a lot of people and not a lot of schools have that. If you have experience with this while you are at school, you’re one step ahead in getting a job.”

Simulations for science, commerce and instruction

The ACE Lab leverages its technology to assist in groundbreaking research across the University — going far beyond just entertainment, sports and video games.

Roughly five years ago, for example, the lab began working with Drexel’s College of Nursing and Health Professions to create a simulated hospital emergency room and palliative care clinic to teach nursing skills to students. Actors, captured as digital characters, perform as patients in realistic health care scenarios.

“It looks exactly like a hospital emergency room or clinic, but they are in VR and don’t need to be restaged over and over again,” Jushchyshyn explains.

That work set the stage for new applications driven by the pandemic. When everything shifted to remote, the ACE Lab worked with the College of Engineering to create a VR learning environment. Jushchyshyn shot photos of equipment and tools in the engineering labs, which were reconstructed by digital media students into 3D replicas to be used by instructors in their online courses.

Drexel dance instructors who found Zoom too limiting during the pandemic asked Jushchyshyn to record them demonstrating correct and incorrect dance forms using motion-capture. An unexpected pedagogical advantage of the recordings was that each pose could be slowed, stopped or magnified.

Meanwhile in Drexel’s School of Biomedical Engineering, Science and Health Systems and the School of Education, engineering students were having difficulty visualizing how flat and two-dimensional diagrams translated into 3D objects, called stereoscopic depth perception. The lab created VR experiences to test students’ spatial awareness using the Purdue Spatial Visualization Test and to enable them to practice translating a 2D object into a 3D object.

“Even in the real world, you can’t interactively hold a 3D object and see a 2D object — but we can do that in VR because of its stereoscopic depth,” Jushchyshyn says. “We did extensive research and a complete study on that.”

The lab has several projects underway with Longwood Gardens, the former du Pont estate and botanical gardens outside of Philadelphia.

In one project, the lab developed an immersive virtual experience of the gardens and assessed its effectiveness scientifically. A Drexel team used neuroimaging technology to measure how engaged visitors were when experiencing the virtual reality version versus visitors strolling the grounds in person. The study measured participants’ blood flow in real time and contrasted the data.

The project began during the pandemic, when visitors were hesitant to venture into crowds and Longwood wanted to reach a broader audience. Longwood brokered the arrangement through Drexel Solutions Institute and involved faculty from Westphal College; Bennett S. LeBow College of Business; the School of Education; and the School of Biomedical Engineering, Science and Health Systems.

“This is what I love doing at Drexel,” Jushchyshyn says. “It’s multidisciplinary. It’s unusual for an R1 research institution to have access to an A-list or triple-A creative design facility. Schools of that caliber are traditionally dedicated art-and-design schools not directly connected to an R1 institution.”

Jushchyshyn recalls presenting his work at a conference and afterward receiving inquiries from art-and-design schools asking how he got access to, say, a medical and nursing school. “We’re literally right in the middle of all of this,” he says. “I feel that Drexel, if not absolutely unique, is exceptionally rare to have that under a single roof, and that is the way Drexel operates. It’s often extremely siloed at other institutions.”

The visitor experience project for Longwood spawned another to develop a virtual learning experience to enhance Longwood’s K-12 educational programming.

“If we can find new ways to share nature with those who don’t have ready access to it and to do it in a meaningful way such as through virtual reality, and to bring it to life for them, we can help people,” says Paul B. Redman, president and CEO of Longwood Gardens.

The Drexel-Longwood partnership also inspired a course, “Beyond the Walls,” taught by Jushchyshyn during the winter 2023 term. Students created a high-definition, computer-animated visualization of various aspects of Longwood’s historic Peirce-du Pont House as another way to enhance visitors’ experience. The course was facilitated by Drexel’s new Innovation Engine, an initiative of the Provost’s Office focused on multidisciplinary, experiential learning among students, faculty and external industry or nonprofit partners.

The labs’ capabilities are being further explored as a tool for learning and providing insights for academia and industry. A study called “Virtual Shoes: Using Virtual Reality to Enhance Safety and Foster Better Design for Older Adults in the Built Environment” secured a $10,000 grant from the AgeWell Collaboratory in Drexel’s College of Nursing and Health Professions.

The researchers are using VR to show design students what it is like for an older adult to view the built environment with diminished eyesight, compared with seeing something with 20/20 vision. The contrast is often startling to the students.

The VR production the lab undertakes is also spilling over into the mainstream — about 50% of its partnership work is with organizations outside of the entertainment industry.

The allure of the “metaverse” is a big draw. An undisclosed, international consulting firm recently turned to the lab to explore what metaverse commerce could potentially look like, for example. And a law firm, BakerHostetler, turned to the ACE Lab in November 2022 to help prep for a virtual conference it hosted. Two attorneys spent a day in the lab wearing motion-capture suits embedded with sensors to collect data that was used to design their avatars.

Pepsi, magic gloves and more

The B.I.G. concert wasn’t the first collaboration between Hyperreal and Jushchyshyn’s labs.

Scott had previously visited Drexel to work on a fully virtual PepsiCo commercial that was made to demonstrate that every detail — hamburger, car, street — could be created in a lab: no actors, no set, no perishables.

“We’re testing out what the future of metaverse in advertising will be,” says Scott. “As we were doing motion capture, Nick was able to broadcast to the creative team at PepsiCo and they were able to make creative decisions. We could show them in real time and adjust cameras and performance remotely.”

More recently, Scott returned to Drexel to test out a pair of high-fidelity motion-capture gloves from a New Zealand company called StretchSense that record data on intricate finger movements. A guitarist was brought into the lab and recorded as he played with the gloves on, moving his fingers along the neck of a guitar. Musicians can use the gloves to capture the process of their performance, akin to a musical signature, which can be copyrighted for the emerging metaverse economy.

“Understanding the latest technologies being developed in an industry gives students and everyone involved in the test the ability to understand how we can use it or improve on it,” Scott says. “Students can understand the complete process. The students get it and that creates really good talent and leaders of the future.”

While other facilities Scott has used on the West Coast may offer more redundancy and personnel, the scale of the Drexel lab and involvement with students has worked well for Hyperreal’s needs, he says.

“It’s more about research and development, testing different things, thinking through a concept and seeing it come to life,” Scott says. “We’re in a laboratory.”