Ahead of Its Time podcast by Setapp: VR gloves

Welcome to the first episode of Ahead of Its Time, an original podcast from Setapp about the tech underdogs no one realized would shape the future.

In this episode, we talk VR. Most people imagine gaming right away, but did you know that one VR glove brought about a major healthcare breakthrough?

Listen to Tom Zimmerman’s story about the inception of the legendary Nintendo Power Glove and what it has to do with playing air guitar. Then, get inspired by how Bob Crockett was convinced by one college dropout to start the company that would later launch DK2 VR Glove, a glove that simulates touch sensation and might be the closest thing to attaining superpowers.

Show notes:

Transcript:

Julia Furlan:

When you think of virtual reality, your next thought is probably gaming. There are a bunch of different VR systems on the market today that hardcore gamers use to enhance their virtual experience. And while it's certainly becoming a mainstay in the gaming industry, VR probably has more applications than you realize, and that's because the real power of virtual reality is its ability to convince your mind that you are in fact experiencing something that isn't real.

Bob Crockett:

I think the most impactful vision that I had in an early virtual reality headset was one of standing on a block and looking down into an endless void. And those sort of things, although, they elicit an emotional response, in my case was a combination of terror and delight.

Julia Furlan:

That's Bob Crockett, the co-founder of HaptX. HaptX is a leading company whose VR glove technology lets you touch and feel things in the virtual world. It can accurately reproduce the feeling of things like the smoothness of a wooden table, the softness of an animal's fur, or even the heat of a flame. Bob and HaptX in essence have taken the first major step towards creating a virtual world that feels as real as the real world. And while the HaptX VR gloves could certainly be for gaming, that's not the vision that Bob and his co-founder Jake Rubin have.

Julia Furlan:

They want to take virtual reality well beyond the realm of entertainment.

Bob Crockett:

We already have early experiments and early partnerships in the area of surgical training, and the extension of that is potentially being able to use our haptic gloves as the input device for a remote robot.

Julia Furlan:

But in some ways, the HaptX Glove is a highly complex descendant of a VR device built for the gaming industry, the Power Glove. The Power Glove released more than 30 years ago was both a groundbreaking piece of VR technology and one of the video game industry's big flops.

Bob Crockett:

I have absolute admiration for the product, even though it's suffered its own series of grief as has every innovation that's a little bit ahead of its time. But, the difference between an earlier innovation like the Power Glove and where we are today is not the difference between good and bad innovation or not getting it and getting it. Technology evolves and we could not have built our club more than 10 years ago. It just would not have been possible.

Julia Furlan:

It was the right idea, the right place. Sometimes it's the right time that's elusive. But when that idea is truly right, time and sometimes technology has a funny way of catching up to it.

Julia Furlan:

I'm Julia Furlan. And this is Ahead of Its Time, an original podcast from Setapp. A show about the tech underdogs no one realized would shape the future. Setapp's versatile app subscription service empowers you to step into a new era of productivity.

Tom Zimmerman:

It's going to be terrible quality. All right. Let me do this. And I'm going to put my big speakers on. I thought you would just like get the soundtrack and edit it up yourself. Do you really want this shitty quality? Hold on, let me stop this. Okay. So this is a piece called T Rose. I'm not sure why I called it that, but that's what it says. And it's mostly a drone piece. It's done with Arp synthesizer. Hi, I'm Tom Zimmerman. I'm an inventor. I invented the original Data Glove, which morphed into the commercial product, the Power Glove.

Tom Zimmerman:

We actually used this soundtrack on our first VPL demo to show people what you could do with a Data Glove. So it has some relevance to VR.

Julia Furlan:

In the late 1980s, Tom was at a tech startup called Visual Programming Language Research, or VPL for short. VPL was the world's first virtual reality company and developed several pieces of tech including the first VR headsets. But Tom's Data Glove was their marque product. And like Tom himself, it was rooted in music.

Tom Zimmerman:

So I was surrounded by music. My dad loved opera and I hated it. And every weekend he'd be painting the house or something and I'd hear opera. Yeah. So as I look back, I had all these pieces of the puzzle of this virtual world. And one of them was watching my dad conduct Beethoven's Ninth. He was so excited. He went out and he got from Radio Shack some big speakers. And he said, "Tom, listen to that bass." And he's standing up and conducting it and like thunder shaking his hands. And I'm just a kid watching this and here's my dad, the accountant, grooving on Beethoven's Ninth.

Julia Furlan:

And while his dad was jamming to classical, Tom was busy grooving on the works of John, Paul, George, and Ringo.

Tom Zimmerman:

So I used to listen on my tube amplifier monophonic, and there's a big 12 inch speaker I have right to my face. So The Beatles are inside that and I'm standing up next to it and I'm singing along and playing with them. And so I'm in my imagination, the fifth Beatle, because they're right in front of me. If I close my eyes, all I can hear is them. And I can feel my guitar in my hands and I'm strumming along with that. And I thought, wouldn't it be cool if, and I find as an inventor that's a great phrase. Wouldn't it be cool if when you made believe you were playing air guitar, you could really hear it coming out of the speakers?

Julia Furlan:

Like those timeless Beatles songs, that question lingered in Tom's mind through school, during his guitar lessons and all the way to his time at college, where he discussed the idea with a roommate.

Tom Zimmerman:

He had this idea that if you touched the different finger combinations, you could play chords. If I really wanted to hear air guitar, I'd have to measure finger bending. So I needed some sensor that can tell when your fingers are bending. So I went down to Canal Street, which is where all these surplus electronics is in New York, picked out up some LEDs, some photo transistors, some rubber shiny tubing. And so I glued it to a gardening glove and that was the birth of the Data Glove.

Julia Furlan:

Tom ran the tubing along the fingers of the gardening glove. At one end of the tube was an LED light, at the other end was a light sensor. When he bent his fingers, the tube bent, choking off a bit of the light to the sensor, changing the signal. Using his Atari 400 computer, those changes in light could be translated as a change in pitch to a musical note. So by bending his fingers, he could play chords. This got Tom thinking. If he could use his hands, maybe he could use other parts of his body to make music too.

Tom Zimmerman:

And actually in developing this, I was in my parents' house in my bedroom and I had done the glove on the hand and I thought, well, let's try my knees. So I took my pants down and put an ACE bandage on my knee and put the flex sensor on it and I'm programming on my Atari 400. And I paused for a moment and I thought this is such a weird scene.

Julia Furlan:

After college, Tom decided to move to California to take a job at the gaming company, Atari. As it turned out, there were major layoff after the gaming crash of 1983. So Tom's time at Atari didn't last long, but his creativity fit perfectly with the bold eccentric what if attitude of California. Tom found a community of like-minded creators. One day, while it attending an electronic music concert at Stanford University, he met someone who would change his life forever, that someone was Jaron Lanier, the man who coined the term virtual reality and who's widely considered the father of the field.

Tom Zimmerman:

And I told him about this glove I had, and he was developing a visual programming language. And it was very clear in a couple minutes, we had a lot in common. And so that was just fortuitous. I talk about successful inventions having three personas, the dreamer, the engineer, and the entrepreneur. And I'm pretty much in the dreamer engineer camps. And he really touches on all three. So meeting him really propelled this prototype into a product.

Julia Furlan:

Jaron Lanier's visual programming language was the software Tom needed to run his gloves. And Tom's glove was the hardware Jaron needed to unlock the potential of his software. So Tom joined Jaron's startup VPL Research and they got to work. They started creating all kinds of applications for the Data Glove and even built a full body data suit outfitted with sensors that could measure arm and leg movement.

Tom Zimmerman:

We looked at a range of applications, medical, entertainment, computer aided design, robotics, telerobotics, telepresence. And we tried pursuing all these customers. But when we started talking to people, they said, forget about programming languages. No one wants to program. We want video games.

Julia Furlan:

1989 was a whirlwind year for Tom and Jaron. Their technology was the talk of January's hugely influential Consumer Electronic Show and their glove was featured on the cover of Scientific American. Soon, there was a deal with the toy company Mattel. They wanted to market the Data Glove as a game controller for Nintendo. They called it the Power Glove. Late that year, just two weeks before Christmas, the Power Glove was front and center in The Wizard, a Fred Savage film about an underdog kid in a major video game competition. With that marketing sizzle from Nintendo and Mattel behind it, the Power Glove sold well.

Julia Furlan:

It was a busy year for Tom. He hadn't really had much time to process everything he'd accomplished in the last 12 months. But one day it all hit him.

Tom Zimmerman:

After the Power Glove came out, I was in New York walking down the street and I passed a Toys-R-Us and I looked in the window and there was my Power Glove. And I thought, this is amazing. I dreamed of something and here it is in reality. And I had this like feeling of the universe saying If you really dream something and are passionate about it and dedicate your heart, mind, and body to it, it'll be realized.

Julia Furlan:

Realized, sure. But its success wouldn't be sustainable. To meet a consumer friendly price point, they had to compromise on functionality and marketing could only distract from the Power Glove's performance issues for so long.

Tom Zimmerman:

The Data Glove, which was $8,000, we sold a lot to universities because they were doing early work in VR and then Mattel came out with the $80 glove. And a question would always be, why is one $80 and one 8,000? Well, the performance obviously was a lot better with the Data Glove.

Julia Furlan:

The rush to market left no time to develop custom games for the Power Glove. Instead, Nintendo adapted it to existing video games, but the glove was meant for use in virtual reality, not as a games controller. And that was a problem. There were lags with some of the older games that the glove was retrofitted to work with. There were games that failed to recognize hand movement. The sizzle around the Power Glove faded and in or 1990, it was discontinued. While it was a failure in its time, Tom sensed that his invention had helped pave the way for virtual reality to come.

Tom Zimmerman:

We learned a hard lesson that people are still trying to learn. With VR, you're putting hand through walls, you're picking up cups that aren't there. And so I think tactical feedback, our haptic feedback will be really essential to make the experience more rich, especially interacting with physical things.

Julia Furlan:

That's exactly the problem HaptX is solving with their Haptic Feedback VR Glove. Over the last 30 years, every facet of VR technology has evolved and improved. But in a rather ironic twist of fate, the very thing that proved to be the Power Gloves kryptonite is now the backbone of the HaptX Glove's versatility. HaptX takes advantage of today's advanced video games software, software that the Power Glove desperately needed to improve its functionality.

Bob Crockett:

High end video games have very, very sophisticated physics engines running underneath them. Now, they provide everything you need to drive our gloves.

Julia Furlan:

Here's Bob Crockett again.

Bob Crockett:

Our gloves could now be used essentially in any video game environment. You don't have to redo video game content to make it haptic ready. It just works.

Julia Furlan:

And so it makes sense that the HaptX demo programs inspire the same sense of play that many people feel when they play video games.

Speaker:

Okay.

Speaker :

How's that feel? Man, look at that.

Speaker 4:

I feel like the Power Glove.

Speaker 5:

So when I first drop you into the virtual environment here [inaudible 00:14:33].

Julia Furlan:

It sounds like this VR enthusiast who HaptX brought in to test drive their gloves might have been a fan of Tom Zimmerman's Power Glove when he was a kid, but he's about to discover just how far the technology has come. He slips on a VR headset and a pair of black gloves fitted with flexible tubes along the back of his hands. These tubes lead to sensors at the end of his fingertips.

Speaker 5:

All right. So now you can reach out and interact with any of those objects, throw out some gang signs.

Speaker 4:

On my God. Oh snap.

Speaker 5:

If you look at over to your left, you got a little fox friend.

Speaker 4:

Hello, little friend.

Speaker 5:

If you turn your palm facing-

Bob Crockett:

And it's got a fox that jumps out and curls up on your hand. And so when you're playing with scale or when you're looming large over a farm or have a tiny animal in your hand, that's when you have the most visceral responses, the most human responses that you're interacting with something that is amazing and magical.

Speaker 4:

That is too weird. It is wiggling in my hand. Okay. Wow. Wow. You got to try this. You got to try this.

Bob Crockett:

It was one of those rare and very special moments when you see how technology intersects with human emotion. Because it was at that point that we could see just exactly how impactful this new technology was.

Julia Furlan:

Fueling that emotion is this glove technology that replicates every kind of physical sensation. The way fingers stop against an object, its feel and texture, its temperature, and the tiniest movements. The field of haptics has been around for many years, but haptics in virtual reality is starting to take on a much more important role. Our sense of touch and the information or haptic data we get from helps us make sense of our world. We need more than just the visuals and the basic physical feedback we get from traditional VR systems. And that is what HaptX is doing.

Julia Furlan:

They are at the forefront of VR haptics, but the forefront of virtual reality isn't necessarily where Bob Crockett imagined himself. In fact, he was late to the field.

Bob Crockett:

I had never tried on a virtual reality headset. And the first time I tried one was back in the days before Oculus was owned by Facebook. And I put that headset on like many others before me and had a life-changing experience. When you see a good virtual reality system for the first time, even though it might be imperfect, you know that you're seeing the future.

Julia Furlan:

Then one day nearly a decade ago, he got an unexpected email from a young man named Jake Rubin.

Bob Crockett:

And so he emailed me, we had some phone conversations and he was coming to me as a 20-year-old college dropout with a vision for an apparatus that you strap yourself into, full body, immersive haptic system and all of a sudden you can't tell the difference between the real world and the virtual world. And as you can imagine, that's a challenging conversation for a 20-year-old who hasn't finished college to have with a college professor to convince them first off that you're not entirely crazy.

Julia Furlan:

Jake's intelligence and drive soon won him over. After that, the challenge for Bob was to think and work at Jake's pace.

Bob Crockett:

Working with Jake was like drinking from a fire hose. He would come prepared with 27 articles that I was to have pre-read before any one of these mental tennis matches. And, of course, I was totally unprepared for this, but what it turned out to be was very enjoyable conversations that start in our offices and then would go over lunch and then would continue into dinner and then transition into the bar at night. And so I started working with Jake and got deeper and deeper into the conversation about what this device would look like.

Julia Furlan:

Bob and Jake founded HaptX in 2012 and got to work creating virtual experiences hoping to win over investors. The logical starting point was to create virtual tactile experiences for the hand, but the first prototype wasn't a glove. That was too complicated. First, they had to build an artificial skin that would accurately simulate our sense of touch. And this is what makes their technology a game-changer for VR.

Bob Crockett:

You actually have to be able to press far into the skin in order to trick your mind into thinking that you're truly interacting with something that's out there. And so the way that we do that is with microfluidics, and that's just a fancy way of saying that we're using air that are routed to little balloons that are at the ends of your fingers and across your palm. And we can inflate these balloons in the right sequence, in the right pattern to the right pressure to replicate any sort of interaction that you would be having with a real object.

Julia Furlan:

Their first prototype was a giant box, which became a key attraction at investor meetings.

Bob Crockett:

We would lug this box around on its own cart. And I'm telling you this thing was about 120 pounds. It was not a small device. And we would inevitably set it up in the conference room of a venture capitalist firm or an investor's office. It would draw a crowd.

Julia Furlan:

Wearing a VR headset, the user placed their hand inside the box and pressed it against this artificial skin and then the virtual experience began.

Bob Crockett:

You feel the thermal sensation, we had hot and cold in different locations across the hand. And so if you picked up an ice cube, you'd put it in your hand and you'd fill the cold. We had a dragon fly in and breathe on you and you could feel the fire coming out of the dragon's mouth. Had a bunch of fruit laid out on the desk nearby and you would grab a piece of free route and put it into your open palm. That was the first time that people could really understand what we were talking about and really see the future. And it was so mind blowing.

Bob Crockett:

We had people who would when the demo was over, they'd take off the headset and they would open the box to see, they'd looked for the banana because they were convinced that we're trying to pull something over on them that feeling was actually a real object, but that's how new it was.

Julia Furlan:

At the heart of it, it's this type of experience, this sense of wonder and disbelief that's at the root of what Bob and Jake are doing. They're creating technology that convinces your mind that you are actually feeling an object that isn't there. And their technology has only gotten better over the last few years. In 2021, HaptX released their first commercial VR glove called the DK2. They struck deals with various companies, including Nissan, to help them develop vehicle prototypes and with the English medical firm Fundamental Surgery for use on its VR medical training platforms.

Bob Crockett:

There are examples of customers using it for simulation and training. So you could imagine that if you can train people in a virtual environment, that can be much safer and much less expensive and much faster than if you have to send firefighters out to an actual burning building, for example, or pilots in an actual cockpit. The next day, it can be an electrician workstation. The next day, it could be something for the space station. It really doesn't matter. You're just changing out the virtual environment. And that's very different from the way things are right now.

Julia Furlan:

Perhaps the most consequential application is one that Tom Zimmerman and Jaron Lanier first imagined for their Data Glove. The HaptX DK2 Gloves can be used to remotely control a pair of robotic hands, and that could have massive implications for telemedicine and healthcare.

Bob Crockett:

The idea is that the robot might be in London and it's got a pair of robot hands that have sensors on it. And when that robot touches something, you feel it on the user side. And you can imagine that having an application in remote surgery in the future where a surgeon can be on one side of the continent and a patient can be on the other side of the continent and perform a remote surgery, those are the sort of things that absolutely will happen. And it hasn't truly been possible until you've been able to have that natural touch interaction.

Bob Crockett:

So, that's why we're very excited to be kind of the key piece of the puzzle for these much larger systems.

Julia Furlan:

And where is HaptX technology headed?

Bob Crockett:

For right now, things are aligned to let us do the job that we set out to do, which is to produce a fully immersive haptic full body system. The gloves will also be one component in a more comprehensive system that's more like Jake described back in 2012, where you strap into an apparatus, put on a VR headset and distinction between virtual and real diminishes to the point of zero.

Julia Furlan:

The technology has changed, but the vision remains the same since Tom Zimmerman developed his Power Glove.

Tom Zimmerman:

I think haptics is going to be a very exciting experience. We have a great power as humans to suspend disbelief. All I have to say is once upon a time, boom and you're there. We're inherently storytellers. And so we have such great facility for imagining things just as long as the technology stays out of the way, and VR has great potential.

Julia Furlan:

And Bob Crockett couldn't agree more.

Bob Crockett:

Yeah. I think that's, for me, one of the things that's so exciting about creating this. It's not just a business tool. It's not just something that will improve gaming. It actually is a new way of interacting with worlds that haven't existed before. The future of VR really is catching up to the promises of VR, first and foremost. It's literally world changing because now you get to create your reality.

Julia Furlan:

I'm Julia Furlan, and this is Ahead of Its Time, an original podcast from Setapp. Working on your next big thing, Setapp's productivity toolkit will help you stay focused and get stuff done. Head over to setapp.com to see if Setapp can help you bring your ideas to life.