Why you should be starting VR/AR/MR design & development right now
tl;dr — This is for all of you that are interested in the phenomena “beyond mobile”. And how this will change the world as we know it. Buckle up Earl, it’s gonna be a bumpy ride.
Fast paced future ahead.
If you haven’t noticed, things are about to change. I’m not talking small change. I’m talking big time, a fundamental shift in the way we interact with technology on-a-daily-basis kind of change. And it will all happen within the next two years…
Hold on, you might think I’m way off here, so for the sake of the inevitable, remove the time frame of this prediction. So now it’s up to you and how to react. Should you or should you not learn about this new fundamental shift? I would go for the latter.
Biggest shift you’ve ever experienced.
This change will be the biggest and maybe even fastest transition in technological medium mankind has ever experienced. We moved from radio to television, we moved onwards towards home PC’s and then we moved into mobile and tablets, we’ve been touched by virtual reality…
Today there is a wide landscape of mediums. Radio, TV, PC’s, Mobile, Tablets and — well – the Internet itself. With every major transition in medium, loads of wealth has been created. Information has become close to free, and distributed across the globe at a speed never witnessed before (for better and worse). Today, all the mediums basically act as information highways, utilising, sound, stills and moving images. Experiences have become more elaborate than ever, and storytelling is at the heart of everything.
This next shift in medium will start to bring magical experiences into your daily life. At least this is how it’s sold and marketed at this time. You might ask yourself how this will affect you and your life, frankly. I have no idea of the widespread impacts of this new era in tech, but I know it will be such a fundamental shift, that everything you’ve come to learn and gotten used to will be flipped on its head. Alice in Wonderland style.
Beyond mobile
We’ve been living in the mobile generation since a bit before the introduction of the iPhone. We all remember Steve Jobs walking onto that stage with a deep grin on his face, showing the “one more thing”… Well it was more of 3-in-1 kind of situation, but anyhow that’s history now, and –sadly — Steve is gone. But what is not gone is his legacy. With the introduction of the iPhone, Apple kickstarted a mobile revolution that has led to more mobile phones in the world than humans. Now it’s time for the next step in that evolution…
Within the next 24 months, companies will come out of hiding (yes, I’m talking about you Magic Leap and possibly Apple) and others will ship their first consumer products (a Microsoft Hololens that is not $4000). And some will be shipping next generation devices (Meta 2, Will.I.am is endorsing). All of the companies above has been in the augmented reality game for a couple of years now; hiding in plain sight. And that long research and development phase is soon over and it’s time to set in motion the next revolution in information technology… Mixed Reality.
Mixed reality
So let’s take a step back, lets begin with VR or virtual reality. It’s something that has been spoofing around since the 70’s, clumsy, big and not very good. But wait, erase that, VR is good, it’s so good that it helps patients with dementia, it let’s creators create in virtual reality. And it helps with telling deep, connected stories. Putting you as the consumer inside the actual content, it’s #contentception, like the people over at Kickpush say.
Now what does this have to do with Mixed Reality (from here on MR)? Simply VR has been the stepping stone for a lot of the technologies that we will see in MR headsets and it’s technology. And while we all been hanging around the HTC Vive and Oculus Rift demos, MR has been left alone to tinker and evolve in secrecy.
But… What is it?
Mixed Reality (MR) is a combination of sensors, super advanced optics and next generation compute power. Simply put, it’s an holographic overlay on your world. Seen through a thin strip of glass that — with time –, will be just like a pair of glasses. And this is important. Because unlike VR, MR will be active with you throughout your day. It will be your, always on companion. A little helper might live in there, feeding on your digital footprint, and helping you clear your tasks and be productive. Like Clippy, but built for the 21th century using deep neural nets and machine learning. Maybe even knowing more about you and your world than you do…
Sounds scary? It is. But it’s also beautiful.
What will Mixed Reality actually do?
With the combination of sensors, inside-out tracking of your environment — mixed reality will render holograms in your world. This is important: it’s not “on top” it’s really “in”; holograms will be able to hide behind your furniture; knowing where to be in your rooms, so when you get back into the bathroom you’ll have a daily calendar pinned on your mirror to give you a quick brief of what’s going on, or even your favourite game next to the toilet for those extra long sessions.
So, not only will it be able to place emulated screens and objects in your bathroom, it will add a whole new world of interactions, apps, games, and experiences we yet have imagined. Yes, I’m talking about the real world as a canvas. Now, you might start to understand the magical experiences this will bring — and like a true illusionist — the best is saved for last.
The use-cases for this technology right now seems endless. The fact that it will replace your mobile is just the first step; it will most likely replace the vast majority of technology you own. Out with screens, tablets, phones, TV’s… That will just be the beginning. I’m not saying those mediums will die, but just as other mediums have dwarfed since the introduction of mobile, so will this have a similar effect.
What can we expect of the devices?
So there is a lot of sources pointing to some obvious things. Full device motion tracking, camera object recognition and tracking with AI support, gesture tracking, inside-out room scale mapping. microphones, cameras, GPS, motion sensors, light sensors, temperature, internet connection and so on… Not to go to much into details, we are expecting quite a lot.
While all this is great news, this will be another level of attention grab- that we have never experienced before. Now the distractions won’t be tucked away in your pocket — only living inside a screen. It will literally be pushed into your eyes. This is where we — as creators and early explorers — need to be aware on how to shape this technology for the better.
One of the biggest advantages that will come with MR is six degrees freedom of movement, with inside-out tracking. So instead of you standing in front of a camera or lasers towers (what you currently need to do with PSVR or HTC Vive), you’ll have the equivalent hardware pointing the other direction, away from you, mapping the world around you in real-time. This is one of the core functionalities with mixed reality. Microsoft is already doing this really good with the Hololens, so we just have to hope that whatever is next is even more high-res.
So why you should care…
With the information I shared so far, there should be enough to at least get you excited. Or really freaking overwhelmed, since this will most likely blow everyone’s minds out. The wow-factor will be insane. Yes, I truly believe that. This is one or maybe the one major shift in medium our generation will experience — just like our parents did with television.
At this point, there are very few people in the world that works on user experience design in VR. Even fewer in MR. This presents as an amazing opportunity for all the 2D designers out there. While you (we) are busy pushing pixel perfect designs to Dribbble, this new world is unfolding in the shadows. Again don’t get me wrong, screens will be around just as radio still is. But the vast majority of people will interact with MR. So that’s where the new jobs will be. And since it’s such an engaging experience no matter if you are just using MR to do work-related tasks, the amount of skilled workers producing content for this new medium will be enormous.
The biggest challenge: we’ll have to try to transition more or less a whole industry from 2D to 3D design. With a deep deep focus on empathy and inclusiveness. These experiences will be much different from the ones you currently consume. They will be integrated into your reality in a way never seen before. Maybe even with people first instead of applications first. It will be all about engaging and communication. Instead of watching a video of your niece taking her first steps, she will be teleported to you and you’ll see her first steps in your context — forever changing your experience with her first steps. Your truth. Your Mixed Reality.
Tools, skill set, emotions, Where do you start?
So currently we all know how to work with our favourite stack of tools (Sketch, Principle, Framer, Photoshop, After Effects). But very few of the people working on 2D design have also mastered the tools used in 3D game design. Cinema 4D, 3D Studio Max or Unity.
We will have to pull up our sleeves and get ready to learn some new tools and practices. Lighting, expressions, textures, motion blur, ray tracing, reflection mapping, low poly modelling. The list grows long, but don’t fall into despair. You already have an advantage, you are already a UX or UI designer; you are not starting from zero. You have a backpack full of experience. Now it’s just about broadening that experience, and mastering a new set of skills to prepare you for the workforce of tomorrow.
Just as with anything that’s new, there is a learning curve, and that’s something you’ll have to overcome. With time, the tools will become easier to use and the transition will be less painful. However, right now there are a couple of things you should start looking into.
- Get yourself a VR setup, (Example HTC Vive & a good enough PC)
- Get to know at least one 3D render tool well, (Example Cinema4D)
- Experiment a lot with Unity3D — I can’t stress this enough.
- Start reading through a lot of resources, I have collected a bunch here.
- Don’t be afraid; overcome your fear to fail and just keep learning.
- Join a local community. Look on Facebook or for Meetups.
Ending words.
Now that you have all that, do shit, experiment, fail, iterate, fail again. And in all that despair, you’ll learn, you’ll improve. And when time comes somewhere in the next 24 months, you’ll find yourself in a competitive advantage. And you wont be stuck pushing pixels to Dribbble, you’ll be finding yourself in a position where you are part of the next big paradigm shift in mediums.
It’s our responsibility as designers to shape this world by all the best means. Building experiences on facts and trust, that helps empower humans to do more incredible things than we’ve managed to do so far. This is your opportunity to join in early on a revolution that will help shape the next generation.
If you want to follow along in my VR/MR journey you can follow me on Twitter or here on Medium where I’ll be posting my journey as it unfolds.
I’m also thinking about starting a community for interested designers and developers that want to get into VR/MR. Let me know on Twitter if that would be of interest to you.
All resources for this post can be found here, and you can read my one year old post about Mixed Reality and Virtual Reality here.
