Preparing for VR: Combining LEAP Motion+Cinemizer OLED headtracking+Kinect Voice Recog in PS3 and Xbox 360 games

Ask not what Virtual Reality can do for you ask what you can do for Virtual Reality....This seems to be  the  mantra behind the burgeoning VR scene..and I wouldn't have it any other way ...

I have been using hmds now for almost two years on a daily basis, watched them evolve from the headcrab that was the HMZT1 through the ST1080 to the the Cinemizer OLED and my soon to arrive Rift ( come on Palmer keep calm and carry on!)  .Hmds are only part of the puzzle that is VR .. Another piece is how on Earth we each  personally interface with VR worlds,to do so we have to let go of legacy controllers like joypads ,keyboards and move beyond them assuming someone has managed to create those worlds ..which is yet another piece of the puzzle..

the moving finger moves...... 

We also have to move beyond the current obsession hardware developers have re headtrackers , hypothetical lag that exists or does not exist on pre release hardware or in software ... that itself does not exist outside pre alpha code..For VR to take off , it will take more than a fast headtracker hard soldered to your hmd circuit board sorry app devs ... For VR to take off the end user expects lag free headtracking, voice recognition, hand, palm, finger, thumb recognition ,the ability to include or add on legacy controllers and most importantly the ability to use all this with not only promised VR apps ( launching whenever they are ready) but with ALL the thousands of apps and games that we the users have spent our lives accumulating.. 

and having moved so does Kratos......!

The problem of lag/latency  if it is discernible when using your headtracking hardware is a problem that exists in that current hardware, to solve it either we wait for the technology to catch up to the idea of lag free headtracking and use brute force  ..or with current headtracker tech we offload the grunt work to the cloud or other hardware,which we can also use to tweak, accelerate ,adapt ,adjust or in some cases even slow down the headtracking.. 

The developers of Oculus have to be careful they do not talk their device out of existence with all the scuttlebutt on latency that is clouding the launch of the developer kits this week ...yes I have one coming too ! 

it's full of stars......that you move ! 

I have used headtracking now for 3 months . never experiencing latency or lag in any scenario , I have used LEAP Motion now for over a month and again never seen any discernible lag . In fact if you look through my diatribes on the Cinemizer OLED headtracker and the LEAP Motion I often talk of these devices being over sensitive and that is on what most core game developers would think is obsolete hardware...

I also do not have much time for the idea that VR hmds should only run specific VR ready code.... I always intended to use my hmds to interface with any form of display, OS, game , app or kitchen appliance/lawnmower ..I need to shoot latency free Force lightning from my Sith hands now ..not when Senior Carmack licenses the engine for Doom 5 to Gearbox or Rovio(laugh but it will happen) and as you will see I can ... 

The ability to shoot Force ligthning from your hands pales into insignificance compared to the true power of multiple combined  latency free VR control input mechanisms

errhm ...with all this in mind I set out to use my i7 X220T running Windows 8 to combine the Cinemizer  OLED headtracker AND LEAP Motion finger tracking on 10 year old hardware .. namely the PlayStation 3 and Xbox 360 ... 

So we are going to use actual games , play them with multiple VR leaning control methods 1. Zeiss headtracking 2. LEAP Motion  finger tracking 3.Kinect voice recognition  and throw in some legacy control too ...

Note I would use the Cinemizer OLED hmd to play these games in 3D and or the smd ST1080 ...to show you it works though I am filming off a normal home Plasma tv...!

Are we ready ? then let's see how we can combine headtracking , voice recog and finger tracking to play Xbox 360 games ...any Xbox 360 game ... and PS3 games ...any PS3 games...

1.Xbox 360+ Cronus+Bullseye + Win 8 running Bullseye+Dangermouse + Leap Motion for finger tracking +Zeiss Cinemizer OLED headtracker + Kinect connected to the console

Xbox 360 gaming with VR leaning controls :  

 

Halo 4 with LEAP Motion finger tracking and trigger action combined with Zeiss Headtracker FOV movement

Halo Anniversary lets us combine Kinect Voice recog , Leap Motion finger tracking . Zeiss headtracking AND play in OLED ghost free 3D on the Cinemizer OLED hmd...

Ikaruga played with LEAP Motion with the thumb click mapped to the trigger...my score's improving already....

Combining LEAP Motion+ Cinemizer OLED headtracking to play Xbox 360 games : 

Hand tracking combined with headtracking to play REZ HD,Panzer Dragoon Orta , Ace Combat 6, Halo Wars, Shenmue 2, Bastion, Radiant Silvergun

this may look good ...but it feels totally awesome when you play using the LEAP and or Zeiss Headtracker!

Panzer Dragoon Orta with LEAP Motion control  NB we could easily map the FOV shoulder movement to the Zeiss Headtracker too 

This is an example of where headtracking does not need to mirror real world head movement ..in a game .. Panzer Dragoon Orta uses rigid shoulder pad view shifts ..we could map these to left right turns of the Zeiss headtracker , and or we could also map the entire FOV control to the headtracker  , whilst we use the LEAP Motion to handle the targeting, weapon selection, or movement..

Rez HD playing with LEAP Motion ... now you can feel the vibes from the 360 controller in your pocket ..whilst playing the game with Finger tracking ...Matron!!!!!

look around or pilot your Ace Combat 6 planes with your fingers,hands or your head!

Before Ikaruga there was Radiant Silvergun...here played with LEAP Motion finger tracking ...I have yet to figure out what to do with the concurrent headtracking!

Combining Leap Motion+Cinemizer OLED headtracker+Kinect Xbox 360+Legacy control : 




This is very nearly the VR control Holy Grail,  using Mass Effect 3 :which lets us combine 3 VR leaning control methods : Kinect Voice Recogntion, LEAP Motion finger tracking and Zeiss Cinemizer OLED headtracking ... Note we are actually using 4 control methods simultaneously if you count the legacy Xbox 360 Pad.. If you really want to push it you could have Kinect on the PC too , with full body tracking in addition to voice, head and hand recog! (but that is for another time !) 

You should be able to see there is 0 lag that can be discerned by the user when combining LEAP Motion and Zeiss headtracker use on Xbox 360 games using this solution. And that we can if our games allow it also add voice recognition to the mix.  The Cronus and Bullseye combination is most impressive , there is no screen judder or delay when using our VR leaning controllers with this set up . I found it simple to tweak settings from Bullseye too . Lastly we can see that LEAP Motion runs parallel to any other form of controller that we want and we can use apps like Dangermouse running alongside them. 


So on to PlayStation 3 gaming with VR leaning controls ...


PS3+Cronus+Windows 8 i7 Lenovo X220T running Bullyseye+Dangermouse+ Leap Motion +Zeiss Cinemizer OLED headtracker +/- any other input we can connect to the PS3 or X220T!

Some of these controllers are not like the others... 

Combining LEAP Motion+Cinemizer OLED headtracking on PS3 to play Child of Eden 

How cool is that? Now imagine..well you do not need to imagine..just attach the Zeiss Headtracker to a Cinemizer OLED or ST1080 hmd and you can do all the above in OLED ghost free 3D ..or LCOS 1080p ghost free 3D... right now. 

playing Child Of Eden in ghost free OLED 3D with LEAP Motion and Zeiss headtracking is simply mind blowing!

  


Combining LEAP Motion + Cinemizer OLED headtracker in PlayStation 3 games test 2



Combining LEAP Motion+Cinemizer OLED headtracker in PlayStation 3 games Metal Gear Solid 3 , Metal Gear Rising , DUST 514,Uncharted 3, Crysis , Wipeout ,DYAD all tested with combined LEAP Motion finger tracking and Zeiss Cinemizer OLED headtracking 

you spin me right round ... 

right round....!

both Zeiss headtracking and LEAP Motion working in conjunction ...

you know for when Raiden goes to the dentist... 

LEAP Motion and DangerMouse,Bullseye and Cronus used to control Cryengine 3 Crysis on PS3 you should experience this in 3D!

Zeiss headtracker and LEAP motion used together to play Crysis on the PS3 ...no conflicts , no slow down ... no discernible lag

Using the Cronus both Cinemizer OLED headtracker and LEAP Motion can be used to play DUST 514 , being able to tweak settings from Bullseye also means we can do far more than with direct connection of a mouse and keyboard to the PS3

With Dust 514 I have already spent time using the the Zeiss headtracker directly connected to the PS3 , taking over the mouse FOV duties. With this new solution though we can do far more .. first we can set up both hands to control the game , in the video I had time to set the LEAP to the left thumbstick , which let me move the character and also to the FOV stick which let me move the field of view .. The Zeiss headtracker ran alongside the LEAP without issue .. this means we have plenty of VR leaning control options to combine and play DUST 514 .. what we are doing here is looking for and at new ways of playing.... moving your FOV around quicker than the next guys headtracker is not going to be enough to kickstart the VR gaming revolution...

VR gaming control is here ...

all that stuff you saw in movies like GAMER ... 

it is in your hands ...

and on your head too if you are using hmds... 

There is 0 lag or latency when using the Zeiss Headtracker and LEAP and both of them together to play MGS3 ! 

Snake responding obediently to the Zeiss headtracker ...as will Gabe .. and Sam too !!!

Using LEAP Motion to turn on my shield and the Zeiss Cinemizer OLED headtracker to steer and accelerate ....

What you see in the above pics are the subtle movements to the mm that LEAP Motion can pick up. Note though, we do not need all this power , what matters in making LEAP and Zeiss headtracking viable VR controllers is how we tweak those small movements and convert them to ingame actions that the gamer and not hardware developer with a slide rule expects ... We can get at this using the Bullseye software, the LEAP sdk, DangerMouse and the Zeiss headtracker sdk. But I purposefully did none of that .. because I want to come at this from a gamers point of view .. they want to connect these devices and play ..not spend hours updating drivers .. Yes you can on the developer side do all those things , but remember the Wii Mote took off not because people were measuring or adjusting it's lag but by the simple fact people were able to pick up and play . I have to say using the Zeiss headtracker and the LEAP is actually simpler than the Wii mote .. no drivers , no lengthy calibration routines ... 

As you can see there is no end to the possibilities of playing games with VR leaning controls and combining them, in Uncharted 3 I would be playing in 3D using the Cinemizer OLED hmd , moving the FOV with the Zeiss Headtracker and moving the character with LEAP motion .. in the video I have just the punch button mapped to the thumb click. LEAP Motion does gestures too, so a punch could actually be a punch ..steady on though we do not want to go the KINECT route ....! 

DYAD is one of the simplest games to demo LEAP Motion with , the response is excellent 0 lag! 

Leap Motion used to play God of War Ascension 




Whilst I was errh testing all this out, my God of War Ascension arrived and as I had everything set up and with the demo already installed,I thought I would give playing it with LEAP Motion a go ... And I was pleasantly surprised, these games tend to twist your thumbs off after a while so mapping one thumbstick to the LEAP was a revelation, suddenly I was jumping , and combo fighting with a twirl of my finger or hand movements or a click of my thumb. More than just a test , I found I was actually playing through a level .. using a VR leaning control method to play what is essentially (the epitome of) a side scrolling beatemup.. 

 Compared to the PS3s own motion control The LEAP is insanely fast even at this developer stage.

Note you could map both hands, 8 fingers, 2 thumbs, add voice recog for other commands using the method here .. Looks like I am finally going to finish some of my PS3 games ... VR style ... We have not even begun to tap the potential of the LEAP and Zeiss Headtracker !

Here is our test setup ...  should you wish to try this out for yourself...

1.Xbox 360+ Cronus+Bullseye + Win 8 running Bullseye+Dangermouse + Leap Motion for finger tracking +Zeiss Cinemizer OLED headtracker + Kinect connected to the console

2.PS3+Cronus+Windows 8 i7 Lenovo X220T running Bullyseye+Dangermouse+ Leap Motion +Zeiss Cinemizer OLED headtracker +/- any other input we can connect to the PS3 or X220T! Note you could connect the Kinect to the laptop and add full body tracking to your PS3 games using this setup...

Theory and Method:

They say you get lag and latency issues on the fastest of hardware , so I set out to use up to 10 year old hardware .. the PS3 and Xbox 360 to show otherwise...note even with direct connection of the Zeiss headtracker to the PS3 there is no discernible lag in any game you play.

The Cronus lets you use mice and other input methods on either console , the flaw is it has to be connected to a laptop or PC, allowing adjustment and user defined control setups through Bullseye software ..

But that flaw is just what we need to connect our VR leaning controllers and to take the load off the consoles looking to eliminate lag...the consoles do not have a clue you are using finger tracking to play God of War Ascension  or Halo 4 ...but hey that's consoles for you ...

Now since the Zeiss headtracker emulates mouse control this gives us headtracking in any game we choose on the consoles as I have shown before with PS3 ,PS2 and PS1 games  and  with Xbox 360 games  ...and yes it does the dof axes too but we wont be using them here

 True VR is going to require more than just headtracking ...we want to use our fingers thumbs and palms ... this is where LEAP Motion comes in ..erhh handy...

But how do we get it working with the Xbox 360 and PS3 ...we need the Bullseye to see it as a Windows controller ...say something simple ...like a mouse for instance...

DangerMouse is a java app that runs from a command line and gives you mouse like control using LEAP Motion only using your fingers in air ...

I had the idea of running this in parallel to the Bullseye software ..and it worked like magic ... the Bullseye and Cronus see the Zeiss headtracker and the LEAP Motion as Windows controllers ..mice in this case ..which through Bullseye we can easily map to any function of the Xbox 360 or PS3 controllers..

As the Cronus turns any controller you connect to your PC into a PS3 or Xbox 360 controller ..we can have multple VR inputs .

We can have BOTH headtracking and finger tracking control in ANY game ..

But that is not enough for VR we also want voice recognition and this we achieve through Kinect attached to the Xbox 360 .. Now the poor old ageing console can barely manage full body recognition but we don't care about that for we have the power of LEAP , what the Kinect can handle well though is speech recog ...

And thus we are ready to test not one but 3 forms of VR control , that will be essential for VR to take off in the consumer arena on aged hardware along with legacy control too..

If this seems a long way around, it is , deliberately so ... I want to introduce points of latency into the control loop to see if a) we can use 3 VR control methods together b) if there is any discernible latency or lag c) Asymmetry my dear Watson is the future ...using the cloud or a higher powered device(our laptop )  to do the grunt work for a lower powered or closed device(our PS3 and Xbox 360) that could not handle it ...and most importantly d) So I can stop asking William Gibson are we there yet?

I could also have substituted GIMX for the Cronus and used a wireless solution ( but that is for yet another day)

So developers, take some time off using your slide rules , grab your favourite game and gaming device , take some time to actually test out all the multiple forms of VR leaning control . Actually play something and then go back to the lab... 

If the development of VR continues along a path where all talk is dominated by problems, problems, problems we may just miss the simple and obvious solutions staring at us ...

I have not even begun to tap the potential of this way of playing games, more to follow !

And good luck with the launch Palmer and Co !

Thanks to Zeiss for the headtracker ,LEAP Motion, DangerMouse , the developers of Cronus 

More to follow .. 

 Purchase  in the UK/EU the Zeiss Cinemizer OLED 3D iPack including iPhone adapter and in the USA the Cinemizer OLED , 

this content is © and originated from thegameveda http://the-games-veda.blogspot.com/  if you are reading it from any other site that has not provided a link back it has been leached