2014 HMD Display Design Theory & Practice : Taking Mobile Head Mounted Display design and use to 5K res and beyond using Asymmetric Cloud Scaling

a 5120 by 1600 res unbroken cloud streamed transparent HMD image of the Pan Am/Met Life building and the original split desktop monitor image behind it you can view anywhere ...without any physical connection to the source PC

Don't worry the actual experience is far more impressive than the poor quality photo my camera can take .... as, are when you start to think about them, the implications for HMD design and use .. 

Last time I was looking at the influence of next gen consoles on HMD development, for my next sermon I am going to be tackling

The impact of Cloud Scaling on  Head Mounted Display design and use 

Taking VR AR and MR To 5K and beyond...and taking it with you 

For HMDs to go mass market .. two HMD barriers need to be broken  ... first the ability to access ultra high res content and second for  the user to be able to access and take that ultra hi res  HMD  experience anywhere .. without dragging the old ball and chain ...their Rig or Laptop with them.. 

This is the year  4K, 8K and 12K resolution content will arrive, for all forms of media ...

Laptops, PCs and Mac Pros are are all pushing 4K content and environments  . We are hitting  near 3K for phones this March with the Samsung S5 for instance .. very soon 4K is going to be considered low res .. with 8K and 12K ideals for video editing and gaming. By the end of this year 4K TVs and content will be replacing 1080p res TVs and content in a big way .. 

However current HMDs max out at native resolutions of 1080p, true 1080p and scaled 1080p.. 

In the future sometime in 2015 or in the future .. Abrash and OVR hope to see 1K res per eye .. a semi 1080p res will not cut it in a world where 4K and above content are seen as a natural and fixed point progression in display and content resolution evolution..

All those young Michael Bay Film students and Carmack Coding Cadets aren't going to be churning out semi 1080p res content on their 12K Mac Pro 2014 Trash Can Editions .. they will want to push 4K content as a differentiating selling point for their masters projects... 

There exists then a disconnect between  evolving common media resolution and HMD native resolution.. the HMD is simply not keeping up with desktop and media resolutions ..

How then to display that 4K and above res content on said VR HMD ? Or indeed any HMD

We can build a micro OLED true native res 4K per eye dual display HMD for several hundred thousand dollars.. we could .. but we won't ..because we don't need to ..and I haven't got my kickstarter for that setup yet..

We could go for sticking two not one phone displays together .. the latest FOV2GO trend .. that's a lot of weight and although a magnitude of cost cheaper than micro OLEDs.. still something you are going to have to wait ( hence the recent  2015 predictions) for the cost to drop ..or enough panels to be left over from the phone industry ... but we won't... because I don't want to steal Valve's thunder ... just yet ..come March though I will be happily engaged building a 3K per eye FOV2GO like ... it's not like HMD design is rocket science ...after all?

We don't need to any of this to display ultra hi res content on a HMD .. First we have to get round a problem:

Even when my PC can push beyond 4K content, connecting a HMD to it will just set the display res to the current 1080p max of the HMD ....or  semi 720p, semi 1080p.. or 1K per eye 2015 Rift...

Solution:

We are going to use Windows 8 cloud scaling to display  5120 by 1600 res  content from my desktop /rig without any physical connection to it .. on the highest native resolution HMD available at present ..the smd ST1080..

I use Splashtop to stream my desktop to my Windows 8 tablet all the time, so I can edit video on my Rig whilst I blog on my tablet ..  all from one screen ...with 2 desktops running on it. One cloud, one local ..

With the arrival of Windows 8.1 with improved options for having Windows 8 "panes" of multiple sizes ... I realised not only could I run two instances of Windows 8 on one tablet screen, but also run them at independent resolutions .. on the same screen

Still with me? My desktop runs at 5120 by 1600 across two monitors  and looks like so :

Using the cloud I can "ghost"  this 5120 1600 resolution in a  1366 by 768 panel on a 12.5 inch  Windows 8 tablet :

Other than the immediate aesthetic benefit of removing the the screen bezel this gives me access to the power of my Rig on a portable pen and touch enabled 12.5 inch tablet ... which for instance lets me edit video at 5120 by 1600 ..in LightWorks with pen and touch .. 

When working this way ..to  my great surprise the screen does not turn to garbage when streamed and a high res desktop can be streamed via the cloud to the low res tablet display with text and detail surprisingly still legible .. and the ability to zoom and pan around the desktop thanks to Splashtop .. let's us control a Windows 8 surface of scaled 5120 by 1600 resolution with pen,touch,trackpad , LEAP ..anything.. 

There are devices with 4K res around the corner ..but cloud scaling gives us  a high powered PC behind that high resolution..so your tablet,phone or PC comes with the power of your desktop..or rig

In use it looks like this : 

So how is this relevant to HMD Use?

If I connect a HMD to my tablet PC (or Android/iOS device)  streaming a 5120 by 1600 desktop .. that scaled 5120 by 1600 desktop will be available for me on that HMD .. elementary .. no ? We just got around that annoying limit of the native resolution of your HMD .. just like we got around the native low resolution of the Windows 8 tablet ..as you can see in the above video and pictures...

By exploiting the second  word in my made up HMD AR VR MR phrase of the day " Cloud Scaling" 

Here is a Google Earth instance at 5120 by 1600 on my desktop .. with the usual screen split  and bezel issue .. 

Now here is the same instance beamed to my smd ST1080  looking through it's 10% transparent LCOS display 

I cannot photograph or film the display as I see it ... a 100 inch detailed and clear image .. but you should be able to see .. the bezel and split are gone, we are viewing ultra high res content without  physical connection between the HMD and the desktop PC .. and we can take the 5120 by 1600 desktop anywhere  with our far more mobile and touch enabled Windows 8 tablet .. 

Of course I can do this with my far more portable Android phone, iOS phone or tablets also.. 

In this picture we are looking through the left eye piece of the ST1080 through to one of the 30 inch monitors behind .. you can  wear the ST1080 as an AR display ..with full 1080p res whilst working on your desktop behind and through it as I attempted a very long time ago here .. 

The 5120 by 1600 display despite being scaled via Splashtop and Windows 8 maintains fidelity ..  

The 100 inch display of the ST1080 providing enough magnification so we can work 

So here is the first test : using the smd ST1080 HMD .. as this is the highest native res head mounted display I have at this point.. and as it is transparent I should be able to show you the desktop rig monitors behind..

This is relevant to HMD design..how?

We can scale ultra high res source content to a lower native res HMD ...to overcome native lower HMD resolution limitations . Not just 720p or 1080p... but beyond and without being physically connected to the source..

An on paper low native res HMD can scale a far higher res image if you  DESIGN for scaling  ...in your HMD 

In fact some HMD designers already have used scaling of higher resolutions and combined it to great effect .. 

We already know this from the way the Cinemizer OLED HMD scales 1080p with a display that remains crystal clear ... On paper thinking of HMDs as people used to back in 2012 ..( and some still do )  it should not be more impressive than the HMZT series .. but it is .. at  scaling a 1080p resolution, exploiting pixel density of it's micro OLED display.. to create a better experience, free of nausea,blur and screen door ..  better and more functional than many higher resolution or field of view  HMDs.. that sit idle in your room right now.. unable even to leave your room with you..but I digress..

We should be able to exploit a degree of ultra high resolution  scaling in all HMDs .. semi 1080p Wide Field of View, true 720p and 1080p as well as scaled 1080p HMDs ..  and combine this with Cloud to connect our HMDs to any hi res source .. 

Why not just wait for true 4K or 8K panels?

Well if you think 720p or 1080p Micro OLED panels are expensive ... a pair of 4K ones will cost more than, even more than Chris Roberts Rig ..budget ..for ...10 or so generations of upgrades...

Like Chris's Rig of Legend .. the price of micro OLEDs and ultra high res phone OLEDs  will decrease with time .. but not within the time frame Abrash and OVR are now talking for the arrival of VR or should I say maturation of VR .. 

Of course I also cannot wait for Korean, Chinese, Japanese display  technology and German optics to catch up to OVRs ideas or my expectations .. a man has said... VR is coming..  for $300 back in 2012 

HMD resolution is likely to keep behind desktop content for a significant amount of time at least until HMDs become mass market, from that point on the Korean, Japanese and Chinese display developers will switch on to making ultra high res micro displays ...if the HMD becomes mass market...

And so to help attain the critical mass level of retail (as opposed to promotional ) success necessary to drive down display panel costs  HMD developers and users can use cloud scaling to keep the scaled if not true resolution of the HMD as a device in line with much of the content most people will own..

The ST1080s true native 1080p res and pixel density are adequate to scale even 5120 by 1600 content and this has many applications , far beyond gaming .. which is where we need to take the use of HMDs to get enough user feedback to make better HMDs for gaming ...it's all in the feedback  ...  

I can edit video at 5120 by 1600 away from my Rig, see the open Sony Vegas Pro editor window AND a hi res output window on the same display without the need for 30Kg of monitors or having to be in front of them at all ...

Your field of view here is being generated not by screen size .. but by numbers of pixels .. in the scaled image .. this is the difference between true fields of view ...like the one your seeing now and the abstract magnified low res so called "VR" views generated by magni-ski-goggle- single/dual phone panel VR .. the true 1080p of the ST1080 allows this .. your scaled image retaining enough detail for it to be of use to you .. 

Now through cloud scaling you can not only edit 4K video in real time thanks to the power of your 12K Mac Pro ...you can leave it at work or home  .. and carry on editing it off site , on the train home or the plane to where ever you need to go .. using a HMD with a larger display res than many multi monitor setups .. weighing 180g 

Abrash et al like to talk of presence , true presence for the common public is only achieved when the image they view IS at the resolution your eyes are seeing right now ..indistinguishable from real life,  this is going to require an astonishing level of rendering power .. something all of Hollywood and PIXAR ,,,and their finest  Chinese and Indian CGI experts cannot produce. We will get there ..but we will get there sooner if we exploit techniques like cloud scaling..

Limitations 

The quality of the experience is limited by the power of your cloud and the Kung Fu of your HMD ..and of course your source 

Our "cloud" is generated by 

Splashtop Streamer on Windows 8.1 from my old Skull Trail PC with a 5970 Black Edition

Splashtop Windows 8.1 App on the streamed to device : Windows 8.1 Lenovo X220T i7 GPU 

Modem Speed the X220T I used has Wifi N dual band 

Wifi speed I use 120 meg Broadband locked to 5ghz, 2.4ghz is not reliable for streaming or cloud work and can be taken out or interrupted by any Bluetooth stream or concurrent data download (which is what makes PSNow so darn impressive for me ..that it does this at 2.4ghz) 

Performance Limitations : 

We are limited by the power of our cloud as much as our hardware .. Here in 2014 there needs to be better optimisation of hardware for cloud streaming, not to mention software .. 

For gaming the streaming limit at the moment will depend on your source hardware ..most streaming services are locked around 720p from the old GAIKAI PC demos to Onlive and PSNow (which I suspect will be less ) . Splashtop and Windows 8.1 do let you stream at higher resolutions but you will see lag with ultra hi res in games. 

But as people keep telling me PC hardware is getting faster all the time..

This was however a test primarily to look at whether we can display ultra high resolutions scaled and streamed to lower res HMDs ..without being connected physically to the hardware..  which we can and...  I do work in 2D this way .. 

With better source GPUS and a 2015 PC  or Mac Pro we should be able to stream easily the 1K per eye content Abrash talks of happening in 2015 not only that we should be able to push extreme resolutions to that 1K per eye display ( I don't have a video as I don't have a 2015 Rift yet.. just like I don't have my Star Citizen combat module ..yet.

Very soon at the rate tablet and phone technology is developing we may be more used to accessing 4K and ultra hi res content without PCs ever being involved.. The first true 4K image being viewed on a HMD may not only come from the cloud but from iOS or Android ..devices..streaming the image.

If you put two phone displays (read phones)  into your FOV2GO prototype ( giving you say 1080 by 1920 per eye )  ..like the one Valve look to have done recently you should be able to test this out now .. In fact you can test it on half of such a device (your phone) now with Splashtop

Personally I am waiting to drop a pair of S5s into mine .. 3K per eye .. and it will still cost less than a Sony HMZT 3 .. wait I am giving the game away .. 

And our "scaled to" display by the siliconmicrodisplay ST1080 Head Mounted Display :

I chose the ST1080 because it has the highest true res atm, trying this with the current dev kit of the Rift at 5120 by 1600 ( because I actually WORK and play at that res) did not yield great results , quite apart from the fact you cannot warp a 2D desktop at present on the Rift :

The best results are arrived at if you can match the scaling of your desktop to the scaling of the the display : but there is nothing to stop you Rift warping a cloud scaled ultra high resolution ... to say an Oculus Rift Dev kit II (of?) when they let us loose on them .. 

This ones for you Mr Abrash : Attaining 1K res per eye 

And on the Rift (until 8K phone displays arrive for <$100(joke Riftmaniacs..joke) we may want to cloud scale less than 4K ..  through cloud scaling the strange (and mysterious) to me screen ratio of the Rift could yet have full 1080p content or Abrash's ideal  1K per eye right now ... and this is what I intend to test next...

2 or 3 iterations in , by 2015 I would expect the Rift to have a 3K panel at least, or 2K .. giving the desired 1K per eye .. but we may not even need to await that long .. or use such an expensive display .. we can cloud scale . or just locally scale 1K content to a <1K native resolution per eye display on the Rift dev kit 1 now

The Cloud and HMDs are two things that are made for each other ...two things I am obsessed with introducing to each other 

It is thus possible to stream and scale ultra high res desktop displays to low res  tablet and HMD displays ..

As long as the pixel density, the physical design of the display used is tight enough you should always be able to stream 

And I get to combine two of my  favourite  things  the cloud and HMD  use... 

There are many applications of this technique ... for both designers of HMDs and users .. get to it and your HMD Kung Fu will be strong...    your ideas like mine will catch up  to the  technology  and you wont be left waiting for the technology  to catch up to your ideas

Head Mounted Display solutions are evolving in many different directions ...true 4K per eye prototypes existed back in 2012 from siliconmicrodisplay ( nobody told you ? naughty endgadget/techcrunch/reddit)..so it is more than likely there will be 4K per eye native res HMDs in a retail space by 2015 .. and those are going to have serious applications ..that make it worth the serious amounts of money they will cost users..  And you know  I will be attempting to stream 12K content to my 4K smd ST4K HMD .. as soon as I get it .. 

As the power of our  PC, Mac,iOs and Android clouds increases, along with the speed and reliability of Wifi and both hardware and code is optimised for streaming ...ultra high resolution cloud scaling and streaming will become an important part of the design of any HMD, common place and an excellent way to get at those high resolutions that will be the norm for media of all kinds ...allowing HMD designers and users to keep pace whilst we wait for Google to drop those 8K per eye Graphene contact lenses ... of course by then I will expect Abrash, Carmack, Luckey et al to have announced the announcement of  the future invention of Virtual Light ... or at least have an E3 2016 prototype for it .or they can come check out the presence in mine, forget VR .. the future is BR .. Brain Reality .. and the catchphrase??  ..  " See you in your Brain" ..well this a subject that makes for great Science Fiction..as anyone whose read Virtual Light By William Gibson... can attest.. meeting and discussing VR with him was a far more enlightening experience back in the 90s than all the VR sages I have met since .. 


Epilogue and thanks

1. thanks to siliconmicrodisplay for delivering a 1080p AR HMD back in 2012 and for the equivalent of what in today's money I am going to pay for my Glyph this Wednesday 
2. thanks to Microsoft and the Windows 8 boys and girls  
3. Splashtop ... simply stunning .. as always 
4. The latest Abrash VR Prophecy as illuminated by the glowing pixel (free) delivery of Endgadget.. cutting and pasting of Abrash's  speech slides was simply magnificent .. as was the drawing of the reader to the  prediction that VR ..will arrive .. perhaps ..in 2015 .. or in ... the future .....   when the tech catches up ... in the FUTCHA! Wow 
5. And lastly  thanks to Michael Bay for making the best Miami Vice film ever..role on Bad Boys III .. perhaps you could do it in VR...then again...


Next time on thegameveda why I am backing Retinal Projection and saving up for MR ...as well as a long hard look at the latest Rift progeny.. 

To join the asymmetric cloud gaming revolution you will need Splashtop Streamer on your PC , the Splashtop THD app for Android or Splashtop 2 for iOS . 

Check out  Splashtop Remote Desktop for iPad - Splashtop Inc. , in the App Store :

this content is © and originated from thegameveda http://the-games-veda.blogspot.com/  if you are reading it from any other site that has not provided a link back it has been leached