HUD – Design & Research 21

by Marton Szabo-Kass - December 12, 2021December 12, 2021

Further automotive AR examples

| continuing my last blog entry on examples

In the past week I was searching for further examples of AR implementation in car’s user interafaces. Currently, after three weeks of research in this topic I have the impression that the industry is mainly focusing on visual augmentation as a help for the driver. Here are some further examples that offer some new aspects and features.

GMC Sierra HD trailer camera

The 2020 GMC Sierra HD pickup truck featured a novel implementation of AR technology. The truck was designed to tow heavy duty trailers and has in sum 15 cameras to help the driver navigate the really large truck. One camera can be mounted at the back end of the trailer looking at the road behind. This image can then be displayed added to the built-in rear camera view of the truck, letting the trailer almost disappear. [3]

I personally find the solution to be a nice gimmick but would actually question its practical benefit. The view for sure doesn’t help manoeuvring with the attached trailer.

Land Rover Clearsight ground view

Looking at special utility solutions, LandRover also implemented a camera augmentation in the Evoque and Defender models – on the main screen. The system works with cameras on the side mirrors and on the front grille and help the driver to see a 180° ground view in front of the car and between the front wheels, below the normal vision field. As LandRover targets off-road enthusiasts, this feature could be welcomed for showing any dangerous obstacles on rough terrains, or simply when climbing steep hills. A similar system is also implemented in the higher-class Bentley Bentaiga SUV. [7]

Besides this “transparent bonnet” system, the company Jaguar-LandRover was also conducting research also on a “transparent pillar” solution. That should help drivers in a more urban environment to see their surroundings in 360°, including objects hidden by the roof’s pillars with the help of cameras and AR. The research was done in 2014 and I couldn’t find any further outcome of the idea. Additionally they have also shown a unique way of AR navigation help: a ghost car projected in front of the driver that has to be followed along the route. [8] [9]

Jaguar transparent pillar and ghost car concept [8]

Panasonic’s state-of-the-art AR HUD

Panasonic Automotive is an other supplier (like Continental, etc.) who is developing onboard systems for automotive OEMs, like an AR Head Up Display with high-end features. Their product was shown on the latest CES 2021 exhibition and is claimed to get implemented in a series production car of an unknown brand in 2024. The system stands out from the other existing HUDs in following features:

AI software for 3D navigation graphics, supporting smooth responses to sudden changes ahead of the car. It also uses information from all the onboard ADAS systems (e.g. a 180° forward facing radar with 90 m range) and generates updates in less then 0,3 seconds. (the spatial-AI, AR navigation platform is a patent by Phiar)
Eye tracking technology to ensure that the driver always sees the projected information on the right place at any head movement.
Vibration control: image stabilization for bumpy roads.
Advanced optics, 4K resolution with laser and holography technology (by Envisics).

[5] [6]

To cover all relevant sensory fields in a car interior interface, next week I want to focus on the topic of haptics and tactile feedback solutions on the market.

Sources

[1] YouTube video from Roadshow: Car Tech 101: The best ways AR is being installed in cars | Cooley On Cars. Retrieved on 11.12.2021.
https://www.youtube.com/watch?v=PHhvCRexjWQ

[2] YouTube video from Roadshow: 2020 GMC Sierra HD: Heavy-duty hauler debuts “Transparent Trailer” tech. Retrieved on 11.12.2021.
https://www.youtube.com/watch?v=U0gZ9HaCWsA

[3] Online article by Road And Track: The 2020 GMC Sierra HD Can Make Your Trailer “Invisible”. Retrieved on 12.12.2021.
https://www.roadandtrack.com/new-cars/future-cars/a26009122/2020-gmc-sierra-heavy-duty-invisible-trailer/

[4] YouTube video by About Cars: Panasonic’s Innovative Augmented-Reality HUD Could Be in Cars by 2024. Retrieved on 12.12.2021.
https://www.youtube.com/watch?v=cLgMnSTSxog

[5] Online article by Panasonic: Panasonic Automotive Brings Expansive, Artificial Intelligence-Enhanced Situational Awareness to the Driver Experience with Augmented Reality Head-Up Display. Retrieved on 12.12.2021.
https://na.panasonic.com/us/news/panasonic-automotive-brings-expansive-artificial-intelligence-enhanced-situational-awareness-driver

[6] Online article by Auganix.org: Panasonic collaborates with Phiar to bring real-world AI-driven Augmented Reality navigation to its automotive solutions. Retrieved on 12.12.2021.
https://www.auganix.org/panasonic-collaborates-with-phiar-to-bring-real-world-ai-driven-augmented-reality-navigation-to-its-automotive-solutions/

[7] Online article by Car Magazine: Does it work? Land Rover’s ClearSight handy X-ray vision tech. Retrieved on 12.12.2021.
https://www.carmagazine.co.uk/car-news/tech/land-rover-clearsight/

[8] Online article by Autocar: Jaguar Land Rover previews transparent pillar technology. Retrieved on 12.12.2021.
https://www.autocar.co.uk/car-news/new-cars/jaguar-land-rover-previews-transparent-pillar-technology

[9] Online article by Jaguar: Jaguar Land Rover Develops Transparent Pillar And ‘Follow-Me’ Ghost Car Navigation Research. Retrieved on 12.12.2021.
https://media.jaguarlandrover.com/news/2014/12/jaguar-land-rover-develops-transparent-pillar-and-follow-me-ghost-car-navigation

by Marton Szabo-Kass - December 5, 2021February 1, 2022

Automotive AR examples

| looking at some state of the art examples of in-car AR systems on the market

MBUX – the newest infotainment system of Mercedes-Benz

2018 was the year when Mercedes Benz introduced their newest infotainment system called MBUX. This uses the front camera (originally used for parking) to create a live stream of the road ahead, combined with graphics of navigation hints or finding adresses. Since then it was continuosly improved and the latest version was revealed in 2021 in the S/EQS-Class models, featuring an AR navigation display and a HUD with distance assist, lane keeping assist and dinamic arrows showing directions.

Video demostrations of the 2021 MBUX system:
https://www.youtube.com/watch?v=hnRbi5UcJnw
https://www.youtube.com/watch?v=DCgy3askMcM

The MBUX AR navigation introduced in 2018

Audi AR HUD

Audi announced their augmented reality HUDs as an optional feature for the newest high-end electric SUV, the Q4 e-tron for 2021. The visual information shown in front of the driver are similar to the MBUX’s content. Audi explicitly defines two areas: the status field (in a visual distance of ca. 3 m) and the AR field (in a visual distance of ca. 10 m), which seems to be bigger than in the German competitor’s solution.

Demo video of the Q4 e-tron HUD:
https://www.youtube.com/watch?v=Ea6o-_smVk8

Hyundai and WayRay

Looking at HUDs further, Hyundai/Genesis was the first brand to implement laser-holographic AR head-up displays in their G80 model, presented by the young AR developer company WayRay in 2019. It is said to have tremendous benefits compared to past HUDs (using reflected LCD screens) in terms of precision and visibility for the driver.

The Swiss startup WayRay claims to be the only company to have implemented holography to HUDs. Their holographic optical elements (HOEs) in their displays should provide unprecedented 3D images while remaining transparent and capable of being bent to windshields. The company presents its uniqueness in the field by covering “deep-tech” holography hardware development (e.g. blue-laser beams) as well as software development, all realised in-house.

They have already received large fundings by Hyundai and Porsche, have presented a 180° AR cockpit experience and offer different add-on solutions for vehicles, boats and airplanes. Their newest project is a shared car concept (“Holograktor”), designed for the “Metaverse” with a complete gaming / working / learning possibility while traveling autonomously. In their cooperation with Pininfarina on a concept car, they proposed solutions of the “True AR” displays also for side windows, providing new ways of passenger infotainment and entertainment experiences.

A report from the FIA Formula E on WayRay’s developments also predicts the use of HUD systems for race cars in the future. The pilots behind the wheels could get visualized ideal racing lines, brake points or a ghost car to chase on the race track.

Hyundai’s In-Car Noise Cancelling

Besides HUDs, Hyundai is pushing the development of AR solutions in cars in other aspects as well. Like in our headphones, noise cancelling also found its way into the car interiors, bringing more comfort to the passengers. According to Hyundai, the earlier systems were only capable of masking steady engine noises, but their newest solution (“Road Noise Active Noise Control”) in the upcoming Genesis GV80 will be capable of deleting different tire noises at different speeds. It uses multiple microphones placed directly into the wheel wells, accelerometers, amplifiers and a digital signal processor. As a result of the complex calculations for each individual wheel, the in-car noise should be reduced by half (3 dB).

Engine sound enhancements

Writing about noises of the car, we also have to take a short look at the opposite effects to noise cancelling – the engine sound enhancement devices. Due to the downsizing of the engine displacements, the roars coming from the combustion got also reduced. To keep the emotions connected to sporty engine sounds though, manufacturers are using additional devices to create compensating sound effects.

These can be pipes from the intake manifold connected to the dashboard walls, in some cases with an extra flap to control the sound throughput only for the sporty driving situations (Toyota, Ford, Porsche).

BMW was known to use engine sound amplification through a synthesised reproduction of the actual engine noise played simply on the car’s speakers.

The Volkswagen Group made it a bit more complicated by adding a special speaker device (“Soundaktor”) below the windshield to produce deep, buzzing tones resembling larger engine sounds. In some models there are also speakers built into the exhaust pipes to alter the natural noises coming from the engine, to make them more emotional or masculin.

3D ADAS system of Arkamys

Beeping noises in a car are existent since many years, with the intention to help drivers. But beeping on itself is not always enough to give an understandable signal about what is happening or dangerous around the driver. The company Arkamys presented an intuitive alerting concept for Advanced Driver Assistance Systems – parking, lane keeping, blind spot and other assistants – by placing many different speakers inside the cabin and generating a 3D sound experience. With this it is possible to signalize the direction where a possible danger can exist, making the recognition and processing of the information easier and more intuitive for the driver.

Electric cars

Electric cars are further good examples where in-car noise generators are used to give the driver and passengers the known feeling of vehicle driving dynamics. Porsche is a perfect example where specific sounds are developed for representing the brand’s identity within the driver experience. They call the system “Porsche Electric Sport Sound” that enhances some natural noises of the drivetrain but also reduces disturbing ones, while implementing sounds to compy with the legal regulations for electric vehicle alerting sounds.

Thinking further about sound augmentation in cars, probably the already most spread system is the parking assistant, giving beeping sound feedback on the remaining distance to obstacles around the car. The design of these systems could probably fill a chapter on its own, but as it is already an everday tool, I won’t go further into detail on it.

The above listed examples are not even close to a complete list of use cases. Therefor I want to further research the current technologies. The next step will then be to look into the reasons for these systems, why they were developed and what practical needs, feelings and experiences are the underlying causes.

Sources

Online article on Wired: With In-Car AR, Drivers Get a New View of the Road Ahead. Retrieved on 05.12.2021
https://www.wired.com/story/in-car-ar-drivers-get-new-view-road-ahead/

Article on Wired: Hyundai’s Luxury SUV Mixes Mics and Math for a Silent Ride. Retrieved on 05.12.2021
https://www.wired.com/story/hyundai-genesis-gv80-suv-noise-cancelling/

Online Article on FIA Formula E: How AR and VR are revolutionising the car industry. Retrieved on 05.12.2021
https://www.fiaformulae.com/en/news/2020/june/ar-vr

WayRay – offical website. Retrieved on 05.12.2021
https://wayray.com/#who-we-are
https://wayray.com/press-area/#media_coverage

Online article on WayRay by CNET and Autocar. Retrieved on 05.12.2021
https://www.hyundai.news/eu/articles/press-releases/hyundai-wayray-unveil-next-generation-visual-technology-at-ces-2019.html

YouTube video by Roadshow: CES 2019: WayRay’s holographic AR windshield is real, hitting the road soon. Retrieved on 05.12.2021
https://www.youtube.com/watch?v=HFIgjQI2E6Y

AutoCar article on the Pininfarina concept car. Retrieved on 05.12.2021
https://www.autocar.co.uk/car-news/new-cars/pininfarina-concept-car-showcased-holographic-ar-display

Online article by AutoZeitung: Mercedes entwickelt MBUX weiter. Retreived on 05.12.2021
https://www.autozeitung.de/mercedes-infotainment-192628.html

Mercedes-Benz MBUX System – online articles and images, retrieved on 05.12.2021
https://www.wired.com/story/in-car-ar-drivers-get-new-view-road-ahead/
https://www.extremetech.com/extreme/314758-2021-mercedes-s-class-2-hud-sizes-level-3-autonomy-4d-sound-5-lcds
https://carbuzz.com/news/new-mercedes-s-class-shows-off-amazing-augmented-reality-display

Audi AR HUD system: online article and Youtube video on Slashgear. Retreived on 05.12.2021
https://www.slashgear.com/the-audi-q4-e-trons-augmented-reality-head-up-display-is-dashboard-genius-09662735/
https://www.audi-technology-portal.de/de/elektrik-elektronik/fahrerassistenzsysteme/audi-q4-e-tron-ar-hud-de/

Online article on GeekDad: Augmented Reality for Your Ears. Retrieved on 01.02.2022
https://geekdad.com/2016/02/arkamys/

Image of Mercedes G350d soundmodule. Retrieved on 01.02.2022
https://www.tuningblog.eu/kategorien/tuning-wiki/soundgenerator-nachruesten-232502/

CarThrottle article on sound enhancers. Retrieved on 05.12.2021
https://www.carthrottle.com/post/5-ways-that-manufacturers-enhance-the-sound-of-their-cars/

The Porsche Sound – online article, retrieved on 05.12.2021
https://newsroom.porsche.com/de/produkte/taycan/sound-18542.html

by Marton Szabo-Kass - November 28, 2021November 29, 2021

AR basics and automotive trends

| a short and basic definition on Augmented Reality, the first implementations in vehicles and current innovation trends

What exactly is Augmented Reality and when was it first used?

To have a clear distinction between related expressions, Paul Milgram’s Reality-Virtuality Continuum from 1994 shows the relation of Augmented, Mixed and Virtual reality in a very comprehensible way. [3] As shown in the illustration below, AR is the evolution of real environments in the direction of complete virtuality, but still having a majority of real content. Augmented Virtuality on the hand would describe systems using more virtual than real models.

Illustration by P. Milgram and H. Colquhoun Jr., in A Taxonomy of Real and Virtual
World Display Integration [4]

To have an official definition, in The Concise Fintech Compendium AR is described as “an enhanced version of the physical, real-world reality of which elements are superimposed by computer generated or extracted real-world sensory input such as sound, video, graphics or haptics.” [1]

Already in 1997 R. T. Azuma stated three essential characteristics of AR systems [2]:

combining reality with a virtual world
interacting in real-time
registering in 3D space

Azuma also described the two basic possibilities of combining virtual inputs with the real world: virtual objects can be added to the real perception or real objects can be hidden by overlaying virtual effects. This may be possible not only for optical perception, but also for sound and haptics. He described systems with speakers and microphones, altering the incoming sound of our surroundings (like today’s noise-cancelling), or gloves with additional haptic feedback of simulated forces. [2] Basically AR could help us to enhance all of our senses, but it is mostly implemented in visual systems. [6]

After reading basics theories on Augmented Reality from the early 1990’s, one wouldn’t think that the first personal AR system was developed in 1968 at the Harvard University by Ivan Sutherland, the “father of computer graphics” – a HMD (Head-Mounted-Display) system. [8]

Regarding vehicles and and the first implementation of AR, we have to go even further back in time. The predecessor of today’s BAE System plc., Elliot Flight Automation along with Cintel claim the development of the first Head-Up-Display (HUD) in operational service in 1961 – for a military aircraft of the British Royal Navy, the Blackburn Buccaneer. [9]

The first HUD in a passenger car is stated to be used in the Oldsmobile Cutlass Supreme Indy 500 pace car made by General Motors in 1988. [10] Following photo depicts this very simple AR solution on the windscreen.

The HUD in the Oldsmobile Cutlass Supreme Indy500 pace car, from 1988.
Source: https://www.autoevolution.com/news/how-to-add-a-head-up-display-to-your-car-136497.html

In the last decades, AR was further decveloped and implemented in many different areas, and with the evolution of displays, projectors and computer graphics, we can have now our own AR applications on our smartphones or passenger cars. While starting to dig deeper into existing automotive AR solutions, I found the following interesting study as a foundation to enclose my topic of interest.

AR innovations in the automotive industry today

A study carried out by the Austrian “innovation intelligence company” StartUs GmbH analysed over 400 startups and created an overview on the most innovative use cases of AR in the automotive industry [7]:

They state that the the total augmented reality automotive market is growing by 177% every year and will reach $5.5 billion by 2022. [7]

From their five areas of innovation my main focus will be on “Experience Enhancement”. The use cases are see-through displays, windshield projectors or various wearables, that can help the driver with additional, immediate information on important events of the surroundings without any distraction. [7]

Existing solutions for this area will follow in my further research.

Sources

[0] Wikipedia – Summaries on Augmented Reality
https://en.wikipedia.org/wiki/Augmented_reality
https://de.wikipedia.org/wiki/Erweiterte_Realität
https://en.wikipedia.org/wiki/Mixed_reality

[1] Schueffel, P.: The Concise Fintech Compendium. Fribourg: School of Management Fribourg/Switzerland, 2017
https://web.archive.org/web/20171024205446/
http://www.heg-fr.ch/EN/School-of-Management/Communication-and-Events/events/Pages/EventViewer.aspx?Event=patrick-schuffel.aspx

[2] Azuma, R. T.: A Survey of Augmented Reality. In: Presence: Teleoperators and Virtual Environments. 6, Nr. 4, 1997, S. 355–385

[4] Milgram, P., Colquhoun Jr., H.: A Taxonomy of Real and Virtual World Display Integration. In: Mixed reality: Merging real and virtual worlds, Springer, 1999, p. 1-26

[5] The basics of Augmented Reality – Interview with an AR expert; Indestry.com; Retrieved on 27.11.2021
https://www.indestry.com/blog/the-basics-of-augmented-reality-interview-with-an-ar-expert

[6] Kipper, G., Rampolla J.: Augmented Reality: An Emerging Technologies Guide to AR; Elsevier; 2013

[7] Online article: How Augmented Reality Disrupts The Automotive Industry; by StartUs Insights Research Blog. Retrieved on 28.11.2021
https://www.startus-insights.com/innovators-guide/how-augmented-reality-disrupts-the-automotive-industry/

[8] Online article by Javornik, A: The Mainstreaming of Augmented Reality: A Brief History; Harvard Business Review; 2016. Retrieved on 28.11.2021
https://hbr.org/2016/10/the-mainstreaming-of-augmented-reality-a-brief-history

[9] Online article by BAE Systems: The evolution of the Head-Up Display. Retrieved on 28.11.2021
https://www.baesystems.com/en/feature/our-innovations-hud

[10] Wikipedia summary on automotive Head-Up Displays:
https://en.wikipedia.org/wiki/Automotive_head-up_display