ISMAR 2014 - Sep 10-12 - Munich, Germany
- Upcoming Events
- Landing Page
- Welcome
- Advance Program
- Awards
- Conference Registration
- Conference Hotels
- Social Program
- Student Volunteers
- Co-Located Events
ISMAR Sessions
- Conference Keynotes
- Papers
- Posters
- Workshops
- Tutorials
- Panels
- Demonstrations
- Volkswagen Tracking Challenge
- Doctoral Consortium
- Industry Day
- BOF Sessions
- Art Exhibition
Call for Participation
- Tracks
- S&T
- MASH'D
- Categories
- Papers & Posters
- Workshop Proposals
- Tutorial Proposals
- Demonstrations
- AR Games Project
- VW Tracking Challenge
- Doctoral Consortium
- Birds-of-a-feather
- Guidelines
- S&T Submission Guidelines
- MASH'D Submission Guidelines
- Reviewing Guidelines
Committees
Supporters and Media
Contact
Social Media
LinkedInPrevious Years
2014 ISMAR All Papers
Rendering
Session :
Location : HS1
Chair : Wolfgang Broll, TU Ilmenau
Papers :
Rendering
Date & Time : September 10 04:15 pm - 06:30 pmLocation : HS1
Chair : Wolfgang Broll, TU Ilmenau
Papers :
Interactive Near-Field Illumination for Photorealistic Augmented Reality on Mobile Devices
Authors: Kai Rohmer, Wolfgang Büschel, Raimund Dachselt, Thorsten Grosch
Abstract :
Mobile devices become more and more important today, especially for augmented reality (AR) applications in which the camera of the mobile device acts like a window into the mixed reality world. Up to now, no photorealistic augmentation is possible since the computational power of the mobile devices is still too weak. Even a streaming solution from a stationary PC would cause a latency, that affects user interactions considerably. Therefore, we introduce a differential illumination method that allows for a consistent illumination of the inserted virtual objects on mobile devices, avoiding a delay. The necessary computation effort is shared between a stationary PC and the mobile devices to make use of the capacities available on both sides. The method is designed such that only a minimum amount of data has to be transferred asynchronously between the stationary PC and one or multiple mobile devices. This allows for an interactive illumination of virtual objects with a consistent appearance under both temporally and spatially varying real illumination conditions. To describe the complex near-field illumination in an indoor scenario, multiple HDR video cameras are used to capture the illumination from multiple directions. In this way, sources of illumination can be considered that are not directly visible to the mobile device because of occlusions and the limited field of view of built-in cameras.
Mobile devices become more and more important today, especially for augmented reality (AR) applications in which the camera of the mobile device acts like a window into the mixed reality world. Up to now, no photorealistic augmentation is possible since the computational power of the mobile devices is still too weak. Even a streaming solution from a stationary PC would cause a latency, that affects user interactions considerably. Therefore, we introduce a differential illumination method that allows for a consistent illumination of the inserted virtual objects on mobile devices, avoiding a delay. The necessary computation effort is shared between a stationary PC and the mobile devices to make use of the capacities available on both sides. The method is designed such that only a minimum amount of data has to be transferred asynchronously between the stationary PC and one or multiple mobile devices. This allows for an interactive illumination of virtual objects with a consistent appearance under both temporally and spatially varying real illumination conditions. To describe the complex near-field illumination in an indoor scenario, multiple HDR video cameras are used to capture the illumination from multiple directions. In this way, sources of illumination can be considered that are not directly visible to the mobile device because of occlusions and the limited field of view of built-in cameras.
Delta Voxel Cone Tracing
Author: Tobias Alexander Franke
Abstract :
Mixed reality applications which must provide visual coherence between synthetic and real objects need relighting solutions for both: synthetic objects have to match lighting conditions of their real counterparts, while real surfaces need to account for the change in illumination introduced by the presence of an additional synthetic object. In this paper we present a novel relighting solution called Delta Voxel Cone Tracing to compute both direct shadows and first bounce mutual indirect illumination. We introduce a voxelized, pre-filtered representation of the combined real and synthetic surfaces together with the extracted illumination difference due to the augmentation. In a final gathering step this representation is cone-traced and superimposed onto both types of surfaces, adding additional light from indirect bounces and synthetic shadows from antiradiance present in the volume. The algorithm computes results at interactive rates, is temporally coherent and to our knowledge provides the first real-time rasterizer solution for mutual diffuse, glossy and perfect specular indirect reflections between synthetic and real surfaces in mixed reality.
Mixed reality applications which must provide visual coherence between synthetic and real objects need relighting solutions for both: synthetic objects have to match lighting conditions of their real counterparts, while real surfaces need to account for the change in illumination introduced by the presence of an additional synthetic object. In this paper we present a novel relighting solution called Delta Voxel Cone Tracing to compute both direct shadows and first bounce mutual indirect illumination. We introduce a voxelized, pre-filtered representation of the combined real and synthetic surfaces together with the extracted illumination difference due to the augmentation. In a final gathering step this representation is cone-traced and superimposed onto both types of surfaces, adding additional light from indirect bounces and synthetic shadows from antiradiance present in the volume. The algorithm computes results at interactive rates, is temporally coherent and to our knowledge provides the first real-time rasterizer solution for mutual diffuse, glossy and perfect specular indirect reflections between synthetic and real surfaces in mixed reality.
Importance Weighted Image Enhancement for Prosthetic Vision: An Augmentation Framework
Authors: Chris McCarthy, Nick Barnes
Abstract :
Augmentations to enhance perception in prosthetic vision (also known as bionic eyes) have the potential to improve functional outcomes significantly for implantees. In current (and near-term) implantable electrode arrays resolution and dynamic range are highly constrained in comparison to images from modern cameras that can be head mounted. In this paper, we propose a novel, generally applicable adaptive contrast augmentation framework for prosthetic vision that addresses the specific perceptual needs of low resolution and low dynamic range displays. The scheme accepts an externally defined pixel-wise weighting of importance describing features of the image to enhance in the output dynamic range. Our approach explicitly incorporates the logarithmic scaling of enhancement required in human visual perception to ensure perceivability of all contrast augmentations. It requires no pre-existing contrast, and thus extends previous work in local contrast enhancement to a formulation for general image augmentation. We demonstrate the generality of our augmentation scheme for scene structure and looming object enhancement using simulated prosthetic vision.
Augmentations to enhance perception in prosthetic vision (also known as bionic eyes) have the potential to improve functional outcomes significantly for implantees. In current (and near-term) implantable electrode arrays resolution and dynamic range are highly constrained in comparison to images from modern cameras that can be head mounted. In this paper, we propose a novel, generally applicable adaptive contrast augmentation framework for prosthetic vision that addresses the specific perceptual needs of low resolution and low dynamic range displays. The scheme accepts an externally defined pixel-wise weighting of importance describing features of the image to enhance in the output dynamic range. Our approach explicitly incorporates the logarithmic scaling of enhancement required in human visual perception to ensure perceivability of all contrast augmentations. It requires no pre-existing contrast, and thus extends previous work in local contrast enhancement to a formulation for general image augmentation. We demonstrate the generality of our augmentation scheme for scene structure and looming object enhancement using simulated prosthetic vision.
P-HRTF: Efficient Personalized HRTF Computation for High-Fidelity Spatial Sound
Authors: Alok Meshram, Ravish Mehra, Hongsheng Yang, Enrique Dunn, Jan-michael Frahm, Dinesh Manocha
Abstract :
Accurate rendering of 3D spatial audio for interactive virtual auditory displays requires the use of personalized head-related transfer functions (HRTFs). We present a new approach to compute personalized HRTFs for any individual using a method that combines state-of-the-art image-based 3D modeling with an efficient numerical simulation pipeline. Our 3D modeling framework enables capture of the listener's head and torso using consumer-grade digital cameras to estimate a high-resolution non-parametric surface representation of the head, including the extended vicinity of the listener's ear. We leverage sparse structure from motion and dense surface reconstruction techniques to generate a 3D mesh. This mesh is used as input to a numeric sound propagation solver, which uses acoustic reciprocity and Kirchhoff surface integral representation to efficiently compute an individual's personalized HRTF. The overall computation takes tens of minutes on multi-core desktop machine. We have used our approach to compute the personalized HRTFs of few individuals, and we present our preliminary evaluation here. To the best of our knowledge, this is the first commodity technique that can be used to compute personalized HRTFs in a lab or home setting.
Accurate rendering of 3D spatial audio for interactive virtual auditory displays requires the use of personalized head-related transfer functions (HRTFs). We present a new approach to compute personalized HRTFs for any individual using a method that combines state-of-the-art image-based 3D modeling with an efficient numerical simulation pipeline. Our 3D modeling framework enables capture of the listener's head and torso using consumer-grade digital cameras to estimate a high-resolution non-parametric surface representation of the head, including the extended vicinity of the listener's ear. We leverage sparse structure from motion and dense surface reconstruction techniques to generate a 3D mesh. This mesh is used as input to a numeric sound propagation solver, which uses acoustic reciprocity and Kirchhoff surface integral representation to efficiently compute an individual's personalized HRTF. The overall computation takes tens of minutes on multi-core desktop machine. We have used our approach to compute the personalized HRTFs of few individuals, and we present our preliminary evaluation here. To the best of our knowledge, this is the first commodity technique that can be used to compute personalized HRTFs in a lab or home setting.
Visibility-Based Blending for Real-Time Applications
Authors: Taiki Fukiage, Takeshi Oishi, Katsushi Ikeuchi
Abstract :
There are many situations in which virtual objects are presented half-transparently on a background in real time applications. In such cases, we often want to show the object with constant visibility. However, using the conventional alpha blending, visibility of a blended object substantially varies depending on colors, textures, and structures of the background scene. To overcome this problem, we present a framework for blending images based on a subjective metric of visibility. In our method, a blending parameter is locally and adaptively optimized so that visibility of each location achieves the targeted level. To predict visibility of an object blended by an arbitrary parameter, we utilize one of the error visibility metrics that have been developed for image quality assessment. In this study, we demonstrated that the metric we used can linearly predict visibility of a blended pattern on various texture images, and showed that the proposed blending methods can work in practical situations assuming augmented reality.
There are many situations in which virtual objects are presented half-transparently on a background in real time applications. In such cases, we often want to show the object with constant visibility. However, using the conventional alpha blending, visibility of a blended object substantially varies depending on colors, textures, and structures of the background scene. To overcome this problem, we present a framework for blending images based on a subjective metric of visibility. In our method, a blending parameter is locally and adaptively optimized so that visibility of each location achieves the targeted level. To predict visibility of an object blended by an arbitrary parameter, we utilize one of the error visibility metrics that have been developed for image quality assessment. In this study, we demonstrated that the metric we used can linearly predict visibility of a blended pattern on various texture images, and showed that the proposed blending methods can work in practical situations assuming augmented reality.
Applications
Session :
Location : HS1
Chair : Dieter Schmalstieg, TU Graz
Papers :
Applications
Date & Time : September 10 02:15 pm - 03:45 pmLocation : HS1
Chair : Dieter Schmalstieg, TU Graz
Papers :
AR-IVI – Implementation of In-Vehicle Augmented Reality
Authors: Qing Rao, Tobias Tropper, Christian Grünler, Markus Hammori, Samarjit Chakraborty
Abstract :
In the last three years, a number of automotive Augmented Reality (AR) concepts and demonstrators have been presented, all looking for an interpretation of what AR in a car may look like. In October 2013, Mercedes-Benz exhibited to a public audience the AR In-Vehicle Infotainment (AR-IVI) system aimed at defining an overall in-vehicle electric/electronic (E/E) architecture for augmented reality rather than showing specific use cases. In this paper, we explain the requirements and design decisions that lead to the system-design, and we share the challenges and experiences in developing the AR-IVI system in the prototype vehicle. Based on our experiences, we give an outlook on future software and E/E architectural challenges of in-vehicle augmented reality.
In the last three years, a number of automotive Augmented Reality (AR) concepts and demonstrators have been presented, all looking for an interpretation of what AR in a car may look like. In October 2013, Mercedes-Benz exhibited to a public audience the AR In-Vehicle Infotainment (AR-IVI) system aimed at defining an overall in-vehicle electric/electronic (E/E) architecture for augmented reality rather than showing specific use cases. In this paper, we explain the requirements and design decisions that lead to the system-design, and we share the challenges and experiences in developing the AR-IVI system in the prototype vehicle. Based on our experiences, we give an outlook on future software and E/E architectural challenges of in-vehicle augmented reality.
Thermal Touch: Thermography-Enabled Everywhere Touch Interfaces for Mobile Augmented Reality Applications
Author: Daniel Kurz
Abstract :
We present an approach that makes any real object a true touch interface for mobile Augmented Reality applications. Using infrared thermography, we detect residual heat resulting from a warm fingertip touching the colder surface of an object. This approach can clearly distinguish if a surface has actually been touched, or if a finger only approached it without any physical contact, and hence significantly less heat transfer. Once a touch has been detected in the thermal image, we determine the corresponding 3D position on the touched object based on visual object tracking using a visible light camera. Finally the 3D position of the touch is used by human machine interfaces for Augmented Reality providing natural means to interact with real and virtual objects. The emergence of wearable computers and head-mounted displays desires for alternatives to a touch screen, which is the primary user interface in handheld Augmented Reality applications. Voice control and touchpads provide a useful alternative to interact with wearables for certain tasks, but particularly common interaction tasks in Augmented Reality require to accurately select or define 3D points on real surfaces. We propose to enable this kind of interaction by simply touching the respective surface with a fingertip. Based on tests with a variety of different materials and different users, we show that our method enables intuitive interaction for mobile Augmented Reality with most common objects.
We present an approach that makes any real object a true touch interface for mobile Augmented Reality applications. Using infrared thermography, we detect residual heat resulting from a warm fingertip touching the colder surface of an object. This approach can clearly distinguish if a surface has actually been touched, or if a finger only approached it without any physical contact, and hence significantly less heat transfer. Once a touch has been detected in the thermal image, we determine the corresponding 3D position on the touched object based on visual object tracking using a visible light camera. Finally the 3D position of the touch is used by human machine interfaces for Augmented Reality providing natural means to interact with real and virtual objects. The emergence of wearable computers and head-mounted displays desires for alternatives to a touch screen, which is the primary user interface in handheld Augmented Reality applications. Voice control and touchpads provide a useful alternative to interact with wearables for certain tasks, but particularly common interaction tasks in Augmented Reality require to accurately select or define 3D points on real surfaces. We propose to enable this kind of interaction by simply touching the respective surface with a fingertip. Based on tests with a variety of different materials and different users, we show that our method enables intuitive interaction for mobile Augmented Reality with most common objects.
AR-Mentor: Augmented Reality Based Mentoring System
Authors: Zhiwei Zhu, Vlad Branzoi, Michael Wolverton, Louise Yarnall, Girish Acharya, Supun Samarasekera, Rakesh Kumar, Glen Murray , Nicholas Vitovitch
Abstract :
AR-Mentor is a wearable real time Augmented Reality (AR) mentoring system that is configured to assist in maintenance and repair tasks of complex machinery, such as vehicles, appliances, and industrial machinery. The system combines a wearable Optical-See-Through (OST) display device with high precision 6-Degree-Of-Freedom (DOF) pose tracking and a virtual personal assistant (VPA) with natural language, verbal conversational interaction, providing guidance to the user in the form of visual, audio and locational cues. The system is designed to be heads-up and hands-free allowing the user to freely move about the maintenance or training environment and receive globally aligned and context aware visual and audio instructions (animations, symbolic icons, text, multimedia content, speech). The user can interact with the system, ask questions and get clarifications and specific guidance for the task at hand. A pilot application with AR-Mentor was successfully developed to instruct a novice to perform an advanced 33-step maintenance task on a training vehicle. The initial live training tests demonstrate that AR-Mentor is able to help and serve as an assistant to an instructor, freeing him or her to cover more students and to focus on higher-order teaching.
AR-Mentor is a wearable real time Augmented Reality (AR) mentoring system that is configured to assist in maintenance and repair tasks of complex machinery, such as vehicles, appliances, and industrial machinery. The system combines a wearable Optical-See-Through (OST) display device with high precision 6-Degree-Of-Freedom (DOF) pose tracking and a virtual personal assistant (VPA) with natural language, verbal conversational interaction, providing guidance to the user in the form of visual, audio and locational cues. The system is designed to be heads-up and hands-free allowing the user to freely move about the maintenance or training environment and receive globally aligned and context aware visual and audio instructions (animations, symbolic icons, text, multimedia content, speech). The user can interact with the system, ask questions and get clarifications and specific guidance for the task at hand. A pilot application with AR-Mentor was successfully developed to instruct a novice to perform an advanced 33-step maintenance task on a training vehicle. The initial live training tests demonstrate that AR-Mentor is able to help and serve as an assistant to an instructor, freeing him or her to cover more students and to focus on higher-order teaching.
Towards Augmented Reality User Interfaces in 3D Media Production
Authors: Max Krichenbauer, Goshiro Yamamoto, Takafumi Taketomi, Christian Sandor, Hirokazu Kato
Abstract :
The idea of using Augmented Reality (AR) user interfaces (UIs) to create 3D media content, such as 3D models for movies and games has been repeatedly suggested over the last decade. Even though the concept is intuitively compelling and recent technological advances have made such an application increasingly feasible, very little progress has been made towards an actual real-world application of AR in professional media production. To this day, no immersive 3D UI has been commonly used by professionals for 3D computer graphics (CG) content creation. In this paper, we are first to publish a requirements analysis for our target application in the professional domain. Based on a survey that we conducted with media professionals, the analysis of professional 3D CG software, and professional training tutorials, we identify these requirements and put them into the context of AR UIs. From these findings, we derive several interaction design principles that aim to address the challenges of real-world application of AR to the production pipeline. We implemented these in our own prototype system while receiving feedback from media professionals. The insights gained in the survey, requirements analysis, and user interface design are relevant for research and development aimed at creating production methods for 3D media production.
The idea of using Augmented Reality (AR) user interfaces (UIs) to create 3D media content, such as 3D models for movies and games has been repeatedly suggested over the last decade. Even though the concept is intuitively compelling and recent technological advances have made such an application increasingly feasible, very little progress has been made towards an actual real-world application of AR in professional media production. To this day, no immersive 3D UI has been commonly used by professionals for 3D computer graphics (CG) content creation. In this paper, we are first to publish a requirements analysis for our target application in the professional domain. Based on a survey that we conducted with media professionals, the analysis of professional 3D CG software, and professional training tutorials, we identify these requirements and put them into the context of AR UIs. From these findings, we derive several interaction design principles that aim to address the challenges of real-world application of AR to the production pipeline. We implemented these in our own prototype system while receiving feedback from media professionals. The insights gained in the survey, requirements analysis, and user interface design are relevant for research and development aimed at creating production methods for 3D media production.
Reconstruction and Fusion
Session :
Location : HS1
Chair : Walterio Mayol-Cuevas, Bristol University
Papers :
Reconstruction and Fusion
Date & Time : September 11 10:00 am - 12:45 pmLocation : HS1
Chair : Walterio Mayol-Cuevas, Bristol University
Papers :
Improved Registration for Vehicular AR using Auto-Harmonization
Authors: Eric Foxlin, Thomas Calloway, Hongsheng Zhang
Abstract :
This paper describes the design, development and testing of an AR system that was developed for aerospace and ground vehicles to meet stringent accuracy and robustness requirements. The system uses an optical see-through HMD, and thus requires extremely low latency, high tracking accuracy and precision alignment and calibration of all subsystems in order to avoid mis-registration and “swim”. The paper focuses on the optical/inertial hybrid tracking system and describes novel solutions to the challenges with the optics, algorithms, synchronization, and alignment with the vehicle and HMD systems. A system accuracy analysis is presented with simulation results to predict the registration accuracy. Finally, a car test is used to create a through-the-eyepiece video demonstrating well-registered augmentations of the road and nearby structures while driving.
This paper describes the design, development and testing of an AR system that was developed for aerospace and ground vehicles to meet stringent accuracy and robustness requirements. The system uses an optical see-through HMD, and thus requires extremely low latency, high tracking accuracy and precision alignment and calibration of all subsystems in order to avoid mis-registration and “swim”. The paper focuses on the optical/inertial hybrid tracking system and describes novel solutions to the challenges with the optics, algorithms, synchronization, and alignment with the vehicle and HMD systems. A system accuracy analysis is presented with simulation results to predict the registration accuracy. Finally, a car test is used to create a through-the-eyepiece video demonstrating well-registered augmentations of the road and nearby structures while driving.
Real-Time Illumination Estimation from Faces for Coherent Rendering
Authors: Sebastian B. Knorr, Daniel Kurz
Abstract :
We present a method for estimating the real-world lighting conditions within a scene in real-time. The estimation is based on the visual appearance of a human face in the real scene captured in a single image of a monocular camera. In hardware setups featuring a user-facing camera, an image of the user's face can be acquired at any time. The limited range in variations between different human faces makes it possible to analyze their appearance offline, and to apply the results to new faces. Our approach uses radiance transfer functions - learned offline from a dataset of images of faces under different known illuminations - for particular points on the human face. Based on these functions, we recover the most plausible real-world lighting conditions for measured reflections in a face, represented by a function depending on incident light angle using Spherical Harmonics. The pose of the camera relative to the face is determined by means of optical tracking, and virtual 3D content is rendered and overlaid onto the real scene with a fixed spatial relationship to the face. By applying the estimated lighting conditions to the rendering of the virtual content, the augmented scene is shaded coherently with regard to the real and virtual parts of the scene. We show with different examples under a variety of lighting conditions, that our approach provides plausible results, which considerably enhance the visual realism in real-time Augmented Reality applications.
We present a method for estimating the real-world lighting conditions within a scene in real-time. The estimation is based on the visual appearance of a human face in the real scene captured in a single image of a monocular camera. In hardware setups featuring a user-facing camera, an image of the user's face can be acquired at any time. The limited range in variations between different human faces makes it possible to analyze their appearance offline, and to apply the results to new faces. Our approach uses radiance transfer functions - learned offline from a dataset of images of faces under different known illuminations - for particular points on the human face. Based on these functions, we recover the most plausible real-world lighting conditions for measured reflections in a face, represented by a function depending on incident light angle using Spherical Harmonics. The pose of the camera relative to the face is determined by means of optical tracking, and virtual 3D content is rendered and overlaid onto the real scene with a fixed spatial relationship to the face. By applying the estimated lighting conditions to the rendering of the virtual content, the augmented scene is shaded coherently with regard to the real and virtual parts of the scene. We show with different examples under a variety of lighting conditions, that our approach provides plausible results, which considerably enhance the visual realism in real-time Augmented Reality applications.
Comprehensive Workspace Calibration for Visuo-Haptic Augmented Reality
Authors: Ulrich Eck, Frieder Pankratz, Christian Sandor, Gudrun Klinker, Hamid Laga
Abstract :
Visuo-haptic augmented reality systems enable users to see and touch digital information that is embedded in the real world. Precise co-location of computer graphics and the haptic stylus is necessary to provide a realistic user experience. PHANToM haptic devices are often used in such systems to provide haptic feedback. They consist of two interlinked joints, whose angles define the position of the haptic stylus and three sensors at the gimbal to sense its orientation. Previous work has focused on calibration procedures that align the haptic workspace within a global reference coordinate system and developing algorithms that compensate the non-linear position error, caused by inaccuracies in the joint angle sensors. In this paper, we present an improved workspace calibration that additionally compensates for errors in the gimbal sensors. This enables us to also align the orientation of the haptic stylus with high precision. To reduce the required time for calibration and to increase the sampling coverage, we utilize time-delay estimation to temporally align external sensor readings. This enables users to continuously move the haptic stylus during the calibration process, as opposed to commonly used point and hold processes. We conducted an evaluation of the calibration procedure for visuo-haptic augmented reality setups with two different PHANToMs and two different optical trackers. Our results show a significant improvement of orientation alignment for both setups over the previous state of the art calibration procedure. Improved position and orientation accuracy results in higher fidelity visual and haptic augmentations, which is crucial for fine-motor tasks in areas including medical training simulators, assembly planning tools, or rapid prototyping applications. A user friendly calibration procedure is essential for real-world applications of VHAR.
Visuo-haptic augmented reality systems enable users to see and touch digital information that is embedded in the real world. Precise co-location of computer graphics and the haptic stylus is necessary to provide a realistic user experience. PHANToM haptic devices are often used in such systems to provide haptic feedback. They consist of two interlinked joints, whose angles define the position of the haptic stylus and three sensors at the gimbal to sense its orientation. Previous work has focused on calibration procedures that align the haptic workspace within a global reference coordinate system and developing algorithms that compensate the non-linear position error, caused by inaccuracies in the joint angle sensors. In this paper, we present an improved workspace calibration that additionally compensates for errors in the gimbal sensors. This enables us to also align the orientation of the haptic stylus with high precision. To reduce the required time for calibration and to increase the sampling coverage, we utilize time-delay estimation to temporally align external sensor readings. This enables users to continuously move the haptic stylus during the calibration process, as opposed to commonly used point and hold processes. We conducted an evaluation of the calibration procedure for visuo-haptic augmented reality setups with two different PHANToMs and two different optical trackers. Our results show a significant improvement of orientation alignment for both setups over the previous state of the art calibration procedure. Improved position and orientation accuracy results in higher fidelity visual and haptic augmentations, which is crucial for fine-motor tasks in areas including medical training simulators, assembly planning tools, or rapid prototyping applications. A user friendly calibration procedure is essential for real-world applications of VHAR.
Recognition and reconstruction of transparent objects for Augmented Reality
Authors: Alan Francisco Torres-Gomez, Walterio Mayol-Cuevas
Abstract :
Dealing with real transparent objects for AR is challenging due to their lack of texture and visual features as well as the drastic changes in appearance as the background, illumination and camera pose change. The few existing methods for glass object detection usually require a carefully controlled environment, specialized illumination hardware or ignore information from different viewpoints. In his work, we explore the use of a learning approach for classifying transparent objects from multiple images with the aim of both discovering such objects and building a 3D reconstruction to support convincing augmentations. We extract, classify and group small image patches using a fast graph-based segmentation and employ a probabilistic formulation for aggregating spatially consistent glass regions. We demonstrate our approach via analysis of the performance of glass region detection and example 3D reconstructions that allow virtual objects to interact with them.
Dealing with real transparent objects for AR is challenging due to their lack of texture and visual features as well as the drastic changes in appearance as the background, illumination and camera pose change. The few existing methods for glass object detection usually require a carefully controlled environment, specialized illumination hardware or ignore information from different viewpoints. In his work, we explore the use of a learning approach for classifying transparent objects from multiple images with the aim of both discovering such objects and building a 3D reconstruction to support convincing augmentations. We extract, classify and group small image patches using a fast graph-based segmentation and employ a probabilistic formulation for aggregating spatially consistent glass regions. We demonstrate our approach via analysis of the performance of glass region detection and example 3D reconstructions that allow virtual objects to interact with them.
Tracking
Session :
Location : HS1
Chair : Georg Klein, Microsoft Corporation
Papers :
Tracking
Date & Time : September 11 04:00 pm - 06:00 pmLocation : HS1
Chair : Georg Klein, Microsoft Corporation
Papers :
Pixel-Wise Closed-Loop Registration in Video-Based Augmented Reality
Authors: Feng Zheng, Dieter Schmalstieg, Gregory Welch
Abstract :
In Augmented Reality (AR), visible misregistration can be caused by many inherent error sources, such as errors in tracking, calibration, and modeling. In this paper we present a novel pixel-wise closed-loop registration framework that can automatically detect and correct registration errors using a reference model comprised of the real scene model and the desired virtual augmentations. Registration errors are corrected in both global world space via camera pose refinement, and local screen space via pixel-wise corrections, resulting in spatially accurate and visually coherent registration. Specifically we present a registration-enforcing model-based tracking approach that weights important image regions while refining the camera pose estimates (from any conventional tracking method) to achieve better registration, even in the case of modeling errors. To deal with remaining errors, which can be rigid or non-rigid, we compute the optical flow between the camera image and the real model image rendered with the refined pose, enabling direct screen-space pixel-wise corrections to misregistration. The estimated flow field can be applied to improve registration in two distinct ways: (1) forward warping of modeled on-real-object-surface augmentations (e.g., object re-texturing) into the camera image, leading to surface details that are not present in the virtual object; and (2) backward warping of the camera image into the real scene model, preserving the full use of the dense geometry buffer (depth in particular) provided by the combined real-virtual model for registration, leading to pixel accurate real-virtual occlusion. We discuss the trade-offs between, and different use cases of, forward and backward warping with model-based tracking in terms of specific properties for registration. We demonstrate the efficacy of our approach with both simulated and real data.
In Augmented Reality (AR), visible misregistration can be caused by many inherent error sources, such as errors in tracking, calibration, and modeling. In this paper we present a novel pixel-wise closed-loop registration framework that can automatically detect and correct registration errors using a reference model comprised of the real scene model and the desired virtual augmentations. Registration errors are corrected in both global world space via camera pose refinement, and local screen space via pixel-wise corrections, resulting in spatially accurate and visually coherent registration. Specifically we present a registration-enforcing model-based tracking approach that weights important image regions while refining the camera pose estimates (from any conventional tracking method) to achieve better registration, even in the case of modeling errors. To deal with remaining errors, which can be rigid or non-rigid, we compute the optical flow between the camera image and the real model image rendered with the refined pose, enabling direct screen-space pixel-wise corrections to misregistration. The estimated flow field can be applied to improve registration in two distinct ways: (1) forward warping of modeled on-real-object-surface augmentations (e.g., object re-texturing) into the camera image, leading to surface details that are not present in the virtual object; and (2) backward warping of the camera image into the real scene model, preserving the full use of the dense geometry buffer (depth in particular) provided by the combined real-virtual model for registration, leading to pixel accurate real-virtual occlusion. We discuss the trade-offs between, and different use cases of, forward and backward warping with model-based tracking in terms of specific properties for registration. We demonstrate the efficacy of our approach with both simulated and real data.
Semi-Dense Visual Odometry for AR on a Smartphone
Authors: Thomas Schöps, Jakob Engel, Daniel Cremers
Abstract :
We present a direct monocular visual odometry system which runs in real-time on a smartphone. Being a direct method, it tracks and maps on the images themselves instead of extracted features such as keypoints. New images are tracked using direct image alignment, while geometry is represented in the form of a semi-dense depth map. Depth is estimated by filtering over many small-baseline, pixel-wise stereo comparisons. This leads to significantly less outliers and allows to map and use all image regions with sufficient gradient, including edges. We show how a simple world model for AR applications can be derived from semi-dense depth maps, and demonstrate the practical applicability in the context of an AR application in which simulated objects can collide with real geometry.
We present a direct monocular visual odometry system which runs in real-time on a smartphone. Being a direct method, it tracks and maps on the images themselves instead of extracted features such as keypoints. New images are tracked using direct image alignment, while geometry is represented in the form of a semi-dense depth map. Depth is estimated by filtering over many small-baseline, pixel-wise stereo comparisons. This leads to significantly less outliers and allows to map and use all image regions with sufficient gradient, including edges. We show how a simple world model for AR applications can be derived from semi-dense depth maps, and demonstrate the practical applicability in the context of an AR application in which simulated objects can collide with real geometry.
Sticky Projections - A New Approach to Interactive Shader Lamp Tracking
Authors: Christoph Resch, Peter Keitler, Gudrun Klinker
Abstract :
Shader lamps can augment physical objects with projected virtual replications using a camera-projector system, provided that the physical and virtual object are well registered. Precise registration and tracking has been a cumbersome and intrusive process in the past. In this paper, we present a new method for tracking arbitrarily shaped physical objects interactively. In contrast to previous approaches our system is mobile and makes solely use of the projection of the virtual replication to track the physical object and "stick" the projection to it. Our method consists of two stages, a fast pose initialization based on structured light patterns and a non-intrusive frame-by-frame tracking based on features detected in the projection. In the initialization phase a dense point cloud of the physical object is reconstructed and precisely matched to the virtual model to perfectly overlay the projection. During the tracking phase, a radiometrically corrected virtual camera view based on the current pose prediction is rendered and compared to the captured image. Matched features are triangulated providing a sparse set of surface points that is robustly aligned to the virtual model. The alignment transformation serves as an input for the new pose prediction. Quantitative experiments show that our approach can robustly track complex objects at interactive rates.
Shader lamps can augment physical objects with projected virtual replications using a camera-projector system, provided that the physical and virtual object are well registered. Precise registration and tracking has been a cumbersome and intrusive process in the past. In this paper, we present a new method for tracking arbitrarily shaped physical objects interactively. In contrast to previous approaches our system is mobile and makes solely use of the projection of the virtual replication to track the physical object and "stick" the projection to it. Our method consists of two stages, a fast pose initialization based on structured light patterns and a non-intrusive frame-by-frame tracking based on features detected in the projection. In the initialization phase a dense point cloud of the physical object is reconstructed and precisely matched to the virtual model to perfectly overlay the projection. During the tracking phase, a radiometrically corrected virtual camera view based on the current pose prediction is rendered and compared to the captured image. Matched features are triangulated providing a sparse set of surface points that is robustly aligned to the virtual model. The alignment transformation serves as an input for the new pose prediction. Quantitative experiments show that our approach can robustly track complex objects at interactive rates.
Dense Planar SLAM
Authors: Renato Salas-Moreno, Ben Glocker, Paul Kelly, Andrew Davison
Abstract :
Using higher-level entities during mapping has the potential to improve camera localisation performance and give substantial perception capabilities to real-time 3D SLAM systems. We present an efficient new real-time approach which densely maps an environment using bounded planes and surfels extracted from depth images (like those produced by RGB-D sensors or dense multi-view stereo reconstruction). Our method offers the every-pixel descriptive power of the latest dense SLAM approaches, but takes advantage directly of the planarity of many parts of real-world scenes via a data-driven process to directly regularize planar regions and represent their accurate extent efficiently using an occupancy approach with on-line compression. Large areas can be mapped efficiently and with useful semantic planar structure which enables intuitive and useful AR applications such as using any wall or other planar surface in a scene to display a user's content.
Using higher-level entities during mapping has the potential to improve camera localisation performance and give substantial perception capabilities to real-time 3D SLAM systems. We present an efficient new real-time approach which densely maps an environment using bounded planes and surfels extracted from depth images (like those produced by RGB-D sensors or dense multi-view stereo reconstruction). Our method offers the every-pixel descriptive power of the latest dense SLAM approaches, but takes advantage directly of the planarity of many parts of real-world scenes via a data-driven process to directly regularize planar regions and represent their accurate extent efficiently using an occupancy approach with on-line compression. Large areas can be mapped efficiently and with useful semantic planar structure which enables intuitive and useful AR applications such as using any wall or other planar surface in a scene to display a user's content.
Real-time Deformation, Registration and Tracking of Solids Based on Physical Simulation
Authors: Ibai Leizea, Hugo Álvarez, Iker Aguinaga, Diego Borro
Abstract :
This paper proposes a novel approach to registering deformations of 3D non-rigid objects for Augmented Reality applications. Our prototype is able to handle different types of objects in real-time regardless of their geometry and appearance (with and without texture) with the support of an RGB-D camera. During an automatic offline stage, the model is processed in order to extract the data that serves as input for a physics-based simulation. Using its output, the deformations of the model are estimated by considering the simulated behaviour as a constraint. Furthermore, our framework incorporates a tracking method based on templates in order to detect the object in the scene and continuously update the camera pose without any user intervention. Therefore, it is a complete solution that extends from tracking to deformation formulation for either textured or untextured objects regardless of their geometrical shape. Our proposal focuses on providing a correct visual with a low computational cost. Experiments with real and synthetic data demonstrate the visual accuracy and the performance of our approach.
This paper proposes a novel approach to registering deformations of 3D non-rigid objects for Augmented Reality applications. Our prototype is able to handle different types of objects in real-time regardless of their geometry and appearance (with and without texture) with the support of an RGB-D camera. During an automatic offline stage, the model is processed in order to extract the data that serves as input for a physics-based simulation. Using its output, the deformations of the model are estimated by considering the simulated behaviour as a constraint. Furthermore, our framework incorporates a tracking method based on templates in order to detect the object in the scene and continuously update the camera pose without any user intervention. Therefore, it is a complete solution that extends from tracking to deformation formulation for either textured or untextured objects regardless of their geometrical shape. Our proposal focuses on providing a correct visual with a low computational cost. Experiments with real and synthetic data demonstrate the visual accuracy and the performance of our approach.
User Interfaces
Session :
Location : HS1
Chair : Steven Feiner, Columbia University
Papers :
User Interfaces
Date & Time : September 11 02:00 pm - 03:45 pmLocation : HS1
Chair : Steven Feiner, Columbia University
Papers :
Grasp-Shell vs Gesture-Speech: A comparison of direct and indirect natural interaction techniques in Augmented Reality
Authors: Thammathip Piumsomboon, David Altimira, Hyungon Kim, Adrian Clark, Gun Lee, Mark Billinghurst
Abstract :
In order for natural interaction in Augmented Reality (AR) to become widely adopted, the techniques used need to be shown to support precise interaction, and the gestures used proven to be easy to understand and perform . Recent research has explored free-hand gesture interaction with AR interfaces, but there have been few formal evaluations conducted with such systems. In this paper we introduce and evaluate two natural interaction techniques: the free-hand gesture based Grasp-Shell, which provides direct physical manipulation of virtual content; and the multi-modal Gesture-Speech, which combines speech and gesture for indirect natural interaction. These techniques support object selection, 6 degree of freedom movement, uniform scaling, as well as physics-based interaction such as pushing and flinging. We conducted a study evaluating and comparing Grasp-Shell and Gesture-Speech for fundamental manipulation tasks. The results show that Grasp-Shell outperforms Gesture-Speech in both efficiency and user preference for translation and rotation tasks, while Gesture-Speech is better for uniform scaling. They could be good complementary interaction methods in a physics-enabled AR environment, as this combination potentially provides both control and interactivity in one interface. We conclude by discussing implications and future directions of this research.
In order for natural interaction in Augmented Reality (AR) to become widely adopted, the techniques used need to be shown to support precise interaction, and the gestures used proven to be easy to understand and perform . Recent research has explored free-hand gesture interaction with AR interfaces, but there have been few formal evaluations conducted with such systems. In this paper we introduce and evaluate two natural interaction techniques: the free-hand gesture based Grasp-Shell, which provides direct physical manipulation of virtual content; and the multi-modal Gesture-Speech, which combines speech and gesture for indirect natural interaction. These techniques support object selection, 6 degree of freedom movement, uniform scaling, as well as physics-based interaction such as pushing and flinging. We conducted a study evaluating and comparing Grasp-Shell and Gesture-Speech for fundamental manipulation tasks. The results show that Grasp-Shell outperforms Gesture-Speech in both efficiency and user preference for translation and rotation tasks, while Gesture-Speech is better for uniform scaling. They could be good complementary interaction methods in a physics-enabled AR environment, as this combination potentially provides both control and interactivity in one interface. We conclude by discussing implications and future directions of this research.
Improving Co-presence with Augmented Visual Communication Cues for Sharing Experience through Video Conference
Authors: Seungwon Kim, Gun Lee, Nobuchika SAKATA, Mark Billinghurst
Abstract :
Video conferencing is becoming more widely used in areas other than face-to-face conversation, such as sharing real world experience with remote friends or family. In this paper we explore how adding augmented visual communication cues can improve the experience of sharing remote task space and collaborating together. We developed a prototype system that allows users to share live video view of their task space taken on a Head Mounted Display (HMD) or Handheld Display (HHD), and communicate through not only voice but also using augmented pointer or annotations drawn on the shared view. To explore the effect of having such an interface for remote collaboration, we conducted a user study comparing three video-conferencing conditions with different combination of communication cues: (1) voice only, (2) voice + pointer, and (3) voice + annotation. The participants used our remote collaboration system to share a parallel experience of puzzle solving in the user study, and we found that adding augmented visual cues significantly improved the sense of being together. The pointer was the most preferred additional cue by users for parallel experience, and there were different states of the users’ behavior found in remote collaboration.
Video conferencing is becoming more widely used in areas other than face-to-face conversation, such as sharing real world experience with remote friends or family. In this paper we explore how adding augmented visual communication cues can improve the experience of sharing remote task space and collaborating together. We developed a prototype system that allows users to share live video view of their task space taken on a Head Mounted Display (HMD) or Handheld Display (HHD), and communicate through not only voice but also using augmented pointer or annotations drawn on the shared view. To explore the effect of having such an interface for remote collaboration, we conducted a user study comparing three video-conferencing conditions with different combination of communication cues: (1) voice only, (2) voice + pointer, and (3) voice + annotation. The participants used our remote collaboration system to share a parallel experience of puzzle solving in the user study, and we found that adding augmented visual cues significantly improved the sense of being together. The pointer was the most preferred additional cue by users for parallel experience, and there were different states of the users’ behavior found in remote collaboration.
A Study of Depth Perception in Hand-Held Augmented Reality using Autostereoscopic Displays
Authors: Matthias Berning, Daniel Kleinert, Till Riedel, Michael Beigl
Abstract :
Displaying three-dimensional content on a flat display is bound to reduce the impression of depth, particularly for mobile video see-trough augmented reality. Several applications in this domain can benefit from accurate depth perception, especially if there are contradictory depth cues, like occlusion in a x-ray visualization. The use of stereoscopy for this effect is already prevalent in head-mounted displays, but there is little research on the applicability for hand-held augmented reality. We have implemented such a prototype using an off-the-shelf smartphone equipped with a stereo camera and an autostereoscopic display. We designed and conducted an extensive user study to explore the effects of stereoscopic hand-held augmented reality on depth perception. The results show that in this scenario depth judgment is mostly influenced by monoscopic depth cues, but our system can improve positioning accuracy in challenging scenes.
Displaying three-dimensional content on a flat display is bound to reduce the impression of depth, particularly for mobile video see-trough augmented reality. Several applications in this domain can benefit from accurate depth perception, especially if there are contradictory depth cues, like occlusion in a x-ray visualization. The use of stereoscopy for this effect is already prevalent in head-mounted displays, but there is little research on the applicability for hand-held augmented reality. We have implemented such a prototype using an off-the-shelf smartphone equipped with a stereo camera and an autostereoscopic display. We designed and conducted an extensive user study to explore the effects of stereoscopic hand-held augmented reality on depth perception. The results show that in this scenario depth judgment is mostly influenced by monoscopic depth cues, but our system can improve positioning accuracy in challenging scenes.
Measurements of Live Actor Motion in Mixed Reality Interaction
Authors: Gregory Hough, Ian Williams, Cham Athwal
Abstract :
This paper presents a method for measuring the magnitude and impact of errors in mixed reality interactions. We define the errors as measurements of hand placement accuracy and consistency within bimanual movement of an interactive virtual object. First, a study is presented which illustrates the amount of variability between the hands and the mean distance of the hands from the surfaces of a common virtual object. The results allow a discussion of the most significant factors which should be considered in the frame of developing realistic mixed reality interaction systems. The degree of error was found to be independent of interaction speed, whilst the size of virtual object and the position of the hands are significant. Second, a further study illustrates how perceptible these errors are to a third person viewer of the interaction (e.g. an audience member). We found that interaction errors arising from the overestimation of an object surface affected the visual credibility for the viewer considerably more than an underestimation of the object. This work is presented within the application of a real-time Interactive Virtual Television Studio, which offers convincing real-time interaction for live TV production. We believe the results and methodology presented here could also be applied for designing, implementing and assessing interaction quality in many other Mixed Reality applications.
This paper presents a method for measuring the magnitude and impact of errors in mixed reality interactions. We define the errors as measurements of hand placement accuracy and consistency within bimanual movement of an interactive virtual object. First, a study is presented which illustrates the amount of variability between the hands and the mean distance of the hands from the surfaces of a common virtual object. The results allow a discussion of the most significant factors which should be considered in the frame of developing realistic mixed reality interaction systems. The degree of error was found to be independent of interaction speed, whilst the size of virtual object and the position of the hands are significant. Second, a further study illustrates how perceptible these errors are to a third person viewer of the interaction (e.g. an audience member). We found that interaction errors arising from the overestimation of an object surface affected the visual credibility for the viewer considerably more than an underestimation of the object. This work is presented within the application of a real-time Interactive Virtual Television Studio, which offers convincing real-time interaction for live TV production. We believe the results and methodology presented here could also be applied for designing, implementing and assessing interaction quality in many other Mixed Reality applications.
Special TVGC Presentation
Session :
Location : HS3
Chair : Christian Sandor, Nara Institute of Science and Technology
Papers :
Special TVGC Presentation
Date & Time : September 11 04:00 pm - 04:30 pmLocation : HS3
Chair : Christian Sandor, Nara Institute of Science and Technology
Papers :
Making Graphical Information Visible in Real Shadows on Interactive Tabletops
Authors: Mariko Isogawa, Daisuke Iwai, Kosuke Sato
Abstract :
We introduce a shadow-based interface for interactive tabletops. The proposed interface allows a user to browse graphical information by casting the shadow of his/her body, such as a hand, on a tabletop surface. Central to our technique is a new optical design that utilizes polarization in addition to the additive nature of light so that the desired graphical information is displayed only in a shadow area on a tabletop surface. In other words, our technique conceals the graphical information on surfaces other than the shadow area, such as the surface of the occluder and non-shadow areas on the tabletop surface. We combine the proposed shadow-based interface with a multi-touch detection technique to realize a novel interaction technique for interactive tabletops. We implemented a prototype system and conducted proof-of-concept experiments along with a quantitative evaluation to assess the feasibility of the proposed optical design. Finally, we showed implemented application systems of the proposed shadow-based interface.
We introduce a shadow-based interface for interactive tabletops. The proposed interface allows a user to browse graphical information by casting the shadow of his/her body, such as a hand, on a tabletop surface. Central to our technique is a new optical design that utilizes polarization in addition to the additive nature of light so that the desired graphical information is displayed only in a shadow area on a tabletop surface. In other words, our technique conceals the graphical information on surfaces other than the shadow area, such as the surface of the occluder and non-shadow areas on the tabletop surface. We combine the proposed shadow-based interface with a multi-touch detection technique to realize a novel interaction technique for interactive tabletops. We implemented a prototype system and conducted proof-of-concept experiments along with a quantitative evaluation to assess the feasibility of the proposed optical design. Finally, we showed implemented application systems of the proposed shadow-based interface.
Layout and Head-Worn Displays VST
Session :
Location : HS1
Chair : Tobias Hoellerer, University of California, Santa Barbara
Papers :
Layout and Head-Worn Displays - Video See-Through
Date & Time : September 12 02:00 pm - 03:30 pmLocation : HS1
Chair : Tobias Hoellerer, University of California, Santa Barbara
Papers :
Creating Automatically Aligned Consensus Realities for AR Videoconferencing
Authors: Nicolas Lehment, Daniel Merget, Gerhard Rigoll
Abstract :
This paper presents an AR videoconferencing approach merging two remote rooms into a shared workspace. Such bilateral {AR} telepresence inherently suffers from breaks in immersion stemming from the different physical layouts of participating spaces. As a remedy, we develop an automatic alignment scheme which ensures that participants share a maximum of common features in their physical surroundings. The system optimizes alignment with regard to initial user position, free shared floor space, camera positioning and other factors. Thus we can reduce discrepancies between different room and furniture layouts without actually modifying the rooms themselves. A description and discussion of our alignment scheme is given along with an exemplary implementation on real-world datasets.
This paper presents an AR videoconferencing approach merging two remote rooms into a shared workspace. Such bilateral {AR} telepresence inherently suffers from breaks in immersion stemming from the different physical layouts of participating spaces. As a remedy, we develop an automatic alignment scheme which ensures that participants share a maximum of common features in their physical surroundings. The system optimizes alignment with regard to initial user position, free shared floor space, camera positioning and other factors. Thus we can reduce discrepancies between different room and furniture layouts without actually modifying the rooms themselves. A description and discussion of our alignment scheme is given along with an exemplary implementation on real-world datasets.
FLARE: Fast Layout for Augmented Reality Applications
Authors: Ran Gal, Lior Shapira, Eyal Ofek, Pushmeet Kohli
Abstract :
Creating a layout for an augmented reality (AR) application which embeds virtual objects in a physical environment is difficult as it must adapt to any physical space. We propose a rule-based framework for generating object layouts for AR applications. Under our framework, the developer of an AR application specifies a set of rules (constraints) which enforce self-consistency (rules regarding the inter-relationships of application components) and scene-consistency (application components are consistent with the physical environment they are placed in). When a user enters a new environment, we create, in real-time, a layout for the application, which is consistent with the defined constraints (as much as possible). We find the optimal configurations for each object by solving a constraint-satisfaction problem. Our stochastic move making algorithm is domain-aware, and allows us to efficiently converge to a solution for most rule-sets. In the paper we demonstrate several augmented reality applications that automatically adapt to different rooms and changing circumstances in each room.
Creating a layout for an augmented reality (AR) application which embeds virtual objects in a physical environment is difficult as it must adapt to any physical space. We propose a rule-based framework for generating object layouts for AR applications. Under our framework, the developer of an AR application specifies a set of rules (constraints) which enforce self-consistency (rules regarding the inter-relationships of application components) and scene-consistency (application components are consistent with the physical environment they are placed in). When a user enters a new environment, we create, in real-time, a layout for the application, which is consistent with the defined constraints (as much as possible). We find the optimal configurations for each object by solving a constraint-satisfaction problem. Our stochastic move making algorithm is domain-aware, and allows us to efficiently converge to a solution for most rule-sets. In the paper we demonstrate several augmented reality applications that automatically adapt to different rooms and changing circumstances in each room.
Presence and Discernability in Conventional and Non-Photorealistic Immersive Augmented Reality
Authors: William Steptoe, Simon Julier, Anthony Steed
Abstract :
Non-photorealistic rendering (NPR) has been shown as a powerful way to enhance both visual coherence and immersion in augmented reality (AR). However, it has only been evaluated in idealized pre-rendered scenarios with handheld AR devices. In this paper we investigate the use of NPR in an immersive, stereoscopic, wide field-of-view head-mounted video see-through AR display. This is a demanding scenario, which introduces many real-world effects including latency, tracking failures, optical artifacts and mismatches in lighting. We present the AR-Rift, a low-cost video see-through AR system using an Oculus Rift and consumer webcams. We investigate the themes of consistency and immersion as measures of psychophysical non-mediation. An experiment measures discernability and presence in three visual modes: conventional (unprocessed video and graphics), stylized (edge-enhancement) and virtualized (edge-enhancement and color extraction). The stylized mode results in chance-level discernability judgments, indicating successful integration of virtual content to form a visually coherent scene. Conventional and virutalized rendering bias judgments towards correct or incorrect respectively. Presence as it may apply to immersive AR, and which, measured both behaviorally and subjectively, is seen to be similarly high over all three conditions.
Non-photorealistic rendering (NPR) has been shown as a powerful way to enhance both visual coherence and immersion in augmented reality (AR). However, it has only been evaluated in idealized pre-rendered scenarios with handheld AR devices. In this paper we investigate the use of NPR in an immersive, stereoscopic, wide field-of-view head-mounted video see-through AR display. This is a demanding scenario, which introduces many real-world effects including latency, tracking failures, optical artifacts and mismatches in lighting. We present the AR-Rift, a low-cost video see-through AR system using an Oculus Rift and consumer webcams. We investigate the themes of consistency and immersion as measures of psychophysical non-mediation. An experiment measures discernability and presence in three visual modes: conventional (unprocessed video and graphics), stylized (edge-enhancement) and virtualized (edge-enhancement and color extraction). The stylized mode results in chance-level discernability judgments, indicating successful integration of virtual content to form a visually coherent scene. Conventional and virutalized rendering bias judgments towards correct or incorrect respectively. Presence as it may apply to immersive AR, and which, measured both behaviorally and subjectively, is seen to be similarly high over all three conditions.
WeARHand: Head-Worn, RGB-D Camera-Based, Bare-Hand User Interface with Visually Enhanced Depth Perception
Authors: Taejin Ha, Steven Feiner, Woontack Woo
Abstract :
We introduce WeARHand, which allows a user to manipulate virtual 3D objects with a bare hand in a wearable augmented reality (AR) environment. Our method uses no environmentally tethered tracking devices and localizes a pair of near-range and far-range RGB-D cameras mounted on a head-worn display and a moving bare hand in 3D space by exploiting depth input data. Depth perception is enhanced through egocentric visual feedback, including a semi-transparent proxy hand. We implement a virtual hand interaction technique and feedback approaches, and evaluate their performance and usability. The proposed method can apply to many 3D interaction scenarios using hands in a wearable AR environment, such as AR information browsing, maintenance, design, and games.
We introduce WeARHand, which allows a user to manipulate virtual 3D objects with a bare hand in a wearable augmented reality (AR) environment. Our method uses no environmentally tethered tracking devices and localizes a pair of near-range and far-range RGB-D cameras mounted on a head-worn display and a moving bare hand in 3D space by exploiting depth input data. Depth perception is enhanced through egocentric visual feedback, including a semi-transparent proxy hand. We implement a virtual hand interaction technique and feedback approaches, and evaluate their performance and usability. The proposed method can apply to many 3D interaction scenarios using hands in a wearable AR environment, such as AR information browsing, maintenance, design, and games.
Head-Worn Displays OST
Session :
Location : HS1
Chair : Tom Furness, HITLab, University of Washington
Papers :
Head-Worn Displays - Optical See-Through
Date & Time : September 12 11:00 am - 12:30 pmLocation : HS1
Chair : Tom Furness, HITLab, University of Washington
Papers :
Performance and Sensitivity Analysis of INDICA: INteraction-free DIsplay CAlibration for Optical See-Through Head-Mounted Displays
Authors: Yuta Itoh, Gudrun Klinker
Abstract :
An issue in AR applications with Optical See-Through Head- Mounted Display (OST-HMD) is to correctly project 3D information to the current viewpoint of the user. Manual calibration methods give the projection as a black box which explains observed 2D- 3D relationships well (Fig. 1). Recently, we have proposed an INteraction-free DIsplay CAlibration method (INDICA) for OSTHMD, utilizing camera-based eye tracking[7]. It reformulates the projection in two ways: a black box with an actual eye model (Recycle Setup), and a combination of an explicit display model and an eye model (Full Setup). Although we have shown the former performs more stably than a repeated SPAAM calibration, we could not yet prove whether the same holds for the Full Setup. More importantly, it is still unclear how the error in the calibration parameters affects the final results. Thus, the users can not know how accurately they need to estimate each parameter in practice. We provide: (1) the fact that the Full Setup performs as accurately as the Recycle Setup under a marker-based display calibration, (2) an error sensitivity analysis for both SPAAM and INDICA over the on-/offline parameters, and (3) an investigation of the theoretical sensitivity on an OST-HMD justified by the real measurements.
An issue in AR applications with Optical See-Through Head- Mounted Display (OST-HMD) is to correctly project 3D information to the current viewpoint of the user. Manual calibration methods give the projection as a black box which explains observed 2D- 3D relationships well (Fig. 1). Recently, we have proposed an INteraction-free DIsplay CAlibration method (INDICA) for OSTHMD, utilizing camera-based eye tracking[7]. It reformulates the projection in two ways: a black box with an actual eye model (Recycle Setup), and a combination of an explicit display model and an eye model (Full Setup). Although we have shown the former performs more stably than a repeated SPAAM calibration, we could not yet prove whether the same holds for the Full Setup. More importantly, it is still unclear how the error in the calibration parameters affects the final results. Thus, the users can not know how accurately they need to estimate each parameter in practice. We provide: (1) the fact that the Full Setup performs as accurately as the Recycle Setup under a marker-based display calibration, (2) an error sensitivity analysis for both SPAAM and INDICA over the on-/offline parameters, and (3) an investigation of the theoretical sensitivity on an OST-HMD justified by the real measurements.
Analysing the Effects of a Wide Field of View Augmented Reality Display on Search Performance in Divided Attention Tasks
Authors: Naohiro Kishishita, Kiyoshi Kiyokawa, Ernst Kruijff, Jason Orlosky, Tomohiro Mashita, Haruo Takemura
Abstract :
A wide field of view augmented reality display is a special type of head-worn device that enables users to view augmentations in the peripheral visual field. However, the actual effects of a wide field of view display on the perception of augmentations have not been widely studied. To improve our understanding of this type of display when conducting divided attention search tasks, we conducted an in depth experiment testing two view management methods, in-view and in-situ labelling. With in-view labelling, search target annotations appear on the display border with a corresponding leader line, whereas in-situ annotations appear without a leader line, as if they are affixed to the referenced objects in the environment. Results show that target discovery rates consistently drop with in-view labelling and increase with in-situ labelling as display angle approaches 100 degrees of field of view. Past this point, the performances of the two view management methods begin to converge, suggesting equivalent discovery rates at approximately 130 degrees of field of view. Results also indicate that users exhibited lower discovery rates for targets appearing in peripheral vision, and that there is little impact of field of view on response time and mental workload.
A wide field of view augmented reality display is a special type of head-worn device that enables users to view augmentations in the peripheral visual field. However, the actual effects of a wide field of view display on the perception of augmentations have not been widely studied. To improve our understanding of this type of display when conducting divided attention search tasks, we conducted an in depth experiment testing two view management methods, in-view and in-situ labelling. With in-view labelling, search target annotations appear on the display border with a corresponding leader line, whereas in-situ annotations appear without a leader line, as if they are affixed to the referenced objects in the environment. Results show that target discovery rates consistently drop with in-view labelling and increase with in-situ labelling as display angle approaches 100 degrees of field of view. Past this point, the performances of the two view management methods begin to converge, suggesting equivalent discovery rates at approximately 130 degrees of field of view. Results also indicate that users exhibited lower discovery rates for targets appearing in peripheral vision, and that there is little impact of field of view on response time and mental workload.
SmartColor: Real-Time Color Correction and Contrast for Optical See-Through Head-Mounted Displays
Authors: Juan David Hincapié-Ramos, Levko Ivanchuk, Srikanth Kirshnamachari Sridharan, Pourang Irani
Abstract :
Users of optical see-through head-mounted displays (OHMD) perceive color as a blend of the display color and the background. Color-blending is a major usability challenge as it leads to loss of color encodings and poor text legibility. Color correction aims at mitigating color blending by producing an alternative color which, when blended with the background, more closely approaches the color originally intended. To date, approaches to color correction do not yield optimal results or do not work in real-time. This paper makes two contributions. First, we present QuickCorrection, a real-time color correction algorithm based on display profiles. We describe the algorithm, measure its accuracy and analyze two implementations for the OpenGL graphics pipeline. Second, we present SmartColor, a middleware for color management of user-interface components in OHMD. SmartColor uses color correction to provide three management strategies: correction, contrast, and show-up-on-contrast. Correction determines the alternate color which best preserves the original color. Contrast determines the color which best warranties text legibility while preserving as much of the original hue. Show-up-on-contrast makes a component visible when a related component does not have enough contrast to be legible. We describe the SmartColor’s architecture and illustrate the color strategies for various types of display content.
Users of optical see-through head-mounted displays (OHMD) perceive color as a blend of the display color and the background. Color-blending is a major usability challenge as it leads to loss of color encodings and poor text legibility. Color correction aims at mitigating color blending by producing an alternative color which, when blended with the background, more closely approaches the color originally intended. To date, approaches to color correction do not yield optimal results or do not work in real-time. This paper makes two contributions. First, we present QuickCorrection, a real-time color correction algorithm based on display profiles. We describe the algorithm, measure its accuracy and analyze two implementations for the OpenGL graphics pipeline. Second, we present SmartColor, a middleware for color management of user-interface components in OHMD. SmartColor uses color correction to provide three management strategies: correction, contrast, and show-up-on-contrast. Correction determines the alternate color which best preserves the original color. Contrast determines the color which best warranties text legibility while preserving as much of the original hue. Show-up-on-contrast makes a component visible when a related component does not have enough contrast to be legible. We describe the SmartColor’s architecture and illustrate the color strategies for various types of display content.
Minimizing Latency for Augmented Reality Displays: Frames Considered Harmful
Authors: Feng Zheng, Turner Whitted, Anselmo Lastra, Peter Lincoln, Andrei State, Andrew Maimone, Henry Fuchs
Abstract :
We present initial results from a new image generation approach for low-latency displays such as those needed in head-worn AR devices. Avoiding the usual video interfaces, such as HDMI, we favor direct control of the internal display technology. We illustrate our new approach with a bench-top optical see-through AR proof-of-concept prototype that uses a Digital Light Processing (DLP) projector whose Digital Micromirror Device (DMD) imaging chip is directly controlled by a computer, similar to the way random access memory is controlled. We show that a perceptually-continuous-tone dynamic gray-scale image can be efficiently composed from a very rapid succession of binary (partial) images, each calculated from the continuous-tone image generated with the most recent tracking data. As the DMD projects only a binary image at any moment, it cannot instantly display this latest continuous-tone image, and conventional decomposition of a continuous-tone image into binary time-division-multiplexed values would induce just the latency we seek to avoid. Instead, our approach maintains an estimate of the image the user currently perceives, and at every opportunity allowed by the control circuitry, sets each binary DMD pixel to the value that will reduce the difference between that user-perceived image and the newly generated image from the latest tracking data. The resulting displayed binary image is "neither here nor there," but always approaches the moving target that is the constantly changing desired image, even when that image changes every 50µs. We compare our experimental results with imagery from a conventional DLP projector with similar internal speed, and demonstrate that AR overlays on a moving object are more effective with this kind of low-latency display device than with displays of similar speed that use a conventional video interface.
We present initial results from a new image generation approach for low-latency displays such as those needed in head-worn AR devices. Avoiding the usual video interfaces, such as HDMI, we favor direct control of the internal display technology. We illustrate our new approach with a bench-top optical see-through AR proof-of-concept prototype that uses a Digital Light Processing (DLP) projector whose Digital Micromirror Device (DMD) imaging chip is directly controlled by a computer, similar to the way random access memory is controlled. We show that a perceptually-continuous-tone dynamic gray-scale image can be efficiently composed from a very rapid succession of binary (partial) images, each calculated from the continuous-tone image generated with the most recent tracking data. As the DMD projects only a binary image at any moment, it cannot instantly display this latest continuous-tone image, and conventional decomposition of a continuous-tone image into binary time-division-multiplexed values would induce just the latency we seek to avoid. Instead, our approach maintains an estimate of the image the user currently perceives, and at every opportunity allowed by the control circuitry, sets each binary DMD pixel to the value that will reduce the difference between that user-perceived image and the newly generated image from the latest tracking data. The resulting displayed binary image is "neither here nor there," but always approaches the moving target that is the constantly changing desired image, even when that image changes every 50µs. We compare our experimental results with imagery from a conventional DLP projector with similar internal speed, and demonstrate that AR overlays on a moving object are more effective with this kind of low-latency display device than with displays of similar speed that use a conventional video interface.
Medical AR
Session :
Location : HS1
Chair : Nassir Navab, TU Munich
Papers :
Medical AR
Date & Time : September 12 04:00 pm - 05:30 pmLocation : HS1
Chair : Nassir Navab, TU Munich
Papers :
Single View Augmentation of 3D Elastic Objects
Authors: Nazim Haouchine, Jeremie Dequidt, Marie-Odile Berger, Stephane Cotin
Abstract :
This paper proposes an efficient method to capture and augment highly elastic objects from a single view. 3D shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and re-solve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or ressort to shading constraints. However, few of them can handle properly large elastic deformations. We propose in this paper a real-time method which makes use of a mechanical model and is able to handle highly elastic objects. Our method is formulated as a energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. The only parameter involved in the method is the Young’s modulus but we show in experiments that a rough estimate of the Young’s modulus is sufficient to obtain a good reconstruction. Our method is compared to existing techniques with experiments conducted on computer-generated and real data that show the effectiveness of our approach. Experiments in the context of minimally invasive liver surgery are also provided.
This paper proposes an efficient method to capture and augment highly elastic objects from a single view. 3D shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and re-solve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or ressort to shading constraints. However, few of them can handle properly large elastic deformations. We propose in this paper a real-time method which makes use of a mechanical model and is able to handle highly elastic objects. Our method is formulated as a energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. The only parameter involved in the method is the Young’s modulus but we show in experiments that a rough estimate of the Young’s modulus is sufficient to obtain a good reconstruction. Our method is compared to existing techniques with experiments conducted on computer-generated and real data that show the effectiveness of our approach. Experiments in the context of minimally invasive liver surgery are also provided.
Improved Interventional X-ray Appearance
Authors: Xiang Wang, Christian Schulte zu Berge, Stefanie Demirci, Pascal Fallavollita, Nassir Navab
Abstract :
Depth cues are an essential part of navigation and device positioning tasks during clinical interventions. Yet, many minimally-invasive procedures, such as catheterizations, are usually performed under X-ray guidance only depicting a 2D projection of the anatomy, which lacks depth information. Previous attempts to integrate pre-operative 3D data of the patient by registering these to intra-operative data have led to virtual 3D renderings independent of the original X-ray appearance and planar 2D color overlays (e.g. roadmaps). A major drawback associated to these solutions is the trade-off between X-ray attenuation values that is completely neglected during 3D renderings, and depth perception not being incorporated into the 2D roadmaps. This paper presents a novel technique for enhancing depth perception of interventional X-ray images preserving the original attenuation appearance. Starting from patient-specific pre-operative 3D data, our method relies on GPU ray casting to compute a colored depth map, which assigns a predefined color to the first incidence of gradient magnitude value above a predefined threshold along the ray. The colored depth map values are carefully integrated into the X-Ray image while maintaining its original grayscale intensities. The presented method was tested and analysed for three relevant clinical scenarios covering different anatomical aspects and targeting different levels of interventional expertise. Results demonstrate that improving depth perception of X-ray images has the potential to lead to safer and more efficient clinical interventions.
Depth cues are an essential part of navigation and device positioning tasks during clinical interventions. Yet, many minimally-invasive procedures, such as catheterizations, are usually performed under X-ray guidance only depicting a 2D projection of the anatomy, which lacks depth information. Previous attempts to integrate pre-operative 3D data of the patient by registering these to intra-operative data have led to virtual 3D renderings independent of the original X-ray appearance and planar 2D color overlays (e.g. roadmaps). A major drawback associated to these solutions is the trade-off between X-ray attenuation values that is completely neglected during 3D renderings, and depth perception not being incorporated into the 2D roadmaps. This paper presents a novel technique for enhancing depth perception of interventional X-ray images preserving the original attenuation appearance. Starting from patient-specific pre-operative 3D data, our method relies on GPU ray casting to compute a colored depth map, which assigns a predefined color to the first incidence of gradient magnitude value above a predefined threshold along the ray. The colored depth map values are carefully integrated into the X-Ray image while maintaining its original grayscale intensities. The presented method was tested and analysed for three relevant clinical scenarios covering different anatomical aspects and targeting different levels of interventional expertise. Results demonstrate that improving depth perception of X-ray images has the potential to lead to safer and more efficient clinical interventions.
Computer-Assisted Laparoscopic Myomectomy by Augmenting the Uterus with Pre-operative MRI Data
Authors: Toby Collins, Daniel Pizarro, Adrien Bartoli, Nicolas Bourdel
Abstract :
An active research objective in Computer Assisted Intervention (CAI) is to develop guidance systems to aid surgical teams in laparoscopic Minimal Invasive Surgery (MIS) using Augmented Reality (AR). This involves registering and fusing additional data from other modalities and overlaying it onto the laparoscopic video in realtime. We present the first AR-based image guidance system for assisted myoma localisation in uterine laparosurgery. This involves a framework for semi-automatically registering a pre-operative Magnetic Resonance Image (MRI) to the laparoscopic video with a deformable model. Although there has been several previous works involving other organs, this is the first to tackle the uterus. Furthermore, whereas previous works perform registration between one or two laparoscopic images (which come from a stereo laparoscope) we show how to solve the problem using many images (e.g. 20 or more), and show that this can dramatically improve registration. Also unlike previous works, we show how to integrate occluding contours as registration cues. These cues provide powerful registration constraints and should be used wherever possible. We present retrospective qualitative results on a patient with two myomas and quantitative semi-synthetic results. Our multi-image framework is quite general and could be adapted to improve registration other organs with other modalities such as CT.
An active research objective in Computer Assisted Intervention (CAI) is to develop guidance systems to aid surgical teams in laparoscopic Minimal Invasive Surgery (MIS) using Augmented Reality (AR). This involves registering and fusing additional data from other modalities and overlaying it onto the laparoscopic video in realtime. We present the first AR-based image guidance system for assisted myoma localisation in uterine laparosurgery. This involves a framework for semi-automatically registering a pre-operative Magnetic Resonance Image (MRI) to the laparoscopic video with a deformable model. Although there has been several previous works involving other organs, this is the first to tackle the uterus. Furthermore, whereas previous works perform registration between one or two laparoscopic images (which come from a stereo laparoscope) we show how to solve the problem using many images (e.g. 20 or more), and show that this can dramatically improve registration. Also unlike previous works, we show how to integrate occluding contours as registration cues. These cues provide powerful registration constraints and should be used wherever possible. We present retrospective qualitative results on a patient with two myomas and quantitative semi-synthetic results. Our multi-image framework is quite general and could be adapted to improve registration other organs with other modalities such as CT.
MASHD: Theory and Evaluation
Session :
Location : HS3
Chair : Henry Duh, HITLab AU, University of Tasmania
Papers :
MASH'D: Theory and Evaluation
Date & Time : September 12 02:00 pm - 03:30 pmLocation : HS3
Chair : Henry Duh, HITLab AU, University of Tasmania
Papers :
nARratives of augmented worlds
Authors: Roy Shilkrot, Nick Montfort, Pattie Maes
Abstract :
This paper presents an examination of augmented reality (AR) as a rising form of interactive narrative that combines computer-generated elements with reality, fictional with non-fictional objects, in the same immersive experience. Based on contemporary theory in narratology, we propose to view this blending of reality worlds as a metalepsis, a transgression of reality and fiction boundaries, and argue that authors could benefit from using existing conventions of narration to emphasize the transgressed boundaries, as is done in other media. Our contribution is three-fold, first we analyze the inherent connection between narrative, immersion, interactivity, fictionality and AR using narrative theory, and second we comparatively survey actual works in AR narratives from the past 15 years based on these elements from the theory. Lastly, we postulate a future for AR narratives through the perspective of the advancing technologies of both interactive narratives and AR.
This paper presents an examination of augmented reality (AR) as a rising form of interactive narrative that combines computer-generated elements with reality, fictional with non-fictional objects, in the same immersive experience. Based on contemporary theory in narratology, we propose to view this blending of reality worlds as a metalepsis, a transgression of reality and fiction boundaries, and argue that authors could benefit from using existing conventions of narration to emphasize the transgressed boundaries, as is done in other media. Our contribution is three-fold, first we analyze the inherent connection between narrative, immersion, interactivity, fictionality and AR using narrative theory, and second we comparatively survey actual works in AR narratives from the past 15 years based on these elements from the theory. Lastly, we postulate a future for AR narratives through the perspective of the advancing technologies of both interactive narratives and AR.
A Theory of Meaning for Mixed Reality Walking Tours
Author: Evan Barba
Abstract :
In the broadest sense, Mixed and Augmented Reality experiences mix sensory and conceptual elements both externally in the world and in the minds of their users. The question of how participants in these experiences derive meaning from these hybrid realities is important for both analysis and design. By focusing on MAR cultural heritage walking tours, this paper develops a theory of meaning-making based on the aboriginal walkabout that accounts for both physical and conceptual experience. Through an interweaving of concepts from anthropology, architecture, design, cognitive science and MAR itself, I demonstrate that his theory is compatible with known principles of brain function and human behavior and thus it serves as a more general theory of meaning-making applicable beyond the MAR walking tours from which it was derived.
In the broadest sense, Mixed and Augmented Reality experiences mix sensory and conceptual elements both externally in the world and in the minds of their users. The question of how participants in these experiences derive meaning from these hybrid realities is important for both analysis and design. By focusing on MAR cultural heritage walking tours, this paper develops a theory of meaning-making based on the aboriginal walkabout that accounts for both physical and conceptual experience. Through an interweaving of concepts from anthropology, architecture, design, cognitive science and MAR itself, I demonstrate that his theory is compatible with known principles of brain function and human behavior and thus it serves as a more general theory of meaning-making applicable beyond the MAR walking tours from which it was derived.
Can mobile augmented reality systems assist in portion estimation? A user study.
Authors: Thomas Stütz, Radomir Dinic, Michael Domhardt, Simon Ginzinger
Abstract :
Accurate assessment of nutrition information is an important part in the prevention and treatment of a multitude of diseases, but remains a challenging task. We present a novel mobile augmented reality application, which assists users in the nutrition assessment of their meals. Using the realtime camera image as a guide, the user overlays a 3D form of the food. Additionally the user selects the food type. The corresponding nutrition information is automatically computed. Thus accurate volume estimation is required for accurate nutrition information assessment. This work presents an evaluation of our mobile augmented reality approaches for portion estimation and offers a comparison to conventional portion estimation approaches. The comparison is performed on the basis of a user study (n=28). The quality of nutrition assessment is measured based on the error in energy units. In the results of the evaluation one of our mobile augmented reality approaches significantly outperforms all other methods. Additionally we present results on the efficiency and effectiveness of the approaches.
Accurate assessment of nutrition information is an important part in the prevention and treatment of a multitude of diseases, but remains a challenging task. We present a novel mobile augmented reality application, which assists users in the nutrition assessment of their meals. Using the realtime camera image as a guide, the user overlays a 3D form of the food. Additionally the user selects the food type. The corresponding nutrition information is automatically computed. Thus accurate volume estimation is required for accurate nutrition information assessment. This work presents an evaluation of our mobile augmented reality approaches for portion estimation and offers a comparison to conventional portion estimation approaches. The comparison is performed on the basis of a user study (n=28). The quality of nutrition assessment is measured based on the error in energy units. In the results of the evaluation one of our mobile augmented reality approaches significantly outperforms all other methods. Additionally we present results on the efficiency and effectiveness of the approaches.
Evaluating Controls for a Point and Shoot Mobile Game: Augmented Reality, Tilt and Touch
Authors: Asier Marzo, Benoît Bossavit, Martin Hachet
Abstract :
Controls based on Augmented Reality (AR), Tilt and Touch have been evaluated in a point and shoot game for mobile devices. A user study (n=12) was conducted to compare the three controls in terms of player experience and accuracy. Tilt and AR controls provided more enjoyment, immersion and accuracy to the players than Touch. Nonetheless, Touch caused fewer nuisances and was playable under more varied situations. Despite the current technical limitations, we suggest to incorporate AR controls into the mobile games that supported them. Nowadays, AR controls can be implemented on handheld devices as easily as the more established Tilt and Touch controls. However, this study is the first comparison of them and thus its findings could be of interest for game developers.
Controls based on Augmented Reality (AR), Tilt and Touch have been evaluated in a point and shoot game for mobile devices. A user study (n=12) was conducted to compare the three controls in terms of player experience and accuracy. Tilt and AR controls provided more enjoyment, immersion and accuracy to the players than Touch. Nonetheless, Touch caused fewer nuisances and was playable under more varied situations. Despite the current technical limitations, we suggest to incorporate AR controls into the mobile games that supported them. Nowadays, AR controls can be implemented on handheld devices as easily as the more established Tilt and Touch controls. However, this study is the first comparison of them and thus its findings could be of interest for game developers.
MASHD: AR Interaction and Creativity
Session :
Location : HS3
Chair : Julian Stadon, FH Salzburg
Papers :
MASH'D: AR Interaction and Creativity
Date & Time : September 12 11:00 am - 12:30 pmLocation : HS3
Chair : Julian Stadon, FH Salzburg
Papers :
Effects of Mobile AR-Enabled Interactions on Retention and Transfer for Learning in Art Museum Contexts
Authors: Weiquan Lu, Linh-Chi Nguyen, Teong Leong Chuah, Ellen Yi-Luen Do
Abstract :
In this paper, we describe an experiment to study the effect of mobile Augmented Reality (AR) on learning in art museum contexts. We created six original paintings and placed them in a mini art museum. We then created an AR application on the iPad to enable the artist to visually augment each painting by introducing animation. We then measured the ability of the visitors to remember the appearance of the paintings after 24 hours, as well as their ability to objectify the paintings. Experiment results show that while AR does improve retention and transfer of such art information, the benefits of AR are mediated by other factors such as interference from other elements of the exhibition, as well as subjects' own prior art experience and training. The use of AR may also produce unexpected benefits, such as providing users with a new perspective of the artwork, as well as increasing their curiosity and encouraging them to experiment with the technology. Such benefits may potentially improve the chances for learning and analytical activities to take place.
In this paper, we describe an experiment to study the effect of mobile Augmented Reality (AR) on learning in art museum contexts. We created six original paintings and placed them in a mini art museum. We then created an AR application on the iPad to enable the artist to visually augment each painting by introducing animation. We then measured the ability of the visitors to remember the appearance of the paintings after 24 hours, as well as their ability to objectify the paintings. Experiment results show that while AR does improve retention and transfer of such art information, the benefits of AR are mediated by other factors such as interference from other elements of the exhibition, as well as subjects' own prior art experience and training. The use of AR may also produce unexpected benefits, such as providing users with a new perspective of the artwork, as well as increasing their curiosity and encouraging them to experiment with the technology. Such benefits may potentially improve the chances for learning and analytical activities to take place.
AR PETITE THEATER: Augmented Reality Storybook for Supporting Children'
Authors: Kyungwon Gil, Jimin Rhim, Taejin Ha, Young Yim Doh, Woontack Woo
Abstract :
In this paper, we present an AR Petite Theater, a story book that enables role-play using augmented reality (AR) technology. It provides an opportunity for children to learn the ability of empathy through interactive reading experience by thinking and speaking in accordance with the character’s role of the story. In general, empathy is one of most important elements for children to make friends at school and to expand their social relations. In particular, it is crucial for early school-age children who have difficulties in getting along with friends due to their egocentric perspective. Through the experiment with 24 six-year-old children, we measured children’s role-playing participation and perspective taking state. As a result, more empathic behaviors were revealed in the AR group. Children in the AR condition were more actively involved in role-playing and showed less unrelated perspectives than children in the non-AR condition. Therefore, we verified that AR Petite Theater had the potential of expanding children’s ability to empathize with others.
In this paper, we present an AR Petite Theater, a story book that enables role-play using augmented reality (AR) technology. It provides an opportunity for children to learn the ability of empathy through interactive reading experience by thinking and speaking in accordance with the character’s role of the story. In general, empathy is one of most important elements for children to make friends at school and to expand their social relations. In particular, it is crucial for early school-age children who have difficulties in getting along with friends due to their egocentric perspective. Through the experiment with 24 six-year-old children, we measured children’s role-playing participation and perspective taking state. As a result, more empathic behaviors were revealed in the AR group. Children in the AR condition were more actively involved in role-playing and showed less unrelated perspectives than children in the non-AR condition. Therefore, we verified that AR Petite Theater had the potential of expanding children’s ability to empathize with others.
Integrating Augmented Reality to Enhance Expression, Interaction &
Authors: Alexis Clay, Gaël Domenger, Julien Conan, Axel Domenger, Nadine Couture
Abstract :
The democratization of high-end, affordable and off-the-shelf sensors and displays triggered an explosion in the exploration of interaction and projection in arts. Although mostly witnessed in interactive artistic installations (e.g. museums and exhibitions), performing arts also explore such technologies, using interaction and augmented reality as part of the performance. Such works often emerge from collaborations between artists and scientists. Despite being antonymic in appearance, we advocate that both fields can greatly benefit from this type of collaboration. Since 2006 the authors of this paper (from a research laboratory and a national ballet company) have collaborated on augmenting a ballet performance using a dancer’s movements for interaction. We focus on large productions using high-end motion capture and projection systems to allow dancers to interact with virtual elements on an augmented stage in front of several hundred people. To achieve this, we introduce an ‘augmented reality engineer’, whose role is to design the augmented reality systems and interactions according to a show’s aesthetic and choreographic message, and to control them during the performance alongside light and sound technicians. Our last production: Debussy3.0 is an augmented ballet based on La Mer by Claude Debussy, featuring body interactions by one of the dancers and backstage interactions by the augmented reality engineer. For the first time, we explored 3D stereoscopy as a display technique for augmented reality and interaction in real-time on stage. The show was presented at Biarritz Casino in December 2013 in front of around 700 people. In this paper, we present the Debussy3.0 augmented ballet both as a result of the use of augmented reality in performing arts and as a guiding thread to provide feedback on arts-science collaboration. First, we will describe how the ballet was constructed aesthetically, technically and in its choreography. We will discuss and provide feedback on the use of motion capture and stereoscopy techniques in a live show and will then broaden the scope of discussion, providing feedback on art-science collaboration, the traps and benefits for both parties, and the positive repercussions it can bring to a laboratory when working on industrial projects.
The democratization of high-end, affordable and off-the-shelf sensors and displays triggered an explosion in the exploration of interaction and projection in arts. Although mostly witnessed in interactive artistic installations (e.g. museums and exhibitions), performing arts also explore such technologies, using interaction and augmented reality as part of the performance. Such works often emerge from collaborations between artists and scientists. Despite being antonymic in appearance, we advocate that both fields can greatly benefit from this type of collaboration. Since 2006 the authors of this paper (from a research laboratory and a national ballet company) have collaborated on augmenting a ballet performance using a dancer’s movements for interaction. We focus on large productions using high-end motion capture and projection systems to allow dancers to interact with virtual elements on an augmented stage in front of several hundred people. To achieve this, we introduce an ‘augmented reality engineer’, whose role is to design the augmented reality systems and interactions according to a show’s aesthetic and choreographic message, and to control them during the performance alongside light and sound technicians. Our last production: Debussy3.0 is an augmented ballet based on La Mer by Claude Debussy, featuring body interactions by one of the dancers and backstage interactions by the augmented reality engineer. For the first time, we explored 3D stereoscopy as a display technique for augmented reality and interaction in real-time on stage. The show was presented at Biarritz Casino in December 2013 in front of around 700 people. In this paper, we present the Debussy3.0 augmented ballet both as a result of the use of augmented reality in performing arts and as a guiding thread to provide feedback on arts-science collaboration. First, we will describe how the ballet was constructed aesthetically, technically and in its choreography. We will discuss and provide feedback on the use of motion capture and stereoscopy techniques in a live show and will then broaden the scope of discussion, providing feedback on art-science collaboration, the traps and benefits for both parties, and the positive repercussions it can bring to a laboratory when working on industrial projects.
VAL: Visually Augmented Laser cutting to enhance and support creativity
Authors: Kristoffer Winge, Rune Haugaard, Timothy Merritt
Abstract :
Laser cutters are increasingly relevant within many user contextsand have become an essential tool for model building and prototyping. While providing precision and flexibility, these tools are typically suited for expert staff in industrial settings. VAL (Visually Augmented Laser cutting) proposes a novel system utilizing spatial augmented reality techniques to provide visual augmentation directly on the work surface. VAL involves projection of the user’s model prior to and during laser cutting providing key benefits including minimizing idle time, reduction of errors, and support for new creative practices. We interview and observe laser cutter users to identify issues and concerns in the shared work context of a design school and describe the design process for our prototype, which aims to address these problems and unmet needs. Initial evaluation suggests VAL reduces complexity and raises user confidence. Our findings extend research on adapting new use contexts and creative practices with industrial fabrication tools.
Laser cutters are increasingly relevant within many user contextsand have become an essential tool for model building and prototyping. While providing precision and flexibility, these tools are typically suited for expert staff in industrial settings. VAL (Visually Augmented Laser cutting) proposes a novel system utilizing spatial augmented reality techniques to provide visual augmentation directly on the work surface. VAL involves projection of the user’s model prior to and during laser cutting providing key benefits including minimizing idle time, reduction of errors, and support for new creative practices. We interview and observe laser cutter users to identify issues and concerns in the shared work context of a design school and describe the design process for our prototype, which aims to address these problems and unmet needs. Initial evaluation suggests VAL reduces complexity and raises user confidence. Our findings extend research on adapting new use contexts and creative practices with industrial fabrication tools.
