Scholars@Duke
Guy Rosman

Guy Rosman

Adjunct Associate Professor in the Department of Surgery
Surgery, Minimally Invasive Surgery
guy.rosman@duke.edu
DUMC 3704, Durham, NC 27710
DUMC 3704, Durham, NC 27710
Office hours By appointment / virtual. Please send an email to schedule.  

Selected Publications

From Dashboards to Dialogue: Evaluating a Conversational AI Coach for Performance Driving Skill Development

Conference Proceedings of the 17th International Conference on Automotive User Interfaces and Interactive Vehicular Applications · September 21, 2025 Full text Cite

Shared Autonomy for Proximal Teaching

Conference ACM IEEE International Conference on Human Robot Interaction · January 1, 2025 Motor skills education often requires experienced professionals who can provide personalized instruction. Unfortu-nately, the availability of high-quality training can be limited for specialized tasks, such as high performance racing. Several recent works ... Full text Cite

ReGen: GENERATIVE ROBOT SIMULATION VIA INVERSE DESIGN

Conference 13th International Conference on Learning Representations Iclr 2025 · January 1, 2025 Simulation plays a key role in scaling robot learning and validating policies, but constructing simulations remains a labor-intensive process. This paper introduces ReGen, a generative simulation framework that automates simulation design via inverse desig ... Cite

Beyond Breathalyzers: Towards Pre-Driving Sobriety Testing with a Driver Monitoring Camera

Conference IEEE Intelligent Vehicles Symposium Proceedings · January 1, 2025 Field sobriety tests and breathalyzers are commonly used to prevent alcohol-impaired driving, but are expensive and time-consuming to administer. We propose a set of sobriety tests which, in contrast, can feasibly be automated and deployed to modern vehicl ... Full text Cite

Cognitive Distraction Detection Using Gaze and Pupil with an Interpretable Approach

Conference IEEE Intelligent Vehicles Symposium Proceedings · January 1, 2025 Cognitive distraction (CD) is one of the major causes of traffic accidents, but there remains room to improve its detection. Most prior research on CD detection has commonly used basic statistical measures (e.g., mean, standard deviation) of driver-facing ... Full text Cite

Probing Multimodal LLMs as World Models for Driving

Journal Article IEEE Robotics and Automation Letters · January 1, 2025 We provide a sober look at the application of Multimodal Large Language Models (MLLMs) in autonomous driving, challenging common assumptions about their ability to interpret dynamic driving scenarios. Despite advances in models like GPT-4o, their performan ... Full text Cite

Hypergraph-Transformer (HGT) for Interaction Event Prediction in Laparoscopic and Robotic Surgery

Conference Proceedings IEEE International Conference on Robotics and Automation · January 1, 2025 Understanding and anticipating events and actions is critical for intraoperative assistance and decision-making during minimally invasive surgery. We propose a predictive neural network that is capable of understanding and predicting critical interaction a ... Full text Cite

Generating Out-of-Distribution Scenarios Using Language Models

Conference Proceedings IEEE International Conference on Robotics and Automation · January 1, 2025 The deployment of autonomous vehicles controlled by machine learning techniques requires extensive testing in diverse real-world environments, robust handling of edge cases and out-of-distribution scenarios, and comprehensive safety validation to ensure th ... Full text Cite

Computational Teaching for Driving via Multi-Task Imitation Learning

Conference Proceedings IEEE International Conference on Robotics and Automation · January 1, 2025 Learning motor skills for sports or performance driving is often done with professional instruction from expert human teachers, whose availability is limited. Our goal is to enable automated teaching via a learned model that interacts with the student simi ... Full text Cite

Think Deep and Fast: Learning Neural Nonlinear Opinion Dynamics from Inverse Dynamic Games for Split-Second Interactions

Conference Proceedings IEEE International Conference on Robotics and Automation · January 1, 2025 Non-cooperative interactions commonly occur in multi-agent scenarios such as car racing, where an ego vehicle can choose to overtake the rival, or stay behind it until a safe overtaking 'corridor' opens. While an expert human can do well at making such tim ... Full text Cite

Personalizing driver safety interfaces via driver cognitive factors inference.

Journal Article Sci Rep · August 5, 2024 Recent advances in AI and intelligent vehicle technology hold the promise of revolutionizing mobility and transportation through advanced driver assistance systems (ADAS). Certain cognitive factors, such as impulsivity and inhibitory control have been show ... Full text Link to item Cite

Concept Graph Neural Networks for Surgical Video Understanding.

Journal Article IEEE Trans Med Imaging · January 2024 Analysis of relations between objects and comprehension of abstract concepts in the surgical video is important in AI-augmented surgery. However, building models that integrate our knowledge and understanding of surgery remains a challenging endeavor. In t ... Full text Link to item Cite

Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models

Conference Proceedings IEEE International Conference on Robotics and Automation · January 1, 2024 As autonomous driving technology matures, end-to-end methodologies have emerged as a leading strategy, promising seamless integration from perception to control via deep learning. However, existing systems grapple with challenges such as unexpected open se ... Full text Cite

Online Adaptation of Learned Vehicle Dynamics Model with Meta-Learning Approach

Conference IEEE International Conference on Intelligent Robots and Systems · January 1, 2024 We represent a vehicle dynamics model for autonomous driving near the limits of handling via a multilayer neural network. Online adaptation is desirable in order to address unseen environments. However, the model needs to adapt to new environments without ... Full text Cite

Can Pupillometry be used to Detect Driver Hazard Awareness?

Conference Conference Proceedings IEEE International Conference on Systems Man and Cybernetics · January 1, 2024 Modern Advanced Driver-Assistance Systems (ADAS) increasingly rely on interactions between vehicle and human driver. To inform these interactions, it is helpful for a vehicle system to have a good understanding of a driver's situational awareness. In this ... Full text Cite

Dreaming to Assist: Learning to Align with Human Objectives for Shared Control in High-Speed Racing

Conference Proceedings of Machine Learning Research · January 1, 2024 Tight coordination is required for effective human-robot teams in domains involving fast dynamics and tactical decisions, such as multi-car racing. In such settings, robot teammates must react to cues of a human teammate's tactical objective to assist in a ... Cite

SAGES consensus recommendations on surgical video data use, structure, and exploration (for research in artificial intelligence, clinical quality improvement, and surgical education).

Journal Article Surg Endosc · November 2023 BACKGROUND: Surgery generates a vast amount of data from each procedure. Particularly video data provides significant value for surgical research, clinical outcome assessment, quality control, and education. The data lifecycle is influenced by various fact ... Full text Link to item Cite

Multi-Abstractive Neural Controller: An Efficient Hierarchical Control Architecture for Interactive Driving

Journal Article IEEE Robotics and Automation Letters · August 1, 2023 As learning-based methods make their way from perception systems to planning/control stacks, robot control systems have started to enjoy the benefits that data-driven methods provide. Because control systems directly affect the motion of the robot, data-dr ... Full text Cite

TEsoNet: knowledge transfer in surgical phase recognition from laparoscopic sleeve gastrectomy to the laparoscopic part of Ivor-Lewis esophagectomy.

Journal Article Surg Endosc · May 2023 BACKGROUND: Surgical phase recognition using computer vision presents an essential requirement for artificial intelligence-assisted analysis of surgical workflow. Its performance is heavily dependent on large amounts of annotated video data, which remain a ... Full text Link to item Cite

MPOGames: Efficient Multimodal Partially Observable Dynamic Games

Conference Proceedings IEEE International Conference on Robotics and Automation · January 1, 2023 Game theoretic methods have become popular for planning and prediction in situations involving rich multi-agent interactions. However, these methods often assume the existence of a single local Nash equilibria and are hence unable to handle uncertainty in ... Full text Cite

Human-Centric Intelligent Driving: Collaborating with the Driver to Improve Safety

Chapter · January 1, 2023 Despite the benefits of autonomous vehicles, their many challenges have made their wide scale deployment and adoption slower than hoped for. In order to help spread the potential benefits of autonomy sooner, as well as to cater to people who will continue ... Full text Cite

Artificial intelligence prediction of cholecystectomy operative course from automated identification of gallbladder inflammation.

Journal Article Surg Endosc · September 2022 BACKGROUND: Operative courses of laparoscopic cholecystectomies vary widely due to differing pathologies. Efforts to assess intra-operative difficulty include the Parkland grading scale (PGS), which scores inflammation from the initial view of the gallblad ... Full text Link to item Cite

SUPR-GAN: SUrgical PRediction GAN for Event Anticipation in Laparoscopic and Robotic Surgery

Journal Article IEEE Robotics and Automation Letters · April 1, 2022 Comprehension of surgical workflow is the foundation upon which artificial intelligence (AI) and machine learning (ML) holds the potential to assist intraoperative decision making and risk mitigation. In this work, we move beyond mere identification of pas ... Full text Cite

Learning an Explainable Trajectory Generator Using the Automaton Generative Network (AGN)

Journal Article IEEE Robotics and Automation Letters · April 1, 2022 Symbolic reasoning is a key component for enabling practical use of data-driven planners in autonomous driving. In that context, deterministic finite state automata (DFA) are often used to formalize the underlying high-level decision-making process. Manual ... Full text Cite

A Deep Concept Graph Network for Interaction-Aware Trajectory Prediction

Conference Proceedings IEEE International Conference on Robotics and Automation · January 1, 2022 Temporal patterns (how vehicles behave in our observed past) underline our reasoning of how people drive on the road, and can explain why we make certain predictions about interactions among road agents. In this paper we propose the ConceptNet trajectory p ... Full text Cite

TIP: Task-Informed Motion Prediction for Intelligent Vehicles

Conference IEEE International Conference on Intelligent Robots and Systems · January 1, 2022 When predicting trajectories of road agents, motion predictors often approximate the future distribution by a limited number of samples. This constraint requires the predictors to generate samples that best support the task given task specifications. Howev ... Full text Cite

HMIway-env: A Framework for Simulating Behaviors and Preferences to Support Human-AI Teaming in Driving

Conference IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · January 1, 2022 We introduce a lightweight simulation and modeling framework, HMIway-env, for studying human-machine teaming in the context of driving. The goal of the framework is to accelerate the development of adaptive AI systems which can respond to individual driver ... Full text Cite

Trajectory Prediction with Linguistic Representations

Conference Proceedings IEEE International Conference on Robotics and Automation · January 1, 2022 Language allows humans to build mental models that interpret what is happening around them resulting in more accurate long-term predictions. We present a novel trajectory prediction model that uses linguistic intermediate representations to forecast trajec ... Full text Cite

Leveraging Smooth Attention Prior for Multi-Agent Trajectory Prediction

Conference Proceedings IEEE International Conference on Robotics and Automation · January 1, 2022 Multi-agent interactions are important to model for forecasting other agents' behaviors and trajectories. At a certain time, to forecast a reasonable future trajectory, each agent needs to pay attention to the interactions with only a small group of most r ... Full text Cite

HYPER: Learned Hybrid Trajectory Prediction via Factored Inference and Adaptive Sampling

Conference Proceedings IEEE International Conference on Robotics and Automation · January 1, 2022 Modeling multi-modal high-level intent is important for ensuring diversity in trajectory prediction. Existing approaches explore the discrete nature of human intent before predicting continuous trajectories, to improve accuracy and support explainability. ... Full text Cite

Challenges in surgical video annotation.

Journal Article Comput Assist Surg (Abingdon) · December 2021 Annotation of surgical video is important for establishing ground truth in surgical data science endeavors that involve computer vision. With the growth of the field over the last decade, several challenges have been identified in annotating spatial, tempo ... Full text Link to item Cite

SAGES consensus recommendations on an annotation framework for surgical video.

Journal Article Surg Endosc · September 2021 BACKGROUND: The growing interest in analysis of surgical video through machine learning has led to increased research efforts; however, common methods of annotating video data are lacking. There is a need to establish recommendations on the annotation of s ... Full text Link to item Cite

Automated operative phase identification in peroral endoscopic myotomy.

Journal Article Surg Endosc · July 2021 BACKGROUND: Artificial intelligence (AI) and computer vision (CV) have revolutionized image analysis. In surgery, CV applications have focused on surgical phase identification in laparoscopic videos. We proposed to apply CV techniques to identify phases in ... Full text Link to item Cite

CARPAL: Confidence-Aware Intent Recognition for Parallel Autonomy

Journal Article IEEE Robotics and Automation Letters · July 1, 2021 Predicting driver intentions is a difficult and crucial task for advanced driver assistance systems. Traditional confidence measures on predictions often ignore the way predicted trajectories affect downstream decisions for safe driving. In this letter, we ... Full text Cite

Computer vision in surgery.

Journal Article Surgery · May 2021 The fields of computer vision (CV) and artificial intelligence (AI) have undergone rapid advancements in the past decade, many of which have been applied to the analysis of intraoperative video. These advances are driven by wide-spread application of deep ... Full text Link to item Cite

Vehicle Trajectory Prediction Using Generative Adversarial Network with Temporal Logic Syntax Tree Features

Journal Article IEEE Robotics and Automation Letters · April 1, 2021 In this work, we propose a novel approach for integrating rules into traffic agent trajectory prediction. Consideration of rules is important for understanding how people behave-yet, it cannot be assumed that rules are always followed. To address this chal ... Full text Cite

Aggregating Long-Term Context for Learning Laparoscopic and Robot-Assisted Surgical Workflows

Conference Proceedings IEEE International Conference on Robotics and Automation · January 1, 2021 Analyzing surgical workflow is crucial for surgical assistance robots to understand surgeries. With the understanding of the complete surgical workflow, the robots are able to assist the surgeons in intra-operative events, such as by giving a warning when ... Full text Cite

Risk Conditioned Neural Motion Planning

Conference IEEE International Conference on Intelligent Robots and Systems · January 1, 2021 Risk-bounded motion planning is an important yet difficult problem for safety-critical tasks. While existing mathematical programming methods offer theoretical guarantees in the context of constrained Markov decision processes, they either lack scalability ... Full text Cite

MAAD: A Model and Dataset for "Attended Awareness"in Driving

Conference Proceedings of the IEEE International Conference on Computer Vision · January 1, 2021 We propose a computational model to estimate a person's attended awareness of their environment. We define "attended awareness"to be those parts of a potentially dynamic scene which a person has attended to in recent history and which they are still likely ... Full text Cite

Behaviorally diverse traffic simulation via reinforcement learning

Conference IEEE International Conference on Intelligent Robots and Systems · October 24, 2020 Traffic simulators are important tools in autonomous driving development. While continuous progress has been made to provide developers more options for modeling various traffic participants, tuning these models to increase their behavioral diversity while ... Full text Cite

Driving through ghosts: Behavioral cloning with false positives

Conference IEEE International Conference on Intelligent Robots and Systems · October 24, 2020 Safe autonomous driving requires robust detection of other traffic participants. However, robust does not mean perfect, and safe systems typically minimize missed detections at the expense of a higher false positive rate. This results in conservative and y ... Full text Cite

Deep Context Maps: Agent Trajectory Prediction Using Location-Specific Latent Maps

Journal Article IEEE Robotics and Automation Letters · October 1, 2020 In this letter, we propose a novel approach for agent motion prediction in cluttered environments. One of the main challenges in predicting agent motion is accounting for location and context-specific information. Our main contribution is the concept of le ... Full text Cite

DiversityGAN: Diversity-Aware Vehicle Motion Prediction via Latent Semantic Sampling

Journal Article IEEE Robotics and Automation Letters · October 1, 2020 Vehicle trajectory prediction is crucial for autonomous driving and advanced driver assistant systems. While existing approaches may sample from a predicted distribution of vehicle trajectories, they lack the ability to explore it-a key ability for evaluat ... Full text Cite

Discovering Avoidable Planner Failures of Autonomous Vehicles using Counterfactual Analysis in Behaviorally Diverse Simulation

Conference 2020 IEEE 23rd International Conference on Intelligent Transportation Systems ITSC 2020 · September 20, 2020 Automated Vehicles require exhaustive testing in simulation to detect as many safety-critical failures as possible before deployment on public roads. In this work, we focus on the core decision-making component of autonomous robots: their planning algorith ... Full text Cite

Artificial Intelligence in Anesthesiology: Current Techniques, Clinical Applications, and Limitations.

Journal Article Anesthesiology · February 2020 Artificial intelligence has been advancing in fields including anesthesiology. This scoping review of the intersection of artificial intelligence and anesthesia research identified and summarized six themes of applications of artificial intelligence in ane ... Full text Link to item Cite

Differentiable Logic Layer for Rule Guided Trajectory Prediction

Conference Proceedings of Machine Learning Research · January 1, 2020 In this work, we propose a method for integration of temporal logic formulas into a neural network. Our main contribution is a new logic optimization layer that uses differentiable optimization on the formulas' robustness function. This allows incorporatin ... Cite

Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving

Conference Robotics Science and Systems · January 1, 2020 Autonomous driving has achieved significant progress in recent years, but autonomous cars are still unable to tackle high-risk situations where a potential accident is likely. In such near-accident scenarios, even a minor change in the vehicle’s actions ma ... Full text Cite

Persistent Surveillance of Events with Unknown Rate Statistics

Chapter · January 1, 2020 We present a novel algorithm for persistent monitoring of stochastic events that occur at discrete locations in the environment with unknown event rates. Prior research on persistent monitoring assumes knowledge of event rates, which is often not the case ... Full text Cite

MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control

Conference Proceedings of Machine Learning Research · January 1, 2020 Reasoning about human motion is a core component of modern human-robot interactive systems. In particular, one of the main uses of behavior prediction in autonomous systems is to inform robot motion planning and control. However, a majority of planning and ... Cite

RGBD-fusion: Depth refinement for diffuse and specular objects

Chapter · January 1, 2020 The popularity of low-cost RGB-D scanners is increasing on a daily basis and has set off a major boost in 3D computer vision research. Nevertheless, commodity scanners often cannot capture subtle details in the environment. In other words, the precision of ... Full text Cite

Infrastructure-free NLoS Obstacle Detection for Autonomous Cars

Conference IEEE International Conference on Intelligent Robots and Systems · November 1, 2019 Current perception systems mostly require direct line of sight to anticipate and ultimately prevent potential collisions at intersections with other road users. We present a fully integrated autonomous system capable of detecting shadows or weak illuminati ... Full text Cite

Probabilistic Risk Metrics for Navigating Occluded Intersections

Journal Article IEEE Robotics and Automation Letters · October 1, 2019 Among traffic accidents in the USA, 23% of fatal and 32% of non-fatal incidents occurred at intersections. For driver assistance systems, intersection navigation remains a difficult problem that is critically important to increasing driver safety. In this ... Full text Cite

Computer Vision Analysis of Intraoperative Video: Automated Recognition of Operative Steps in Laparoscopic Sleeve Gastrectomy.

Journal Article Ann Surg · September 2019 OBJECTIVE(S): To develop and assess AI algorithms to identify operative steps in laparoscopic sleeve gastrectomy (LSG). BACKGROUND: Computer vision, a form of artificial intelligence (AI), allows for quantitative analysis of video by computers for identifi ... Full text Link to item Cite

Uncertainty-aware driver trajectory prediction at urban intersections

Conference Proceedings IEEE International Conference on Robotics and Automation · May 1, 2019 Predicting the motion of a driver's vehicle is crucial for advanced driving systems, enabling detection of potential risks towards shared control between the driver and automation systems. In this paper, we propose a variational neural network approach tha ... Full text Cite

Variational end-to-end navigation and localization

Conference Proceedings IEEE International Conference on Robotics and Automation · May 1, 2019 Deep learning has revolutionized the ability to learn 'end-to-end' autonomous vehicle control directly from raw sensory data. While there have been recent extensions to handle forms of navigation instruction, these works are unable to capture the full dist ... Full text Cite

Variational Autoencoder for End-to-End Control of Autonomous Driving with Novelty Detection and Training De-biasing

Conference IEEE International Conference on Intelligent Robots and Systems · December 27, 2018 This paper introduces a new method for end-to-end training of deep neural networks (DNNs) and evaluates it in the context of autonomous driving. DNN training has been shown to result in high accuracy for perception to action learning given sufficient train ... Full text Cite

ShadowCam: Real-Time Detection of Moving Obstacles behind A Corner for Autonomous Vehicles

Conference IEEE Conference on Intelligent Transportation Systems Proceedings ITSC · December 7, 2018 Moving obstacles occluded by corners are a potential source for collisions in mobile robotics applications such as autonomous vehicles. In this paper, we address the problem of anticipating such collisions by proposing a vision-based detection algorithm fo ... Full text Cite

Surgical Video in the Age of Big Data.

Journal Article Ann Surg · December 2018 Full text Link to item Cite

Task-Specific Sensor Planning for Robotic Assembly Tasks

Conference Proceedings IEEE International Conference on Robotics and Automation · September 10, 2018 When performing multi-robot tasks, sensory feedback is crucial in reducing uncertainty for correct execution. Yet the utilization of sensors should be planned as an integral part of the task planning, taken into account several factors such as the toleranc ... Full text Cite

Information-Driven Adaptive Structured-Light Scanners

Journal Article IEEE Transactions on Computational Imaging · September 2018 Full text Cite

Artificial Intelligence in Surgery: Promises and Perils.

Journal Article Ann Surg · July 2018 OBJECTIVE: The aim of this review was to summarize major topics in artificial intelligence (AI), including their applications and limitations in surgery. This paper reviews the key capabilities of AI to help surgeons understand and critically evaluate new ... Full text Link to item Cite

The Manhattan Frame Model-Manhattan World Inference in the Space of Surface Normals.

Journal Article IEEE Trans Pattern Anal Mach Intell · January 2018 Objects and structures within man-made environments typically exhibit a high degree of organization in the form of orthogonal and parallel planes. Traditional approaches utilize these regularities via the restrictive, and rather local, Manhattan World (MW) ... Full text Link to item Cite

Hybrid control and learning with coresets for autonomous vehicles

Conference IEEE International Conference on Intelligent Robots and Systems · December 13, 2017 Modern autonomous systems such as driverless vehicles need to safely operate in a wide range of conditions. A potential solution is to employ a hybrid systems approach, where safety is guaranteed in each individual mode within the system. This offsets comp ... Full text Cite

On the Role of Representations for Reasoning in Large-Scale Urban Scenes

Conference IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · August 22, 2017 The advent of widely available photo collections covering broad geographic areas has spurred significant advances in large-scale urban scene modeling. While much emphasis has been placed on reconstruction and visualization, the utility of such models exten ... Full text Cite

Machine learning and coresets for automated real-time video segmentation of laparoscopic and robot-assisted surgery

Conference Proceedings IEEE International Conference on Robotics and Automation · July 21, 2017 Context-aware segmentation of laparoscopic and robot assisted surgical video has been shown to improve performance and perioperative workflow efficiency, and can be used for education and time-critical consultation. Modern pressures on productivity preclud ... Full text Cite

Persistent surveillance of events with unknown, time-varying statistics

Conference Proceedings IEEE International Conference on Robotics and Automation · July 21, 2017 We consider the problem of monitoring stochastic, time-varying events occurring at discrete locations. Our problem formulation extends prior work in persistent surveillance by considering the objective of maximizing event detections in unknown, dynamic env ... Full text Cite

Duckietown: An open, inexpensive and flexible platform for autonomy education and research

Conference Proceedings IEEE International Conference on Robotics and Automation · July 21, 2017 Duckietown is an open, inexpensive and flexible platform for autonomy education and research. The platform comprises small autonomous vehicles ('Duckiebots') built from off-the-shelf components, and cities ('Duckietowns') complete with roads, signage, traf ... Full text Cite

Information-driven adaptive structured-light scanners

Conference Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition · December 9, 2016 Sensor planning and active sensing, long studied in robotics, adapt sensor parameters to maximize a utility function while constraining resource expenditures. Here we consider information gain as the utility function. While these concepts are often used to ... Full text Cite

Real-Time Depth Refinement for Specular Objects

Conference Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition · December 9, 2016 The introduction of consumer RGB-D scanners set off a major boost in 3D computer vision research. Yet, the precision of existing depth scanners is not accurate enough to recover fine details of a scanned object. While modern shading based depth refinement ... Full text Cite

RGBD-fusion: Real-time high precision depth recovery

Conference Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition · October 14, 2015 The popularity of low-cost RGB-D scanners is increasing on a daily basis. Nevertheless, existing scanners often cannot capture subtle details in the environment. We present a novel method to enhance the depth map by fusing the intensity and depth informati ... Full text Cite

Multi-region active contours with a single level set function

Journal Article IEEE Transactions on Pattern Analysis and Machine Intelligence · August 1, 2015 Segmenting an image into an arbitrary number of coherent regions is at the core of image understanding. Many formulations of the segmentation problem have been suggested over the past years. These formulations include, among others, axiomatic functionals, ... Full text Cite

Coresets for visual summarization with applications to loop closure

Conference Proceedings IEEE International Conference on Robotics and Automation · June 29, 2015 In continuously operating robotic systems, efficient representation of the previously seen camera feed is crucial. Using a highly efficient compression coreset method, we formulate a new method for hierarchical retrieval of frames from large video streams ... Full text Cite

Fleye on the car: Big data meets the Internet of Things

Conference IPSN 2015 Proceedings of the 14th International Symposium on Information Processing in Sensor Networks Part of Cps Week · April 13, 2015 Vehicle-based vision algorithms, such as the collision alert systems [4], are able to interpret a scene in real-time and provide drivers with immediate feedback. However, such technologies are based on cameras on the car, limited to the vicinity of the car ... Full text Cite

A mixture of Manhattan frames: Beyond the Manhattan world

Conference Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition · September 24, 2014 Objects and structures within man-made environments typically exhibit a high degree of organization in the form of orthogonal and parallel planes. Traditional approaches to scene representation exploit this phenomenon via the somewhat restrictive assumptio ... Full text Cite

Aerial reconstructions via probabilistic data fusion

Conference Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition · September 24, 2014 We propose an integrated probabilistic model for multi-modal fusion of aerial imagery, LiDAR data, and (optional) GPS measurements. The model allows for analysis and dense reconstruction (in terms of both geometry and appearance) of large 3D scenes. An adv ... Full text Cite

Fast regularization of matrix-valued images

Conference Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics · January 1, 2014 Regularization of matrix-valued data is important in many fields, such as medical imaging, motion analysis and scene understanding, where accurate estimation of diffusion tensors or rigid motions is crucial for higher-level computer vision tasks. In this c ... Full text Cite

Coresets for k-segmentation of streaming data

Conference Advances in Neural Information Processing Systems · January 1, 2014 Life-logging video streams, financial time series, and Twitter tweets are a few examples of high-dimensional signals over practically unbounded time. We consider the problem of computing optimal segmentation of such signals by a k-piecewise linear function ... Cite

Perspective photometric stereo with shadows

Conference Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics · September 25, 2013 High resolution reconstruction of 3D surfaces from images remains an active area of research since most of the methods in use are based on practical assumptions that limit their applicability. Furthermore, an additional complication in all active illuminat ... Full text Cite

Active contours for multi-region image segmentation with a single level set function

Conference Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics · September 25, 2013 Segmenting the image into an arbitrary number of parts is at the core of image understanding. Many formulations of the task have been suggested over the years. Among these are axiomatic functionals, which are hard to implement and analyze, while graph-base ... Full text Cite

On globally optimal local modeling: From moving least squares to over-parametrization

Chapter · January 1, 2013 This paper discusses a variational methodology, which involves locally modeling of data from noisy samples, combined with global model parameter regularization. We show that this methodology encompasses many previously proposed algorithms, from the celebra ... Full text Cite

Group-valued regularization for motion segmentation of articulated shapes

Chapter · January 1, 2013 Motion-based segmentation is an important tool for the analysis of articulated shapes. As such, it plays an important role in mechanical engineering, computer graphics, and computer vision. In this chapter, we study motion-based segmentation of 3D articula ... Full text Cite

Patch-collaborative spectral point-cloud denoising

Journal Article Computer Graphics Forum · January 1, 2013 We present a new framework for point cloud denoising by patch-collaborative spectral analysis. A collaborative generalization of each surface patch is defined, combining similar patches from the denoised surface. The Laplace-Beltrami operator of the collab ... Full text Cite

Polyakov action minimization for efficient color image processing

Conference Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics · December 20, 2012 The Laplace-Beltrami operator is an extension of the Laplacian from flat domains to curved manifolds. It was proven to be useful for color image processing as it models a meaningful coupling between the color channels. This coupling is naturally expressed ... Full text Cite

Articulated motion segmentation of point clouds by group-valued regularization

Conference Eurographics Workshop on 3D Object Retrieval Eg 3dor · December 1, 2012 Motion segmentation for articulated objects is an important topic of research. Yet such a segmentation should be as free as possible from underlying assumptions so as to fit general scenes and objects. In this paper we demonstrate an algorithm for articula ... Full text Cite

Sparse modeling of shape from structured light

Conference Proceedings 2nd Joint 3dim 3dpvt Conference 3D Imaging Modeling Processing Visualization and Transmission 3dimpvt 2012 · December 1, 2012 Structured light depth reconstruction is among the most commonly used methods for 3D data acquisition. Yet, in most structured light methods, modeling of the acquired scene is crude, and is executed separately from the decoding phase. Here, we bridge this ... Full text Cite

Patch-space Beltrami denoising of 3D point clouds

Conference 2012 IEEE 27th Convention of Electrical and Electronics Engineers in Israel Ieeei 2012 · December 1, 2012 The Beltrami framework has been shown to be an effective and efficient denoising filter for color images, treating them as two dimensional manifolds embedded in a hybrid spatial-spectral space. Recent work using this framework on the patchspace of an image ... Full text Cite

Fast regularization of matrix-valued images

Conference Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics · October 30, 2012 Regularization of images with matrix-valued data is important in medical imaging, motion analysis and scene understanding. We propose a novel method for fast regularization of matrix group-valued images. Using the augmented Lagrangian framework we separate ... Full text Cite

Over-parameterized optical flow using a stereoscopic constraint

Conference Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics · January 16, 2012 The success of variational methods for optical flow computation lies in their ability to regularize the problem at a differential (pixel) level and combine piecewise smoothness of the flow field with the brightness constancy assumptions. However, the piece ... Full text Cite

Group-valued regularization framework for motion segmentation of dynamic non-rigid shapes

Conference Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics · January 16, 2012 Understanding of articulated shape motion plays an important role in many applications in the mechanical engineering, movie industry, graphics, and vision communities. In this paper, we study motion-based segmentation of articulated 3D shapes into rigid pa ... Full text Cite

Group-valued regularization for analysis of articulated motion

Conference Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics · January 1, 2012 We present a novel method for estimation of articulated motion in depth scans. The method is based on a framework for regularization of vector- and matrix- valued functions on parametric surfaces. We extend augmented-Lagrangian total variation regularizati ... Full text Cite

On semi-implicit splitting schemes for the beltrami color image filtering

Journal Article Journal of Mathematical Imaging and Vision · June 1, 2011 The Beltrami flow is an efficient nonlinear filter, that was shown to be effective for color image processing. The corresponding anisotropic diffusion operator strongly couples the spectral components. Usually, this flow is implemented by explicit schemes, ... Full text Cite

Augmented Lagrangian for Polyakov action minimization in color images

Conference Aip Conference Proceedings · December 1, 2010 Full text Cite

Nonlinear dimensionality reduction by topologically constrained isometric embedding

Journal Article International Journal of Computer Vision · August 1, 2010 Many manifold learning procedures try to embed a given feature data into a flat space of low dimensionality while preserving as much as possible the metric in the natural feature space. The embedding process usually relies on distances between neighboring ... Full text Cite

On reconstruction of non-rigid shapes with intrinsic regularization

Conference 2009 IEEE 12th International Conference on Computer Vision Workshops Iccv Workshops 2009 · December 1, 2009 Shape-from-X is a generic type of inverse problems in computer vision, in which a shape is reconstructed from some measurements. A specially challenging setting of this problem is the case in which the reconstructed shapes are non-rigid. In this paper, we ... Full text Cite

On semi-implicit splitting schemes for the beltrami color flow

Conference Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics · August 27, 2009 The Beltrami flow is an efficient non-linear filter, that was shown to be effective for color image processing. The corresponding anisotropic diffusion operator strongly couples the spectral components. Usually, this flow is implemented by explicit schemes ... Full text Cite

Efficient Beltrami image filtering via vector extrapolation methods

Journal Article SIAM Journal on Imaging Sciences · January 1, 2009 The Beltrami image flow is an effective nonlinear filter, often used in color image processing. It was shown to be closely related to the median, total variation, and bilateral filters. It treats the image as a two-dimensional manifold embedded in a hybrid ... Full text Cite

A new physically motivated warping model for form drop-out

Conference Proceedings of the International Conference on Document Analysis and Recognition ICDAR · December 1, 2007 Documents scanned by sheet-fed scanners often exhibit distortions due to the feeding and scanning mechanism. This paper presents a new model, motivated by the distortions observed in such documents. Numerical problems affecting the use of this model are ad ... Full text Cite

Efficient Beltrami filtering of color images via vector extrapolation

Conference Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics · January 1, 2007 The Beltrami image flow is an effective non-linear filter, often used in color image processing. It was shown to be closely related to the median, total variation, and bilateral filters. It treats the image as a 2D manifold embedded in a hybrid spatial-fea ... Full text Cite