Intraoperative Visual Guidance and Control Interface for Augmented Reality Robotic Surgery

Augmented Reality (AR) is a natural development from virtual reality (VR), which was developed several decades earlier. AR complements VR in many ways. Due to the advantages of the user being able to see both the real and virtual objects simultaneously, AR is far more intuitive, but it's not completely detached from human factors and other restrictions. AR doesn't consume as much time and effort in the applications because it's not required to construct the entire virtual scene and the environment. In this book, several new and emerging application areas of AR are presented and divided into three sections. The first section contains applications in outdoor and mobile AR, such as construction, restoration, security and surveillance. The second section deals with AR in medical, biological, and human bodies. The third and final section contains a number of new and useful applications in daily living and learning.


Introduction
Minimally invasive surgery (MIS) with its small artificial incisions imposed on the patient skin for operation is becoming a preferred surgical treatment comparing with the conventional open surgery.The open surgeries usually create large traumas on the patient bodies to expose their internal anatomy for operational access.The MIS offers its distinct advantages of small injuries, less blood loss and pain and faster recovery during the surgery procedures (Puangmali et al., 2008).However, the indirectly accessed operating causes problems such as restricted vision and difficult hand-eye coordination through the small key holes.The recent development of the modern optics, computer graphics, computer vision and robotics provides possible chances to resolve the problems described above.
Virtual Reality (VR) is a technology providing users an artificial environment to simulate the physical world and objects for displaying, training or gaming etc. Users are allowed to interact with the VR environment through human-computer interface that consists of computer graphic models and various sensors.The major applications of virtual reality in surgery can be divided into three areas: virtual humans for training, virtual telemedicine shared decision environments for training of multiple players and the fusion of virtual humans with real humans for performing surgery (Rosen et al., 1996).VR can create individualized treatment needs and training protocols by providing an environment in which the intensity of feedback and training can be systematically manipulated and enhanced (Anderson et al., 2002;Wang et al., 1999;Cai et al., 2003).VR simulators also allow users to compare their performance with that of their peers (Pearson et al., 2002;Watterson et al., 2002).VR assisted surgical planning facilitates preoperative and postoperative diagnosis with sufficient details of 3-D medical models which are commonly constructed from the computed tomography (CT) or magnetic resonance (MR) images.For example, surgical planning was conducted using a three-dimensional VR based interface to provide a quantitative osteotomy-simulated bone model and prediction of postoperative appearance with photorealistic quality (Xia, et al., 2001).Additionally, there were medical researches based on VR technology specifically conducted to assist surgeons during neurologic, orthopedic, arthroscopic, and urologic surgery (Akay & Marsh, 2001;Burdea et al., 2000;Heng et al., 2004;Kuo et al., 2001;Chui et al., 2006).Recently, there is an increasing emphasis on data fusion involving integration of virtual patient with real patient as a navigational aid in surgery (Rosen et al., 1996;Wen et al., 2010).
Augmented reality (AR) does not rely solely on artificially created virtual environments but expands the virtual computer-generated models with real world environment.AR is characterized as being a fusion of real and virtual data within a real world environment (Milgram et al., 1994).Medical augmented reality is proposed to create the virtual scenes on the related physical tissues, organs or patient skin to assist surgeons visualizing the internal anatomy or invisible surgical planning such as operational distance, trajectories etc.. Due to the advancement in tracking, visualization and display technology, computer-aided medical procedure based AR solutions was examined in the context of minimally invasive surgery (Navab et al., 2007).AR system has been developed to enhance the endoscopic or laparoscopic view to extend surgeons' visibility of hidden anatomical structures that were beneath the then surgical scene.This augmentation facilitates the surgical operation to avoid risk regions like arteries and nerves.On the other hand the pathology regions are easily to be reached.The needle's insertion happened within the target organ was guided to reach the tumor volume (Konishia et al., 2005;Thoranaghatte et al., 2008;Lee et al., 2010).Thus a more intuitive visual assistance during the surgery procedure is provided for surgeon's response.Integration of laparoscopic and 3D-ultrosound images that provided anatomical information inside the target organs was studied by Konishia et al. (Konishia et al., 2005).With head-mounted display (HMD) based AR interface, numerous studies have demonstrated the potential efficiency of MRI-guided tumour extraction and therapy (Liao et al., 2010).Projector-based visualization system for clinically intraoperative navigation was reported (Hoppe, 2003).Grimson et al. (Grimson et al., 1996) showed its potentials to surgical navigation systems that enhance physical scenes with the augmentation of overlaying virtual internal structures.Another medical application of marking the surgical object with a virtual-pen was based on direct AR interface (Seo et al., 2007).It was proposed to supersede the ink pen which was widely used in the surgical environment.With the virtual-pen, the surgical markings were achieved by directly projecting them onto human body.The introduction of AR to surgical treatment creates a virtual medium between preoperative plan and intraoperative environment.
In order to enhance manual dexterity and accuracy of instrument manipulation, robotic assistance has been introduced in the MIS to solve operation constraint problems.The da Vinci™ Surgical System, a master-slave integrated robot, has been applied for treating a range of surgical conditions improving operational flexibility and accuracy (Guthart & Salisbury, 2000).For the sophisticated minimally invasive operations like coronary artery bypass and mitral valve repair, robot assisted surgery is expected to achieve satisfying clinical outcomes (Devernay et al., 2001).Surgical robot also demonstrated its distinct advantages in speed, accuracy and consistency for the larger liver tumor treatment with RFA.Yang et al. investigated design and implementation of a novel robotic system for RFA considering both kinematics and clinical requirements (Yang et al., 2010).Other advantages offered by the robotic RFA include minimizing radioactive exposure and overcoming visual restriction as well as spatial constraint for minimally invasive approaches like the laparoscopic or percutaneous surgical procedure.Integrating the imaging guidance and the surgical robot assistance into minimally invasive operation brings the preoperative planning and intraoperative manipulation into perspective.

Spatial AR interface for robot assisted MIS
Depending on different ways of augmenting the environment, there are three approaches in AR display devices: Optical See Through devices, Video See Through devices, and Direct Augmentations.Direct Augmentations is also called spatial augmented reality (SAR) using projector-camera system to present the virtual information directly in the 3D real world environment (Raskar et al., 2001;Bimber & Raskar, 2004).Based on the above different approaches of augmentations, four kinds of AR platform are popularly used in present surgical simulation and treatment: head-mounted display (HMD) based AR (Cakmakci & Rolland, 2006), semi-transparent mirror based AR (Liao et al., 2010) and projector based AR (Bimber & Raskar, 2004).
Comparing with the other augmentation approaches, SAR offers the distinct advantages that allow users not wearing heavy helmet, overcoming the limited vision area, sharing the AR intraoperative information and communicating freely with their fellow users.The projector-based AR, which is considered to be the most natural way combining the virtual environment and real environment together, is expected to serve as an effective integrated interface to overcome the aforementioned constraints of the MIS.In addition, the surgical robot and interactive hand gesture operation, considered as dynamic components in the real environment, can be easily integrated in this AR interface for robot assisted MIS surgery.
The general work flow using this advanced user interface is illustrated in Fig. 1.It first involves the handling of 3D models that can be derived from the CT images and the preoperative surgical planning.The construction of the static 3D AR environment includes Augmented Reality -Some Emerging Application Areas 194 registration, geometric and radiometric correction.There is an iterative process with AR real-time model rendering and feedback from the real world environment, including hand gesture interaction with the AR model, surgical robot and the feedback of signal from related medical devices during the surgical operation.When the AR model, displayed over the real organ of the patient by the projector-camera system, receives any "stimulus" from the real environment, there will be changes responding in real time on the model to guide the surgeons' decisions and operation.

Registration
Registration is a main challenge in AR technology (Pablo et al., 2010).Correct and consistent registration method plays an important role in estimating the position of either the virtual objects relative to the object in the real world or of the virtual objects relative to a common coordinate system.A number of registration methods have been developed ranging from analysis of the real world environment to evaluations of intentionally introduced fiducials (Massoptier & Casciaro, 2008;Teber et al., 2009).
Marker based methods are commonly used to aid intraoperative registration in robot assisted surgeries (Yang et al., 2010).A marker based registration method for projectorcamera system is introduced in this chapter.With the markers attached onto the patient's skin, spatial relationship between the markers and target organ can be derived from CT image of the patient.As shown in Fig. 2, three markers M 1, M 2, M 3, are attached onto the patient body to construct a space reference fame T. This marker based frame is used to construct the spatial relationship that connects the stereovision frame, world coordinate and CT image frame.The spatial relationship enables the virtual model displayed by the projector-camera system to register with the target organ within the patient body.The two steps to overlay a virtual model of the target organ precisely on the patient with correct position, scale and angle in the projector-camera system are described as follows.
The first step involves establishing the spatial relationship between the markers M i (i=1, 2, 3) and the target organ O. CT imaging of the patient with three markers attached is performed prior to the surgery.Appropriate markers deployment is important.The markers should facilitate the registration and avoid inconveniencing the surgery.Their deployment is dependent on the following considerations.Firstly, the surgical tools' motion should not be obstructed by the markers.The surgical path or motion is given a higher priority in preoperative planning compared to that of markers' deployment.Since the markers are used primarily for constructing the space reference frame T, this can be achieved as long as the stereovision system can detect the markers during registration.Preoperative planning including obstruction prevention will be discussed in section 5.1.Secondly, the markers should be placed beyond the expected projection region to avoid violating the projected virtual model.
For the second step, the initial calibration of projector-camera system has been described by Bimber & Raskar (Bimber & Raskar, 2004).It establishes the correspondence between the pixels in camera and projector as a lookup table.With the stereo vision system, the reference frame T consisted of three markers is achieved from the estimation of space coordinate of the three markers.As mentioned above the spatial relationship between the markers and the target organ based on the image coordinate is derived from the CT images.In this case, the spatial relationship between target organ and the projected virtual model is established as Eq. ( 1 (1) Since the coordinate of the top-left corner L 2 of the projected image is easily detected, the position of L 2 can be adjusted to its expected coordinate.Assume that the L 1 and L 3 are the top-left corner of the source image and the captured image respectively, and L 1 '(x, y) and L 3 '(x, y) are the expected position of the top-left corner of the source image and the captured image respectively.The expected coordinates of the projected image can be acquired by calculating the spatial relationship with the homography between the camera and the world coordinate.The relationship between L 1 ' (x, y) and L 3 ' (x, y) is shown as Eq. ( 2) and Eq. ( 3): where is a affine transformation.A testing projection is conducted to detect the differences between L 3 and L' 3 .With the piecewise linear method described above, and can be replaced by the and for corresponding piecewise regions.
Suppose that the patient is lying on a surgical bed and the camera is placed along the vertical axis from the target organ relative to the surgical bed.A virtual organ model with the same size and angle located as the real one is expected to be projected on the patient body.This bird-eye view of the image of the virtual model can be derived from the CT images.Finally, the geometric correction for the projected image is conducted to register the virtual model with the real target organ.

Correction for projection
In order to create an overall projected image that appears correct and seamless on arbitrary surfaces in the real world, geometric correction plays an important role in construction of the SAR environment.This task is challenging for projector-based displays because of the following problems: 1. Perspective distortion arising from projection. 2. Geometric distortion arising from lens distortion, arbitrary physical surfaces, etc.. 3. Real-time geometric correction.This is a challenging problem and it may occur during the dynamic projection in spatial space in which screen surface is changing in real-time because of variance of projector's location or angle.
In addition, the display quality of the projected image is modulated by the colored and textured arbitrary surface sacrificing its photometric quality.Human eyes are sensitive to this artifact.Radiometric correction can be used to relax this restriction with the SAR system by compensating the spatially varying reflectance.

Geometric correction
Assume the screen surface is planar and projection light is perpendicularly to the surface.The projection on the screen surface appears hardly distorted to the observers in front of the screen.When an image is projected on a planar screen surface that is not perpendicular to it, the projective distortion corrupts the projected image.When an image is projected onto a screen surface with irregular unevenness, geometric factor of the screen surfaces also affects distortion of the projected image.
For a planar projection, homography is used to represent the transformation from the source image to the projected image in the world coordinate, from the projected image to the captured image in the camera as Eq. ( 4) and Eq. ( 5): (w x , w y , 1) T is an image point projected on a planar surface.(p x , p y , 1) T and (c x , c y , 1) T are the corresponding points located in the projector's and camera's respectively.From H p and H c , a relation from the source image to the captured image can be established in Eq. ( 6): The geometric correction process is illustrated in Fig. 3.There are two types of correction models: viewer dependent and viewer independent.For viewer-dependent correction, the source image should be pre-wrapped as T -1 S, where S is the source image and T is the mapping function between the source image and the captured image.For viewerindependent correction, the source image should be wrapped as P -1 S, where P is the mapping function between the source image and the projected image.Eq. ( 7) and Eq. ( 8) illustrate the correction process: www.intechopen.comAlthough Eq. ( 6) is not directly applicable when the screen is non-flat, the function H p H c −1 can be found easily.An idea of replacing H p H c −1 by piecewise linear mapping function that relates the camera and the projector is proposed in this research.With the piecewise linear method, the source image can be marked piecewise by defined feature points.
Consequently, the source image can be operated "piece" by "piece", in which the correction of projected image on non-planar screen surface is approximated by connecting all these piece regions.Quadrilateral piece is used as unit region to compute the homography between source image and projected image, captured image and source image.The region connection could be compensated by bilinear interpolation.
The point-correspondence between the image pairs could be acquired from the result of registration.For every piece region, four feature points are established with the respective relationship defined by Eq. ( 9), ( 10) and (11).i is the i th piecewise region in the source image, projected image and the captured image.The other parameters have the same meaning as that of the equation ( 4), ( 5), ( 6).www.intechopen.com An example of piecewise geometric correction for chessboard projection is shown as Fig. 4. From Fig. 4, we can observe that the number and location of the selected feature points play important role on the final result.

Radiometric correction
Radiometry distortion is due to angle of projection, screen texture and the environment lightings.Photometric models of interaction with spectra reflectance between projector and camera were proposed by S. K. Nayar et al. (Nayar et al., 2003) and K. Fujii et al. (Fujii et al., 2005).Based on the photometric models, a practical method to apply radiometric compensation by the projector-camera system is presented in this section.Image C = [C R C G C B ] T acquired by camera from the projected image P = [P R P G P B ] T is affected by surface reflectance and environment lighting.Assume projector and camera each have three color channels (R, G, B).The equation ( 12) illustrates the radiometric transformation from projector to camera via the screen surface based on the above photometric models: where : The vector F is the reflectance of surface.E is the contribution of the environment lighting to the captured image.The color mixing matrix M describes the radiometric interaction between the projector and the camera.
In comparison with the brightness of environment lighting, projector is considerably brighter when performing the projection on the textured and colored surface (Fujii et al., 2005).The color mixing matrix M can be derived by projecting two gray images with gray values of 255 and 0 respectively for monotonic radiometric calibration.To compute the M .By comparing of images with differences in one of the three channels, the other elements in the M ij are acquired for the corresponding channel: Based on the piecewise linear method described in the section A, the matrix F, E and M can be replaced by F i , E i and M i as ( 9), ( 10) and (11).

Therapeutic model design
The dynamic augmented virtual model displayed on the AR interface during surgery is derived from the therapeutic model developed in the preoperative phase.The preoperative plan is based on medical diagnosis, and includes the therapeutic model and surgical planning.The therapeutic model is a stereoscopic cyber-graphic model reconstructed from CT images with relevant organs and tissues segmented.Besides the three dimensional profile of the target tumour and blood vessels, other important circumjacent tissues and organs around the interested region are included in the model as specified by the surgeons.Additionally, markers deployment should also be examined for this therapeutic model design to make sure they are beyond the surgical working path.For the therapeutic model design based on the robot assisted surgery, optimization algorithms are usually applied to enhance the efficiency of robotic implementation and to decrease complexity during the surgery.For example, an ablation model with a "Voxel Growing" algorithm (Yang et al., 2009) is applied for a prototyped robot assisted radiofrequency ablation (RFA) surgery.This algorithm automatically adapts the different tumour's irregular 3-D profile to produce the ablation points.In order to further reduce the number of required ablation and minimize burning of healthy tissue, the "grown region" is optimally aligned to best fit the geometry of the target region.The input for the ablation model is the surface data of the tumour, which are acquired from CT scan in advance.Surgical path planning is another important consideration for the preoperative planning.This method involves the construction of spherical elements defined by inscribed cubes.
Each sphere represents the ablation area, and its inscribed cube represents a voxel element.
Voxel elements propagate layer by layer along defined directions according to its surface edges of the present layer, covers the entire tumour area coinciding with its contour.As voxel elements grow, the coordinates of centres of every voxel elements are produced and being assigned as the ablation coordinates.After the therapeutic model and the surgical plan are prepared, the model is projected onto the patient in the spatial AR environment.

Intraoperative tracking and hand gesture recognition
Intraoperative tracking is to establish the real-time spatial awareness of medical instrument navigation and hand motion necessary for visual guidance and intraoperative control during the surgery.Intraoperative tracking of the operating trajectory of the medical instrument provides surgeons real-time feedback of the operation procedure.This real-time feedback information is usually analysed and dealt with in consideration of the intraoperative process as well as the previous medical preplanning.For example in RFA surgery, when the RFA needle is inserted into the patient body, the distal end of needle is tracked to detect the needle's trajectory.The AR environment is updated in real-time producing a virtual needle part, which is projected on the patient surface, following the needle's treatment plan.The surgeon can visualize the operation within the patient's body and surgical processes.
Numerous tracking methods have been developed for surgical environment including vision based tracking and magnetics based tracking.(Dutkiewicz et al., 2004;Felfoul et al., 2008).After taking into consideration the consistent surgical luminance, distinct background content and usability, a vision-based approach based on Continuously Adaptive Mean-Shift (CamShift) algorithm (Ling et al., 2010) is adopted for intraoperative tracking.CamShift algorithm is built upon the Mean-Shift algorithm to track objects based on probability distribution (Wen et al., 2010;Mustafa & Kalajdziski, 2007).Back projection is a way to find the extent to which the pixel distribution of a region coincides with a histogram model.It can be used to compute the probability distribution image from the original image with the colour histogram.The pixel values in the input image are scaled using the values in the corresponding bin of the color histogram and stored in the probability distribution image.
The function computes the probability of each element value according to the empirical probability distribution represented by the histogram.If C is the color of the pixel and F is the probability that a pixel belongs to the object being tracked, then the probability map gives P (C|F).The probability distribution image is used in the algorithm for tracking.
Intraoperative Visual Guidance and Control Interface for Augmented Reality Robotic Surgery

201
The procedure of the Mean-Shift tracking algorithm is described as follows:: 1.An initial location of a search window is chosen.2. Compute the mean location or centroid in the window based on the probability distribution image.The centroid (x c , y c ) is calculated as follows: Firstly, the zeroth and first moments of the image are calculated as: where I(x, y) is the probability value at coordinate (x, y) in the image, ranging over the entire search window.The coordinates of the centroid are then computed from the moments as Eq. ( 19) and Eq. ( 20).The CamShift algorithm also has the ability to update the size of the search window to fit the object being tracked as the frames change (Ling et al., 2010).
Fig. 6.RFA needle is tracked based on an AR environment.
For intraoperative interaction, hand gesture is adopted to control display of the AR model and robotic device.Surgeon's hand gestures are detected by a stereovision system over or beside the AR environment.This interface has advantage on sterilization which is an important requirement in operating room.The vision based hand gesture can be integrated into the AR system using projector-camera system or conventional tracking device.The AR model projected on the patient surface can be scaled and rotated using hand gestures.Additionally, hand gestures can also be used to guide robotic execution.
In order to enhance the speed and accuracy of hand gesture recognition, an innovative 3D model based method is introduced.This method is based on the analysis of model features produced by a space palm plan and 3D convex hull points.Users are allowed to pose the defined hand gestures without wearing markers or data glove.Due to human hand's flexibility, stereovision system is used to resolve ambiguity on recognizing the hand features.
As shown in Fig. 7, the 3D model is constructed including the space palm plan which is relatively stable, and a brace model of five fingers with the key joints and finger tips.Before starting the hand gestures recognition, the 3D hand model is initialized by characterizing the model with the specific hand parameters such as palm size, fingers layout of different users.This initialization can be achieved by directly measuring the key parameters of the user's hand under the vision system with a defined pose.These key parameters include hand length and width, the ratio between the palm and the fingers, and the palm's location relative to the stereovision system.Palm plan can be acquired by regression of the hand palm coordinates shown as Fig. 7 (b).There are several possible ways of tracking the user's hands (Mahmoudi & Parviz, 2006).In consideration of the background of hand operation based on this AR interface is covered by the surgical cloth which is green in color.CamShift based method used for medical instrument tracking can also be integrated with hand tracking.When the hand tracking is implemented, the palm plan is invoked to change its pose in the space coordinate.
Following with the hand tracking, the 3D hand model is attached to the user's hand wherever the hand moves.In this case the parameters of the 3D hand model are analysed in real-time and the hand gesture recognition is achieved in real-time.

Robotic implementation
While the introduction of AR in computer aided surgery enhances visualization of preoperative surgical plan, it alone does not necessarily translate to precision and accuracy in intraoperative execution.In the case of minimally invasive surgery, the surgeon is not only deprived of direct visual information but also subjected to dexterous constraints in the manoeuvring of laparoscopic tools or percutaneous ablation devices.Performing such operation manually even with AR guidance can be non-trivial and often expose to uncertainties and inconsistent outcomes.For instance, the challenges in large tumor treatment with percutaneous ablation technique have been discussed (Abolhassani et al.,2007;Choi, et al., 2000;Goldberg, et al., 1998).This lack of intraoperative consistency poses a serious bottleneck for the effectiveness surgical outcome.Robotic assistance provides the necessary augmentation in dexterity to provide precision and accuracy even in space constraint situation (Hyosig & Wen, 2001;Kuo & Dai, 2009;Stoianovici et al., 1998;Taylor & Stoianovici, 2003;Taylor, 2008).The Transcutaneous Robot-assisted Ablationdevice Insertion Navigation System (TRAINS) is developed for ablation treatment in liver tumor (Chang et al., 2011;Yang et al., 2010).Technical details and demonstrate the implementation of robotic modules for AR robotic system are presented as follows.
The integration of the robotic modules requires establishment of the spatial relationship between the multi-body system and surgical field.It begins with the construction of a kinematic model for the manipulator and frame assignments to the various entities for the AR applications.Frame assignment of the multi-body robotic mechanism can be established based on the Denavit-Hartenberg (D-H) convention described by (Fu, 1987).Fig. 8 (left) illustrates the D-H frame assignments for TRAINS prototype.The joint variables and link parameters tabulated in Table 1.
The kinematic model can subsequently be expressed in homogenous transformation matrix as shown in Eq. ( 21). which represents the orientation and position of the end effector as a function of joint variables from their respective axis of control.ss s q l c c q l l q c q s l c l c Pc s s q l s c q l l q c q s l s l s cc q l lq where s j =sin (q i ), c ij =cos(q i ), sij=sin (q i +q j ) and c ij =cos(q i +q j ).
Direct vision is usually absence in minimally invasive surgery or percutaneous interventional procedure.Hence registration between diagnostic medical images, surgical field and its entities has to be done.Preoperative image coordinates can be readily transformed to spatial coordinates based on the system configuration of the imaging module.Intraoperative spatial information however, requires a tracking and navigation system.TRAINS uses a vision-based system for tracking and navigation purpose.Light emitting fiducial markers are used for the registration of entities in the real world to the interventional plan.Fig. 8 (right) depicts the robotic setup in a porcine surgery.

Discussion
The use of the intraoperative visual guidance and AR control interface in surgical procedures provide surgeons with a very exciting viewable integrated information platform.With this platform, the surgeon can concurrently conduct an intraoperative diagnosis, surgical operation and collection of patient's medical information on a single interactive augmented interface.This augmented reality based interactive interface allows surgeons to not only experience a new type of technology that extends their visibility without looking at any other screen or wearing any head-mounted device during the surgery, but also perform the operation with the combination of preoperative and intraoperative medical information.With regards to the surgical robot, not only does it contribute to higher accuracy of the surgical operation and optimizes the procedural flow, it is also easy for this intraoperative AR interface to incorporate the surgical robot assistance into the surgery, which includes the display of the robotic operation inside the patient and interactive control by the surgeons.
Although interactive spatial AR interfaces have been used for some applications such as games and advertisements, the intraoperative visual guidance and interactive AR interface for surgery is still a challenging problem.The reasons for this are as followed.First of all, an accurate display of the human organs and tissues is essential in surgery.The visual corrections, including geometric corrections of a projected virtual organ model, are important steps in constructing the spatial augmented reality environment, especially when projecting on an arbitrary surface.Other than the influence of irregular geometric surfaces, the projection is also affected by dynamic motions in the AR environment such as the patient's aspiration, unexpected surface motions or occlusions etc.Secondly, registration of an AR model with the real organ is especially difficult because of the non-rigid soft tissue.Thirdly, in usability tests with the medical tool, the ablation needle is as assumed to be a rigid needle without bending during the insertion.However, this assumption may not always be true.Therefore, other than

Conclusion and future work
The intraoperative visual guidance and control AR interface is an attempt to integrate the preoperative diagnostic information with real-time augmented visual information produced by the patient's reactions, surgeon's decisions, and robotic implementation.This interface provides surgeons with a mean to extend a person's visual limitations during the MIS and to engage interactively during the surgical robot assisted surgery.Augmented reality technology has the potential as a therapeutic intervention which combines the virtual augmented physical model with locally detected real-time medical information including geometric variance and respiration rate.The interface is coupled with the augmented physical model, surgeon's operation, and robotic implementation through vision based tracking and hand gesture recognition.
This intraoperative AR interface will benefit from a stereoscopic vision system with higher accuracy and higher frame rate which plays an important factor in vision based tracking and recognition.While the current tracking algorithm worked satisfactorily at the workstation used for testing, users still had problems with robustness when operating in an environment with a colourful background.In addition, this interface could be expanded to assimilate with more medical devices and equipments such as ultrasound that would integrate more real-time augmented medical information.In this scenario, the interface is a powerful integrated medical AR platform centralized in the network of all the other medical devices and equipment in the operating room.

Fig. 1 .
Fig. 1.Work flow of the advanced interface for augmented reality surgery.

Fig. 4 .
Fig. 4. Geometric correction on the curved surface with a chessboard pattern ((a) and (d) are side views.(b) and (e) are front views.(c) and (f) are the corrected projection for (a) and (d) respectively from the front views).
and Control Interface for Augmented Reality Robotic Surgery 199 for the color case, normalization of the elements in the M is conducted to eliminate the influence of the surface reflectance.The diagonal elements in the M are evaluated to unity, M dd = 1( d = R, G, B)

Fig. 5 .
Fig. 5. Projected correction for Augmented Reality environment construction ((a) is a source image of liver model.(c) is the distorted image projected on a phantom body.(b) and (d) are the corrected images.) the center of the window to the location derived from the previous step.4.Return toStep 2 until convergence is achieved.
and Control Interface for Augmented Reality Robotic Surgery 203
Intraoperative Visual Guidance and Control Interface for Augmented Reality Robotic Surgery 205 vision-based tracking technology, other types of intraoperative feedback such as real-time CT scanning etc. are needed to generate more reliable surgical information.Additionally, in order to realize a dynamic observational perspective from any location around the AR environment, more projectors and cameras are required.