Haptic Guidance Based on All-Optical Ultrasound Distance Sensing for Safer Minimally Invasive Fetal Surgery

By intervening during the early stage of gestation, fetal surgeons aim to correct or minimize the effects of congenital disorders. As compared to postnatal treatment of these disorders, such early interventions can often actually save the life of the fetus and also improve the quality of life of the newborn. However, fetal surgery is considered one of the most challenging disciplines within Minimally Invasive Surgery (MIS), owing to factors such as the fragility of the anatomic features, poor visibility, limited manoeuvrability, and extreme requirements in terms of instrument handling with precise positioning. This work is centered on a fetal laser surgery procedure treating placental disorders. It proposes the use of haptic guidance to enhance the overall safety of this procedure and to simplify instrument handling. A method is described that provides effective guidance by installing a forbidden region virtual fixture over the placenta, thereby safeguarding adequate clearance between the instrument tip and the placenta. With a novel application of all-optical ultrasound distance sensing in which transmission and reception are performed with fibre optics, this method can be used with a sole reliance on intraoperatively acquired data. The added value of the guidance approach, in terms of safety and performance, is demonstrated in a series of experiments with a robotic platform.


Introduction
Instruments for Minimally Invasive Surgery (MIS) are typically long and slender, to provide access to tissues deep within the body, with access typically performed through small incisions. With MIS, there tend to be shorter hospital stays, faster recoveries, and cosmetic benefits, as compared to open surgeries [1]. For fetal surgery, minimally invasive fetoscopic techniques are particularly appealing. When an open surgical approach is taken, a rather large uterine incision is made, which is believed to lead to an increased likelihood of preterm birth [2]. Indeed, a recent study from Petersen et al. indicated that increases in access diameter might lead to increases in the likelihood of premature delivery before 28 weeks [3]. Fetoscopic interventions, where the placenta and fetus are accessed through small incisions in the amniotic sac, are therefore the preferred approach.
Current developments in instrumentation for MIS in general, and fetoscopy in particular, aim at minimizing the instrument diameter [4]. Such developments would reduce the invasiveness even further. However, as instrument diameters decrease, the instruments become more compliant and hence more difficult to handle. Together with a reduced diameter, the instrument also becomes sharper, which can lead to a larger rate of accidental puncturing or damage to delicate tissue. This risk is aggravated by poor visibility, because in fetoscopy visibility is compromised by the turbid amniotic fluid, as well as limited illumination possibilities, limited field of view and low resolution by available fetoscopes. This combination implies that the surgeon must constantly ensure to maintain a safe distance between the instrument and the surrounding tissue.
The treatment of the twin-to-twin transfusion syndrome (TTTS) is a fetal procedure where the surgeon requires particularly precise control over the distance between his/her tools and the environment. TTTS is a condition where interconnections (anastomoses) between the blood circulatory systems of monochorionic twins are present on their shared placenta. These anastomoses cause a disproportionate blood supply and nutrition between fetuses. If left untreated, morbidity and mortality rates exceed 70% [5]. Treatment is typically done via an MIS intervention where the surgeon enters the uterus with a slender (ø3 mm) fetoscope, equipped with a therapeutic laser fibre, in order to locate and subsequently sever the anastomoses by means of laser coagulation (Fig. 1). Afterwards, a process called solomonization involves lasering in a continuous line between the previously treated anastomoses [6]. Figure 1 shows the principle of this therapy. During this therapy, it is important that a lasering distance of around 10 mm is maintained with respect to the placenta surface [7]. This distance yields an optimal focus for the laser beam. When ablating from larger distances, the laser beam is attenuated too much, rendering the therapy inefficient. For smaller distances there is an increased risk of puncturing the placenta, which would cause bleeding and could lead to termination of the intervention; additionally, the ablation site might be overexposed [8].
The above factors induce a heavy cognitive load for the fetal surgeon. To alleviate the burden, a smart guidance system can provide support, allowing the surgeon to focus on complicated tasks at decision level, which require human intelligence. This paper develops a haptic guidance scheme that aims at increasing the safety and the interventional outcome for laser-based coagulation tasks in TTTS procedures. More specifically, the proposed guidance installs a forbidden region virtual fixture (VF) over the placenta surface. This VF is embodied by a virtual elastic wall that can be haptically perceived, thus protecting the placenta from the sharp instrument tip. This also helps the surgeon to maintain the optimal lasering distance of 10 mm. The stiffness of the VF should be tuned such that it is perceived in between soft guidance and touching a rigid wall. This way the surgeon feels effective assistance, without sacrificing all control.
As the environment in fetal surgery is highly deformable, accurate prior knowledge on the environment is not available. Therefore, the VF has to be generated from intraoperatively acquired environment data. Concretely, the presented method relies on all-optical ultrasound (AOUS) distance sensing technology. AOUS is a recent imaging method that is suitable for real-time distance sensing during fetal surgical procedures, as is argued in the following sections.
The primary contributions of this work include the novel distance sensing technology, an associated method for estimating the placenta surface, and a model-mediated control approach. Together these elements are integrated into a haptic guidance system targeted at effective safety and performance increase in TTTS procedures, relying on online captured environment knowledge only. Table 1 summarizes the characteristics of this system, in accordance with specifications from literature and specifications put forward by clinical experts.
The rest of this paper is organized as follows. In Sec. 2, related work from the state of the art is discussed. The different components of the proposed guidance method are described in Sec. 3. Section 4 describes the experimental setup to validate all components and presents the results. The discussion of the proposed guidance method follows in Sec. 5. Finally, the paper is concluded in Sec. 6.

State of the Art
In the past and across different surgical disciplines, various technologies for intraoperative guidance have been proposed with the aim to improve safety and efficacy of MIS. Typically, these technologies consist of a means to measure or calculate the distance to fragile anatomical body regions and subsequently a means to supervise the instrument motion in the neighborhood of these regions. This behavior can be enforced by conveying proximity information to the surgeon through either visual, auditory or haptic cues, or any combination thereof.
pose to the pose within the preoperative dataset, a process referred to as "registration", an estimate of the distance to fragile or critical anatomic zones can be obtained. Guidance schemes can be planned in the preoperative stage. This alleviates the need for direct observation of the dangerous zones and allows using more or less traditional instruments which need not be equipped with dedicated sensors.
Orthopaedic surgeons have been early adopters of such safety-enhancing, preplanned guidance [12], but in other fields that deal with static, quasi-rigid structures it has been applied as well. For instance, Goffin et al. present personalized drill guides for spinal surgery [13]. These guides mechanically constrain instrument motion along the envisioned drilling axes, thus avoiding delicate vertebral arteries. Luz et al. show that in mastoidectomy safety can be enhanced and effort decreased by issuing auditory and visual cues when the image-guided navigation system detects that tools approach dangerous regions [14]. Furthermore, robotic systems such as the commercial Stryker Mako (previously Acrobot [12]) and the Renishaw NeuroMate [15] systems can also provide haptic guidance based on preoperative scans. The latter systems confine a cutting device to a predefined safe region, respectively, in orthopaedic surgery and in neurosurgery. This type of guidance is a so-called forbidden region VF, where haptic sensory information is overlaid with virtual forces that shield a forbidden region [16,17].
If, on the other hand, the targeted tissue is soft and deformable, one cannot rely only on preoperative models to derive useful guidance actions. In such cases, the availability of realtime intraoperative sensing becomes a prerequisite for guidance. A noteworthy example of this is brain surgery, where the brain shift and tissue resection should ideally be compensated [18]. To this end the deformation process can be modeled and non-rigid registration is used to match rich preoperative data with intraoperative images [19]. By doing so, detailed intraoperative information becomes available for navigation and guidance. An alternative approach is to estimate the error from the measured deformation and reshape a predefined forbidden region VF accordingly [20,21]. In yet other approaches, preoperative data is not used at all. Instead, guidance information is generated purely from intraoperative measurements. For example, Nakazawa et al. suggest generating a forbidden region by delineating it with dedicated instrument motions, prior to the actual therapy [22]. Also in the context of liver surgery, guidance systems are hindered by tissue deformation and require appropriate measures in order to safely use the available data. As an example, Song et al. employ locally rigid registration between intraoperative ultrasound and preoperative images to realize an image guidance system for this context [23].
Haptic guidance has been proposed also for dynamic environments that move or deform, e.g. in the presence of physiological motion. For proper functioning, the guidance scheme must adjust in phase with the dynamic phenomenon. This calls for real-time measurement of and feedback on the environment. Beating-heart surgery forms an inspiring example. By adjusting a VF to move synchrously with the heartbeat, control over the relative motion between instrument tip and cardiac wall can be obtained [24,25]. The dynamic VF can for instance be defined based on a point cloud streamed from an external stereo camera [26].
Bebek et al. employ multiple sonomicrometry crystals placed inside the body [27]. Three- [28]. The limited sampling frequency and processing delay of 3D US limits the quality of haptic guidance schemes that can be set up. This shortcoming can be overcome by exploiting the periodicity of the heartbeat [28,29]. Ren et al. suggest registering preoperative data of an entire period of the heart cycle to the intraoperative scene and using ECG feedback to determine the phase in real time. By doing so, this approach circumvents intraoperative distance measurements of the environment [30]. Francoise et al. provide an example for applying VFs with a non-periodically moving environment, in the context of bone milling [31].
Also when working close to (or below) the boundaries of human motion capability, guidance schemes become of interest. When working on fragile anatomic structures with dimensions well below the millimetre scale, guidance could prevent involuntary motions and tremors from damaging the structures. Due to the limited accuracy of external imaging modalities and the impossibility of getting registration errors sufficiently low, one cannot rely on preoperative data in this scenario. Intraoperative distance measurement is thus essential. For example, for eye-surgery, Becker et al. generate VFs with submillimetre accuracy. They propose the use of surface reconstruction from an external stereomicroscope [32]. Balicki et al. showed in earlier work how the distance between a needle and the retinal surface could be measured and controlled based on Common Path Optical Coherence Tomography (CP-OCT) via a fibre that is integrated inside the needle [33].
Referring to the discussed works, fetal surgery embodies a particularly difficult case for accurate guidance. It suffers from several complications that are not present in other surgical areas and hence requires specific attention. The uterus is unstructured, deformable and dynamic; the amniotic sack is further populated by free-floating bodies (fetuses). Damage to a fetus, the placenta, the amniotic sac, or to the mother is to be avoided at all times. Numerous factors, such as the position of the mother, affect the shape of the uterus. Within this highly variable environment, slender and sharp instruments have to be navigated with millimetre precision for treating fragile structures. As argued before, this all happens under very poor visibility due to the turbid amniotic fluid and the limitations imposed by current small-diameter fetoscopes. Figure 2 provides an overview of the haptic guidance system that has been developed. The guidance scheme follows a 'comanipulation' paradigm [34]. The surgical instrument is attached to a haptic manipulator. The pose of the instrument is controlled simultaneously by this robotic device and the human operator. The implemented guidance scheme resists motion commands that would move the instrument inside forbidden regions. Such forbidden regions are computed on-the-fly from measurements of the placenta surface.

Materials and Methods
A dedicated sensor, based on fibre-optic AOUS technology, is employed to obtain real-time information on the placenta by measuring the distance between the instrument tip and the placenta. Section 3.1 details the principle of this sensor. Subsequently, Sec. 3.2 explains how this distance signal is combined with the current instrument pose to estimate the placenta pose in the world reference frame. This placenta estimation algorithm is employed to set up a model-based haptic guidance controller. Aside from helping to overcome some particular characteristics of the sensor data, the model-based controller allows to render a reliable and stable forbidden region VF above the placenta surface. It will be shown that this feature helps the operator to maintain the instrument at a safe distance to the placenta surface and to execute the surgical task more efficiently. The proposed haptic comanipulation scheme is explained in Sec. 3.3. The interaction of all components is depicted in Fig. 3.

All-optical ultrasound distance sensor
Determining the distance between the instrument tip and the placenta in fetal surgery presents certain difficulties. Preoperative data cannot easily be used due to the complex and deformable nature of the environment. Approaches based on visual servoing from the fetoscopic view are not reliable, as the vision and depth perception are of poor quality. Often there are too few anatomical landmarks in view. 3D abdominal ultrasound imaging can be used for distance sensing, but it requires segmentation that is difficult to automate. The achievable update rates and precision are also too low for haptic guidance, limiting the effectiveness of guidance schemes that rely on this data source. Stereocameras or other sensors are typically too bulky so that they do not fit within the 3 mm outer diameter of the fetoscope (Table 1), with only recent efforts to reduce this form factor [35]. Fibre-based distance sensors form a good option as they can be readily integrated into slender instruments. While CP-OCT has a limited sensing range, AOUS has been found to provide a more acceptable range for the same small form factor. Additionally, the update rates for AOUS distance sensing are only limited by the switching time of the exciting laser, so that distance can be measured at rates above 1 kHz. Also, amniotic fluid has proven to be a good medium to transfer ultrasound waves. For these reasons, this work proposes to use the principle of AOUS for distance sensing in fetal surgery.

Sensor principle-AOUS
is an emerging technology well-suited for highly miniaturized ultrasound devices. Here, sound is generated at the coated tip of an optical fibre via the photo-acoustic effect: light incident on the coating material is absorbed and the ensuing safe heat rise causes a pressure change which propagates as sound [36]. To receive sound, interferometric methods can be used, such as the Fabry-Pérot hydrophone [37].
For this work, an AOUS device consisting of two optical fibres, as previously described [38,39], was used. Briefly, the transmitter comprised a 300 μm core diameter fibre coated with a CNT/PDMS composite to generate sound. The receiver comprised a single mode fibre coated with a concave Fabry-Pérot cavity [40]. The device was interrogated using a system previously described [38]. Briefly, it comprised two lasers: a Q-switched Nd: YAG laser (SPOT-10-500-1064, Elforlight, UK) with a pulse width of 2 ns, pulse energy of 20 μJ, and repetition rate of 100 Hz, to generate ultrasound, and a tunable continuous wave laser (Tunics T100S-HP CL, Yenista Optics, France) to interrogate the hydrophone. The tunable laser was connected to the hydrophone via a circulator allowing the reflection of the Fabry-Pérot cavity to be monitored with a photodiode. bias point of the hydrophone. The received ultrasound was digitized at 100 MS/s at 14 bits (PCI-5142, National Instruments). In order to measure the distance to the target surface, the received signal s(i), indexed with i, was compared to a threshold function t(i) The index i t of the first returned value above the threshold was converted to a distance d, using the following formula: with s = 10 MS/s the sample rate, c s the sound speed and This way false readings were avoided at close distance, where cross-talk (sound transmitted directly to the receiver) was high.
The ultrasound data was displayed in real-time as an M-mode image at 100 Hz along with the measured device/placenta separation. The measured separation values were sent to a secondary computer via TCP/IP protocol with a 100 μm precision for processing and filtering, with the goal of estimating the placenta pose.

Sensor validation-
In order to validate the distance measurements provided by the ultrasound device, a series of experiments was carried out. The following protocol was used. The device was mounted on a 1D motorized translation stage and device was moved to the surface of a tissue mimicking phantom made of gel wax [41]. The surface was found using both the received ultrasound image and by observation. The device was moved away from the surface in a series of movements each consisting of 5 mm travel at 0.5 mm/s, followed by a rest period of 5 s. The ultrasound image and distance was recorded over this movement.
The protocol was repeated for angles of 0°, 15°, and 30° to the normal of the phantom surface. No further angles beyond 30° were used as the signal return from 30° was already too poor for distance estimation.
The accuracy of the measured distance was dependent on the accuracy of the speed of sound, which is well known for water and human tissues, including amniotic fluid [42]. For the presented measurements, a value of 1500 m/s was used for c s . When perpendicular to the phantom surface, the furthest distance that could be measured was 45 mm. Beyond this, the received signal was not strong enough to be distinguished from noise. The maximum achievable distance quickly reduced as the angle between the probe and the normal to the surface was increased, at 15° the maximum distance was 25 mm and at 30° there was insufficient signal for distance measurements.
The distance measured by the probe was assumed to be in the line of sight, thus when the probe was not perpendicular to the surface, the measured distance should vary as where d n is distance normal to the surface and ϕ is the angle between the probe and the normal. It was found that at 15° to the normal the relationship between the measured distance and the normal distance was d m = d n , suggesting that, due to the divergence of the ultrasound beam and strong specular reflection at the phantom surface, the perpendicular distance was in fact being measured.
In addition to controlled distance sensing at specific angles, a single vessel phantom was scanned. The probe was moved laterally above the phantom surface at a constant speed of 0.5 mm/s. The top surface of the vessel surface and the surrounding phantom were easily resolved (Fig. 4). However, the gradual height increase at the vessel edges was poorly resolved and there was a gap in the recorded distance. This was expected given the poor reflected signal received at high angles in the previously described experiment.

Placenta model estimation
To overcome the discrepancy between the precision and measurement frequency obtained from the AOUS distance sensor and the precision and measurement frequency required for proper haptic control, a model-mediated [43] control approach is adopted in the presented comanipulation system. Rather than relying on the raw measurements, the guidance scheme (presented in Sec. 3.3) is developed based on a model of the placenta that is obtained through fusion and filtering of the available data. The estimated placenta model is continuously updated. It is robust to measurement outliers and data gaps. It also encodes the directionality of the surface. This allows preventing motion into the surface, without restricting motion parallel to the surface. Motion away from the surface is always permitted. Furthermore the model allows control at update rates that exceed the measurement frequency.
A planar placenta model is assumed to describe the placenta surface sufficiently well in the vicinity of the photocoagulation site, as shown in Fig. 5. For point labels the name of the corresponding position vector in the world frame is used. The planar placenta model is represented by its normal, the unit vector n, and a point p p , located at the intersection between the instrument axis, the unit vector i, and the placenta surface. The measured distance d is then the distance from the instrument tip p i to p p . For now, the assumption is made that is possible to measure the distance along i, disregarding the fact that the divergent beam from the distance sensor could also be measuring the distance along n. Now, it is possible to express the placenta pose in the world reference frame by a homogeneous coordinate transformation matrix T: Note that since all points and frames are expressed relative to the same world frame (frame {w} in Fig. 5), without loss of generality and for notational convenience, a leading superscript, denoting the frame in which all points/transformations are expressed, is omitted.

The 3 × 3 rotation matrix
captures the orientation of the placenta as a orthonormal frame with unit vectors t, s and n.
Note that for t and s any two mutually orthogonal unit vectors in the placenta plane, perpendicular to n, can be taken. For obtaining R it is thus sufficient to know n.
The placenta pose, expressed by T, requires knowledge of p i , i, d, and n. The distance sensor and sensorized haptic manipulator that holds the instrument provide raw measurements for p i , i, and d. However, n, and more generally R, cannot be measured directly, as a single 1D distance measurement at a single instrument pose does not contain any information on the placenta orientation. To determine n, measurements for multiple instrument poses need to be collected. Provided that the instrument motion is sufficiently rich, i.e. with components along t and s, it will become possible to estimate n accurately.
In order to not interfere with the normal workflow, rather than imposing a predefined rich motion pattern to acquire information on n, we propose to use an Extended Kalman Filter (EKF). The EKF estimates n based on measurement updates at each time instant, combined with the measurement history that is contained in the previous estimate of the placenta pose.
A built-in model of a planar placenta helps the EKF to retain an estimate of n in case the instrument motion isn't sufficiently rich. In addition to solving the problem that n is not directly observable from the measurements, the EKF also refines all measurements, making them more robust against noise and outliers. This EKF is described below.

EKF for placenta model estimation-
The governing relations of the EKF are given as [44] x t = g x t − 1 + ϵ t , (6) with x t the current state vector and x t-1 the state vector at the previous time instant t -1. The time step between t and t -1 is indicated as Δt. The state transition model is described by the function g(). ϵ t expresses the noise on the state transition. Further, z t is the measurement vector at time t, h() the measurement model and δ t the measurement noise vector. These components will be elaborated below.

Europe PMC Funders Author Manuscripts
The state vector is defined as This vector contains all parameters necessary for a complete state description. This includes the linear velocity of the instrument tip v = ẋ ẏ ż T , as knowledge of the change in the tip position is required. Furthermore, it contains the orientation of i and n, parametrized as the intuitive altitude and azimuth angle pairs (α, β) and (θ, γ), respectively: i = cos α cos β cos α sin β sin α T , (9) n = cos θ cos γ cos θ sin γ sin θ T . (10) Figure 6 visualizes this parametrization of i and n. Finally, the orientation change of the instrument is included in the state vector as (α, β) and the distance to the placenta as d.

State transition model
If an instrument motion with a constant velocity and a locally planar placenta are assumed, all state transition functions can be straightforwardly defined, except for d. Referring to Fig.   7, this transition can be modeled as follows: with h the distance d t projected onto n t-1 , λ the angle between h and d t . In analogy with Eq.
(9), it follows that i t can be expressed as: The above summarizes to the following state transition model:

State transition noise
All state transitions are assumed to be affected by mutually independent Gaussian noise. While for most state parameters the standard deviation of the noise is assumed to be constant, for a pair of azimuth-altitude angles the standard deviation of the azimuth depends on the current value of the altitude angle. This will be shown next for (θ, γ) in n, but a similar reasoning holds for (α, β) and for α, β . The impact of a change in the altitude angle θ to a change in the orientation of n, is a constant: ∂n ∂θ = sin 2 θ cos 2 γ + sin 2 θ sin 2 γ + cos 2 θ = 1, (15) while the impact of a change in the azimuth angle γ to a change in the orientation of n is a function of θ: ∂n ∂γ = cos 2 θ sin 2 γ + cos 2 θ cos 2 γ = cos θ . (16) Therefore, if the noise on θ is modeled as having a Gaussian probability distribution with 0, σ θ 2 , the noise on γ needs to be formulated as 0, σ γ , in order to have an orientation noise that does not favor any particular direction. The exact value of the standard deviations needs to be tuned in function of expected disturbances with respect to the state transition model.

Measurement vector
The measurement vector z contains the measurement for all states, except for θ and γ as no direct measurements on the placenta normal can be obtained. In the rest of the text, the tilde symbol ˜ is used to indicate a measured value for a certain variable. The sensors in the haptic manipulator are assumed to be able to measure the instrument direction i and instrument twist vector t = ẋ ẏ ż w x w y w z T (17) with w n the measured angular velocity of the instrument around the axis n of the world reference frame {w}. The distance sensor embedded in the instrument provides the measured distance d .
The measured altitude angle α and azimuth angle β can be obtained from i as To avoid jumps in β when the 0 ↔ 2π border is crossed, β has to be unwrapped to β uw , by comparing it to its previous value β uw, prev : β uw = β uw, prev − π + β − β uw, prev + π mod 2π , (20) with mod the modulo operator.
The change in the measured altitude angle α and azimuth angle β can then be determined from the adapted instrument twist t a , which is obtained after subtracting the rotation component around i from t, as instrument rotation about its own axis does not contribute to a change in the instrument altitude or azimuth angle: This yields the following measurement vector:

Measurement model
The measurement model is defined as follows: The lack of measurements for θ and γ will be accounted for by the mechanics of the EKF.
Thus, the EKF allows determining the unknown n without directly measuring it.

Measurement noise
For the measurement noise the same remarks hold as for the state transition noise. All measurements are assumed to be affected by mutually independent Gaussian noise. For the azimuth-altitude pairs α, β and α, β the standard deviation of the azimuth angle is again dependent of the current value of the altitude angle. The exact value of the standard deviations needs to be tuned in function of the sensor characteristics.

Verification of the EKF for placenta model estimation-A simulation was
set up to verify the surface estimation algorithm. In the simulation, an instrument tip is programmed to make small, realistic oscillatory motions with respect to a planar, angulated surface. These motions had an amplitude in the range of 10 mm and a frequency not exceeding 1 Hz. The measurements are sampled with a frequency of 1 kHz and Gaussian noise is introduced on all measured values. For the initialization of the state estimate vector, the first measured values are used for ẋ t , ẏ t , ż t , α t , α t , β t , β t , d t . This is not possible for θ and γ, so an offset of 45° with respect to the given ground truth is taken for their initial value. The EKF estimate covariance matrix is initialized as an identity matrix.
The simulation was repeated several times under different conditions for the instrument motion pattern, the artifially created noise variances, θ and γ. Results for a representative simulation run are shown in Fig. 8. From Figs. 8(a)-8(c), it follows that the offset for the estimate of the normal was reduced to a few degrees in about 2 s. Figure 8(d) indicates that a reliable distance measurement can be obtained thanks to the filter, despite the large measurement noise of the simulated distance sensor.
Although a detailed discussion of the performance of the surface estimation algorithm is outside the scope of this paper, simulation results showed the ability of the algorithm to quickly and robustly identify the surface. Depending on the choice for parameters such as the applied instrument motion, the placenta orientation, the noise and the initialization, the results differ, but robustly converge to the ground truth. Only when λ (see Fig. 7) exceeds 75°, i.e. when the instrument is almost parallel to the placenta surface, the EKF experiences problems converging. In this situation, small disturbances on the instrument pose result in large jumps in the measured distance, introducing large discrepancies between the predicted and the measured distance. Therefore, situations where the instrument is almost parallel to the placenta are to be avoided.

Haptic guidance
The distance sensor and placenta estimation algorithm provide the placenta pose T, without requiring prior knowledge. The haptic guidance algorithm uses T to implement a forbidden region VF. There are several reasons why T, expressed in the world reference frame, is preferred over the raw instrument-placenta distance as input for the haptic guidance: • When the distance measurement runs at a lower frequency than the instrument tracking measurements from the haptic manipulator, T still allows to compute the distance for every sample of the tracking data.
• If there is a time delay in the distance measurement with respect to the instrument tracking, the surface estimation algorithm can synchronize both data streams by introducing a delay in the instrument tracking data stream. The resulting T can then be used by the haptic guidance algorithm, in combination with the most recent instrument tracking data.

•
Next to the position, T also contains the orientation of the placenta surface. This allows the haptic guidance algorithm to know in what direction it has to guide the instrument, to move it orthogonally away from the placenta surface.
• If the raw distance measurement were used, outliers in the measured values would cause jumps in the guidance, while these outliers are filtered out by the EKF that produces T.

•
In the case that the distance sensor misses samples because of unpredictable ill conditions, the haptic guidance can still continue working with T.
Based on T and the current instrument pose, there are different options to implement the guidance that prevents the axial distance between the instrument tip and the placenta from falling below the threshold distance d th of 10 mm. The most straightforward approach is illustrated in Fig. 9(a). A spring with stiffness k i and rest length d th is virtually attached to the instrument tip, along the direction i, while the placenta is considered to be infinitely stiff. Whenever the virtual end p v of this spring touches the placenta surface, the spring compresses and pushes the instrument back. The force that this VF would apply on the instrument can be written as with d the actual distance from the instrument tip to the placenta surface: d = h/cos λ. This implementation, however, results in undesirable behavior when cos λ → 0. In this case, the instrument is almost parallel to the surface and small changes in λ result in large, abrupt changes in d and F i , which is disturbing to the operator. It also interferes with the intention of the operator, as the generated force has components tangential to the placenta surface, which unnaturally move the instrument tip over the placenta plane.
A better approach is shown in Fig. 9(b). Here, a virtual rigid rod of length d th is attached to the instrument tip p i . The virtual end of this rod is point p v , at position: When the instrument's virtual tip p v touches the placenta surface, represented as a virtual wall with stiffness k w , the wall pushes back with a force: where h v is the distance from a point p p on the placenta surface to the virtual tip p v , projected onto n: This forbidden region VF avoids the problematic dependency of the force on 1/cos λ. At the same time the force is limited to the n-direction, resulting in an intuitive virtual wall feeling that resembles the interaction forces/torques that would take place upon contact with a real wall.
The virtual guidance force F w at the instrument tip results in a corresponding force F ee and moment M ee at the instrument handle [11]. The haptic manipulator is to exert this amount of force at its end effector upon the user: with r the vector from the instrument handle (coinciding with the end effector of the manipulator) to the instrument tip. These forces are visualized in Fig. 10. The human operator perceives the output F ee and M ee from the haptic manipulator. For him/her, it feels as if touching a real wall installed to protect the placenta. The proposed configuration allows the operator to cooperatively control the instrument with the manipulator, ensuring safety, but not hindering functional motion parallel to the placenta surface.

Experiments
This section introduces a number of experimental results that demonstrate the functionality of the proposed system. The experimental setup is shown in Fig. 11. For these experiments, a dummy instrument with a 3 mm outer diameter was used. The developed AOUS distance sensor is embedded in the instrument tip. The operator holds the instrument by the handle. At the same time the instrument is held by the haptic manipulator, in this case a Virtuose6D (Haption S. A., Laval, France). In a synergetic fashion both the operator and the Virtuose6D manipulate the instrument with respect to a placenta mock-up inside a water tank. The water provides a medium for propagation of the US waves, similar to the amniotic fluid inside the uterus. For convenience a replica of a placenta was made. The custom-made mock-up was more or less planar and was made out of gel wax for its good US reflection properties [41].
Prior to the experiments the location and orientation of the instrument as it is attached to the Virtuose6D were calibrated. Via the Virtuose6D it is then possible to acquire the instrument pose and twist at 1 kHz. The distance sensor reported the distance at 100 Hz. At time steps when no distance measurement was available, an estimation of the distance measurement was generated from the current placenta model and instrument pose, allowing the full EKF to run at 1 kHz. The measurements from the Virtuose6D and the distance sensor were synchronized up to 1 ms. Due to the current constraints for sensing the distance under an angle, the operator was asked not to exceed a 10° angular misalignment between the instrument axis and the placenta surface normal during all experiments. Such a perpendicular orientation is also desirable from a clinical perspective.
In order to verify the feasibility and performance improvement of the proposed guidance scheme, a comparison is presented of haptic versus visual guidance. To simplify the task, the operator is presented with a very much "idealized" scenario, when compared to the clinical conditions. Figure 11 depicts how the operator has a clear, sideways view upon the instrument and a cross-sectional view upon the placenta. In clinical practice, the surgeon never has such a good view allowing him/her to gauge the distance between the instrument tip and the placenta surface. Additionally, the tasks are limited to translational motions, which are more comprehensible than pivoting motions around an incision point. It is anticipated that if beneficial effects can be shown in such an idealized scenario, the real, more challenging scenario would benefit even more from haptic guidance.
In all following experiments the stiffness parameter k w for the virtual wall was set to 0.9 N/mm, in order to have a protective layer that is a good compromise between the operator being in control and the guidance giving corrective cues. The threshold distance was set at d th = 12 mm, such that sufficient resistive force is generated by the time the instrument tip reaches the clinically relevant 10 mm distance to the placenta.

Validation of placenta estimation algorithm
The placenta estimation algorithm earlier verified in simulation was now validated on the experimental setup. The operator moved the instrument at a close distance over a perfectly flat surface, holding the instrument perpendicular to this surface, within a 10° margin. The results of a representative validation run are shown in Fig. 12. The algorithm was every time able to estimate the orientation of the placenta normal in about 20 s, even though it started from a very poor initialization (70° off). With a proper, supervised initial guess, the convergence time can be reduced. Figures 12(a)-12(c) provides an overview of the evolution of respectively the azimuth, elevation, orientation error and distance values for a representative case. As soon as the estimation of the normal converged, the filter was capable to reliably filter the outliers out of the distance measurements ( Fig. 12(d)). Note that the amplitude of the noise on the distance measurements was rather low compared to the one used in the simulation. Table 2 provides an overview of the tuned standard deviations used for the Gaussian the state transition noise and measurement noise during the validation of the placenta estimation algorithm.

Haptic guidance -feasibility
A series of experiments was set up to examine the feasibility of the haptic guidance system to provide an additional safety measure during fetal interventions. In the experiments an artificial situation was created where the operator was asked to move towards the surface carelessly, simulating inattention. A scenario with and without haptic guidance was foreseen. Without guidance, the operator would always come into contact with the surface (Fig.  13(a)); while with guidance the involuntary motion of the operator was successfully countered in such a way that the operator could eventually retract the instrument without any contact with the placenta (Fig. 13(b)).
A subsequent experiment shows the operator moving around near the placenta surface in a relatively rough manner. Figure 13(c) shows how also here the haptic guidance manages to protect the placenta at all times.

Haptic guidance -task performance
The impact of the proposed haptic guidance on the performance of a laser ablation task in TTTS treatment was experimentally evaluated next. The solomonization phase of the laser ablation process is essentially a line-tracing task [6]. To replicate solomonization, the operator was asked to move the instrument tip across the placenta surface following an ideally straight line between point A to point B and back (see Fig. 14). Both points lie at the same distance to the placenta replica so that ideally the operator would maintain a constant distance of 10 mm to the surface. The primary objective for successful task completion was keeping distance (performance); time was a secondary objective. Three participants were asked to participate in this task. The participants had no prior experience with haptic feedback. Each of the 3 operators executed this task 3 times with haptic guidance and 3 times without it. In cases with haptic guidance, the surface had a slight curvature of κ = 1/400 m −1 , thus challenging the guidance to cope with an imperfect surface. While the number of experiments/participants are too low for statistical analysis, it is already possible to provide some first insights. Figure 15 presents representative results for the line-tracing task without and with guidance. Table 3 summarizes the results over the different experiments. It can be seen that the haptic guidance substantially improves the task completion time and the capability of the operator to maintain the targeted 10 mm distance.

Discussion
The experiments demonstrated a successful integration of the presented distance sensing modality, the placenta estimation algorithm and haptic guidance within a phantom experimental environment. The system provides effective haptic guidance to the operator, protecting the delicate placenta surface, while allowing free motion along the surface. From Fig. 13, it can be derived that the created virtual wall, with a stiffness of k w = 0.9 N/mm, is sufficiently perceivable to alert an unwary operator and effective avert otherwise dangerous impacts. At the same time the operators reported that the guidance was subtle enough not to interfere with their intended motion along the surface and could be overridden if required.
For the completion of a line-tracing task, at a prescribed distance from the placenta surface, the forbidden region VF proved to be advantageous as well. Figure 15 illustrates that the provided guidance enables the operator to obtain a more accurate and smooth distance control. From the results in Table 3 one can see that the completion time for the task was reduced by 1/3 thanks to the guidance, while at the same time the distance errors were reduced by 2/3. These results indicate that haptic guidance may have significant added value for this type of applications.
The distance sensor combined with the placenta surface estimation algorithm suffice to provide on-the-fly guidance solely based on data acquired in real time (and a nonspecific planar placenta model). This is necessary for fetal procedures, as the environment is so dynamic and deformable that preoperative models have only very limited value and an accurate registration of this data is practically impossible. Figure 12 indicates that for a flat surface the orientation and distance can be accurately estimated. The convergence time for this estimation is higher than for the simulated case of Fig. 8. This can be attributed to, amongst others, the rudimentary characterization of the measurement noise, the lower frame rate of the distance measurements (100 Hz versus 1 kHz) and to inaccuracies in the calibration of the instrument tip position with respect to the end effector of the haptic manipulator. On the other hand, it is encouraging to see that the EKF still converges to satisfactory results despite these factors. That robustness is especially relevant, as in clinical practice small diameter instruments will always slightly bend due to interaction forces at the incision point, thereby invalidating the tip calibration to some extent.
The assumption of the planar placenta surface introduces a small offset between the measured distance and the EKF estimation of the distance in the case that the real surface is slightly curved, as is shown in Fig. 15(b). For moderate curvatures this offset could be counteracted by reducing the model reliance of the EKF. This entails a trade-off between recognizing surface curvature and rejection of measurement noise. For large curvatures, the EKF could be possibly extended with a higher order surface model.
In order to evaluate the distance sensing in a fair manner, a tissue mimicking gel wax phantom was used as the target. With this signals were detected at distances up to 45 mm from the surface. An even larger range can be achieved by increasing the sensitivity of the hydrophone, for example, by increasing the sensor thickness. Improvements in ultrasound generation may additionally lead to increased range. Unfortunately, the angle between the probe and the surface had a dramatic effect on the recorded signal. This was thought to be due to the specular nature of the reflection from the surface. The surface used was smooth and therefore will not scatter ultrasound in a diffuse manner. For a diffuse reflector, it is likely that signal received at higher angles would improve.
Because of these problems with the reliability of the distance sensor under large angles, the current work adopted the simplification that the instrument is kept approximately perpendicular to the placenta surface. In clinical practice, this is not possible, because the incision point dictates the orientation of the instrument. Therefore, future work will aim at improving the allowed angulation range. The directionality of the ultrasound probe can be modified [45] to focus the signal, in order to give only the line of sight distance. Conversely, omnidirectionality of the sensor can be pursued. Thus, the perpendicular distance will always be measured, even at high angles, as the strong specular reflection of the signal will come from the perpendicular direction. It is also possible to equip the instrument with local degrees of freedom at the tip and control the tip curvature in order to realize continuous perpendicularity [46]. This is interesting for both sensing and therapeutic purposes. Lastly, in the software, a modeling approach that takes into account the directionality of the ultrasound beam can also provide a solution. A combination of these approaches will be investigated to ensure robust performance of the system in all situations, including these with a large angulation.
The accuracy of the distance measurement is dependent on four things: Systematic offset due to delays in the system, the speed of sound estimate, the ultrasound resolution, and jitter. The first, the systematic offset can be measured and accounted for. The speed of sound is well characterized in many biological tissues, including amniotic fluid [42]. The ultrasound resolution is dependent on the ultrasound bandwidth of the system, for the system used the probe resolution was 60 μm in the axial direction. This could be improved by creating higher bandwidth devices. However, the resolution already exceeds the requirements. Finally, jitter gives small temporal displacements in the signal, caused by timing jitter in the system, the laser, electronics, etc. This was measured to be 10 ns, corresponding to a distance of 7.5 μm.
In all, the accuracy of the distance measurements exceeds the requirements for maintaining a virtual wall and guidance in surgery.
Occasionally the signal of the distance sensor dropped. In order to differentiate a null signal, i.e. no distance measured from an actual signal, a negative value was assigned to null signals. In this way, they could be easily distinguished from data and allow distance tracking even with an unstable ultrasound signal. Surface estimation was still possible even when large parts of the data points were being dropped.
For this experiment the ultrasound frame rate was limited to 100 Hz due to the need to display the M-mode ultrasound image. Improvements to the software would allow distance measurement at a rate of more than 1 kHz.
If AOUS distance sensing is used at the same time as conventional ultrasound from outside the body, ultrasound transmitted by the piezoelectric array will be received on the distance sensing device. However, device synchronization could be implemented to avoid interferences. An additional feature of the technology used for the distance sensing probe is that it can also potentially be used for device tracking in conjunction with traditional ultrasound imaging [47].

Conclusion
In this paper, a haptic guidance system was developed that is able to successfully create a forbidden region VF that warrants safety and improves performance in TTTS laser therapy. This system functions without any requirement of prior knowledge on the environment. The three main components of the guidance system were explained in detail. First of all, a distance sensing technology was introduced: AOUS distance sensing. This allowed to capture data on the uterine environment on the fly. Secondly, a placenta estimation algorithm, based on the EKF, was developed, which allows to determine the location of the VF in real time. The last component of the system, a model-mediated haptic guidance approach that robustly incorporates the available environment data, was implemented on a robotic platform. All elements were separately and jointly validated. The preliminary results show that hazardous situation can be countered and procedural skills can be improved for novice users.
The obtained system satisfies most of the desired characteristics, as listed in Table 1. Mainly the range and accepted angles for the distance measurement have to be improved. These improvements will be targeted in future work. As soon as this has been realized, the complexity of the task can be increased. A more complex placenta surface will be used and users will operate endoscopically, through an incision point. The group of test subjects will be extended to include different levels of expertize.  Comanipulation setup for fetal surgery, with the robotic manipulator, the surgeon, and the patient. Parametrization of i with altitude angle α and azimuth angle β (green) and parametrization of n with altitude angle θ and azimuth angle γ (red). Representation of the distance prediction, showing the instrument pose and the associated distance measurement at two consecutive time instants. The representation is a projection onto a plane parallel to n t-1 and i t .  Representative result for the simulation of the placenta surface estimation. In (a) and (b) the estimation of the altitude θ and the azimuth γ of the placenta normal n is visualized, respectively. The angular error in the estimation of n with respect to the ground truth n th is shown in (c). This error equals arccos(n · n th ). The estimated distance is depicted in (d).  Different haptic guidance approaches. In (a) a virtual spring is implemented between point p i and point p v . This spring will compress when p v touches the rigid placenta surface. In (b) the virtual point p v , which is rigidly connected to the instrument, compresses the elastic placenta surface. Instrument-placenta distance as a function of time. (a) and (b) show the distance when the operator moves carelessly towards the surface, respectively without and with assistance from the haptic guidance. In (c) the operator moves around near the placenta surface in a relatively rough fashion, while the guidance ensures sufficient clearance between the instrument tip and the placenta. Definition of the line-tracing task. The operator is asked to move the instrument from point A to point B while maintaining a constant distance of 10 mm to the placenta surface. Results of the line-tracing task in a representative experiment; (a) distance to surface when tracing a line without haptic guidance and (b) with haptic guidance. Characteristics of the proposed haptic guidance system for TTTS. Fetoscope: Diameter 7-12 Fr (2.33-4 mm) [3] Distal bending range 0-60° [9] Distance measurement: Sensing modality Fetoscope mountable

Registration based
No, intraoperative data only Range 0-50 mm Angle of incidence 0-90°L aser: Optimal distance 10 mm [9] Optimal angle of incidence 0° [9] Controller: Virtual wall stiffness 200-2000 N/m [10,11] Update rate 1 kHz [10] Gruijthuijsen et al. Page 44 Table 2 Values of the standard deviation of the state transition noise and the measurement noise used in the placenta estimation algorithm.  Table 3 Results of line-tracing task, with the task completion time t and the error with respect to the 10 mm target distance; employed metrics: root-mean-square error e rms averaged over the different experiment runs, minimum error e min over all runs, maximum error e max over all runs, average minimum error e min;av over the runs and average maximum error e max;av over the runs.