Camera Augmented Mobile C-arm - Chair for Computer Aided ...

Camera Augmented Mobile C-arm
Student: Lejing Wang (2809335)
Advisor: Jörg Traub
Director: Prof. Nassir Navab
Abstract:
Traditional mobile C-arm that can take intra-operative X-ray images is popular in the
operating room. But X-ray images that are displayed on the monitor are in the
different space from the patients. For positioning and repositioning C-arm, it needs
the skills, time and radiation. This paper presents Camera Augmented Mobile C-arm
(CAMC) system that can almost solve the problems mentioned before. The first
section of the paper will introduce CAMC system from basic construction and
principle. Then CAMC will be described from three different aspects corresponding
to three different surgical applications in three more or less independent sections.
After this, the Advanced CAMC system attaching 2 nd camera will be studied. In the
end of this paper, it is the conclusion and the ideas for extending CAMC system in the
near future are proposed.
Introduction:
The mobile C-arm is an essential everyday tool for most interventional procedures.
Although the traditional mobile C-arm just can take 2D X-ray images, modern mobile
C-arms like Siremobil-Iso-C3D (figure 1) that can do 3D reconstruction are already
available. The basic motivations of the CAMC system are the followings: first of all,
Current C-arm with 3D reconstruction capability requires the reproducible motion of
the C-arm, but we also want low cost C-arm without reproducible motion can do 3D
reconstruction. Secondly, the surgeon performs the operation by looking at the screen
where the medical image is presented usually. In this case, surgeons need to build the
relation between the medical images like X-ray images and patient, which is not
convenient and requires much experience. What we want is to augment the surgeon’s
view by overlaying the medical images and optical images of patients, which is a kind
of medical augmented reality. In addition to, we want to speed up the procedures like
positioning and repositioning mobile C-arm, simplify their execution and reduce the
necessary radiation during the operation.
Figure 1: Siremobil-Iso-C3D

Camera Augmented Mobile C-arm (CAMC) is constructed by attaching one camera
and two mirrors to the gantry of C-arm (Figure 2) [1]. With this simple construction
and the offline calibration for double mirrors, CAMC can really provide on-line
recovery of the projection geometry used by 3D reconstruction for low cost C-arm,
real time overlay of X-ray and optical image and speeding up positioning and
repositioning mobile C-arms with less radiation.
Figure 2: The basic construction of CAMC
Here, the basic idea and principle of CAMC will be shortly described. CAMC uses
one camera to recover the motion of the C-arm and therefore the X-ray projection
geometry during its rotational run around an object of interest [3]. The CCD camera
attached to the C-arm is virtually at the same location as the x-ray source. This is done
by a double mirror construction. A two step calibration routine has been done only
once at the moment the camera is attached [1]. Visual servoing is used to do
positioning and repositioning with pre-operative CT-data [4].
Real-time recovery of the projection geometry using CAMC
Real-time recovery of the projection geometry is useful in two points by CAMC: First
point is 3D tomographic reconstruction in medical imaging. Moreover, it also can do
overlay of 3D reconstruction and 2D optical images. The interesting observation that
even though the motion parameters are better estimated using the external sensor, the
projection geometry is more accurately estimated by the integrated optical camera [3].
At the end of this section, you can directly see the better result from CAMC compared
to using the external sensor.
3D reconstruction
Before the recovery of the projection geometry, the basic principle and steps for 3D
reconstruction using X-ray C-arm is shortly described here. A sequence of images is
captured during the C-arm rotation around the object of interest, and then all the 2D
projection images are filtered, back-projected into the volume and combined to form
the volume data [2]. For each 2D image the precise projection geometry has to be
determined for the 3D reconstruction process. The one way to recover X-ray
projection geometry is the direct estimation, in which a calibration phantom with Xray
opaque markers that is positioned such that it is visible in each frame has to be
used. The biggest advantage of the direct estimation is that it will get the most
accurate estimation of the projection geometry for each X-ray frame. The projection
matrices are obtained with this method as gold standard– all other methods are
compared to it [2]. However, marker points visible in the sequence of X-ray images
that overlay the object (patient’s anatomy) will influence the result of the 3D

econstruction in the medical imaging application. But this problem can be solved in
an offline procedure if the motion of the C-arm is reproducible.
The projection geometry has to be determined online if the motion of the C-arm is not
reproducible. The followings define the framework for online recovery of X-ray
projection geometry. First, we briefly introduce the X-ray projection geometry.
The X-ray projection geometry is represented by P a 3*4 homogeneous matrix of
projection [3].
U = P X, (1)
where U = [u; v; 1] T and X = [x; y; z; 1] T are the homogeneous coordinates of an
image pixel (2D) and corresponding 3D voxel of the world coordinate system.
P = [AR AT] (2)
where R, and T are extrinsic parameters. A is the intrinsic parameters.
In our case, we will model the X-ray imaging system by a pinhole camera model. The
intrinsic parameters of X-ray source will not be as constant as it can be assumed for
standard CCD camera. It is interesting to note that the intrinsic parameters of a
moving X-ray system depend in general on its extrinsic parameters [3].
For fixing the intrinsic parameters of X-ray source, we propose using virtual detector
plane. The intrinsic parameters of CCD camera only depend on the manufacturing of
the camera and the relation between image plane, focal point, and optical axis stay
fixed. In the case of C-arm X-ray system, focal point (X-ray source) and image plane
(X-ray detector) are far apart from each other, and the weights of both X-ray source
and detector the C-arm will cause minor torsion during the motion. Besides, the
optical axis (normal dropping from X-ray source on the detector plane) depends on
the orientation of the detector plane [3].
Figure 3: Pinhole camera model: Differences between CCD camera (top) and X-ray
imaging system (bottom)
We found a solution to solve this problem by introducing the concept of a virtual
detector plane [3]. We attach a number of makers to the X-ray source, and then Force

the markers projected to the image at pre-defined positions by image transformation
3*3 matrix H. The virtual detector plane is defined for all images by image
transformation such that these markers are projected to pre-defined positions.
The image transformation is described by a 2D-2D planar transformation H.
Pi VD = A VD · Ei = (Hi · Ai ) · Ei = Hi · Pi (3)
Pi VD is the projection matrix with the fixed intrinsic parameters for i-th frame. The
fixed intrinsic parameters are represented by A VD . The equation 3 gives us the way
how to use the original projection matrix in order to get a projection matrix with the
changed intrinsic geometry.
Another necessary component for recovering the projection geometry is the reference
frame. A pair of images (X-ray and optical) is taken for an arbitrary frame. The
projection geometry is determined simultaneously for both X-ray P X ref) and the
camera (P c ref) used as references.
The last thing is needed to do for recovering the projection geometry is the motion
estimation. An optical marker system, not visible in the X-ray image, is used to
determine the projection geometry of the CCD camera for each frame. The motion of
X-ray source between current and reference frame is computed from the estimated
CCD camera projection matrices.
P ^X i =P X ref · M X i with P c i =P c ref · M c i (4)
The one thing you should notice is that P ^X i projects 3D voxels onto the virtual
detector plane. We need to transform P ^X i to the original projection matrix Pi by using
the equation 3.
Experimental Results
The set of projection matrices that has been computed for each method is used to
reconstruct a test phantom. It consists of two cylindrical objects in an acrylic cover.
Attached to it is a ring made from titanic alloy also in an acrylic cover.

Figure 4: Experimental results of 3D reconstruction (from top to bottom): CAMC,
CHOU, DANI, gold standard and the test phantom used in the experiment
The results are shown in figures 4. The 3D reconstructed volumes are visualized as
orthogonal Maximum Intensity Projections (MIP). Both the gold standard of using
direct estimation and the CCD camera approach lead to better reconstruction quality
than the external tracking system method [3].
3D-2D Overlay using CAMC
The projection geometry of X-ray recovered by CAMC can be used to project the 3D
reconstruction data from CAMC to 2D images (simulated X-ray images) without
taking the actual X-ray image again. The surgeon can move C-arm freely and observe
the simulated X-ray images without X-ray exposure from different viewpoints if the
patient is immobilized.
This process also can be used in order to visualize the 3D reconstruction combined
with the camera images. Due to using optical imaging parameters as 3D volume
rendering parameters, then C-arm geometry and the result of 3D reconstruction from
CAMC are defined in the same coordinate system as the optical camera, which makes
the overlay the 3D reconstruction and camera images possible.
Figure 5: The superimposition of the reconstructed volume on top of optical images
Figure5 shows the experimental results of the overlaying the 3D reconstruction and
camera images of using pig’s knuckle. The left one is the camera optical image, the
middle one is the X-ray image and right one is the overlay image.
Real time overlay of X-ray and optical image using CAMC

Clinician usually does operation by looking at the screen where the X-ray images are
displayed. In this case, the medical (X-ray) images and patients are in different space,
which is not convenient. We want to put the X-ray images and patients into the same
space. One way is to overlay medical images and patient optical images. In the
previous section 2D-3D overlay, we have already got the overlay of the 3D
reconstruction and patient camera images. But the problem is that 2D-3D overlay
need 3D reconstruction first, which means inevitable much X-ray exposure. What we
want here is merging both X-ray and optical images without pre-computed 3D
reconstruction.
Without pre-computed 3D reconstruction, merging both X-ray and optical images
requires they are taken from same viewpoint. The X-ray source and the camera center
should be at the same place, which is not physically possible. However, we propose to
use two mirrors to bend the principle axis of the CCD camera in such a way that both
principal axes are aligned like Figure6 [1].
Figure 6: Conceptual design of the double mirrors
With the double mirrors construction, totally blending X-ray and optical images using
CAMC with double mirrors requires 1) Precise alignment of the principle axes of both
X-ray and optical imaging systems, 2) 2D-2D co-registration for correcting the
difference in the intrinsic parameters.
We can use a single offline calibration procedure to meet the both requirements, and
the calibration procedure includes 5 steps [1].
1. Position 4 or more markers on the X-ray detector plane and one or more extra
markers that are further from the plane
2. Take one X-ray image
3. Compute the image plane transformation based on the in-plane corresponding
markers (Figure7)

H
I I
Figure 7: Calibration, left one is the optical image and the right is the X-ray image
4. Move camera until the out of plane markers are superimposed, which aligns the
principle axes (Figure 8)
camera
XX-Ray
Figure 8: Calibration, superimposing out of plane markers
5. Fix the camera and record the final estimated planar transformation
With double mirrors, this system can continue blend the video images with the fixed
X-ray image. If the region of interest is immobilized, the clinician can work under the
guidance of this augmented video and no registration of the patient was required. The
clinician also can control the blending coefficient, between 0 (only X-ray image) and
1(only video image)
Experimental Results
In the experiment, the set of metallic objects are used. The objects are placed on the
table such that they occlude each other (Figure9)
Figure 9: The set of metallic objects
The Figure 10 shows us the results. The occluded parts can be seen in the X-ray
image and the combined image.

Figure 10: The left is the original video, the center is the overlay and the right is the
pure X-ray image
Visual Servoing of CAMC
Visual servoing are mostly used in robotic systems for control of robot manipulators
that is based on visual perception. Visual servoing involves the use of one or more
cameras and a computer vision system to control the position of the robot's endeffector
relative to the workpiece as required by the task. Here, it will control the Carm
using the visual information taken from the on-board camera.
The problem of positioning mobile C-arms, e.g. for down the beam techniques, as
well as repositioning during surgical procedures currently requires time, skill and
additional radiation. The Camera-Augmented Mobile C-arm (CAMC) with visual
servoing is able to speed up the procedure, simplify its execution and reduce the
necessary radiation. The advantage here is the use of visual servoing with no radiation
for positioning a mobile C-arm to obtain target images defined for the X-ray imaging
[4]. We propose two visual servoing algorithms providing elegant solutions for
performing down the beam techniques and repositioning with no additional radiation
for surgeon as well as for the patient. In the rest of this section, the basic principle of
using visual servoing and working steps will be described.
For Down-the-Beam positioning of the C-arm, we use a pre-operative CT to define
the Down-the-Beam axis. Intra-operatively we estimate the current pose of the C-arm
with regards to the CT data (figure 11). Position-based visual servoing is used to
move the C-arm towards the desired position. When we are close to the optimal
position, we generate the virtual target images and then witch to image-based visual
servoing [4].
Figure 11: Down-the-Beam C-arm positioning
The repositioning task is done by (optical) image-based visual servoing. A target
image from the current C-arm position is taken. When the C-arm is repositioned, the

current positions of the fiducials are compared to the target positions and joint
increments of the C-arm are computed (figure 12)[4].
Figure 12: Intra-operative repositioning
Experimental Results of C-arm Repositioning
The standard radiographic examination of the ankle for fracture detection will be
studied by using cadaver. Intraoperatively the precise mortise view is mandatory to
proof the correct reposition of the ankle [4].
Figure 13: ankle joint repositioning; 1: reference video; 2: reference X-ray; 3: blended
reference images; 4-8: images acquired during movement from start to reference
position of C-arm.
Advanced system: Integrating 2nd camera to CAMC
Camera Augmented Mobile C-arm is really useful for many interventional procedures.
However, this CAMC, in which no depth control is possible, just provides an accurate
positioning and guidance of instruments in 2D by doing real time overlay of X-ray
and optical images. This makes CAMC limited to applications where depth did not
matter, like in im-nail locking. Based on this CAMC, the advanced system is
developed by integrating a 2nd camera (Figure 14). The second camera is attached
orthogonal to the gantry such that its view is aligned with the X-ray image after a 90
degrees orbital rotation of the C-arm. This advanced system is also capable of depth
control during trauma surgery and orthopedic procedures using only one additional Xray
image and a second video camera that is rigidly attached to the C-arm [5].

Figure 14: The C-arm with two attached optical cameras.
CAMC uses second camera to track the instrument attaching optical markers. Using
invariance of cross ratio under projective transformation, we estimate the position of
the tip in the image. After one time calibration of the newly attached second video
camera we are able to show the instrument tip in the orthogonal X-ray view [5].
Notation Since we have images at different positions of the C-arm superscript 0
denote cameras and images at a 0 degree orbital rotation and superscript 90 denote
cameras and images acquired by the C-arm after an orbital rotation around 90 degree.
Furthermore subscript x is used for X-ray, g for gantry mounted, and o for the
orthogonal mounted cameras.
To get best results for the depth navigation, we have to ensure that after an orbital
rotation the optical center and axis of the X-ray gantry and the orthogonal camera are
aligned. The gantry mounted (1st) camera and X-ray source have already been
calibrated as described before. For aligning optical center and axis of X-ray to
orthogonal mounted camera, it can be done by physically aligning the gantry mounted
and orthogonal mounted camera after rotation [5]. After aligning optical center and
axis, 2D planar transformation remains to be computed for correcting intrinsic
parameters. Hgx maps the X-ray image I 90 x to the gantry mounted image I 90 g and Hog
maps the image of the gantry mounted camera I 90 g to the image of the orthogonal
mounted camera I 0 o. Finally, Hgx · Hog maps the X-ray image I 90 x to the orthogonal
mounted camera image I 0 o.
Experimental Results
With second camera, we still need to track the instruments. In the experiment, the
instrument is extended by three markers collinear arranged on the instrument axis. We
use retro-reflective circular markers that are illuminated by an additional light source
attached to the orthogonal camera (Figure 15 middle). According to these illuminated
markers and the invariance of cross ratio, the tip of the instrument can be tracked.
Figure 15 show us the results from the cadaver study.

Figure 15: First camera for lateral positioning (left), Second camera for depth tracking
(middle) and Superimposition of depth tracking onto the X-ray image (right)
Conclusion
Camera Augmented Mobile C-arm not only provides robust 2D AR-visualization, but
also 3D AR-visualization with 2nd camera. First cadaver experiment demonstrated
that the advanced system can be easily integrated into the clinical workflow while
reducing the radiation dose compared to other methods [5]. By using visual servoing,
the number of positioning steps is comparable to clinical routine but without any
radiation exposure. However X-ray images can be obtained at any time to check the
reached position or to check patient movement [4].
Future possible extension of CAMC
CAMC can overlay the medical images and patient images in real time, which
augments the surgeon‘s view and benefits many interventional procedures, but these
images are displayed by the monitor, so surgeons need to check monitor during
operation. What we want in the future is to let surgeons can concentrate on the patient
without checking monitor during operation.
References:
1. Navab, N., Mitschke, M., Bani-Hashemi, A.: Merging visible and invisible:
Two camera-augmented mobile C-arm (CAMC) applications. In: Proc. IEEE
International Workshop on Augmented Reality, San Francisco, CA, USA
(1999) 134–141
2. N. Navab, A. Bani-Hashemi, M. S. Nadar, K. Wiesent, P. Durlak, T. Brunner,
K. Barth, R. Graumann. 3D reconstruction from projection matrices in a Carm
based 3D-angiography system, MICCAI, 1998.
3. M. Mitschkea, N. Navab: Recovering Projection Geometry: How a cheap
camera can outperform an expensive stereo system, IEEE, 2000.
4. N. Navab, S. Wiesner, S. Benhimane, E. Euler, S.M. Heining; Visual Servoing
for Intraoperative Positioning and Repositioning of Mobile C-arms; Proc. Of
MICCAI 2006
5. J. Traub, T.H. Heibel, P. Dressel, S.M. Heining,R. Graumann, N. Navab; A
Multi-View Opto-Xray Imaging System Development and First Application in
Trauma Surgery; Proc. Of MICCAI 2007