Imaging and printing in plastic and reconstructive surgery part 2: emerging techniques

Introduction

A plethora of imaging modalities has been used in plastic and reconstructive surgery to aid preoperative planning, intraoperative guidance and medical education.^1,2 Conventional tomographic imaging modalities such as computed tomography angiography (CTA) and magnetic resonance angiography (MRA) remain relatively affordable and commonly accessible.^3–8 As a result, clinicians have investigated novel technologies to expand their use such as image-guided navigation systems, augmented reality (AR), virtual reality (VR), holograms and machine learning (ML).

In the second of a two-part series, we evaluate emerging 3D imaging and printing techniques based on CTA and MRI.

Methods

We reviewed the published English literature from 1950 to 2017 from well-established databases such as PubMed, MEDLINE®, Web of Science and EMBASE. We included all studies that analyse 3D imaging and printing techniques used in surgery, especially plastic and reconstructive surgery. We used search terms such as ‘3D imaging’, ‘CTA’, ‘MRA’, ‘3D image software’, ‘simulation surgery’, ‘stereotactic navigation-assisted surgery’, ‘augmented reality’, ‘virtual reality’, ‘hologram’, ‘automation’, ‘machine learning’, ‘artificial intelligence’, ‘preoperative planning’, ‘intraoperative guidance’, ‘education’, ‘training’ and ‘customised implant’. We also retrieved secondary references found through bibliographical linkages.

Through our literature review, we qualitatively analysed hardware and software programs used for image-guided navigation-assisted surgery, AR, VR, holograms and ML, evaluating their cost (affordability arbitrarily being defined as costing less than AU$500) and up-to-date clinical applications. Papers were assessed using Oxford Centre for Evidence-Based Medicine levels of evidence.⁹

Results and discussion

Recent technological advances have led to the use of image-guided navigation systems, AR, VR, holograms and ML in surgical planning.

Image-guided navigation systems

An image-guided navigation system tracks surgical instruments in real time and matches their location to the preoperative CTA and MRI for viewing them intraoperatively.^10,11 The earliest system used external stereotactic frames fixed to the skull or other bony landmarks.¹² Modern frameless navigation systems using fiducial markers,¹³ surface landmarks¹⁴ and surface-matching laser registration¹⁵ are faster, safer and more convenient.¹⁶ As a result, stereotactic navigation is used routinely in neurosurgery,¹⁷ spinal surgery,¹⁸ orthopaedic surgery,¹⁹ craniofacial surgery,²⁰ ear, nose and throat surgery²¹ and endovascular surgery.²²

In plastic surgery, Rozen and colleagues demonstrated that registration systems using fiducial markers—six to seven in deep inferior epigastric artery perforator (DIEP) and nine to 10 in anterolateral thigh (ALT) flaps—are reliable for viewing CTA-derived perforator anatomy.^23–26 Durden and colleagues developed a novel electrocautery pen attached to a stereotactic frame and reported a global error of 2.1–2.4 mm during DIEP flap harvest.²⁷ However, the longer, heavier diathermy handle may compromise surgical dexterity and requires its large reference frame to be fixed to the operating table.

In an interesting application, Chao and colleagues developed a robot (KUKA Lightweight Robot; KUKA, Augsburg, Germany) that can perform osteotomy on a 3D-printed acrylic fibula with the aid of stereotactic navigation.²⁸ Out of 18 robotic osteotomies executed, it reported average linear variation of 1.3 ± 0.4 mm and angular variation of 4.2 ± 1.7 degrees. It remains to be seen how this can be translated in vivo but its potential is intriguing.

Overall, navigation systems are seldom used in soft-tissue surgery due to lack of reliable bony landmarks and have been superseded by augmented and virtual reality platforms (see Table 1).

Augmented reality, virtual reality and holograms

In comparison with two-dimensional (2D) imaging modalities, AR, VR and holograms provide natural 3D visual perception and haptic feedback respectively. First described by Boeing engineers Caudell and Mizell in 1992,²⁹ in AR real-time virtual images are superimposed on the view of one’s real environment.³⁰ These images can be displayed directly onto an object in real life, also known as the projection method, or indirectly onto a portable device, such as a head-mount display or smartphone.³¹ In contrast, in VR, one’s entire visual perception is completely shrouded by a computer-simulated graphics environment.³⁰

Virtual reality

Virtual reality is an attractive platform on which to generate anatomically accurate surgical simulations in order to perform preoperative planning or medical training and enable visual communication with multidisciplinary team members and patients.³² Arora and colleagues have shown that mental practice using VR simulators can significantly improve the surgical skills of novice surgeons in laparoscopic cholecystectomy (p < 0.05).³³ However, currently most VR surgical simulators are pre-programmed, offer only limited interactions and exhibit such low image quality that it impedes the immersive experience.^34–36

Augmented reality

Augmented reality produces an extended ‘layer’ or field of view that leads to intuitive real-time 3D visualisation of anatomical structures. Currently, most AR devices are expensive, slow and complicated. Nonetheless, their potential application has been explored in numerous surgical specialities including calibrating stereotactic instruments in neurosurgery,³⁷ fashioning craniofacial implants in maxillofacial surgery,³⁸ enhancing visualisation in laparoscopic surgery³⁹ and sentinel lymph node biopsy in head and neck cancer⁴⁰ and breast cancer surgery.⁴¹ In plastic and reconstructive surgery, AR appears to be most useful for preoperative planning, intraoperative image navigation and surgical training (see Table 2 and Figure 1).⁴²

Hummelink and colleagues described a projection-based direct AR technique using an affordable hand-held projector and proprietary software suites in three case series.^43–45 In the first series, they projected a 3D-reconstructed CTA image of DIEPs onto the abdominal wall and demonstrated its high accuracy (84.3 vs 56.9%, p = 0.03).⁴⁵ In the following series, they extended this application by including the location of inguinal lymph nodes.⁴⁴ In the latest series, they calculated the required flap volume and dimensions using 3D surface scanning and projected the combined 3D-reconstructed image to aid flap design and planning.⁴³ One of the major limitations of this technique is operator dependence, since the projector must be held steadily above the patient at the correct height without significant tremor. Sotsuka and colleagues attempted to resolve this by mounting the projector onto a fixed handstand⁴⁶ but its reliability remains to be seen.

In animal studies, Jiang and colleagues developed a highly accurate (3.5 mm) direct AR technique for raising thoracodorsal artery perforator flaps that does, however, require invasive positioning of the image registration system via percutaneous screws.⁴⁷ Gan and colleagues developed a compact direct AR technique consisting of a mini-projector and a near-infrared camera to detect skin perfusion after tail vein injection of ICG dye.⁴⁸ However, their system is too small for clinical application.

Recent advances in AR and VR

Conventional AR devices require large stereoscopic towers for image registration and viewing that are inconvenient and occupy space in the operating theatre. However, there are now wearable devices that can carry sufficient computing power for AR and several investigators have developed bespoke wearable devices for surgical application.^49,50 Mela and colleagues report a device capable of projecting fluorescent angiography, 2D ultrasound and 3D CTA with depth perception in a compact, user-friendly interface.⁴⁹ Similarly, Liu and colleagues developed a compact, wireless, battery-operated device for hands-free viewing of fluorescent angiography for sentinel lymph node biopsy and tumour cell localisation.⁵⁰ The latest and most promising wearable AR device was Google Glass (Alphabet, Mountain View, CA, USA). In plastic and reconstructive surgery, clinicians reported its benefits for viewing images and recording videos.^51–54 Unfortunately, in 2015 Google Glass was taken off the market due to persistent software bugs and privacy concerns.⁵⁵

Fig 1.Projection of ALT perforators preoperatively using CTA-based direct augmented reality performed using OsiriX software (Pixmeo) and Philips PicoPix pocket projector (Koninklijke Philips NV, Amsterdam, The Netherlands)

The purple line delineates traditional anatomical landmarks. The mark on the line correlated exactly with the location of the ALT perforator. ALT = anterolateral thigh perforator CTA = computed tomographic angiography

Recently, researchers in Australia have developed a high-resolution, immersive 3D AR and VR environment using integrated supercomputers and multiple projectors with a cylindrical matrix of stereoscopic panels.⁵⁶ This bespoke CAVE2^TM (Monash University, Clayton, Victoria, Australia) consists of 80 high-resolution, stereo-capable displays producing an 8-metre diameter, 320-degree panoramic view (see Figure 2). Medical images can be processed relatively easily by a dedicated laboratory technician and the clinician can view them realistically in a 3D manner as if they are ‘walking through’ the anatomy. Currently, the set-up is too large to be portable and it is also expensive, but as lithium-ion batteries improve and technology becomes more mobile, the potential of such technology being transferred to a portable head-mounted display appears enticing.

Holograms

A hologram exhibits reflective auto-stereoscopic (that is, no wearable device) 3D visuals that contain hogels (holographic elements instead of pixels or voxels), where each hogel contains up to one million different perspective views. Hackett and colleagues evaluated the role of holograms in teaching cardiac anatomy to 19 volunteers (10 intervention versus nine control) and found a superior overall test performance after using it (89% vs 68%, p < 0.05).⁵⁷ Furthermore, volunteers demonstrated a trend in lower mental effort required in learning (4.9 vs 6.0, p = 0.16). Recently, Makino and colleagues have added tactile feedback to holograms by using concentrated ultrasonic energy.⁵⁸ However, this technology has yet to advance beyond the prototypic stage.

Machine learning

Machine learning is a branch of artificial intelligence that uses a computer algorithm to aid clinical decision making and to predict clinical outcomes based on knowledge acquisition from data mining of historical examples without explicit programming.^59–61 The algorithm statistically analyses each hypothesis, compares multiple combinations and yields data models that are descriptive or predictive in nature. Machine learning has already transformed popular search engines such as Google (Google LLC, Mountain View, California, USA)⁶² and speech recognition software on smartphones such as Siri (Apple Inc, Cupertino, California, USA).⁶³ Owing to an ever-growing volume of digitalised clinical data, ML presents a superior form of data interpretation to the traditional statistical methods.⁶⁴

Machine learning techniques can be classified according to their mathematical structure: predictive, where learning is supervised by using pre-labelled data sets⁶⁵; descriptive, where learning is unsupervised and similar data points are clustered⁶⁶; and reinforcement, where ideal behaviour is determined by computer based on a simple reward feedback system on their actions.⁶⁷ Evidently, it is difficult for non-statistically inclined clinical investigators to analyse how an algorithm has reached its conclusion.^68,69 As a result, when using ML, clinicians need to collaborate with data scientists who can accurately evaluate the validity of the output obtained.⁷⁰

In the last decade, investigators have applied ML to improve clinical challenges in various fields within plastic surgery as a diagnostic and predictive tool. In melanoma detection, Safran and colleagues conducted a systematic review of 50 different ML screening techniques and found a mean sensitivity of 87.60 per cent (95% confidence interval: 72.72–100) and a mean specificity of 83.54 per cent (60.92–100).⁷¹ Encouragingly, there was no statistically significant difference between ML and dermoscopy examination by experienced professionals.⁷² In craniofacial surgery, Mendoza and colleagues used a statistical shape model to help diagnose non-syndromic craniosynostosis from CT.⁷³ The algorithm yielded a sensitivity of 92.30 per cent and a specificity of 98.9 per cent, similar to the trained radiologists.

In burns surgery, Yeong and colleagues developed an ML algorithm to analyse reflectance spectrometry images and assess burns area and depth.⁷⁴ They demonstrated an average predictive accuracy of 86 per cent. In free flap reconstructions, Kiranantawat and colleagues developed an ML-based smartphone application, SilpaRamanitor, that can predict vascular compromise from 2D photographs with an overall sensitivity of 94 per cent, a specificity of 98 per cent and an accuracy of 95 per cent.⁷⁵ In hand surgery, Conforth and colleagues developed an algorithm capable of estimating the likelihood of tissue-engineered peripheral nerve graft take at 92.59 per cent accuracy.⁷⁶ In aesthetic surgery, Gunes and colleagues developed an automated classifier of facial beauty by analysing 165 images of attractive female faces as graded by human referees.⁷⁷

Fig 2.Evolution of 3D imaging and printing techniques from 3D-reconstructed images and a basic 3D-printed model (A1–4) to clinically useful 3D printing applications in perforator-based breast reconstructive surgery (B1–6) and advanced image analysis technology such as augmented reality and CAVE2^TM facility (C1–6)

(A1) 2D-reconstructed CTA image of the abdominal wall vasculature. (A2) 3D-reconstructed CTA image of the same patient in A1. (A3) Segmented image of the DIEA of the same patient in A1. (A4) 3D-printed model of the DIEA in A3
(B1) Segmented image of the abdominal wall and DIEA that spurned the idea of creating a template for preoperative planning. (B2) A patient-specific bespoke ‘DIEP template’ is 3D printed and placed on the patient’s abdominal wall to help locate the DIEA perforator and its pedicle. (B3) This information is used for flap design. As 3D printing techniques advanced, we were able to create both a standard DIEP template (B4) and a ‘perforasome template’ (B5), which can additionally identify each perforasome (B6)
(C1–6) Augmented reality can significantly reduce the time and labour cost involved in 3D printing by enabling direct viewing and real-time interaction with the image data. (C1) Our published direct ARC (augmented reality CTA) technique set-up using a hand-held projector demonstrating 2D-reconstructed (C2) and 3D-reconstructed (C3) images on the patient’s abdomen. As technology advances, we envision that greater software processing power will enable display of greater anatomical information, such as the intramuscular course of a DIEP (C4) and translation into a user-friendly, interactive platform for clinicians (C5). (C6) The latest CAVE2^TM facility (Monash University, Clayton, Victoria, Australia) housing 84-million pixel stereoscopic display with powerful real-time motion tracking capability will enable interactive, seamless visualisation of relevant anatomy for preoperative planning and collaborative discussion
CTA: computed tomographic angiography DIEA: deep inferior epigastric artery DIEP: deep inferior epigastric artery perforator. Reprinted with permission, Chae and colleagues^78–80

Conclusion

Most studies of image-guided navigation systems, AR, VR, holograms and ML have been presented in small case series and they remain to be analysed using outcomes-based validation studies. Image-guided navigation systems are used less frequently in soft tissue surgery, in comparison with orthopaedic and neurosurgery, due to unreliable landmarks being available for image registration. Augmented reality platforms such as CAVE2^TM which leads to intuitive real-time 3D visualisation of anatomical structures, appear promising. Machine learning is a rapidly emerging, disruptive technology that may become highly useful as a diagnostic and predictive tool. Together, they illustrate an exciting future where clinicians will be armed with numerous intuitive technologies for surgical planning and guidance.

Disclosures

The authors have no conflicts of interest to disclose.

Funding

The authors received no financial support for the research, authorship and/or publication of this article.