11 January 2023
A consortium that brings together the Champalimaud Foundation and several Portuguese companies will receive funding from the Portuguese Plano de Resiliência e Recuperação to improve, through digital technologies, the precision of breast cancer surgery.
The name of the project is “MetaBreast - Metaverse for Breast Cancer Surgery”. Developed specifically in the field of breast cancer, it may one day also be applicable in other surgical contexts. For now, its goal is to “develop and test a new medical device to visualize malignant tissue in real time inside the patient’s breast and within the operating room”, states the document presenting the project. One of MetaBreast’s participants is the Champalimaud Foundation’s (CF) Breast Unit, represented by breast cancer surgeon Pedro Gouveia.
“MetaBreast technology will seamlessly integrate advanced computer vision with clinical imaging data, allowing for live, digital, non-invasive, image-guided precision surgery for breast cancer”, adds the same statement.
In addition to the CF, the other participants in the project are the Portuguese companies INESC-TEC Porto, BMD Software (a spinoff from the University of Aveiro) and IT People. And now, MetaBreast has been approved to receive 2.7 million euros in funding from the Portuguese Plano de Resiliência e Recuperação, 1.4 million of which will be allocated to the CF.
MetaBreast is one of the many projects integrating “Health from Portugal” (HfP), a newly-launched megaproject that aims to reform health. HfP was promoted by a consortium of more than 90 members of the Health Cluster Portugal (a not-for-profit association that brings together universities, hospitals and health-related companies, among others).
Health from Portugal is spearheaded by the Portuguese company Prologica, and aims at the creation, development and production of advanced services and products for the health market. It will receive a total of 94 million euros from the so-called “Agendas Mobilizadoras para a Inovação Empresarial do Plano de Resiliência e Recuperação”, an initiative launched by the Portuguese government in 2021.
Concerning MetaBreast itself, it is specifically “a project for planning surgeries related to breast cancer”, says Pedro Gouveia. More to the point, the idea is that, thanks to augmented reality, virtual reality and artificial intelligence technologies, visualising a breast tumor and its margins during surgery can become akin to seeing it directly, inside the patient’s body – “through the skin”, as it were. This would increase surgical accuracy, potentially allowing for complete extraction of breast tumours during a single surgical act.
To do this (and here comes the metaverse), it will be essential to use augmented reality headsets connected to the internet to be able to superimpose the patient’s medical images (namely magnetic resonance images and mammograms) on a precise representation of the patient’s body and its position on the operating table.
The idea has already been tested in the last few years by Pedro Gouveia at the Champalimaud Foundation. In January 2020, his team used an augmented reality headset for the first time, in the Breast Unit’s operating room, to give the surgeon a virtual image of the tumour to be extracted and its accurate location. Now, thanks to the new funding, the MetaBreast project – which will last until the end of 2025 – can be further advanced.
It should be noted that accurately visualising and locating the tumour was only possible by “fusing” medical images of the tumor with a 3D personalised model of the surface of the patient’s torso, with the patient lying supine on the operating table. This fusion is a delicate task, which relies on complex algorithms and computing power to make the medical images, taken in different positions and conditions, superimposable on the 3D model (just to give an example, the breast needs to be compressed, deformed, to obtain a mammogram).
The first step of this new phase of the MeatBreast project will be the creation of an online database of “pseudo-anonymised” data (pseudo-anonymisation ensures higher data protection) from several hundred breast cancer patients, including annotated images, photos of their torso and other relevant individual attributes. Patient registration will be performed at the FC by the Breast Unit, in a space called the “Medical Metaverse Room” inside its brand-new Digital Surgery LAB. In that room, Pedro Gouveia explains “there is an operating table to register the patients in a supine position, and spatial 3D cameras to create 3D images of the patients’ torso.”
Also at the Digital Surgery LAB, 2D photographs of the patients’ torso will be taken with the help of a robot called Pink, developed for another Breast Unit project – the Cinderella project –, led by breast surgeon Maria João Cardoso and which aims to improve and personalise the aesthetic results of breast cancer surgeries.
Taking all that data, it will be possible to generate a database of more than 10,000 “digital”, “synthetic” patients. “The creation of this online image-based data structure, which will be used for research, will be the job of BMD Software”, says Pedro Gouveia. INESC-TEC Porto will be in charge of developing the algorithms and artificial intelligence needed to fuse medical images and 3D models. Lastly, IT People, a company specialised in graphic computing, will help develop “a new way of visualising image data through augmented reality and virtual reality”, he adds.
In yet another space of the Digital Surgery LAB, known as the “Immersive Surgical Arena” and housed in the former warehouses of Doca Pesca, right beside the FC buildings, engineers will work at the design and testing of MetaBreast’s new medical device. This second room “mimics an operating room”, points out Pedro Gouveia.
The ultimate step of the project will consist in testing the device in the operating room on breast cancer patients, through a clinical feasibility study.
According to the presentation document mentioned above, “MetaBreast envisages a new surgical era by developing a radically new digital diagnostic tool (medical device) within a medical metaverse interface for the operating room of the future, to digitally anchor radiological data into the patient’s body”.
