Abstract

Many modern MedTech innovations rely on deep learning systems, yet their internal decision-making processes remain largely opaque. This raises an important question: when does accuracy alone suffice, and when does medicine require deeper understanding?

Introduction

MedTech is built on the idea that better tools lead to better outcomes, and few tools have arrived with more promise than deep learning (DL). Inspired by the neural architecture of the brain, DL systems train from labelled examples to detect patterns using layers of interconnected artificial ‘neurones’. Such approaches have transformed medicine with the emergence of algorithms that can detect tumours and interpret ECGs. Yet behind this promise lies a deeper problem that stems from the complexity of DL: it is a black box. DL models can only be understood in terms of their input and output as the internal algorithm linking the two remains virtually unknown. Given the high accuracy of these approaches, an important question is whether it matters that we don’t fully understand how they reach their conclusions. The answer depends entirely on medical context. In research, opacity may be acceptable for accelerated discovery, but in clinical use, it becomes a risk we cannot afford to ignore.

Research – When Observation May be Sufficient

Many great breakthroughs in medicine began not with a theory but with a pattern someone was interested to follow. A powerful example is the recent development of Casgevy, a CRISPR-Cas9 therapy that treats sickle cell disease (SCD). This treatment works by silencing the BCL11A gene in a patient’s stem cells to promote the re-expression of foetal haemoglobin (HbF).

On the surface, this seems like a strange solution to fixing the sickle-shaped red blood cells that block vessels in adults – how can HbF re-expression correct SCD? This idea didn’t originate from first-principles engineering, but simple observation. In the 1950s, clinicians noticed that SCD patients who continued producing HbF beyond infancy were largely protected from the disease. Nobody could explain the molecular mechanism for decades, but that observation alone drove the hypothesis that reactivating HbF could be therapeutic. Although this incidental finding led to a highly celebrated FDA drug approval, its trajectory for discovery is not unique. Penicillin, for example, was also recognised as a powerful therapeutic long before its underlying mechanism of action was understood. 

This is the paradigm that DL is beginning to replicate, but at a much faster pace. By scanning large datasets from thousands of samples, these models can identify candidate drug targets in hours without underlying mechanistic knowledge. In 2020, MIT researchers used DL in exactly this way by analysing a molecular library with thousands of possible bacterial targets. This led to the discovery of Halicin, a novel antibiotic that is effective against drug-resistant bacteria. While a clinician noticing a patient pattern and an algorithm processing genomic data may seem worlds apart, in both cases the pattern precedes the mechanism, and a human scientist still performs validation. The critical difference is velocity; DL dramatically accelerates the pace at which new discoveries in MedTech can emerge.

Crucially, medicine has also never required complete mechanistic understanding as a prerequisite for clinical benefit. FDA approval does not even strictly require a known mechanism of action. Furthermore, paracetamol has been prescribed for over a century without fully understanding how it relieves pain, and nobody has seriously argued that we should stop using it. I would apply that same logic here: in the impact-driven world of medical research, opacity is a worthy sacrifice for the rapid development of treatments that work. This highlights that while DL does not necessarily deliver understanding, it delivers direction. DL acts as a useful compass that can point researchers towards novel hypotheses.

Clinical Deployment: When the Compass Points the Wrong Way

The same compass is far less reliable when the stakes shift from hypothesis generation to patient diagnosis and treatment. Here, opacity can become dangerous in ways that can directly impact patient care.

A clear example of this is considering the DL models built during the COVID-19 pandemic to try and detect COVID-19 from chest X-rays. On internal testing, one model performed exceptionally well, achieving an AUC of 0.992 (measure of diagnostic accuracy where 1.0 is perfect). Researchers attempted to understand how this model was working through saliency mapping, a technique that highlights which parts of an image most influence the model’s decision. However, this revealed a critical flaw: the model was almost entirely ignoring the lungs. It was instead relying on positioning markers, projection labels, and formatting differences that varied systematically between patients. As the COVID-positive and COVID-negative images had largely come from different sources, the model had simply learned to identify the dataset, not the disease. Deployed at a new hospital, the AUC collapsed to 0.76 and the model’s high accuracy had disappeared. 

In this example, the flaw in the DL model was only caught because it was visible through saliency mapping. Many other DL applications in clinical medicine are not image-based at all and lack parallel approaches to help explain how they work. Models predicting sepsis risk, patient mortality or treatment response using thousands of variables in electronic health records offer no image to interrogate. When such a model flags a patient as high-risk, there is no window into how the black box determined this to reveal potentially similar methodological problems.

Although this is concerning, the use of explainable AI methods here does reflect a broader and encouraging trend. Researchers are increasingly developing sophisticated methods to understand how DL models actually work, which could help to tackle the black box problem head-on. Another prominent example is SHAP (Shapley Additive Explanations), which aims to quantify how much each input variable contributes to a given prediction. However, these tools remain imperfect as SHAP values can be unstable across different model runs. Furthermore, neither approach resolves the fundamental question of whether a model is reasoning correctly or simply producing a correct output for the wrong reason. 

These limitations are not merely technical inconveniences, but carry real consequences that sit at the heart of medical practice:

1. Liability: When a human clinician misdiagnoses, the reasoning is recoverable: a failure in judgement or training that can be reconstructed and corrected. However, when a DL system misdiagnoses, there is no such trail as the error is structural and silent. Legal frameworks built around duty of care and documented clinical reasoning are not designed for this.
2. Patient autonomy: Patients have a right to understand the basis of decisions made about their health. If a DL system clears a mammogram or recommends against a statin, and neither the clinician nor the system can explain why, meaningful informed consent is compromised. An algorithm may be correct more often than a human, but correctness and transparency are not the same standard.
3. The nature of error: While a doctor’s misdiagnosis is individual and contextual, an algorithm’s error is systematic and scalable by nature. This is particularly troubling when we consider who is most likely to be harmed. Medical training has long been criticised for centring on a narrow demographic with the classic textbook patient being male, white, and middle-aged, leaving clinicians less equipped to generalise to others. As DL models inherit these biases through training on unrepresentative historical data, they risk promoting inequality at scale.

Conclusion

Knowing a system works well is not the same as knowing when it doesn’t. In clinical medicine, this distinction fundamentally matters. Although explainable AI offers great promise, it is currently insufficient to meet the accountability that clinical medicine demands. While embracing DL as an engine for hypothesis generation in research will accelerate discovery, bedside deployment requires more than statistical accuracy. It demands systems that safeguard patient autonomy, preserve accountability, and prevent scalable harm to vulnerable groups. To further complicate matters, the rapid pace of DL research means the timeline from benchside experimentation to bedside deployment is rapidly shrinking. This makes the exact boundary of where we can accept algorithmic opacity incredibly difficult to define. 

Definitions

1. Black box = A system in which the internal decision making process is not transparently interpretable by humans, even if its inputs and outputs are known
2. Saliency mapping = A technique used in image-based deep learning models that highlights which regions of an input image most influenced the model’s prediction
3. Area Under Curve (AUC) = Metric summarising the performance of a diagnostic model across all classification thresholds with values ranging from 0 to 1.0 (perfect discrimination)
4. Explainable AI = Class of methods designed to make the behaviour and decision-making processes more interpretable to humans
5. Shapley Additive Explanations (SHAP) = Method that estimates the contribution of each input variable to a model’s individual prediction

References

1. Lettre, G. and Bauer, D.E., 2016. Fetal haemoglobin in sickle-cell disease: from genetic epidemiology to new therapeutic strategies. The Lancet, 387(10037), pp.2554-2564.
2. Stokes, J.M., Yang, K., Swanson, K., Jin, W., Cubillos-Ruiz, A., Donghia, N.M., MacNair, C.R., French, S., Carfrae, L.A., Bloom-Ackermann, Z. and Tran, V.M., 2020. A deep learning approach to antibiotic discovery. Cell, 180(4), pp.688-702.
3. DeGrave, A.J., Janizek, J.D. and Lee, S.I., 2021. AI for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7), pp.610-619.