When Algorithms Reach Their Limits: New Approaches to Analyzing Spatial Transcriptomics
Spatial transcriptomics enables researchers to simultaneously capture gene activity and the spatial organisation of individual cells. This makes it possible to study complex tissue structures and so-called spatial niches, an important approach in biomedical and translational research. But how reliable are the bioinformatics tools used to analyse these structures? A research team led by Dr Naveed Ishaque, group leader for Computational Oncology in Roland Eils’ Digital Health department at the Berlin Institute of Health at Charité (BIH) compared different methods in a comprehensive benchmarking study and encountered fundamental challenges in evaluating such tools. These experiences led to the development of SACCELERATOR, a new approach that combines different analytical methods and incorporates the expertise of tissue specialists into the analysis. In this interview, Dr Naveed Ishaque explains the limitations of conventional benchmarking approaches, why human expertise remains essential for interpreting spatial transcriptomics data, and what the findings mean for translational research.
Why did you decide to publish this project despite its challenges?
It was precisely the problems we encountered that motivated us to share our experiences. We often see benchmarking studies that make definitive statements about the performance of computational tools, without the groundwork needed to evaluate them fairly. For some research tasks, e.g. protein structure prediction, there is a long history that has helped establish communities that have defined clear aims, evaluation procedures, and ground truths. For other research areas pushed by novel technologies such as single-cell and spatial transcriptomics, there is a comparative lack of maturity. Because of our own experiences, we thought it would be more useful to share an honest account of the challenges than to present benchmark rankings at face value.
Was there a point when you thought the project had failed?
Yes! Despite the final story, we actually did want to perform a traditional benchmark. We slowly started to observe inconsistencies with our expectations, but the clearest issue we faced was when we tried to reproduce the results of a well-established tool, BayesSpace (Zhao et al., Nature Biotechnology, 2021, https://doi.org/10.1038/s41587-021-00935-2), applied to a well-studied dataset of the human brain. We could not reproduce the published results, which made us question our own setup. So we looked at 13 other studies that had run BayesSpace on the same dataset, and none of them reproduced the original numbers either, and they were inconsistent with each other. That was the moment we pivoted our study to focus on reporting more about the issues, and this precise example ended up being the focal point of our first major result and figure.
What surprised you most about the results?
I think the biggest surprise I encountered was the lack of care taken in establishing “ground truths”. These ground truths are supposed to represent plausible biology as determined by experts and subsequently used to evaluate the performance of computational methods, with the idea that the best computational methods mimic human behaviour most closely. We were surprised to find that around half of the datasets we evaluated had computer generated ground truths, which now means the benchmarking exercise only reveals which methods are most similar to those that created those so-called ground truths. While this revealed the seemingly obvious result that the computational tools which generated the ground truths scored highest in the benchmark, we were surprised that this wasn’t always the case.
For one mouse brain dataset (Shi et al., Nature, 2023, https://doi.org/10.1038/s41586-023-06569-5), the ground truth was generated with a tool called BASS (Li et al., Genome Biology, 2022, https://doi.org/10.1186/s13059-022-02734-7). In published benchmarks BASS scored near the top, as you would expect, but not at the very top. In effect, another tool matched the BASS-derived ground truth slightly better than BASS itself did!
Can different analysis settings lead to different conclusions?
Absolutely. I previously highlighted our experiences in comparing our results running the BayesSpace tool against others who did the same (see response to question 2). In addition, by changing a tool's settings, and only choosing settings the developers themselves recommend, we saw some tools swing from best in class to worst in class. In some cases, the choice of settings mattered more than the choice of the tool.
How reliable are the so-called ground truths?
In my opinion, the ground truths are probably the major limiting factor in conducting meaningful benchmarking studies. For evaluating spatial transcriptomics data, the limitations are two-fold. First, the ground truth definition is static and doesn’t consider different scales of biology. We found that some brain datasets defined the entire isocortex ,the outer layer of the brain responsible for higher-level thinking and perception, as a single region in the ground truth, however, some computational tools identified layering patterns in the isocortex region, which would result in lower reported performance of those tools as they did not perfectly match the scale of the ground truth features, despite layering patterns in the isocortex being biologically meaningful and well understood. Second, we found that many ground truths were not defined by human experts. Some were defined computationally (see response to question 3) and others were defined by non-experts. In these situations, performance metrics now indicate how similar the results are to other tools, and to non-experts.
What are the consequences of inaccurate analyses for patients?
Identification of spatial niches and tissue structures is highly relevant for patients. For example, in principle, immune cells found within a tumour niche may indicate that immunotherapies work for that patient, but if those immune cells are found outside of the tumour, perhaps in a stromal niche, then immunotherapy may not be effective. Therefore, defining these spatial niches and tissue structures provides very important context in which the very same cells can behave differently in different niches, and that can affect whether a therapy works.
Why is human expertise still essential despite advances in AI?
While I strongly believe that AI is and will continue to transform biomedical research, one limitation is that current AI is limited to our current understanding. While we have a strong and well documented understanding of some research areas (e.g. protein structures), newer areas such as spatial transcriptomics are not well understood. This means that we still need to treat the postulations of AI with caution. Given that modern biomedical research is highly multidisciplinary, there is no substitute for a team of experts (even if it is just to validate with authority that the results of AI methods are indeed correct). To address this challenge, we developed SACCELERATOR, a benchmarking and analysis framework that unifies diverse niche and spatial structure detection tools for spatial transcriptomics data. A core element of SACCELERATOR is the involvement of tissue experts, pathologists and histologists in an expert-guided workflow. SACCELERATOR can run more than 20 computational tools and evaluate results on spatial transcriptomics data and provides a guide on how best to move forward with interpretation and further analysis. Using this approach, we identified a robust molecular subdivision within a thalamic nucleus in the brain that had not been described at that granularity and identified a possible transition from dysplasia to neoplasm in a colon cancer sample.
Was the SACCELERATOR framework the project's biggest success?
No. While a lot of work and thought went into the development of the platform, we strongly believe that sharing open and honest experiences about the challenges we faced is more valuable. These negative results and problems are usually never highlighted, so benchmarking studies that are meant to guide users end up causing confusion, because the caveats are buried.
Did this project change the way you think about scientific research?
Absolutely. The problems we faced were not just our problems. They are real problems that the community faces. Having presented the results of our studies at various talks and conferences, I was surprised by the level of engagement and discussion that our results generated. In particular, when presenting at EMBL Heidelberg, many researchers were glad to know that the issues they faced were not isolated and echoed the same concerns that we encountered. This experience has served as a wake-up call for me. While I have understood the importance of good research practices, this study really showed me the value of honest and thoughtful reporting.
What are the implications of your findings for translational research?
Spatial transcriptomics is being extensively used in defining new paradigms for precision medicine and biomarker identification. Inaccurate analysis could therefore mean that subsequent clinical trials follow up on false leads or promising targets are simply missed. We are excited to be collaborating in an effort led by the group of Nikolaus Rajewsky at the Berlin Institute for Medical Systems Biology (MDC-BIMSB), involving the BIH and the Charité Comprehensive Cancer Center to establish a Spatial Molecular Tumor Board.