Skip to main content

Towards optimal deep fusion of imaging and clinical data via a model-based description of fusion quality.

Publication ,  Journal Article
Wang, Y; Li, X; Konanur, M; Konkel, B; Seyferth, E; Brajer, N; Liu, J-G; Bashir, MR; Lafata, KJ
Published in: Med Phys
June 2023

BACKGROUND: Due to intrinsic differences in data formatting, data structure, and underlying semantic information, the integration of imaging data with clinical data can be non-trivial. Optimal integration requires robust data fusion, that is, the process of integrating multiple data sources to produce more useful information than captured by individual data sources. Here, we introduce the concept of fusion quality for deep learning problems involving imaging and clinical data. We first provide a general theoretical framework and numerical validation of our technique. To demonstrate real-world applicability, we then apply our technique to optimize the fusion of CT imaging and hepatic blood markers to estimate portal venous hypertension, which is linked to prognosis in patients with cirrhosis of the liver. PURPOSE: To develop a measurement method of optimal data fusion quality deep learning problems utilizing both imaging data and clinical data. METHODS: Our approach is based on modeling the fully connected layer (FCL) of a convolutional neural network (CNN) as a potential function, whose distribution takes the form of the classical Gibbs measure. The features of the FCL are then modeled as random variables governed by state functions, which are interpreted as the different data sources to be fused. The probability density of each source, relative to the probability density of the FCL, represents a quantitative measure of source-bias. To minimize this source-bias and optimize CNN performance, we implement a vector-growing encoding scheme called positional encoding, where low-dimensional clinical data are transcribed into a rich feature space that complements high-dimensional imaging features. We first provide a numerical validation of our approach based on simulated Gaussian processes. We then applied our approach to patient data, where we optimized the fusion of CT images with blood markers to predict portal venous hypertension in patients with cirrhosis of the liver. This patient study was based on a modified ResNet-152 model that incorporates both images and blood markers as input. These two data sources were processed in parallel, fused into a single FCL, and optimized based on our fusion quality framework. RESULTS: Numerical validation of our approach confirmed that the probability density function of a fused feature space converges to a source-specific probability density function when source data are improperly fused. Our numerical results demonstrate that this phenomenon can be quantified as a measure of fusion quality. On patient data, the fused model consisting of both imaging data and positionally encoded blood markers at the theoretically optimal fusion quality metric achieved an AUC of 0.74 and an accuracy of 0.71. This model was statistically better than the imaging-only model (AUC = 0.60; accuracy = 0.62), the blood marker-only model (AUC = 0.58; accuracy = 0.60), and a variety of purposely sub-optimized fusion models (AUC = 0.61-0.70; accuracy = 0.58-0.69). CONCLUSIONS: We introduced the concept of data fusion quality for multi-source deep learning problems involving both imaging and clinical data. We provided a theoretical framework, numerical validation, and real-world application in abdominal radiology. Our data suggests that CT imaging and hepatic blood markers provide complementary diagnostic information when appropriately fused.

Duke Scholars

Altmetric Attention Stats
Dimensions Citation Stats

Published In

Med Phys

DOI

EISSN

2473-4209

Publication Date

June 2023

Volume

50

Issue

6

Start / End Page

3526 / 3537

Location

United States

Related Subject Headings

  • Tomography, X-Ray Computed
  • Radiography, Abdominal
  • Nuclear Medicine & Medical Imaging
  • Neural Networks, Computer
  • Liver
  • Hypertension
  • Humans
  • 5105 Medical and biological physics
  • 4003 Biomedical engineering
  • 1112 Oncology and Carcinogenesis
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Wang, Y., Li, X., Konanur, M., Konkel, B., Seyferth, E., Brajer, N., … Lafata, K. J. (2023). Towards optimal deep fusion of imaging and clinical data via a model-based description of fusion quality. Med Phys, 50(6), 3526–3537. https://doi.org/10.1002/mp.16181
Wang, Yuqi, Xiang Li, Meghana Konanur, Brandon Konkel, Elisabeth Seyferth, Nathan Brajer, Jian-Guo Liu, Mustafa R. Bashir, and Kyle J. Lafata. “Towards optimal deep fusion of imaging and clinical data via a model-based description of fusion quality.Med Phys 50, no. 6 (June 2023): 3526–37. https://doi.org/10.1002/mp.16181.
Wang Y, Li X, Konanur M, Konkel B, Seyferth E, Brajer N, et al. Towards optimal deep fusion of imaging and clinical data via a model-based description of fusion quality. Med Phys. 2023 Jun;50(6):3526–37.
Wang, Yuqi, et al. “Towards optimal deep fusion of imaging and clinical data via a model-based description of fusion quality.Med Phys, vol. 50, no. 6, June 2023, pp. 3526–37. Pubmed, doi:10.1002/mp.16181.
Wang Y, Li X, Konanur M, Konkel B, Seyferth E, Brajer N, Liu J-G, Bashir MR, Lafata KJ. Towards optimal deep fusion of imaging and clinical data via a model-based description of fusion quality. Med Phys. 2023 Jun;50(6):3526–3537.

Published In

Med Phys

DOI

EISSN

2473-4209

Publication Date

June 2023

Volume

50

Issue

6

Start / End Page

3526 / 3537

Location

United States

Related Subject Headings

  • Tomography, X-Ray Computed
  • Radiography, Abdominal
  • Nuclear Medicine & Medical Imaging
  • Neural Networks, Computer
  • Liver
  • Hypertension
  • Humans
  • 5105 Medical and biological physics
  • 4003 Biomedical engineering
  • 1112 Oncology and Carcinogenesis