UW-Madison researchers have developed an automated tool that uses image data to systematically predict lesion-level response to treatment. This allows the opportunity to guide treatment for an individual. For example, by identifying individual tumors that would be non-responders means they could be treated with radiation rather than a chemotherapy that would be effective for other tumors in that patient. The invention leverages longitudinal information to assign probabilities for a given lesion to be in a certain state (e.g., responding, not-responding), which further allows for discrimination between different response categories in a given time in the future (e.g., will likely disappear, will likely persist). The proposed methodology is equally applicable to all metastatic cancers and possibly other diseases that express heterogeneous response to treatment (e.g., multiple sclerosis).