The emergence of immune checkpoint inhibitors (ICIs) has significantly advanced cancer treatment. However, only 15-30% of the cancer patients respond to ICI treatment, which stimulates and enhances host immunity to eliminate tumor cells. ICI treatment is very expensive and has potential adverse reactions; therefore, it is crucial to develop a method which enables to accurately and rapidly assess a patient's suitability before ICI treatment. We complied germline whole-genome sequencing (WES) data of 37 melanoma patients who have been treated with ICIs and sequenced in our lab previously, and the WES data of other 700 ICI-treated cancer patients in public domain. Using these data, we proposed a novel double-channel attention neural network (DANN) model to predict cancer ICI-response and validate the predictions. DANN achieved a mean accuracy and AUC of 0.95 and 0.98, respectively, which outperformed traditional machine learning methods. Enrichment analysis of the DANN-identified genes indicated that cancer patients whose in-born genomic variants might mainly affect host immune system in a wide-ranging manner, and then affect ICI response. Finally, we found a set of 12 genes bearing genomic variants were significantly associated with cancer patient survivals after ICI treatment.