Abstract: Due to the difference of the orbits of the observatories and multi-modal astronomical data, the solar eruption observed by SDO and ACE will impact the geomagnetic field in 3∼4 days and 0.5-2 hours respectively. Compared with the single solar wind data from ACE, the EUV image-related data from SDO contains earlier magnetic storm information, and could be applied to earlier forecast the grade of geomagnetic storm, geomagnetic index Kp and other parameters. This study aims to strongly connect Kp index and solar wind, coronal hole, solar extreme ultraviolet images together and establish multimodal and multi-scale models for Kp index prediction by virtue of deep learning and a great deal of data sourced from SDO and ACE. To avoid problems of ignoring physical meaning and sample balance on solar wind data, feature construction is designed and feature selection is carried to establish data set of years. Mainly focusing on data from 2011 to 2019, several deep learning Kp prediction models are verified experimentally. Direct Kp index prediction of solar wind (DWKp), Kp prediction by way of indirect multi-scale solar wind based on multimodal EUV images (MMKpI), and Kp prediction taking account of coronal hole location information (MLoCH), all achieve better and similar performance in 3h prediction range. As the prediction range extended to 6h-12h, DWKp model shows the satisfied performance, and MLoCH model obtains the predetermined performance while the range stretches to 15h-33h. Furthermore, the MLoCH even achieves the advance time to 72h with CC around 0.23.