Computer-assisted rectal clinical diagnosis is of great significance for the early detection and treatment of rectal cancer. In data processing, it is quite challenging to achieve automatic segmentation due to the blurred boundary between lesions and healthy rectal tissue. To overcome such difficulties, we propose a rectal tumor segmentation method by using a modified U-net, thereby improving the diagnostic efficiency and accuracy. Firstly, the central coordinates of the rectal part to extract the region of interest are determined. Then, the tumor region is determined in the CT image via the YOLOv3 algorithm. Finally, the residual connection and attention mechanism are introduced to reduce the possibility of misjudgment of healthy rectal tissue as lesions to improve the accuracy of the traditional U - net model, and we use the modified U-net model to segment the rectal tumor region. The experiments show the Dice coefficient of this method can reach 83.45 %, which is about 7% higher than the traditional U-net method, and this shows the validation and merits of the proposed algorithm.