Increasing evidence indicates that end-stage renal disease (ESRD) is associated with the morbidity of cancer. However, whether different dialysis modality and sex effect modify the cancer risks in ESRD patients remains unclear. A total of 3,570 newly diagnosed ESRD patients and 14,280 controls matched for age, sex, index month, and index year were recruited from the National Health Insurance Research Database in Taiwan. The ESRD status was ascertained from the registry of catastrophic illness patients. The incidence of cancer was identified through cross-referencing with the National Cancer Registry System. The Cox proportional hazards model and the Kaplan–Meier method were used for analyses. A similar twofold increase in cancer risk was observed among ESRD patients undergoing hemodialysis (HD) or peritoneal dialysis (PD) after adjusting for other potential risk factors. Patients with the highest cancer risk, approximately fourfold increased risk, were those received renal transplants. Urothelial carcinoma (UC) had the highest incidence in HD and PD patients. However, renal cell carcinoma (RCC) had the highest incidence in the renal transplantation (RT) group. In addition, female patients undergoing RT or PD had a higher incidence of RCC and UC, respectively. Male patients under HD had both higher incidence of RCC and UC. In conclusion, different dialysis modality could modify the cancer risks in ESRD patients. We also found sex effect on genitourinary malignancy when they are under different dialysis modality.