Absenteeism by doctors in public healthcare facilities in rural Bangladesh is a form of chronic rule-breaking and is recognised as a critical problem by the government. We explored the factors underlying this phenomenon from doctors' perspectives. We conducted a facility-based cross-sectional survey in four government hospitals in Dhaka, Bangladesh. Junior doctors with experience in rural postings were interviewed to collect data on socio-demographic characteristics, work and living experience at the rural facilities, and associations with professional and social networks. Multiple logistic regression was used to determine the factors associated with rural retention. Of 308 respondents, 74% reported having served each term of their rural postings without interruptions. The main reasons for absenteeism reported by those who interrupted rural postings were formal training opportunities (65%), family commitments (41%), and a miscellaneous group of others (17%). Almost half of the respondents reported unmanageable workloads. Most (96%) faced challenges in their last rural posting, such as physically unsafe environments (70%), verbally abusive behaviour by patients/caregivers (67%) and absenteeism by colleagues that impacted them (48%). Respondents who did not serve their entire rural posting were less likely to report an unmanageable workload than respondents who did (AOR 0.39, 95% CI 0.22-0.70). Respondents with connections to influential people in the local community had a 2.4 times higher chance of serving in rural facilities without interruption than others (AOR 2.40, 95% CI 1.26-4.57). Our findings demonstrate that absenteeism is not universal and depends upon doctors' socio-political networks. Policy interventions rarely target unsupportive or threatening behaviour by caregivers and community members, a pivotal disincentive to doctors' willingness to work in underserved rural areas. Policy responses must promote opportunities for doctors with weak networks who are willing to attend work with appropriate support.