Delineation of structures and estimation of landmarks in mammograms is a critical step in the evaluation of image quality in breast cancer screening, but requires the estimation of the uncertainty of the predicted landmarks to refer uncertain cases to clinicians. Of particular importance – and the focus of this work – is on the pectoral muscle, where the variability in muscle visibility across images introduces significant uncertainty. While graph convolutional networks (GCN) have been demonstrated to accurately predict landmarks by explicitly leveraging structural relationships between landmarks, they typically lack the ability to provide accurate uncertainty estimates for the landmarks. To address this shortcoming, in this work we propose a novel GCN-based approach that not only locates key points along the muscle boundary but also provides accurate uncertainty estimates, capturing both the aleatoric and epistemic uncertainties. Our method was evaluated on in-house annotated mammograms demonstrating comparable accuracy to human annotators, while at the same time providing highly accurate uncertainty estimates, confirming its potential for identifying cases that require human review. We further validate our proposed approach on the publicly available CSAW-S and INBreast datasets, demonstrating its robustness to domain shift, as well as its potential to detect incorrect or untypical annotations.