Eleanor Milne's success artistically and professionally can be linked to her  positive experiences growing up. Born in Saint John, New Brunswick, in 1925 to W. Harold Milne and Irene Eleanor Mary (Gilhooly) Milne, Eleanor Milne was the second of three children, between her brother William Joseph and her sister Barbara. Her father was a third-generation naval architect who designed the icebreakers still used in the north. W. Harold Milne was also the superintendent of the Saint John Dry-Dock. Her mother loved to paint, and from the age of two Milne was encouraged to practise art. She credits her mother with sparking her creativity. Her parents treated all three children equally, and during childhood Milne had the opportunity to go with her father to the Saint John Dry-Docks. She has said her early contacts with "male culture" made her feel comfortable working around men.

Talent runs in the Milne family, with Eleanor Milne's brother William a naval architect who now teaches at Memorial University, and her sister Barbara Lambert and brother-in-law Martin Lambert both architects. Eleanor Milne's family moved to Montréal in 1938, where she entered boarding school at the Sacred Heart Convent. As a young girl Milne dreamed of marrying a rancher and raising horses, but when her artistic career took off she dedicated herself to her art instead. Her sister Barbara has five children, so Milne continues to be surrounded by members of the younger generation.