If other species of human were still around, would we be able to breed with them?

eg what would happen if a Homo sapien (one of us) had intercourse with and impregnated a neanderthal or Homo ergaster or H. ergaster? Would the baby be normal and healthy or deformed? Or would the egg be fertalized in the first place?