Well, sperm on its own cannot grow into a human being, a fertilised egg can. Potential for life really only applies to the later. By your definition a mud puddle could hold the potential for life, if you hold the development of our species are generally accepted to be true.
Having no sex is a 100% surefire method of birth control.
I'm not against abortion, in fact I think it's a reasonable procedure, considering the evidence and in the correct circumstances. I do think it is often too easily accepted by people somewhat abstract from it, there are a wealth of things to consider, and even some rational things to deny before it can be embraced. So I'm not denouncing those tho support abortion, only saying I find the more vocal groups that support it to be idiots, hypocrites and often clueless. There is a right and a wrong way to support abortion, and most people, in my experience, are in the latter group.
I have personal experience with it, and I found something profoundly discomforting about the entire thing. I'd just rather it was considered holistically rather than supported or campaigned against based entirely on one dimensional and emotional responses that seem to me to be knee jerk and ridiculous.