lauralouisell - First, it's hilarious for me to hear that "less than 5 million" is low! In our case my husband's sperm count was 0, which I guess is pretty much as low as you can go.

He has congenital bilateral lack of vas deferens, so his little guys were all dressed up with no way to go anywhere. Like CT_Michele, he also did the procedure to extract the sperm directly from the testicle and they used ICSI to fertilize the eggs.
The first time we did IVF was two years ago, and even in that time the extraction method was improved for this cycle that we just did in February. Although I was 27 then and have absolutely no issues, our first two tries with the first sperm samples did not take (one fresh, one FET). But there was clearly an improvement in results from the beginning. The first time we got 7 embies, this time we got 17! We did a day-5 transfer 2/23 of 2 blastocysts and one took.
So for us, third try was the charm for our BFP with male factor.
Good luck!!