I've edited tons of drums that weren't tracked to a click, it's not really hard if the drummer is at least in time with himself.
It's particularly easy in Pro Tools, just use "Identify Beat" to tempo map small sections, then Beat Detective on those sections but don't fill gaps or crossfade. Do this for the whole song without ever filling.
Then when you are done, set all the drum tracks to "Ticks" instead of samples but with elastic audio completed disabled to avoid stretching and take those small sections and decide on an "average" BPM for that part of the song. Say you did the verse in 4 chunks, and they were 142, 144, 141, 142. I would set all of those to 144. Now Pro Tools will move all the regions around (still perfectly on the grid) to line up with the new gridlines at that tempo. Always pick faster rather than slower so avoid bad sounding edits.
Anyways do that for the whole song and you'll be left with something perfectly edited to the grid with consistent tempos for each section. It makes BDing particularly easy as well because now every section you do is already confined to exact bar|beat segments since you used them to tempo map.