My opinion on this is that more referees would only bring more disagreements and generate endless controversies and bickering in a game that already has enough of those. Increasing the number of refs would automatically reduce the tolerance towards mistakes, which would STILL happen. The referee is the one game director and his every decision should be respected. These guys are pros, they do make mistakes but so do players, trainers, gardeners, etc.
One thing I could imagine working is one referee per half of the pitch - with a strictly limited field of intervention - so as to prevent for instance that the ref finds himself 50 meters behind the action in case of a fast counter attack. But I can see how this would be tricky to coordinate.
You probably know that experiments have been made with additional refs standing behind the goal line, in charge of surveying the penalty area. From what I hear those have been rather inconclusive so far.
Also don't forget that the linesmen are empowered to intervene at any time to signal things the central ref might have overlooked. These three guys are actually enough to cover all the angles of vision required.
