Punishment should be served by authorised personel, not other inmates, that's flawed. Still publications I read written by criminologist experts say that if criminal's brain associates crime with pain, discomfort the human is less likely to commit it again.
Other publications also said that sole sight of effect of some actions like car crash, tortures made on criminals affects the brain and should stop human from speeding, stealing etc. But that publication is old and from before the internet era, so i doubt it's effective nowadays when we see gore in everyday life.
Perfect jail system would be something like "the harder you work in prison the better living conditions will be offered for you". If you are lazy fuck, die of starvation, if you want to rehabilitate you will be rewarded.