"I saw that, if there was a wall of stone between me and my townsmen, there was a still more difficult one to climb or break through before they could get to be as free as I was". This says that nobody wanted to do what he did to be as free as he was so there was a metaphorical wall between him and all the other people that weren't as unique as him. He says that even if they're is a wall between him and the townspeople, there would still be another wall that would make it difficult for people to make that huge transition from common life and being unique.

TaylorFrost2016 just now Via Flipboard