The biggest difference is not the content itself, but the way its presented. Instead of reading it like a traditional book, you actually have to read right to left. Obviously this makes a lot of sense in Japan, but it doesn't really translate well here. When you take a very basic rule like reading left to right and flipping that on people, it's an automatic hurdle that turns people off.
I know more than a handful of people that won't even give manga a second look because of this. I'll go through an explanation of what a specific manga is all about, and certain people seem really interested. Then I get them to take a look at my copy of the manga, and they are automatically turned off. Reading the opposite direction doesn't just bother them, it's something they simply aren't willing to do. That simple fact keeps them from enjoying the manga content, or so they say.
Is it this tiny hurdle that holds manga back? Could this be why anime is more accepted when you present the two entertainment forms to newcomers? A lot of people are more open to books than animation to begin with, even if the books have more pictures than words. Unfortunately it seems like that fact is negated by the reversed reading direction. I completely understand why things are the way they are, but I think it's terrible that people give up on manga for that one reason.