Look at ST FC and STIV (what a surprise time travels) and you know how a Star Trek movie can be a success for Trekkies and meanstream.
I think it could do very well. I don't really know how else to say it, though, but they might have to dumb it down a bit. Not in terms of the story or the depth, but some of the Trek lingo is hard to follow for some. That's a big reason my friends don't watch it; It's hard for them to figure out what's going on with the ship, or what alien people or technology is threatening the crew.
Tell me one single Star Trek movie that was hard to follow? I even would say Star Trek movies were up to the present more simple to understand than a lot of other movies. From the beginning on you know who is the enemy, who are the heroes and what is the danger ... and this will stay like that until the end of the film. There were only few big surprises, only few storyturns in Star Trek movies.
Lost is a lot more difficult to follow, Lord of the Rings was a lot more complex, but both Lost and LotR were successful. Complex story =/= no success.