Spirits Within suffered from trying to make the characters look too realistic, and ended up with a creepy zombie-like effect. But in Final Fantasy VII: Advent Children, they took a step back from the realism a little, and the overall effect was - conversely - far more realistic and believable. They also paid more attention to details like facial expression, body movement and skin, which helped too. It'll be good to see the Blu-Ray edition of Advent Children when it comes out - with revamped visuals and a half-hour of extra content, it should be fantastic...
Anyway, back to Trek: mixing photo-realistic animation and live action is probably, for now, too costly and time-consuming for use in episodic TV. But for film, it's a realistic option. After all, Gollum from The Lord of the Rings is almost entirely a digital creation, and a virtually seamless piece of work.