It probably didn't happen overnight. I'd figure that they didn't give their fleets up per se, but combined them under starfleet. i would imagine the early Federation Starfleet having a lot of ships from not only Earth, but from Vulcan, Tellar and Andoria. As they came together to design and build "Starfleet" ships, the older models were phased out for a more uniform style fleet.
I wasn't aware that twelve Klingons constituted a swarm...