Hmmm, well, it's not the adapter... I just tried using a USB card reader that handles microSD and got identical performance... Gotta be something weird in Mavericks...
