Tolkien touches on some of this in his musings in the new
The Nature of Middle Earth book. (Recommended, by the way!)
The elves were more tied to the world itself, but "wore down" gradually. They started off much hardier, stronger, and quicker than men, and faster to heal. If they exceeded men in those areas during the War of the Ring -- and they did! -- then how much more powerful would the Elves of the First Age be, unweakened by the passage of literal millennia?
Better armor is an interesting thought. I can't remember if it comes up explicitly in the book but it would make sense for elven armor to be better than mannish armor, though. Everything else the elves do seems to exceed the works of men, and I imagine the weapons and armor forged by Fëanor in Aman would be top flight even by Elven standards.
Those are all the story-internal reasons. External to the story, I think "there were giants on the earth in those days" was just more suited to Tolkien's personality and world view than "look how much better we are now than those losers."