The "caveman" as he is often called was a Homo sapiens. Yea, Neanderthals were pushed out of existence. Some did evidently breed with humans before they vanished.
Fact is, the first Home Sapiens if plucked out of time at the dawn of mankind is no different that the rest of us. The total population of humans back then was low. Over 5300 years ago, Ötzi the Iceman as we now call him lived with more advanced tools than we thought they had back then. He has taught us a lot about that era of humans. Many textbooks were rewritten because of what we learned for just that one rare example.
We really don't know if they had much written language. It would have to be chiseled in stone to survive 6,000 years unless it happened to be caught in a glacier like Ötzi and that is exceeding rare.
Before the dawn of modern humans the lineage goes back until the emergence of H. erectus 1.89 million years ago and then hominids grew tall, evolved long legs and became completely terrestrial creatures.
Humans are pretty amazing. There are tribes of humans that can run down a deer to the point the deer is exhausted, and they can kill it, and bring back the meat to the village. No other animal can travel as far as humans except a dog and maybe a good horse but a dog needs more water.
Every year in England there is a race between horse mounted riders, and a human on foot.
https://en.wikipedia.org/wiki/Man_versus_Horse_Marathon
The horse mounted humans win most of the time, but sometimes the humans are faster. If the race was longer, humans can win more, but it too hard on horses to compete longer. The horses are protected by humane rules, humans can push on further. If they put in a long swim, humans would win every time.