Things won't just regress back even if you lost your memory. Spending time alone for 100 million years would make him had a generally cold personality even if he forgot about it. He would be largely apathetic or even appeared unstable. The mental damage had already been done and it would not just disappear even when the memory itself disappeared.
I have read a novel with the same plot at rrl I can't remember it's name. But in that novel mc's sister is a reincarnated goddess. And mc is traped in a dimension for millions or billions of years.