It was a physical place because Nitocris of Assyria buried the key with herself but Darius I found it and opened it which led to the distribution of Gil's treasures around the world. That's where the conceptual fuckery of Gil having the prototypes of weapons pre-Darius I comes from too.
As a servant it's a pocket dimension, who knows what's it like inside since the only person we know who has seen it is Shinji.
I'm not sure Karl is a fully reliable source on it either, but we'll probably never know for sure. It would surely be strange for such things to have completely disappeared in only 300 years after Camelot. Yeah, it was the last vestige of the Age of Gods, but stuff stuck around even afterwards.