There were once two temples called Niburi-bō and Urumi-bō that that used to exist in this area.
What you see before you now are the ruins of Niburi-bō Temple, which has been moved and is known as Kanjō-ji today. If you look across the road you will see another similar rock formation; this was once a temple called Urumi-bō. This temple has also since moved locations and has been renamed Shōyō-ji.
The reason behind these temples being relocated has to do with the origins of the name of Mt. Akahage.
Back when these two temples existed, Mt. Akahage was not “hage” or bald. Instead, it was covered in lush greenery, and this is when Emperor Go-Daigo came to Oki and lived in Niburi-bō and Urumi-bō. After this, however, there was a large bushfire that swept over the mountain and rendered the temples and of the trees to ashes.
After this, tall trees were unable to grow any longer, forcing the temples Niburi-bō and Urumi-bō to move locations and be respectively renamed as Kanjō-ji and Shōyō-ji.