If you follow the mountain road to Oku-sha, you will see a stone signpost at the fork. Some part of it seems to have been mysteriously scraped off.
They say there were Buddhist Sanscrit characters engraved on the signpost, which were scraped off in the Meiji period. Up until that point, the Shinto gods and Buddha were enshrined together in Japan, but the Meiji government prioritized ancient Japanese Shinto over Buddhism and decided that Buddhist elements should be eliminated, banning asceticism along with it.
From that point on, Kenkō-ji Temple became Togakushi Shrine, and the sub-shrines were changed from Hōkō-in, Chū-in, and Oku-in to Hōkō-sha, Chū-sha, and Oku-sha, respectively. There were riots to destroy Buddhist statues and abandon temples all over Japan, and Kenkō-ji couldn’t escape that. However, some Buddhist statues have survived to this day, slipping through the government’s surveillance by being hidden in the attic or being sent over to other open temples. The Buddhist statues scattered all over Japan return to Togakushi once every seven years for public viewings.