If you see houses with paper lanterns, stone monuments, or pine trees in their yards, then they were once oshi homes.
Unfortunately, of the 86 that existed during the Edo Period, only 20 remain. Of those, only 4 are still in service, at the time of writing this guide in 2019.
In other words, the remainder are currently being used as ordinary houses. Banjōya is one of those. Unfortunately, you can’t go inside, but you can gaze upon it from the main street.