You can walk to Hiji Castle Ruins in about 10 minutes from Yogai Station, and the castle ruins near the sea are now Shiroshita Park.
The flounder caught in this area is known as Shiroshita Karei, and the peak season for this brand-name fish is from April to July. There are several restaurants in the town that serve Shiroshita Karei, but the price is a bit high, as you might expect from a high-class fish.