Hvar town is well worth a visit. Its entire focus is the tourist trade and ferries from the mainland run regularly with arriving and departing holiday makers. The town quay is expensive and generally used for large motor yachts. There are mooring buoys on the west side of the harbour, taking lines ashore, which are more reasonably priced.
Hvar town is a party town. If on the Town Quay raise your passarelle at night to discourage boarders. Not a problem on the buoys.
Anchoring free in the harbour area will give you an entertaining time as the super yachts try to dodge you on arrival. The ferry to and from the islands is about 100kn per person so even for 2 a return trip is a significant proportion of the mooring fees.
Lovely town, clean, friendly, good restaurants, supermarkets and pharmacy. Castle is worth the climb if only for the views.