The castle is located on the highest point of the city. It was built by the Knights Templar on the remains of an old Moorish citadel. It was built with walls of carved stone. Most of the rooms had barrel vaults. It is a sober and solid construction. It was the residence of Benedict XIII, who turned it into a palace and papal library when he moved to Peñíscola in 1411.