The castle may have been built in the Punic period, but only remnants of pottery have been found from this culture. What have been found, however, are the remains of a Roman fortress. It was to become really important in the Nasrid period, when it was famous for its prison and dungeon. Dethroned sultans, ministers fallen out of favour and military leaders were all imprisoned there. The fortress was handed over by Caid Ibn al-Hay to the Catholic Monarchs in 1489, after its capitulation. In 1808, it was bombarded by the English fleet. A highlight inside the castle is the Neo-classical pavilion, which is entirely different from the rest of the building.