The building was erected on the initiative of Charles III in the 18th century. The work was commissioned to Juan de Villanueva, but it was finally carried out by Manuel Serrano. This is a civil building that served as a place of lodging for the Infantes Gabriel and Antonio. The palace was conceived as an integral part of a plaza. The uniform arches and the austerity of the general appearance of the building are the most striking features of the exterior of this palace.