Its construction started around 1515 at the expense of Alonso Pimentel, Count of Benavente, with the participation of the stonemason García de Entrambasaguas. It was finished in 1520. Originally it had the character of a fortress and the intention of a royal palace. The Count also ordered the construction of gardens and ponds that have not survived to this day. In 1555 it was the seat of the Court, in 1559 accommodated the King Felipe II and in 1601 it was a temporary residence of the Royal Family of Felipe III. The daughters of the King Felipe III Ana Mauricia and María were born here. It reached its greatest splendour in the 17th century, when it was owned by the Viceroy of Naples Juan Alfonso de Pimentel, Count-Duke of Benavente. It suffered a fire in 1716. In 1789 it served as an orphanage. Since 1990 it is the headquarters of the Library of Castile-Leon.