The castle was originally built in 1350 by Dirk van Herlaer along the river Maas. Ammersoyen was a unique castle as it was built using a fixed plan, which was unlike other castles built during this era. The design included four wings that were constructed around a center court. Each corner had its own heavy tower for extra protection. The castle included a gatehouse and was originally surrounded by a moat. At the time, it was one of the finest defensive structures in the country. In 1386, the castle was lost to Duke of Gelderland who gave the castle to his illegitimate son. He then sold the castle in 1424 to Johan van Broekhugen, Lord of Waarenburg. For the next four hundred years, the castle only exchanged hands through inheritance. Throughout history the castle was besieged several times with 1513 and 1574 being some of the more notable events. The castle suffered the most damage in 1590 when the castle owner Joris van Arkel was killed from his injuries. After his death, the castle fell into ruin until the 17th century when the Van Arkel family finally raised enough money to restore the castle. Thomas van Arkel paid the French 7,000 guilders to save the castle in 1672 when France swept through Holland and burned many castles along the way. The castle may have survived, but Thomas remained in debt and was never able to finish the castle renovations. After his death, the castle was inherited by another family. The castle was then sold to the Roman Catholic Church in 1876 and was used as a convent. During World War II, the castle was used as shelter for village residents. Once the war was over, the castle was used as a village hall until it was purchased by the Gelderland Castle Trust in the late 1950s. It has since been restored to its former medieval glory.