In 1940 Gosford House was taken over by the War Office for use as a billet for troops. The King's Own Scottish Borderers spent a considerable period of time at Gosford, and many huts were erected to house the large numbers of men based there. The K.O.S.B. were part of 155th Brigade, 52nd Division, which also included the 7th/9th Royal Scots, the latter also spending many months in the Gullane area.

The large number of German prisoners being taken following the Normandy landings in June 1944 meant that an expansion of P.o.W. facilities was required. Consequently, in 1944 Gosford Camp became Camp No. 16, a prisoner of war camp for around 3,000 German troops. The prisoners were accommodated 50 men to each hut but even then the vast numbers held there meant that until Christmas 1944, when numbers in the camp were reduced, approximately 400 men had to sleep on palliasses in tents. Despite this, provision was made to make life as comfortable as possible for the prisoners. The facilities at the camp included a theatre, a cinema, a football pitch and a sports ground. There was also a camp band with 30 musicians playing a wide variety of instruments. A number of the prisoners with artistic talents painted various scenes in the hut and many views of home towns were drawn from memory. Small gardens were made in front of the huts which also helped improve the living conditions.

Every day, at 6.00 a.m. and 6.00 p.m., a roll-call was taken with all 3,000 men standing in 6 rows on the main camp road. The roll-call was taken by a British officer and four soldiers and was probably considered a routine procedure.