Owners of the derelict Strathmartine Hospital say they are fighting a constant battle against youths breaking into the site and setting fires.
On Wednesday evening firefighters spent an hour-and-a-half tackling a small fire in one of the newer blocks on the site, which is in an unsafe condition and due for demolition.
Plans for the redevelopment of the site for housing are being considered by Angus Council.
Robert Evans, of property consultants Muir Smith and Evans, who represent the owner, said the site has a history of wanton vandalism and thefts.
He said: “We do recognise there is a problem and there is security on site, but with the scale and location of this site it is extremely difficult.
“It is a constant battle. All of the openings have been closed up and the buildings are boarded up but there are signs of people getting tools out and going up armed with crowbars and drills.”
He said once planning permission is secured some of the buildings, excluding the listed main building, could be demolished quite quickly.
He said: “A lot of those buildings could be demolished just now, but the cost would be extremely high, so we would need to secure full planning permission first and we are still at the in principle stage.”
Local campaigner Karen McAulay said she and other residents are constantly on the lookout for signs of fireraising at the site and are dreading the warmer weather.
She said: “Summer fills me with dread. I just put my head in my hands and wait to see what will happen.“I love that site and I never thought I would say this but it should be demolished.
“It is an accident waiting to happen and when it does it will be serious.”
Arbroath Fire Station manager Fraser Scott said desks and rubbish had been piled up in a corner before being set alight.
He said the culprits were putting themselves and the firefighters at risk by forcing them to enter the building.
Mr Scott said: “We would advise people to stay out. Derelict buildings are not a playground.
“They are very, very dangerous and there have been incidents in the past where people have been injured or lost their lives because of reckless behaviour.”
A police spokeswoman said: “Derelict buildings and properties that have fallen into disrepair, whether large or small, are not safe environments to be in.
“Whether it’s children who wrongly think such buildings serve as a playground or place to explore, or the less well intentioned looking to steal property, the message is the same keep out.
“Derelict buildings can be, and often are, very unsafe and pose a genuine danger to anyone who ventures in. Police officers are aware of this issue and will continue to give the area their attention.”