Thanks to you both for your replies. I'll try the councillor route if the landlord decides to take no action (which is possible). He seems to be angry about what's been going on, but in the end he's getting a regular income from these tenants, so we're not sure what to expect. The main problem seems to be the agency, who obviously don't want anything to change as they're making money out of the situation too and don't want to rock the boat.
We've tried recording everything and taking it to the council, but they seem to only want to take things so far before they give up. They feel we need to be physically threatened before they take action. Anyway, fingers crossed.