Sometimes when you complete a wave during this quest the stage will advance early resulting in a popup saying you should return to Valus Odiil to report your success, but then immediately more goblins will spawn. One person claims that the problem involves all the goblins spawning in at once. https://www.reddit.com/r/oblivion/comments/1k5nr7t/extra_goblins_during_killing_field_quest_glitch/