The wilderness level will never be fair, no matter what way you turn it around. We can start averaging the wilderness levels where 2 people stand but that's just going to be more confusing.
You've said that you couldn't attack back? Can you provide me more details on that?