There is no denying the whole decorating system is really wonky.
I find two particular issues:
1) refresh time of the guild hall. Removing a decoration does not automatically set the decoration count back. As cloudsareyum mentioned, sometimes logging in and out will reset the count and sometimes it will not. I have sometimes had to leave the game completely and come back several hours later. I have also had this be an issue the other way, adding a decoration did not increase the count and so I have put more than the allowed number in an area with the result that when I did need to remove one I could not replace it. There does not seem to be any consistent way to refresh the count which is really frustrating when you are working right up to the allowable limit.
2) the placement point of a decoration. Sometimes it is the anchor point. For instance, pirate flags count from the center of the base which means you can sometimes put things at the top of the flag and not have the two objects count as in the same "area." On the other hand, it seems long pieces of race track have the anchor point at one end but the count point at the other. So sometimes I have been able to game the system by turning the track around. For Ominous walls the anchor point is in the center but I have never figured out where the count point is located - sometimes turning them around allows me to add more decorations and sometimes it does not. And don't get me started on ice walls. I may be placing something on one level and an ice wall 3 levels above me will turn red, meaning it is in the count zone when it should be quite a distance outside.
The only work around I can suggest is logging in and out, try turning large items around and, if you can, get 5 other guildies to log into your guild hall to force a new guild hall instance and refreshed count. And if the place you are decorating is near an entry point sometimes you can log in and quickly sneak in a decoration before other decorations in the area have loaded.