The following is from one looking at this problem from the outside in so there's bound to be a lot of hypothesizing. Personally, clowns aren't all that scary but they can be a little intimidating at times but mainly for the big unknown man aspect rather than anything else. That does not stop me from being stubborn though. If this childhood story of mine is the truth. I only remember flashes of this so most of this story is second hand. Apparently there was a birthday party (don't remember who for) where all the kids were assembled on the grass and there was a clown up front. You know, the typical scene. But we were all around four or five rather than sniping older kids prone to bullying the clown. So, the clown says "sit down everyone" and all the kids sat down and quietly stared up at him (this is a flash of memory). Except me. Apparently I stood there in the midst of the pack of kids and refused to obey orders. Even from a stranger wearing in what is effectively a mask. Stubborn, no?
So clowns aren't too scary for me. Thus the question. Why are clowns scary? Why are they scary enough to become fictional monsters? Why are they cast as fictional serial killers or maniacs? Why are they shot before questioned in police dramas? Why are they depicted as drug crazed? In fact, are we scared of clowns or do we just hate them? There's probably a little of both feelings involved, as is usually the case when people are afraid. Hate what you're afraid of because it dared to make you afraid. Kill it so you never have to be afraid again. That sort of thing.
Still, fear seems integral to all the emotional responses we have to a clown. So, why do we fear clowns? Here's some I can understand, some that could also be sneakily drawn on to create a really scary clown.
The masked man
His or her real features are almost unrecognisable under the wig and makeup and strange puffy clothing. You're unlikely to be able to identify him or her on the street afterwards unless they were a relative. I believe the fear of the masked one response is a fear mainly induced by male clowns rather than female ones. Possibly for the lower voice, height, build and large hands. Females tend to be less threatening even if they are of heavier build or quite tall as they tend to be softer, rounder, have smaller hands, a higher and more normal voice to a child (children's voices are all high) and it is possible they colour coordinate a little better merely from intense and long lasting practice.
The clown is likely to be someone you don't know at all. Add to that, you can't see what he or she really looks like so there's no real way of identifying the person or anything about them. Stranger danger isn't always taught. Sometimes - I'd actually say quite often if I could get away with it - it is instinctual for a child to fear someone unknown.
Our early facial recognition abilities see them as mutants or other
The lines on their faces are in the wrong places, are the wrong shape and over emphasis or enlarge features beyond easy recognition. We have to consciously look past the paint in order to see the person beneath but that is something learnt. It isn't an instinctual skill at all and so the younger the child the more likely they are to see the painted face as the clown's real face. Now, I don't care how nicely you curve those lines and how big and shiny your eyes are made out to be, if you don't look near human you're going to come across as quite scary to some young children. Purely because they can't see past the paint. Clowns can become aliens and monsters because of their paint.
The inability to predict their actions
As the facial features of a clown are covered it is very hard to tell whether they're frowning, laughing, crying, glaring etc. So you don't know if they're annoyed or upset. And if you don't know then all those quick movements can be read anyway you instinctively do. They could be signs of aggression which would induce fear. That's pretty natural. Also, because you can't properly see any of those emotional indicators you find it hard to tell just what that clown is about to do. Will they grab you? Or will they try to play a trick on you? Do you need to run away from them yet? Why not run now to get out of harms way so as to save worrying about it?
They invade your space
A clown is desperate to entertain at times and will bound up to a small child in hope of a good response or just call out to them and demand a response of an unprepared child. Meanwhile the small child feels suddenly dwarfed and threatened, resulting in tears and cries for a parent. And as you know, some childhood traumas stick. Alternatively, something like stage fright can hit as the child struggles to comprehend just what's being asked of him or her. Especially if the eyes of other children and parents are on them, waiting.
They're loud and boisterous when you aren't and often demand you be too
This means they're likely to call upon you if you're not responding properly, on their part purely to pull you into the game and entertain. Also, because they're loud and boisterous they seem to be even more incomprehensible than when just standing about in heavy makeup, bright puffy clothes, strange shoes and a bright wig. Why are they jumping about all of a sudden? Why are they yelling? What's so exciting? What's going on? In all, it is an unpleasant overload of the senses that confuses the mind.
They perform tricks you can't understand
If being incomprehensibly loud and boisterous wasn't enough clowns also perform tricks you just can't understand and rarely do they explain them to you. This leaves you confused and befuddled (or confuddled) and struggling to catch up with just what's happening before your eyes. On top of that, it seems they are disobeying all the rules of the world you'd learnt up until then. Things don't appear out of thin air just because you wished for them. You'd already learnt that when wishing for cake and ice cream and even your mother had said "No, it'll rot your teeth". Not to mention what happens when you forget to tell someone you lost a tooth and your tooth is still there in the morning even though you were wishing for money, money, money so you could buy a lolly from the school canteen. It just isn't right that the clown's pulling coins from nowhere without losing his or her teeth! They must be fearfully strong magic wielders who want to keep all their secrets to themselves. Enter a little hate along with that fear.
They're laughter is spooky
This is because a clown often has to laugh even after a kid has kicked him or her, or a kid has destroyed the show and reduced their pay, or a parent has become loud and aggressive during the show. The laugh is false. They aren't happy. Yet the makeup says they're still smiling and wide-eyed happy. The problem here is that children can tell when laughter is faked even if they can't say just what the problem is or why they feel suddenly creeped out or scared. This is where the creepy clown laughter that's become so famous comes from. Forced, false and possibly evil laughter from a strange unreadable man (generally) wearing a mask who's intent on getting your attention no matter what and in doing so grabs you and draws you into his game.
I think I'm beginning to see why people are scared of clowns and why clown trauma seems to be a lasting one. And that's just with normal clowns, the average kiddy garden party variety.
Then comes things like real serial killers dressing as clowns to pick victims, clowns as the chosen shapes of fictional monsters, clowns as chainsaw wielding maniacs on television and in the movies, clowns returning from the dead and killing people, dead clowns paired with spooky ice cream trucks playing the ice cream song off key... The clown has a lot to contend with just to make someone laugh. And it seems that we laugh the most when the clown is safely fenced in with the other actors and performers in a circus ring - far from where they can reach us and pick us out from all the rest sitting there watching. We feel much safety in numbers while sitting around a circus ring, more than we ever do while at a garden party.
So, to end this as my clown analysis has become long, I'd like to point out that if you're creating a scary clown character it is easy to make them truly terrifying. Imbue them with real power. Give them strange clown makeup with dulled, worn or unusual colouring. Make the lines on their faces harsh and inhuman. Make their facial expressions barely readable beneath the makeup yet hinting at the opposite of what the makeup suggests (if happy makeup then frowning features, if frowning makeup then an evil grin). Make them laugh in all the wrong way, grab at people, be unpredictable and invasive. Everything that terrifies the child. Because that terror is likely still festering away in your audience, there for playing with.