It doesn't decrease...
Look at it this way.
Those minuses only count for that particular skill. They don't make your IQ lower for anything else.
When you want to use the skill, you roll dice. If you get a number lower than your skill level, you succeed.
So if you have 13 in Art and want to paint something, you roll three dice.
If you get a number lower than 13, you did it.
That's why you want to boost that number, so you succeed more.
You may ask why the minuses, it's because by default, most skills are Attribute - 6/5. So Attribute - 3 is a marked improvement.
“She'd become a governess. It was one of the few jobs a known lady could do. And she'd taken to it well. She'd sworn that if she did indeed ever find herself dancing on rooftops with chimney sweeps she'd beat herself to death with her own umbrella.”
― Hogfather