Why dont more muslims who dont agree with terrorism, stand up!?

im not asking this to offend, im asking because i want to know. do Muslims believe that this 'holy war' (even if you are not physically fighting in it), is supposed to happen? is it part of a 'prophesy?' so even if your not being a 'martyr,' or killing 'infidels', deep down theres still a belief that this is justified? ...Or are non-extremists afraid of the threats thrown their way from extremist parties? or am i completely off, and i have no idea what im talking about. educate me.