Since I'm far from an expert, I mainly look at the difficulty level.
And I must admit, I love my eye candy when it comes to shooters
Ikaruga also impressed me because it made me think differently about how to play. Trigger Heart Excelica was kinda like with the lock on and throw technique, but it didn't really require you to use it.
I look for things that you do mainly. Although I also look for a bit of innovation other than "Shoot, dodge, different weapon(different flavor of killing things)" I own games that do this but not too many.
I look for unique spins on the formula. Like Ikaruga's polarity system. Or R-Type's movable force unit. I tend to prefer horizontals rather than verticals. Difficulty is not really an issue, but I would rather see a game with thought-out level design (any of the R-Types, except Final, and Ikaruga) rather than just endless swarms of enemies (Mars Matrix, though it's a great shooter too).
I like intereseting character design. Even if they could be a certain cliche, as long as the characters are interesting, I'm game. This is why I'm always initially intrigued by a certain shump, like my facination of parodius or pretty much all of psikyo's shumps. Of course gameplay is important, I just appreciate it when the designers make some interesting looking characters or ships (in the case of R-type since there is technically no peeps). This was one of my probs with Chaos Field, other than it being a slightly boring title. I think the other thing that gets me sucked into a shump is when there is an attitude about the whole thing. This is why I love parodius so much and it's bizarre sense of humor and why I enjoy the psikyo shumps, with their nice medieval themes or steam punk themes. I think the last thing that really gets me into the game is neat boss fights and sweet powerups. While both can be cheap, against the player and against the computer respectively, It's sweet to see a giant robot or monster come and blast a bunch of bullets at you or to take out a whole armada of ships with one laser blast.