In no particular order...
Hotel Dusk <-God tier adventure game, amazing OST, fantastic dialogues, innovative puzzles, unique art style. It's pure bliss. The sequel Last Window is great too, but playing it without playing this first would be foolish.
Trace Memory <-Another amazing adventure game by CiNG with all the charm you could ever want.
Ghost Trick <-Some of the best animation you'll ever see and a soundtrack that rocks the house, crazy cool puzzle adventure with a wild story.
Castlevania: Order of Ecclesia <-Combines the "metroidvania" style design along with areas influenced by traditional games in the series. Great protagonist, tons of variety, and legitimately challenging bosses. The second best "metroidvania" game in the franchise right behind Symphony of the Night.
Phoenix Wright - Ace Attorney <-Crazy humor, creative game design, and great music. A classic game that everyone should at least try once.
Mario Kart DS <-The best game in the series. Super tight controls, fair AI, and you've got to love the Mission Mode with the bosses.
Professor Layton & The Curious Village <-This is not the best game in the series, but it's where you should start. A surreal creepy yet childlike charm permeates the entire affair, while puzzles pop out at you from every angle of the evolving plot.
Ninja Gaiden: Dragon Sword <-The best and most accurate use of touch controls on the entire platform. Furious action, great graphics, and tough bosses makes this one feel like a console game squeezed into the tiny DS.
Dragon Quest IV <-Super fun remake of the NES game, with everything you remember from it polished to a gleaming sheen, with much of the tedium cut out. I had a lot more fun with this one overall than part V on the DS.
Mega Man ZX <-People rag on this because the map is hard to use, and they're right, it is. Everything else about the game though is fantastic. Essentially this is a Mega Man game done in the "metroidvania" format. If that sounds fun, don't miss this. Great OST too.
Soma Bringer <-I know this is a Japanese game, but an English fan patch is available. This game is Secret Of Mana meets Diablo, with a sci-fi flair thrown on top. Many people give up on this one too quickly, and admittedly the beginning of the game is slow. Stick with it, Soma Bringer gets better and better the longer you play it. Production values are through the roof because this was developed by Monolith Soft after all.
Brothers In Arms DS <-This game plays quite differently than the mainline series. It's much more third person action than third person tactics. Very fun stuff.
Kirby: Canvas Curse <-Kirby rolls and you have to draw where he goes. Starts off simple but gets hectic fast. Really cool art styles aplomb and highly unique gameplay.
Trauma Center: Under the Knife <-Awesome challenge brought forth by awesome original game design. Turbo scalpel slice like a pro.
Unsolved Crimes <-A hard boiled detective adventure with cool DS puzzles that no one has played other than me. I enjoyed it.
Rune Factory 3: A Fantasy Harvest Moon <-Irons out the annoyances of the first two games and has a much faster pace. Soooo much stuff to do you won't get bored. And howabout them waifus.
Front Mission DS <-Freaking amazing strategy game that is pure nerd porn for mech lovers. The DS version adds a whole new campaign that the SFC version didn't have. You're practically getting two games in one here. I LOVE THIS ONE
Orcs & Elves <-Super quick playing turn based dungeon crawler that is mega fun yet also requires tactical strategy to succeed.
So there's 18 at least. If I had played more of the immense library of the DS I'm sure I could come up with 30 no problem. Give me a few more years.