I think that, as is often the case with dichotomies, this is a false one. A relationship can be perfectly serious while having multiple partners, even if the partners aren't mutual.
Do I think that attraction is "all that matters"? No, of course not. I love my girlfriend, and I want her to be happy. She loves me too, and brought up the concern that we have different sexual needs. If physical attraction was all that mattered, we simply wouldn't be in a relationship any more. But we are. And it's awesome.