Depending on the part of the country you are in, people with an accent may get asked a lot where they are from. Almost everywhere they go, even in passing. It can get annoying for folks, so my recommendation is to not ask in passing (at the supermarket or in a phone call to a service provider for example).
In a social or work setting where you will spend time with someone, by all means ask straight out. And as mentioned above, it is good to ask with a positive comment, i.e. 'cool accent where is that from?'.
I never fault people for asking. But it is a bit like when we had a Great Dane. We'd be out and about with the dog a lot. People loved seeing him and generally it was fun. But everyone had the same couple jokes pertaining to its size, comparing it to horse and riding it, which we heard literally a thousand times over the course of the dog's life. ARGH!
G