Most definitely.
Worst case: They have no idea if the sitter or a member of his/her family has a compromised immune system.
Even a healthy person has the right to know in advance of exposure. Others don't want the extra responsibility.
I personally would probably say yes, but I'd like to know what I'm in for.