Restraining Order against Lucky Talisman

[Zaaland]

I was playing a game where I drew Curse Everything is Relative and one of my opponents played Lucky Talisman to make me loose a level instead of him . I tried to play Restraining Order but he said he was playing it against the curse and not me. I say if he directs it at me then it should be considered against me and Restraining Order should work. I lost the table ruling 3-1 but it makes so much more sense my way. Also would be curious if it would be different if it was a curse I played against him like Curse Duck of Doom and he used a Wishing Ring.