Yes, I think it can work, if when the object passes the portals the portals gets displaced by -m/M times their distance, where m is the mass of the object and M is the combined mass of the portals. (By “work” I mean it doesn't need there to be a privileged frame of reference for it to be described.)

(I'm assuming Galilean relativity; I'm not sure it can be made to work in special relativity as well.)