Motion tracking dolly shots - Blender markers go crazy

Hi everyone,

I’m testing out motion tracking in Blender with different camera moves just so I can learn how to shoot footage in a vfx-friendly manner.

In this shot I “dolly” (its just a shoulder rig) over the table and pan towards the wall while pushing in. There isn’t much parallax at the end of the move but I got a good camera solve and the trackers stick well to the markers but as soon as one of them moves out of frame, the 3D floor goes crazy. It should be a very easy track but it might be that I just need to get used to this tracker (I’ve used AE and Mocha Pro). Should I just use a nodal tracker? I still get the issue of the floor moving all over the place.

Here is the video of what happens: FloorCrazy