I ended up with writing a script that finds the camera and attaches it to a constraint component, as suggested by @Prot .
The script clears the constraint list and creates a brand new one with only the camera in it.
It uses the [ExecuteInEditMode] attribute, but it still works very badly when in the editor(since the editor/prefab camera isn’t Camera.main), and could be removed. In the end I didn’t care and it still works fine when in play mode.
Here’s the script for anyone interested:
public class FindCameraAndAddToConstraint : MonoBehaviour
private LookAtConstraint constraint;
private List<ConstraintSource> clist;
private ConstraintSource source;
source = new ConstraintSource();
source.sourceTransform = Camera.main.transform;
source.weight = 1f;
clist = new List<ConstraintSource>();
constraint = GetComponent<LookAtConstraint>();