wayland/surface: Fix can_scanout_untransformed() for transform+viewport

The buffer needs to match the untransformed layout size.

While on it simplify the check from floats to ints where possible - and
improve logging a bit.

Part-of: <https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/2394>