Yeah, I’ve actually seen that happen in my pool and couldn’t figure it out at first either. The ground was totally level, and the pump wasn’t even running, but the water always seemed to drift toward one side like there was a current.
In my case, it turned out to be mostly wind. Even when it feels calm, a steady breeze, especially if your yard has open space on one side — can slowly push the surface water in one direction. It’s subtle, but it adds up over time and creates that “pull” you’re talking about.
Also, if your return jets are angled a certain way, they can create a circular motion that keeps going a while even after the system shuts off. I didn’t expect it to last, but water keeps moving longer than you'd think.
Another thing I looked at was shade and sun exposure. If one end of your pool warms up more than the other, it can cause tiny shifts in water movement, not strong, but enough to notice if you're looking closely.
If you’ve ruled out slope or leaks, it’s probably something small like that. Definitely odd when you first spot it, though. You're not alone on this one.