Go to Settings> Privacy> Rights Manager> Geodata. There, find the application that needs to turn off access to geodata and disable it.
Starting with Android 6.0, application permissions management is available in the operating system. Depending on the firmware and OS version, the name of the settings section, which is responsible for managing access to the device location, may differ. As well as the path to it.
For example, on stock Android 10, the path looks like this: Settings → Location → Application Permissions.