I found that rsync -rtcvP SRC DEST works very well with FAT32.
Instead of using the popular archive (-a) transfer mode this one uses the recursive+times+checksums flags.
The checksums (-c) flag is the essential bit. Comparison by checksums is more robust than comparison by timestamp and file size. Settings this flag avoids transferring existing files to the destination. The transfer will be a bit more CPU intensive, but depending on the contents this might be negligible.
The times (-t) flag preserves modification times on the destination.