Your vague memory is probably that ls -l gives file size, while du give "disk usage" - the number of blocks used. On my computer the blocksize is 4k, so du only reports multiples of this size. (In particular, the default behavior is to report units of historical blocksize, so it only reports multiples of 8.)

A huge difference that I doubt you forget is how they define the size of directories - just metadata vs recursively. But that means that du is expensive. I use it all the time, but not everywhere.