In certain chaotic systems, you might find that floating point numbers lead to fundamental problems. Quoting from the work of Coveney & HIghfield [0]:
… digital computers only use a very small subset of the rational numbers—so-called dyadic numbers, whose denominators are powers of 2 because of the binary system underlying all digital computers—and the way these numbers are distributed is highly nonuniform. Moreover, there are infinitely more irrational than rational numbers, which are ignored by all digital computers because to store any one of them, typically, one would require an infinite memory. Manifestly, the IEEE floating-point numbers are a poor representation even of the rational numbers. Recent work by one of us (PVC), in collaboration with Bruce Boghosian and Hongyan Wan at Tufts University, demonstrates that there are major errors in the computer-based prediction of the behaviour of arguably the simplest of chaotic dynamical systems, the generalised Bernoulli map, for single precision floating point numbers. For a subset of values of the model’s solitary parameter, very large errors accrue that cannot be mitigated by any increase in the precision of the numerical representation. For other parameter values, double precision reduces the sizeable errors substantially (Milan Kloewer, private communication with PVC). However, this leaves open the question as to whether double precision floating point numbers are themselves sufficient to handle the far more exquisite complexity of real world molecular dynamics and fluid turbulence, which originate in dynamical systems that are many orders of magnitude more complicated. The spectrum of the unstable periodic orbits of the map is badly damaged regardless of the precision of the floating point numbers [65].
I have had the pleasure of listening to Bruce Boghosian at a conference in the past. He gave a talk on inequality and wealth distributions (see Is Inequality inevitable? published by the Scientific American).