A musty smell is a warning sign, not just an annoyance. The EPA states plainly that "a moldy odor suggests that mold is growing in the building and should be investigated". Mold grows wherever moisture collects, so the smell is really telling you that something in the house is staying wet.
Often you can smell it before you can see it. The EPA notes you "may suspect hidden mold if a building smells moldy, but you cannot see the source". That hidden source is frequently below the floor, in a space most homeowners never look at.

