We don't assume time changes, we know it does. For example, say we assemble two hyper precise clocks on a table. We set them to the same time, and hang them on the wall, one closer to the floor, the other closer to the ceiling. The clock closer to the floor will, over a long enough time line, always run slower than the clock above it. Similarly, the top of your body ages faster than your lower extremities. The difference is very tiny, but from ground level to the high atmosphere, the effects become more pronounced.