In most real-world settings, a transportation plan requires modifications during execution. A thorough evaluation of transportation planning methods thus requires testing and comparison in a dynamic environment. We give conditions on a simulation environment that follow from this requirement, and propose a multi-agent simulator meeting these conditions. In addition, we propose a new measure that captures robustness in such dynamic settings. The multi-agent simulator and the robustness measure are then used to compare three different transportation methods (two multi-agent planners and one online optimization approach) in settings with release time uncertainty and truck breakdown incidents.