Activity monitoring through the detection and tracking of moving objects in video streams acquired by a moving platform. The method relies on a graphical representation of moving objects, that allows deriving and maintaining a dynamic template of each moving object by enforcing their temporal coherence. This inferred template along with the graph representation used in our approach allows us to characterize object trajectories as optimal paths in a graph. The tracker allows with multiple objects, partial occlusions, stop and go motion in very challenging situations. This graph-based approach differs from the usual techniques based on the spatial representation of the objects, allowing representation of both spatial properties and their change in time.