As stars form, their position on the HR diagram (or, equivalently, a Color-Magnitude diagram) changes. They start out very cool but physically very large and very bright; as they collapse under gravity and become fully supported by hydrogen fusion power, they become smaller and dimmer.
The practical upshot of all of this is that we can determine the ages of stars based on their HR diagram locations - for a given luminosity and color, there is an associated age. This is particularly useful for low-mass objects, which take extremely long times to reach the main sequence: An M0 star probably takes 200 Myr to reach the main sequence (Dotter et al. 2008), while a brown dwarf will never reach any kind of main sequence, and will slowly cool and dim forever.
Typically, people use theoretical stellar evolution models like Baraffe et al. (1998), but in practice it is also possible to make empirical relations from known young stars with parallaxes, by fitting polynomials to them. The diagram below shows a set of fifth-order (x5) polynomials that were fit to the single-star members of nearby young associations, as they appeared in Riedel et al. (2011).
As expected, we see multi-magnitude differences in the luminosity of Epsilon Chameleon members (pink) versus similarly-colored members of TW Hydra (yellow) or Beta Pictoris (blue). On this basis, I presumed the nearby M dwarf AP Columbae was probably older than Beta Pictoris, but younger than AB Doradus (which annoyingly lies within the range of high-metallicity main sequence stars).
Of course, these polynomials are not perfect - they are dependent on the quality of the parallaxes and whether a star actually is a single, “normal” member of the group... and the polynomials are only useful between the boundaries for which there is data to fit. As of the time when I made these polynomial fits, there were only two young brown dwarfs -- both members of TW Hydra -- with parallaxes AND Johnson V photometry, which is why all the other lines terminate at the middle and hotter M spectral types. If I replaced the Johnson V colors with something redder (I or J-band) it would be a lot easier to produce empirical isochrones for brown dwarfs... although at the moment, extremely few young brown dwarfs are known.