There are many metrics available to measure the goodness of a classifier when working with ordinal datasets. These measures are divided into product-moment and association metrics. In this paper, the behavior of several metrics is studied in different situations. In addition, two new measures associated with an ordinal classifier are defined: the maximum and the minimum mean absolute error of all the classes. From the results of this comparison, a pair of metrics is selected (one associated to the overall error and another one to the error of the class with lowest level of classification) to guide the evolution of a multi-objective evolutionary algorithm, obtaining good results in generalization on ordinal datasets.