Markov multistate models in continuous-time are commonly used to understand the progression over time of disease or the effect of treatments and covariates on patient outcomes. The states in multistate models are related to categorisations of the disease status, but there is often uncertainty about the number of categories to use and how to define them. Many categorisations, and therefore multistate models with different states, may be possible. Different multistate models can show differences in the effects of covariates or in the time to events, such as death, hospitalisation, or disease progression. Furthermore, different categorisations contain different quantities of information, so that the corresponding likelihoods are on different scales, and standard, likelihood-based model comparison is not applicable. We adapt a recently developed modification of Akaike's criterion, and a cross-validatory criterion, to compare the predictive ability of multistate models on the information which they share. All the models we consider are fitted to data consisting of observations of the process at arbitrary times, often called 'panel' data. We develop an implementation of these criteria through Hidden Markov models and apply them to the comparison of multistate models for the Health Assessment Questionnaire score in psoriatic arthritis. This procedure is straightforward to implement in the R package 'msm'.