PURPOSE: Real-World Data (RWD) studies are increasingly used to support regulatory approvals, reimbursement decisions, and changes in clinical practice for novel cancer drugs. However, few studies have systematically appraised their quality or compared outcomes to pivotal trials. METHODS: All RWD studies (2010-2019) for drugs approved by the Food and Drug Administration (FDA) and European Medicines Agency (EMA) from 2010 to 2015 for solid organ tumours in the non-curative setting were identified. Quality assessment was undertaken using the Newcastle Ottawa Scale. Survival differences between each RWD study and the pivotal trial were determined using a related sample Wilcoxon signed-rank test. RESULTS: 293 RWD studies for 45 of the 57 drug indications approved by the FDA/EMA were identified. The most common tumour types were prostate cancer (29%, n = 86) and melanoma (15%, n = 43). A quarter of the studies had industry funding. No high-quality studies were identified, and 78% were low quality. Comparative survival analysis between RWD and pivotal trials was possible for 224 studies (37 drug indications). Differences in median survival between the RWD studies and their corresponding trial ranged from -32 months to 21 months (IQR -4·2 months to 1·6 months). Low-quality studies were more likely to report superior survival outcomes (23%) compared to higher quality studies (8%) (p = 0.02). CONCLUSION: RWD study quality for novel cancer drugs is low and of insufficient rigour to inform reimbursement decisions and clinical practice. RWD studies seeking publication should provide a completed quality assessment tool on submission. Greater investment in properly designed RWD studies is required.