This paper reports upon the development of a test of second language (L2) connected speech comprehension. Despite the importance of connected speech to L2 listening comprehension, there remains the absence of a theoretically and empirically sound means of measuring learners’ understanding of it. Thus, the development of the reduced forms test was undertaken to address this need. The assessed material was delivered through a dictation that contained a wide range of frequently occurring reduced forms. To ensure the trait purity of the instrument, the dictation consisted of a series of short decontextualised sentences that were of great lexical and syntactic simplicity. The test underwent piloting with Japanese university students who were from a false beginner to upper intermediate proficiency level. During the test development, both the Classical Testing Theory and Item Response Theory approaches to test evaluation and item selection were utilised. The second version of the connected speech dictation test was administered to 548 participants. The findings showed that all of the items fitted the Rasch model and, therefore, the test is considered a valid measure of reduced forms L2 English listening. Furthermore, the results indicated that Japanese L2 learners have difficulty recognising even the most frequently used English words when they are spoken in fluent native speaker discourse. It was concluded that the teaching of reduced forms should constitute a more important part of the L2 listening curriculum.