Speech perception entails the mapping of the acoustic waveform to linguistic representations. For this transformation to succeed, the speech signal needs to be tracked over various temporal windows at high temporal precision in order to decode linguistic units ranging from phonemes (tens of milliseconds) to sentences (seconds). Here, we tested the hypothesis that cortical processing of speech-specific temporal structure is modulated by higher-level linguistic analysis. Using fMRI, we measured BOLD signal changes to 4 s long speech quilts with variable temporal structure (30, 120, 480, 960 ms segment lengths), as well as natural speech, created from a familiar (English) or foreign (Korean) language. We found evidence for the acoustic analysis of temporal speech properties in superior temporal sulcus (STS): the BOLD signal increased as a function of temporal speech structure in both familiar and foreign languages. However, activity in left inferior gyrus (IFG) revealed evidence for linguistic processing of temporal speech properties: the BOLD signal increased as a function of temporal speech structure only in familiar, but not in foreign speech. Network connectivity analyses suggested that left IFG modulates the processing of temporal speech structure in primary and non-primary auditory cortex, which in turn sensitizes the analysis of temporal speech structure in STS. The results thus suggest that acousto-linguistic transformation of temporal speech structure is achieved by a cortical network comprising primary and non-primary auditory cortex, STS, and left IFG.