1.1. Phonetic cues and cue weighting

Phonological contrasts are known to be multi-dimensional, as distinctions are frequently achieved along various articulatory and acoustic parameters (Ladefoged, 1980). For example, the obstruent voicing contrast in English [±voicing] can be found in up to 16 different acoustic properties, such as voice onset time (VOT), closure duration, preceding vowel duration, and formant and f0 transitions, among other cues (Lisker, 1986). One hallmark of phonetic cues is that they have different “weights,” in that different cues contribute to the phonological contrast to various degrees (e.g., Repp, 1982). For example, in the English word-final voicing contrast, the duration of the preceding vowel is found to be more widely used than glottal pulsing or the fading-out of formants (Greenlee, 1980; Krause, 1982; Wardrip-Fruin & Peach, 1984; Nittrouer, 2005).

Cues are also perceptually integrated in the processing of a phonological contrast and can “trade” with each other (Fitch et al., 1980; Repp, 1982; Kingston et al., 2008). Primary and non-primary cues can be traded between each other to strengthen a contrast, either in an enhancing relationship (strong primary cue + strong secondary cue) or a compensatory relationship (weaker primary cue + stronger secondary cue). This occurs perceptually without changing the label of the category (Fitch et al., 1980; Francis et al., 2000; Iverson et al., 2005; Shultz et al., 2012).

Cue weights in perception and production are highly variable. They can differ between speakers of the same language (Guion, 1998; Jiang et al., 2020; Pfiffner, 2021; Hauser, 2023) and further within different speaker populations (Coetzee et al., 2018; Pfiffner, 2021, 2025). For example, Coetzee et al. (2018) show that the word-initial plosive voicing contrast in Afrikaans is produced with different weights given to VOT and the following vowel’s f0, which varies by speaker. Some speakers produce the contrast through VOT alone, and others devoice all plosives and rely on differences in the following vowel’s f0. Pfiffner (2021, 2025) shows that these differences in Afrikaans are further influenced by particular social factors of the speaker and listener; women, both younger and older, are found to devoice more often than men, for example. This also translates to perception, where in the absence of any prevoicing, the following vowel’s f0 at onset is a significant factor in the perception of phonological voicing, where plosives followed by low f0s are perceived as phonologically voiced, and plosives followed by high f0s are perceived as phonologically voiceless. However, this effect is greatest when listeners are hearing female talkers in comparison to hearing male talkers (Pfiffner, 2025).

One of the many unanswered questions in studies of phonetic cues is whether these trading and weighting relationships can be extended to other modalities, such as between acoustic and visual dimensions. If visual information is among the total cues employed by the speaker in production and available to listeners to recover intended speech gestures and phonological categories, then visual cues should also be perceptually integrated, weighted, and traded with each other and acoustic cues. This may then have important implications for how studies analyze sound change. In this study, we begin to fill this gap by examining the co-occurrence of visual and acoustic properties in the production of the sibilant contrast in Mandarin. This particular contrast is known to be neutralizing in particular regional varieties of Mandarin (e.g., Taiwan), so by examining which visual and acoustic properties are readily available to listeners and how much they vary in a particular contrast, we can better understand how mergers progress in both the acoustic and visual dimensions. Further, this provides background information that can then be used to test multimodal perceptual cue weighting and integration in future studies. In the following sections, we discuss audiovisual integration in speech and the Mandarin sibilant contrast/merger. Then, we introduce the present study, its results, and implications for sound change.

1.2. Audiovisual cue integration in speech

Acoustic accounts of sound change are widely accepted, but it has long been known that speech perception is multimodal, so a pure auditory perceptual model may not suffice in accounting for all sound changes. Listeners are also found to be sensitive to cues in the visual dimension (Sumby & Pollack, 1954; McGurk & MacDonald, 1976; Mayer et al., 2015; Ménard et al., 2016) or coupling sensorimotor information with the speech signal, such as tactile movement of the jaw and facial muscles (Gick & Derrick, 2009). Cues from all modalities are argued to be integrated without conscious awareness (Marks, 2004) to improve detection of speech and intelligibility. For example, Sumby and Pollack’s (1954) seminal study demonstrated that visual exposure to speakers’ facial movements, in addition to auditory cues, significantly improves speech intelligibility under noise when the amount of information is held consistent. When there is less noise, the contribution of visual cues is reduced and non-significant. Similar results are also reported by Grant and Seitz (2000), where visually matched and unmatched videos were played simultaneously with speech under noise. Compared to the audio-only condition, the detection of speech was improved to around 1.6 dB in the audio-visual (matched) condition. Since there was no difference between audio-visual (unmatched) and audio-only condition, it was concluded that the visual dynamic movements of articulators enhanced the accuracy of speech perception.

Introducing conflicting cues between audio and visual dimensions further supports the role of visual input in speech perception, as illustrated by the McGurk Effect (McGurk & MacDonald, 1976; Sekiyama & Tohkura, 1991), where audio-only inputs of [ba] or [ɡa] are accurately perceived, but when audio utterances are dubbed with incongruent videos, most listeners report hearing a fusion syllable of perceptual cues in both dimensions. For example, [da] is reported when the lip movement of [ba] is overlaid onto the auditory signal of [ɡa]. Among the three critical cues (i.e., place of articulation, manner of articulation, and voicing) that constitute [ba], [da], and [ɡa], the place and manner of articulation are visible, and voicing is audible. When visual information of [ba] (i.e., presence of lip closure) replaces that of [ɡa] and is combined with manner and voicing, listeners perceive [da].

Finally, brain imaging studies have shown that areas once believed to be sensitive only to auditory cues also respond to visual stimulation in a similar way (Calvert et al., 1997, 2001). Such activities also happen at the early stage of auditory processing, suggesting a more plastic, multimodal speech processing in humans (Musacchia et al., 2006). For example, lipreading with no auditory input stimulates activities in the auditory cortex measured by functional magnetic resonance imaging (Calvert et al., 1997), as well as meaningless pseudo-speech expressions, such as lip or jaw movement. However, non-linguistic facial expressions do not activate any response in the area. Musacchia et al. (2006) found that audio-visual stimulation of congruent or conflicting cues may influence auditory processing as early as 11 ms and last as long as 30 ms after exposure. Taken together, these studies show that subcortical auditory processing is also sensitive to visual input, implying a multi-modal nature of speech perception in underlying neurological and physiological mechanisms.

Despite the numerous studies of audiovisual cues in speech perception and their implications for cue relationships between modalities, there is a lack of research examining the co-occurrence of these cues in production and their subsequent cue weighting in perception. The present study will address the co-occurrence of visual and audio properties in the production of the Mandarin sibilant contrast, described in the following section.

1.3. The Mandarin sibilant contrast and merger

In Mandarin, there is traditionally a three-way contrast of sibilants between alveolars (/s/, /ts/, /tsʰ/), retroflexes (/ʂ/, /ʈʂ/, /ʈʂʰ/), and palatoalveolars (/ɕ/, /tɕ/, /tɕʰ/). These sets of sounds include fricatives (/s/, /ʂ/, /ɕ/), affricates (/ts/, /ʈʂ/, /tɕ/), and aspirated affricates (/tsʰ/, /ʈʂʰ/, /tɕʰ/), all of which include a period of frication noise. Affricates have an additional transient burst preceding the frication noise. Aspirated affricates in Mandarin do not reliably have a low-frequency, low-amplitude noise following the frication period (Tseng et al., 2011, p. 791–793); instead, they are frequently articulated as one combined gesture (see discussion in Section 2.4). Alveolars are typically characterized by a high amplitude, high frequency frication noise, in contrast with retroflexes, where the energy is more concentrated in a lower frequency range. Although there is a disagreement among existing studies for the place of articulation (POA) of the two sets of sounds in Mandarin, ranging from the dental region to alveolar ridge for the alveolars, and retroflex or post-alveolar for the retroflexes (Hauser, 2023, and see a discussion in Chang & Shih, 2015), the fact that there is a difference in POA is widely accepted. Palatoalveolars, on the other hand, can be seen as a sequence of a sibilant and the high front vowel [i] in complementary distribution with the other two sets of sibilants.

The traditional three-way contrast between retroflexes (/ʂ/, /ʈʂ/, /ʈʂʰ/), alveolars (/s/, /ts/, /tsʰ/), and palatoalveolars (/ɕ/, /tɕ/, /tɕʰ/), is notable because it is being reduced in some regional varieties of Mandarin, especially in Southern regions and Taiwan (Cheng, 1985; Lee-Kim & Chou, 2022). Specifically, /ʂ/, /ʈʂ/, and /ʈʂʰ/ have been observed to be both acoustically and perceptually more similar to /s/, /ts/, /tsʰ/ respectively, due to a change in spectral properties. This includes an increase in the center of gravity (CoG) in the frication portion of the retroflexes for speakers that participate in the merger in comparison to those that do not (Chang, 2012).¹

One dimension that has not yet been considered in this contrast and merger are potential differences in visual properties. Traditionally, retroflexes are described as curling the tongue tip backwards to lengthen the front resonating tube, thus lowering the filtered frequency (Fant, 1960, 1971) in the frication proportion. However, it is also possible for rounding gestures to achieve similar acoustic properties by lengthening the front resonating tube, and previous studies have described Mandarin retroflexes as rounded (e.g., Chang, 2012). However, to our knowledge, these visual cues have never been tested and quantified. Previous studies of this contrast further focused solely on their acoustic properties (e.g., Hauser, 2023; Lee-Kim, 2022; Chiu et al., 2020), potentially leaving the rounding properties unaccounted for.

Existing experiments have suggested the possibility of an additional gesture; for example, an ultrasound study by Chiu et al. (2020) found variable degrees of tongue retroflexion in Taiwan Mandarin sibilants, ranging from complete articulatory distinction between alveolars and retroflexes to complete articulatory overlap. However, in the same study, such low-frequency CoG was given a lesser weighting by native speakers in perception. The authors reported that compared to [i] and [o], the following vowel [a] introduced more overlap in CoG between retroflexes and alveolars among four of the seven speakers they tested. It is possible that the following vowel had an anticipatory effect of tongue lowering, which causes less tongue retroflexion and more fronted contact closer to the alveolar ridge. However, this might not be the only answer, as speakers exhibiting context-specific merger patterns had a mismatch between acoustic results and reconstructed tongue position using ultrasound data. Chiu et al. further examined formant frequencies and found that these speakers had a lowered F2 in vowel onsets, which pointed to the possibility that there are additional articulatory gestures of lip rounding that may enhance or maintain this contrast.

1.4 The present study

The current study aims to fill the gap by documenting the visual properties and variability of lip rounding in the production of the Mandarin sibilant contrast. We further investigate the relationship between acoustic and visual properties co-occurring in speech production.

In an audiovisual production experiment, we employ the non-intrusive, non-obstructive, easily adaptable, and openly available computer vision tool, OpenFace 2.0, an independent video processing toolkit using Python to analyze facial movements and behavior (Baltrušaitis et al., 2018). The methods have been adopted and expanded in a laboratory phonology context (Krause et al., 2020, 2024). For example, Broś and Krause (2024) examine lip apertures in stop lenition in Canary Islands Spanish, finding that aperture, alongside acoustic measurements, can be used to probe the degree of lenition. Furthermore, they found that lip aperture captures focus-related, sentence-level prosodic differences in lenition more clearly than acoustic measures, while acoustics are more sensitive to stress-based differences. This lends further support for studying phonological contrasts in the audiovisual domain, as visual cues can contribute information independent of acoustic cues. Also, these studies in general demonstrate the feasibility and reliability of using motion capture and computer vision tools to study articulation and visual properties of phonological change. Our study therefore employs a similar audiovisual production paradigm to answer our research questions about the visual properties in this contrast, their variability, and their relationship to acoustic cues. Our research questions are as follows:

1. What are the visual properties of alveolar and retroflex sibilants? What patterns can we see between the two sets of sounds among different visual dimensions?

2. What is the relationship between acoustic properties and (visual) lip rounding? Is there a cue enhancing relationship in articulation whereby the visual parameters strongly predict acoustic properties?

2.1. Stimuli

All stimuli in the experiment were monosyllabic CV words in Mandarin, balanced for tone (Tone 1, 2, 3, and 4). Due to lexical constraints, only 34 tokens were included (Appendix A), as well as 10 additional fillers. All items were repeated three times. Participants were presented with the Chinese characters, and for characters with multiple readings, we also presented Pinyin,² a romanized representation of Chinese phonology, above the character. Word frequency was controlled to ensure minimal difficulty recognizing the character in production. Because the stimuli were single Chinese characters, frequency was extracted using character-frequency values from the SUBTLEX-CH-CHR corpus (Cai & Brysbaert, 2010). One target character (玼) was not listed in the corpus and assigned a value of 0, but it was still selected due to lexical constraints.³ Across all 44 items, mean raw character frequency was 41,980 (SD = 88,486), corresponding to 896 characters per million (SD = 1,889). Alveolar (n = 10) and retroflex items (n = 12) were matched in frequency: Frequency statistics for each member of the contrast (/s, ʂ, ts, ʈʂ, tsʰ, ʈʂʰ/) are given in Appendix B. Frequency did not differ significantly between alveolar and retroflex items in a Welch t-test on log-transformed frequency (i.e., log(raw frequency+1), t(12.29) = –0.38 , p = .711) or in a Wilcoxon rank-sum test on raw frequency (W = 65 , p = .767); the same pattern held for characters per million. All tokens were sibilant-initial and followed by homorganic vocalic nuclei (represented by /i/ in Appendix A).⁴ Two additional pairs between alveolars and palatoalveolars were formed, excluding retroflexes due to lexical constraints. Filler words had non-sibilants onsets, including stops, fricatives, and approximants, and only non-homorganic vocalic nuclei were used.

2.2. Procedure

Recordings were made in a sound-attenuated booth. Participants were seated about 40 inches away from a teleprompter, which consisted of an 11-inch screen and a piece of glass to reflect lights emitted by the screen, and a Razer Kyio Pro webcam that was mounted behind the glass. The whole teleprompter system was mounted on a tripod that was adjusted to the height of the participant’s eye level so that their face stayed in focus at the center of the camera’s view. An additional AKG C535 EB directional microphone was placed approximately 20 inches away to record high-quality audio for acoustic analysis. It was placed below the teleprompter and outside the view of the webcam so that the tracking of facial landmarks would not be affected. All participants were told to keep their upper body still during the experiment, although the facial feature extraction algorithm that we used is capable of accurate measurements despite slight head movements or variations in the distance between the speaker and the camera (Krause et al., 2020).

Audio was recorded using Praat (Boersma & Weenink, 2023) and a Steinberg UR22 audio interface at a sampling rate of 44.1kHz and a depth of 16 bit. Before video recording, the camera was calibrated following the methods of Krause et al. (2020). The calibration (or camera resectioning) is deemed necessary because it helps recover the exact pose and location of the camera in relation to the captured objects in three-dimensional space (Quan & Lan, 1999; Fraser, 2001; Lu, 2018). To calibrate, we took 30 pictures of an angled chessboard and fed the pictures into a Python script to estimate four parameters,⁵ which were later specified in OpenFace (Baltrušaitis et al., 2018) to accurately track the video. An extra copy of index audio was recorded using the webcam through Open Broadcaster Software 29.0.2 at 1080P and 60 frames per second, encoded in H.264 compression standard and .mp4 container format. Audio and video were aligned using Adobe Premiere Pro with reference to the index audio so that the start time and duration of the high-quality audio recording was synchronized with the video recording.

Each session consisted of one familiarization block with four trials and 13 test blocks. Stimuli were pseudo-randomized within each block. The familiarization trials were not included in the analysis. In each trial, participants were presented with a countdown timer from three seconds to zero, followed by a metronomic beep and a two-second presentation of stimuli. The metronomic beep was designed to be the starting point of visual parameter tracking. Participants were instructed to utter the word as soon as they saw the character but to keep their mouth closed before and after stimuli presentation. There was a short break at the end of each block. The experiment took an average of 15 minutes to complete and produced 132 audio and video tokens (i.e., 34 target items plus 10 fillers, with 3 repetitions each) per participant.

2.3. Participants

Thirty native Mandarin speakers participated in the study (19 female and 11 male speakers), all undergraduate students from China who were studying abroad in the United States. All were born and raised in China until they were 18 years old and continued to use Mandarin daily. Participants were also fluent in English or had at least had 10 years of exposure to formal instruction in English. Participants also reported use of local Chinese varieties, though their dominant language for communicating with friends and family was Mandarin, and they classified themselves as native Mandarin speakers. Finally, no one reported any cognitive, hearing, or speech difficulties, and all had normal or corrected-to-normal vision. Participants were paid 10 US dollars for their participation.

2.4. Data Processing

Audio recordings were annotated, aligned, and segmented by the first author using Praat (Boersma & Weenink, 2023). For each trial, the metronomic beep and the utterance were marked. The utterance was further segmented to isolate the frication portion of the sibilant. For both aspirated (n = 715) and unaspirated (n = 1239) retroflexes and alveolars, the frication portion began at the end of the transient burst visible in the waveform and ended at the onset of voicing and visible formant structure. We did not distinguish any subsegmental boundaries between the burst noise and aspiration for the aspirated affricates (ʈʂʰ and tsʰ) because no boundary was often visible. This is not unusual for Mandarin, where the internal boundary between frication and aspiration is not clearly marked (see Tseng et al., 2011, p. 791–793, who reported that approximately 10% of Mandarin tokens surveyed had a clear boundary). Further, cross-linguistically, it is not uncommon for aspiration noise to be partially or completely blended or subsumed by high-amplitude higher-frequency frication noise in both stops and affricates, which suggests a substantial gestural blending between abducted glottis and the stricture at the oral cavity (Stevens, 2000). We observed these patterns in our data (Figure 1) and thus analyzed the entire frication portion of the aspirated affricates.

Figure 1: Example spectrograms of an aspirated [ts^hi2] showing no clear boundary between frication and aspiration periods in comparison to unaspirated [tsi2].

Following previous research on sibilants (e.g. Lee-Kim & Chou, 2022; Rao & Shaw, 2021), tokens were high-pass filtered at 300 Hz, and four spectral moments were calculated from the middle 50% of frication. The four moments calculated for analysis include spectral mean (center of gravity, or CoG), spectral dispersion (SD), skewness, and kurtosis. It should be noted that CoG is prone to changes in sampling rate, or parameter settings for CoG calculation. Alternative approaches include examining the whole spectral slice (e.g., Lee-Kim et al., 2022), or Multitaper Spectral Analysis (e.g., Shadle, 2023). However, CoG has been widely used in previous studies (e.g. Chang, 2012; Chiu et al., 2020; Rao & Shaw, 2021), and it provides an easier way to perform inferential statistics and estimate multiple effects, such as visual cues.

We further segmented audiovisual-aligned videos into chunks that corresponded to the exact timestamp of each trial using the MoviePy library in Python and then imported videos to OpenFace (Baltrušaitis et al., 2018) to track and extract facial landmarks, head-turns, and head poses. A scheme of facial landmarks from the OpenFace documentation (Baltrušaitis, 2019) is shown below in Figure 2a, followed by the scheme overlaid on a real person producing round (left) and unround (right) sibilants in Figure 2b.

Figure 2a: The scheme of critical landmarks, adapted from Baltrušaitis (2019), where red dots are the focus of the current study.

Figure 2b: The scheme of critical facial landmarks applied to a real person’s articulations.

With a reference point of the position of the camera, coordinates inside the three-dimensional space were recorded for the top and bottom points of the upper and lower lips, and the outer and inner sides of the two lip corners. We then found the midpoint between lip edges and obtained a new set of horizontal and vertical apertures, using point numbers 51, 62, 57, 66, 48, 60, and 54 and 64, respectively (Figure 2a). Excluded were data entries that had an average inferential confidence reported by OpenFace below 0.95 or head-turns that exceeded two standard deviations away from the position of the camera. Corresponding to TextGrid annotation, video frames were marked as either pre-speech, consonant, or vowel phase. We used the frames from speech onset to offset (i.e., excluding pre-speech) for the main analyses. The visualization of the rounding trajectory included all three phases. This leaves 1,954 alveolar and retroflex sibilant tokens overall, 56,755 video frames for the main analyses, and 144,725 video frames for the trajectory analysis. Palatoalveolar tokens served as additional fillers and are not analyzed here.

For variables of visual properties, we first computed Euclidean distances (Formula 1) in both directions using the midpoint between the inner and outer edges, i.e., the midpoints between 51 and 62, 57 and 66, 48 and 60, and 54 and 64, respectively. To account for the thickness of the lips, the distances were measured between midpoints and not landmarks at the edges. The distances form the vertical and horizontal apertures of the mouth. It should be noted that although the reported distance uses millimeters as the unit of measurement, it is only meaningful across participants in this study and should not be entirely equated to millimeters in the real world (Krause et al., 2020).

$\mathit{\text{Distanc}}{e}_{\mathit{\text{ab}}}=\sqrt{{\left(X_a-X_b\right)}^2+{\left(Y_a-Y_b\right)}^2+{\left(Z_a-Z_b\right)}^2}$

Formula 1: Euclidean Distance formula used to find the distance between a(X_a, Y_a, Z_a) and b(X_a, Y_a, Z_a).

We then quantified the roundness. We assume that the shape of the mouth in its resting position can be approximated to an ellipse (e.g., the right panel of Figure 2b), and when the lips are rounded, their shape approximates a circle (the left panel of Figure 2b). Under this assumption, to derive roundness, we calculated the eccentricity (Formula 2) of this ellipse, which is equal to the square root of 1 minus the square of the semi-minor axis (i.e., the shortest radius of the ellipse) divided by the square of the semi-major axis (i.e., the longest radius of the ellipse). In the context of the current study, the semi-major and semi-minor axes correspond to half of the horizontal and the vertical distances that were mentioned in the preceding paragraph, respectively. The eccentricity of ellipses, by definition, ranges from 0 to 1. For a perfect circle, where the major and the minor axes are equal, the eccentricity is 0. The value is larger than 0 for any ellipse. Therefore, the less rounded, the higher the eccentricity value. This could be counterintuitive if we adopt eccentricity as the direct representation of roundness, as more rounded tokens would have smaller values. Therefore, we decided that the roundness in this study will be defined as 1 minus the eccentricity (Formula 3). It then still ranges from 0 to 1, but more rounding will lead to larger roundness values. There are other potential ways to derive roundness (i.e., circularity, see Montero & Bribiesca, 2009, for a review), such as the variance of distances from the centroid to each critical landmark on the lips, which will be smaller in value for more rounded tokens. The eccentricity method was chosen for its simple application to the form of data available to us, i.e., coordinates in the three-dimensional space. As a result, we include the variables of horizontal aperture, vertical aperture, and roundness score (1 — eccentricity) in our analysis.

$e=\sqrt{1-\frac{{\beta }^2}{{\alpha }^2}}$

Formula 2: The eccentricity of an ellipse, where ɑ (i.e., the semi-major axis) is half of the horizontal distance, and β (i.e., the semi-minor axis) is half of the vertical distance.

$\mathit{\text{Roundness}}=1-e$

Formula 3: Roundness of sibilants, where e is the eccentricity in Formula 2.

2.5. Analysis

Data analysis was conducted in R (R Core Team, 2024).⁶ Our first research question asked about the visual properties and variabilities of the retroflex-alveolar contrast. To answer this, we first report the summary statistics of the horizontal and vertical apertures, as well as derived roundness scores. Then we examine individual variability by plotting speakers on a spectrum of differences in roundness scores, operationalized with Cohen’s d (Cohen, 1988; see e.g. Brunelle et al., 2020) between the two categories, i.e., retroflex vs. alveolar. Male and female speakers are analyzed separately for their potentially different baseline in overall lip size. To test if the two categories can be differentiated by roundness score, we fit a linear mixed-effects regression model to predict the visual property with category (i.e., retroflex vs. alveolar), manner of articulation (i.e., affricates vs. fricatives) and the interaction between them (see the results section, as well as the supplementary materials for model details). To account for differential baselines of subjects and item-specific effects, by-participant and by-item random intercepts, as well as by-participant random slope, were added to the model.

For our second research question, the relationship between acoustic and visual properties in this contrast, we first establish whether acoustic differences are realized among the participants we sampled, given that there is an ongoing merger in particular varieties of Mandarin (e.g., Taiwan Mandarin). Although all of our participants are from mainland China, where the merger is more limited and stigmatized (Chang et al., 2013; Duanmu, 2007; Lin, 2007), we consider the step necessary as a sanity check for the status of the primary acoustic correlate, i.e., whether CoG has undergone merger among our participants in recent development of the sound change. We divided our participants by gender and examined individual productions with respect to the spectral distance between the two categories using Cohen’s d. Gender groups were also analyzed separately because there are known gender differences in sibilant production, where male speakers tend to have lower CoG measurements in comparison to female speakers. Further, it is common to see gender variation in a sound change (Trudgill, 1974; Labov, 1990, 2001), so it is helpful to analyze by gender in case any of our speakers are participating in the merger. Separate linear mixed-effects regressions with random intercepts for participants and items were fitted to predict the four spectral moments by category.

We then paired visual data with acoustic data and performed Pearson correlation tests to find which acoustic properties best correlate with visual properties. As mentioned above, previous studies focused on CoG as the primary acoustic cue, so we expect that CoG would best correlate with any visual properties. We then present a scatterplot of changes in CoG against changes in the roundness score with fitted regression lines separated by gender, which is predicted to show a positive correlation between changes in both measurements, as more changes in the length of the front resonating tube should result in more changes (i.e., lowering) of frequency in the frication (see the background section for details). We also fitted a linear mixed-effects regression model predicting CoG by measurements in the visual domain. For a cue enhancing relationship among audiovisual cues, we would expect changes in CoG to be significantly predicted by roundness.

To show that acoustics and visual properties may together support the distinction, we also fitted a conditional random forest model for all speakers to categorize property (i.e., alveolars vs. retroflexes) by both CoG and roundness score. While the method of using the generalized linear mixed-effects model for predicting categories with multiple cues and assessing cue weights has been widely used in previous studies (see, e.g., Jiang et al., 2020; Shultz et al., 2012) to assess weightings between different acoustic cues in production and perception (e.g., Berg, 1989; Doherty & Turner, 1996; Yang & Sundara, 2019), the analysis in the current study is limited by the fact that the acoustic and visual correlates are intrinsically correlated, which would introduce collinearity that may make the interpretation of regression models difficult. Therefore, we fitted the random forest model, which is less affected by collinearity, to better counteract the correlation between predictors.

To account for multiple comparisons, p-value adjustments were conducted using the Benjamini-Hochberg method within families defined by research questions to control the false discovery rate at q = .05. Tests for RQ1 (1 LMER model) were adjusted together, then for RQ2 (three correlation tests, five LMER models). We report both the raw p-values and adjusted p-values (referred to as BH-corrected p) in the following sections. Notably, the random forest model outputs (e.g., accuracy, variable importance, model metrics, etc.) are not p-values and were not considered in the correction.

3.1. Visual properties and variability

Our first research question asks about the visual properties of the two classes of sounds, as no previous studies (to our knowledge) have measured them. Figure 3 shows the mean values⁷ and distributional envelopes for horizontal aperture, vertical aperture, and roundness in alveolars and retroflexes. In general, alveolars show larger horizontal apertures than retroflexes (50.26 vs. 47.57 mm), whereas retroflexes are significantly more open in the vertical direction than alveolars (17.15 vs. 14.18 mm). Together, these differences yield a higher mean roundness score for retroflexes than for alveolars (0.08 vs. 0.04). The linear mixed-effects model predicting roundness also showed a significant main effect of property (F(1,32.35) = 15.56, p < .001, BH-corrected p = .001), with larger roundness scores for retroflexes than alveolars (b = 0.039, SE = 0.010, df = 33.41, t = 3.94, p < .001), indicating that retroflexes are indeed characterized by greater roundness.

The violin plots (Figure 3) indicate that alveolars and retroflexes differ in their overall lip configuration, but the size of the category separation is relatively modest when the raw aperture measures are considered separately, i.e., 2.69 mm and 2.97 mm for horizontal and vertical differences, respectively. Alveolars show a slightly larger mean horizontal aperture than retroflexes, whereas retroflexes show a slightly larger mean vertical aperture. In both dimensions, however, the two categories overlap broadly, and the mean differences are small relative to the amount of within-category spread. This suggests that the visual contrast is not carried by a clean separation in any single aperture measure. Rather, the categories are only partially differentiated in horizontal and vertical distance, with substantial token-to-token variability within each class.

Figure 3: Differences in the mean horizontal and vertical apertures, as well as roundness.

The roundness plot shows a clearer contrast. Compared with the alveolars, the retroflexes’ distribution is shifted upward, with a longer upper tail. This means that retroflexes are generally produced with greater lip rounding, but also with much more variability (SD = 0.07) in how strongly that rounding is realized across tokens. By contrast, alveolars are concentrated in a narrower scope of low roundness scores (SD = 0.02), indicating a more stable articulatory pattern with relatively little rounding. In other words, the figure shows that retroflexes differ from alveolars more in the broader and right-skewed rounding distribution, less so by forming a separate category. This pattern is consistent with the idea that retroflexes employ the rounding gesture more strongly overall, while also allowing greater within-category variation in the magnitude of that gesture.

After examining the overall visual properties of alveolars and retroflexes, we next asked whether the visual contrast was shown uniformly across participants. Figure 4 therefore shows the variability in roundness among participants, measured by difference between alveolars and retroflexes in roundness for each sibilant pair, operationalized with Cohen’s d values. Most by-participant by-pair comparisons (i.e., 30 participants, with 3 pairs of sibilants each) still showed a robust visual contrast: Across the 90 comparisons, 75 fell in the stable-contrast range (d ≥ 0.8), 9 showed partial merger (0.2 ≤ d < 0.8), and 6 were categorized as full or reversed merger (d < 0.2). In raw values, the retroflex-alveolar difference in roundness ranged from 0.004 to 0.275 for male speakers and from -0.006 to 0.079 for female speakers. Thus, for most speakers, retroflexes remained more rounded than alveolars, although the strength of this visual contrast varied across individuals.

Figure 4: Individual roundness variation in production of three groups of contrast. Female (n = 19) and male subjects (n = 11) were ranked by the degree of rounding merger (measured by Cohen’s d), averaged across the three sibilant pairs.

The visual merger cases included participants 15 and 29, both female. Participant 15 showed negative raw differences for all three sibilant pairs, with Cohen’s d values between –0.92 and –0.19, placing all three pairs in the full or reversed merger range. Participant 29 likewise showed three full or reversed merger pairs, with negative raw differences and Cohen’s d values between –0.79 and –0.39. These patterns indicate that, for these speakers, retroflexes were not produced with systematically greater roundness than alveolars, suggesting that rounding was not used as a reliable articulatory difference for the contrast.

By contrast, several speakers showed much stronger visual differentiation across pairs, including participants 4, 8, 22, and 28 among the female speakers, and participants 2 and 11 among the male speakers, all of whom displayed large positive raw differences and large Cohen’s d values. Other speakers, such as participants 7, 9, 13, 16, and 27, showed reduced, but still generally positive, contrasts in one or more pairs, indicating a weaker visual difference. However, because the number of speakers was unbalanced between female and male speakers, we do not interpret these patterns as systematic evidence for a gender effect here.

3.2. Trajectory of lip rounding

Figure 5 shows the articulation trajectory of the two classes of sounds by each individual onset along the three visual measurements. All timepoints were normalized relative to the duration of each syllable, with 0% and 100% being the stimuli onset and speech offset (i.e., the onset and offset of trials), respectively. The dotted green lines mark the median speech onset, and the dashed blue lines mark the median timepoint of the boundary between the sibilant and the nucleus. The sibilant-vowel boundary was at 19.901%, 38.504%, and 41.759% in normalized time of the syllable (i.e., speech onset to offset), corresponding to 79 ms, 183 ms, and 214 ms of mean onset duration for /ts-tʂ/, /tsʰ-tʂʰ/, and /s-ʂ/ pairs, respectively. Aperture values at the 0% and 100% indicate the average vertical and horizontal apertures of lips at the resting position.

Figure 5: Fitted trajectories of (a) horizontal and vertical apertures, and (b) roundness scores for syllables with alveolars and retroflexes, split by place and manner of articulation. The green vertical dotted lines mark the median speech onset. The blue vertical dashed lines mark the median sibilant-vowel boundary. The shaded ribbons show central 95% confidence intervals, i.e., 2.5%–97.5%.

Notably, compared to alveolars, retroflexes showed continued increases in vertical aperture and roundness after the initial lip opening, resulting in a higher vertical trajectory and a later peak in roundness. In the analysis of peak alignment, alveolar affricates and fricatives reached peak roundness at 45.6% and 43.0% of the normalized tracked trial interval (i.e., trial onset to speech offset), whereas retroflex affricates and fricatives peaked later, at 53.6% and 49.8%, respectively. However, because mean speech onset occurred at 60.1% of the interval, these retroflex peaks still preceded speech onset by 8.8 (time) percentage points for affricates and 5.5 percentage points for fricatives. Across all tokens, the overall peak occurred at 48.9%, or 11.2 percentage points before speech onset. Moreover, 67.0% to 81.4% of tokens reached peak roundness during the pre-speech interval, indicating that speakers begin preparing the rounding gesture before audible frication begins. This also further confirms that the two sibilant classes differ visually, with the contrast being articulatorily planned and becoming visible well before speech onset and remaining especially clear near the end of the syllabic period.

The graph also shows larger rounding values for retroflexes on the following vowel. Therefore, to fully analyze the time-course differences, we conducted a post-hoc analysis of rounding by syllable phase (consonant vs. vowel) by fitting a GAMM model⁸ during the speech onset-to-offset window to predict roundness score, implemented using bam in the mgcv package (Wood, 2023). We included property (alveolar vs. retroflex), phase (consonant vs. nucleus), manner of articulation (fricatives vs affricate) and their interactions as fixed effects, as well as smooth terms over normalized time, including by-property subsegment interaction and participant-specific smooths to capture individual variation, and an AR(1) error structure to account for autocorrelation with residuals from adjacent timepoints within each token. This model is intended to test the category and manner differences globally and the fine-grained differences over time, especially whether the trajectories diverge and at which phase, and what effects contribute to such divergence.

The GAMM showed a moderate fit that explained 48.7% of the deviance. No evidence of inadequate basis dimension was found in gam.check (all k-indices = 1.00, all p ≥ .49). The parametric terms showed a significant positive main effect of retroflex property (β = 0.0616, SE = 0.0030, t = 20.21, p < .001, BH-corrected p < .001), a negative coefficient for nucleus relative to consonant in the reference alveolar-affricate condition (β = –0.0084, SE = 0.0025, t = –3.42, p < .001, BH-corrected p = .001), and a significant interaction between property and phase (β = –0.0343, SE = 0.0034, t = –10.15, p < .001, BH-corrected p < .001), indicating that the retroflex advantage in roundness over alveolars was present, though it was smaller in the nucleus than in the consonant phase. There was no main effect of manner (β = –0.0019, SE = 0.0040, t = –0.48, p = .635, BH-corrected p = .750), and no interactions involving manner were significant. Smooth terms further showed no reliable time effect for alveolar consonants (edf = 1.001, F = 3.487, p = .062, BH-corrected p = .099), but significant time-varying effects for retroflex consonants (edf = 1.003, F = 140.748, p < .001, BH-corrected p < .001), alveolar nuclei (edf = 3.967, F = 18.008, p < .001, BH-corrected p < .001), and retroflex nuclei (edf = 7.456, F = 129.657, p < .001, BH-corrected p < .001).

The results indicate that the roundness difference emerges primarily in the consonant, though it is also present in the following vowel, and that retroflexes show a higher and more variable rounding trajectory over the whole portion of the syllable. This confirms that participants have employed extra articulatory gestures other than tongue retroflexion. Since the vowel qualities in the current study are tightly controlled (and, importantly, unround), we hypothesize that the additional rounding gesture may be temporally spread from the primary acoustic features in the course of a syllable, but we cannot generalize. More data is needed to test if the rounding difference still persists with more vowel qualities, particularly round vowels. Analyses in the study therefore use the average value of roundness across the syllable, instead of roundness during the frication phase alone.

The finding that rounding differences persisted into the syllable nucleus suggests that the labial gesture associated with retroflexes is not confined to the frication interval but carries over into the following vowel. This can be explained by a sequential combination of protrusion and rounding gestures: Lip protrusion may begin early to lengthen the front cavity and shape the frication acoustics, while the more visibly rounded configuration becomes strongest as the gesture stabilizes and continues into the vowel where rounding first fades away, followed by declining protrusion. This, in turn, provides a potential articulatory link between more retroflex rounding observed in the current study and the formant-onset differences for the following vowel in Mandarin, especially lower F2 onsets after retroflexes: If retroflexes are produced with greater and more persistent lip rounding, the following vowel begins from a more rounded labial configuration and therefore may contribute to a different formant onset. This hypothesis is partially corroborated by other production studies in Mandarin (e.g., Hauser, 2023), where F2 (especially F2 onsets) in the vowel period also differ between retroflexes and palatoalveolars. In addition, the aforementioned pre-speech peaks of roundness are also consistent with an anticipatory account rather than an unrelated gesture, which suggests that speakers often establish the rounding gesture before audible frication begins as part of speech planning for the upcoming CV sequence, with the same labial configuration continuing to shape both the onset consonant and the early portion of the vowel. However, ultrasound imaging and sagittal lip recordings will be needed to disentangle the relative timing of protrusion, rounding, tongue posture and transition, and frication.

3.3. The enhancing relationship between audio and visual correlates

We now turn to the relationship between acoustic and visual properties, beginning with showing that there is a significant acoustic contrast between the two categories among our subjects. However, since our participants came from across Mainland China, where the contrast may have interacted with local language varieties, we expected to see a range of individual variation. To further analyze this, CoG differences analyzed as Cohen’s d between retroflexes and alveolars were plotted for male and female speakers (Figure 6), which show that the retroflex-alveolar contrast is robustly maintained across speakers. Distance was also converted to Cohen’s d based on the difference in CoG between alveolars and retroflexes for each speaker-pair combination. At the level of speaker-pair combinations, 89 of 90 fell in the stable contrast range (d ≥ 0.8), and only one observation fell in the partially merged category. All 30 participants had mean acoustic distances above the maintained-contrast threshold when distances were averaged across the three pair types. In raw CoG units, speaker-pair distances ranged from 1168.95 to 7985.16 Hz, with a mean of 3861.22 Hz. Mean speaker-level Cohen’s d values ranged from 1.35 (Participant 16) to 15.03 (Participant 11). Overall, the figures indicate a clear-cut boundary between retroflexes and alveolars along CoG, with only one relatively attenuated speaker-pair combination showing partial merger, with no evidence for contrast collapse.

Figure 6: Individual acoustic variation in production. Female (n = 19) and male subjects (n = 11) were ranked by the average spectral distance between alveolars and retroflexes across contrasts, measured by Cohen’s d.

Figure 7 visualizes the four spectral moments and labels the mean of each distribution inside the boxplots. The clearest separation is in CoG values, with a mean for alveolars of 8453.80 Hz and 4879.35 Hz for retroflexes. The differences between the two categories are closer for spectral dispersion (2098.62 vs. 2082.00 Hz), skewness (1.05 vs. 1.21), and kurtosis (4.36 vs. 4.34). We fitted separate linear mixed-effects models with property, manner, and their interaction as fixed effects, and by-participant random intercepts, by-participant random slope for property, manner, and their interaction, as well as by-item random intercepts. For kurtosis, the maximal model failed to converge, so the final model retained a by-participant random intercept and slope for property, along with a by-item random intercept. The results showed that only CoG showed a significant effect of property (F(1,40.17) = 241.54, p < .001, BH-corrected p < .001) with lower CoG for retroflexes than alveolars (b = –3779.73, SE = 246.74, df = 42.38, t = –15.32, p < .001). By contrast, the effect of property was not reliable for spectral dispersion (F(1,35.07) = 0.49, p = .490, BH-corrected p = .490), skewness (F(1,33.48) = 3.52, p = .069, BH-corrected p = .089) or kurtosis (F(1,32.76) = 0.72, p = .403, BH-corrected p = .454). These results indicate that CoG retains the robust acoustic cue distinguishing alveolars from retroflexes. Although other spectral moments failed to reach significance, the strongest energy for retroflexes is still more concentrated in the lower portion of the spectrum, in comparison to alveolars.

Figure 7 Differences in four spectral moments between alveolars and retroflexes. Values outside the 2.5th and 97.5th percentiles were excluded.

We further visualized and tested the correlation between changes in roundness scores and changes in CoG values (Figure 8). Across the 90 speaker-pair combinations, the raw CoG difference and the raw roundness difference were positively correlated, but the association across both genders was not significant (r = 0.139, t(88) = 1.32, p = .190, BH-corrected p = .190). When examined separately by gender, the positive association was numerically larger and significant among female speakers (r = 0.441, t(55) = 3.64, p = .0006, BH-corrected p = .0018) than among male speakers (r = 0.296, t(31) = 1.72, p = .095, BH-corrected p = .142), which shows a modest but non-significant positive correlation between raw CoG differences and roundness differences. This indicates that in the current sample, the overall visual-acoustic association was driven more strongly by the female speakers.

Figure 8: Difference in CoG plotted against difference in roundness. Each data point is a by-speaker-by-onset pair.

Although the male regression line in Figure 8 appears steeper, which may indicate a larger effect of rounding per unit on CoG, the fitted regression slope and the Pearson correlation coefficient do not measure the same thing. The slope reflects the expected change in roundness for a one-unit change in CoG in the original measurement units, whereas the correlation coefficient is unit-free and reflects how tightly the observations cluster around a linear trend. Thus, a steeper line for male speakers can coexist with a larger correlation coefficient for female speakers.

Due to the unbalanced gender numbers, we refrained from interpreting the larger correlation coefficient or slope as any main gender effect. Generally, the plot and the tests indicate that larger acoustic differences between alveolars and retroflexes were associated with larger differences in lip roundness, such that speaker-pair contrasts with greater distinction in CoG also tended to show more roundness contrast. This indicates that, on average, participants who were more distinct in the acoustics were also more distinct in the visual dimension. Also, note that when the difference in roundness is close to 0, the difference in raw CoG is still substantial (i.e., 1208 Hz), which indicates the effect of tongue retroflexion gesture independent of lip rounding. Setting tongue retroflexion aside, this indicates that changes in CoG may be at least partially attributed to increased vertical aperture. These results further corroborate our expectation that there is an enhancing relationship between the primary acoustic correlate and visual property of roundness.

To further answer our second research question, we fitted a linear mixed-effects model predicting CoG with property and roundness, with by-participant random intercepts and random slopes for roundness, and by-item random intercepts (Table 1). We expect rounding to contribute to the distinction along CoG by a negative regression coefficient, due to its lengthening of the front resonating tube that lowers the filtered frequency. The decision to include roundness instead of horizontal and vertical apertures was made based on the possibility of non-independence between horizontal and vertical apertures, as both apertures may be affected by each other. A larger horizontal aperture may limit the range in which vertical aperture may vary. Although this may be solved by including interaction terms between the two apertures, using a single variable of roundness score best answers the effects of rounding on acoustics.

Table 1 summarizes the fixed effects in the model predicting CoG from property and z-score standardized roundness. We assumed that the effect of roundness on CoG could vary across speakers, and therefore we fit a model with a by-participant random slope for roundness, which was the maximal structure that still converged without singular fit. To assess the contribution of roundness itself, we additionally compared two random intercepts-only models that differed only in whether roundness was included as a fixed effect. Model performance worsened significantly when roundness was removed (χ²(1) = 22.63, p < .001), with higher AIC and BIC values (AIC: 32305.43 vs. 32284.80; BIC: 32333.32 vs. 32318.27) and a lower log-likelihood (–16147.72 vs. –16136.40). We therefore interpret the model that includes roundness and the participant-specific random slope. Property was strongly significant: At mean roundness, alveolars had a significantly higher CoG than retroflexes (β = 3445.654, SE = 100.200, t = 34.388, p < .001, BH-corrected p < .001), consistent with our previous findings that the contrast is still maintained. The coefficient for roundness was also negative and significant (β = –714.526, SE = 193.527, t = –3.692, p = .0008, BH-corrected p = .0008), indicating that greater roundness was associated with lower CoG values. The omnibus test for roundness was likewise significant in the model ANOVA (F = 13.632, p = .0008), showing that roundness contributed to CoG differences overall. Taken together, these results confirm that CoG is the primary correlate to the sibilant contrast, and they support our hypothesis that the visual property of rounding effectively enhances sibilant contrast together with the acoustic property of CoG.

Recall that the acoustic and visual properties showed different variability among participants, creating different continua of less to more distinct. Individual speakers may thus have employed rounding to different degrees to achieve the contrast in CoG, essentially assigning different loadings to acoustic (i.e., CoG) and visual (i.e., roundness and apertures) correlates in production. Therefore, we further tested whether the inclusion of visual correlates (i.e., roundness) improves the identification of sibilant category overall. Existing studies in cue weighting (e.g., Jiang et al., 2020; Yang & Sundara, 2019; Shultz et al., 2012; Holt & Lotto, 2006; Berg, 1989; Christensen & Humes, 1996; Doherty & Turner, 1996) have employed inferential methods such as ANOVA and generalized linear mixed-effects models with different acoustic parameters to assess the contribution of individual acoustic correlates to the contrast, in both production and perception. However, as mentioned, the current study is limited by the collinearity between roundness and CoG. Therefore, we performed a conditional random forest model, which is less affected by collinearity issues, to categorize retroflexes from alveolars using both CoG and roundness. Using party::cforest (Hothorn et al., 2006) in R, the forest was grown with 500 trees with two predictors considered at each split. Model performance was evaluated with out-of-bag predictions rather than a separate training and test partition: For each tree, 63.2% of the data was randomly sampled without replacement for training, and the remaining cases served as out-of-bag test cases. This sampling was random and not additionally stratified by speaker or contrast.

The model showed high classification performance, with an out-of-bag accuracy of 97.6% and similarly strong precision (0.9695), recall (0.9868), and F1 (0.9781) values. It correctly classified 859 out of 892 alveolar tokens and 1,048 out of 1,062 retroflex tokens. Conditional variable importance indicated that classification was driven overwhelmingly by CoG (conditional importance = 0.381), whereas roundness slightly contributed unique information after CoG was taken into account (conditional importance = 0.000762). This indicates that CoG contributed primarily to the identification of the contrast among the speakers in the current study. We interpret this in relation to the fact that most speakers in our dataset maintained significant differences between alveolars and retroflexes in acoustics. In the context of where the loss of retroflexion is more prevalent (e.g., Taiwan Mandarin), visual measures may contribute more substantially than the current study shows.

4.1. Articulatory markedness of Mandarin retroflexes

As mentioned previously, certain varieties of Mandarin have an ongoing sibilant merger from retroflexes to alveolars. The merger is notable for two reasons. First, broadly speaking, retroflex neutralization tends to be unidirectional, where retroflexes merge towards the alveolars, but not the other way around.⁹ Second, since Mandarin has a three-way contrast, the merger in Mandarin is not only unidirectional but also asymmetrical, as retroflexes almost never merge with palatoalveolars. This is perhaps because alveolars and retroflexes are both produced with the tongue tip, while palatoalveolars are produced with the tongue blade, and furthermore palatoalveolars use F2 at the onset of the following vowel as a distinct acoustic property from alveolars and retroflexes (Hauser, 2023). But this still does not explain why retroflexes are the sibilant class that merges with alveolars rather than vice versa.

The facts of this merger suggest that some unique property of retroflexes may be driving the merger. The unidirectionality can be explained regarding the differences in general markedness (Maddieson, 1984; Greenberg, 1987). Retroflexes are produced by curling the tongue tip backwards while creating a stricture (fricatives) and a complete blockage of airflow followed by the narrow construction (affricates). In comparison to alveolars, which do not possess tongue retroflexion, the required articulatory complexity and efforts are generally higher.

The current study further expands on the articulatory complexity by finding an additional lip rounding gesture for retroflexes, adding to the existing gestural efforts employed inside the oral cavity. The general tendency towards the reduction of articulatorily complex or “marked” segments may have motivated the sibilant merger, among other factors, such as contact with local Chinese varieties (Ing, 1984; Kubler, 1985).

The findings also align with existing studies in multiple articulatory configurations where, phonetically, the same contrast and acoustic properties can be achieved through different articulatory gestures, such as the case of bunched vs. retroflex /ɹ/ (Delattre & Freeman, 1968). In our case, the Mandarin sibilant contrast can be achieved through differences in tongue position, lip rounding, or both.

4.2. Implications for sound change

It is well understood that the dynamic nature of cue relationships can lead to sound change over time, as seen in recent cases of tonogenesis in Afrikaans (Coetzee et al., 2018; Pfiffner, 2021, 2025) and Seoul Korean (Kang & Han, 2013; Kang, 2014; Bang et al., 2018), or in registrogenesis in Eastern Cham (Brunelle, 2009) and Chru (Brunelle et al., 2020). In these cases, a word-initial voicing contrast is changing towards a primary cue of f0 (tonogenesis) or combinations of f0, F1, and phonation type (registrogenesis; Brunelle & Kirby, 2016). Several recent studies have found that changes in cue strength of the same contrast correlate with age, where younger speakers’ perception and/or production of the same contrast is significantly different from older speakers’ perception and/or production, indicating a population-level sound change (Gao, 2016; Bang et al., 2018; Brunelle et al., 2020; Jiang et al., 2020).

Within accounts that relate sound change to the distribution of phonetic cues, the addition of visual correlates in production may increase the likelihood for listeners to fail in perceptual reanalysis and not recover the speaker’s intended category. For example, the listener-oriented model proposed by Ohala (1981, 1983, 1989, 1993) views the exchange of speech as a series of mappings of acoustic cues onto phonological categories. Sound change arises from the listener’s “innocent misinterpretation” along the noisy channel between the speaker and the listener, which can introduce biases against certain acoustic cues and make listeners match acoustic cues onto other phonological categories that were not originally intended by the speaker. Later studies found that the reanalysis is performed not only on the phonological categories but also gestures (e.g., French nasalized vowels, see Beddor, 2009, 2012; Carignan, 2018) and in the visual domain of cues (e.g., Havenhill, 2019; Havenhill & Do, 2018).

The articulatory variation may take on different visual properties, and this variability in visual cues may differentially affect variants in a sound change in progress, such as the case of [θ] and [f] mentioned above (McGuire & Babel, 2012), which could be why the sound change progresses in this direction. Similarly, in regards to the length of resonating cavities, Havenhill (2019) and Havenhill and Do (2018) found that there are at least two articulatory patterns for back vowels used by American English speakers who partake in the Northern Cities Vowel Shift (NCVS); although lip rounding and tongue backing were found to have similar effects on F2, variants of /ɔ/ that were produced with more tongue backing but less lip rounding were both visually and perceptually more similar to variants of /ɑ/.

The source of such individual variability may lie in anatomical differences and speaker’s active control for articulatory configurations to maintain constant acoustic output. Brunner et al. (2009) found that individual differences in palatal morphology can shape speaker-specific tongue configurations. In an electropalatographic and acoustic study of 32 speakers, those with flatter palates showed reduced variability in tongue height, likely because a given change in tongue height produces a larger change in constriction area for flatter palates than for more domed palates. On the other hand, acoustic variability did not increase relative to speakers with more domed palates. This suggests that anatomical differences in palatal shape may have pushed speakers toward different lingual configurations, even when they converged on similar acoustic targets.

In the case of Mandarin, the contrast between alveolars and retroflexes is primarily in CoG, which can be manipulated by tongue position and/or the degree of lip rounding, both of which affect the length of the front cavity and therefore the resonant frequencies. Lip rounding provides both a visual cue and an additional source of variability in acoustics that may influence the possible trajectory of change. Chiu and colleagues (2020) found that there is a continuum of articulatory gestures for Mandarin retroflexes, ranging from tongue retroflexion to tongue fronting. This partially resulted in variable acoustic cues in the frication portion of the sibilants. Such variability in tongue gestures, along with the lip rounding found in the current study, may also arise due to anatomical differences in tongue shape or the height or shape of the alveolar/post-alveolar regions. The variability of retroflexion is then further coupled with the variability of lip gestures that we report among retroflexes, leading to an even less robust contrast and resulting in the total cues available to listeners being less stable across the board.

Therefore, considering the additional individual variation in audiovisual correlates observed in the production study, it is possible for us to hypothesize potential pathways to prevent or induce this merger in perception. Mandarin speakers may fall into any of the four categories of articulatory gestures and resulting cue weighting relationship for retroflex sibilants: One may employ both lip rounding and tongue retroflexion, leading to comparable cue weights between different acoustic cues and between acoustic and visual cues. As rounding cues are discovered to be in an enhancing relationship with acoustic cues, both primary and secondary cues contribute to the contrast. Therefore, this articulatory configuration may be the most robust and least prone to sound change, as it offers the clearest distributional separation along multiple cues for listeners. Alternatively, one may primarily rely on tongue retroflexion and employ little rounding, resulting in little to no visual and less acoustic contrast. As tongue retroflexion tends to have more variable articulation (see, e.g., Tabain, 2009, on variability in the place of articulation of retroflexes mediated by stress in Arrente), this second configuration is more likely to be incorrectly recovered by perceptual reanalysis in perception, possibly most prone to undergo a merger. A third option would be for speakers to resort to rounding as a primary cue; however, this would involve a reweighting of cues that is not currently supported by our overall production results, and, at the same time, the availability of visual cues is not always consistent and can be influenced by the noisy channel along the visual dimension, such as attention level (e.g., Andersen et al., 2009; Tiippana et al., 2004) or the medium of speech exchanges (i.e., auditory or audiovisual), which may also make rounding-only configurations more susceptible to merger. Finally, if both visual and acoustic cues are no longer available, younger generations or language acquirers will merge the two categories.