Field of View is an important parameter for ultrasound analysis. It defines the extent of the comparable contours observed on the graphs window at any given moment. It is measured as a horizontal angle (angle aperture), which corresponds with the virtual image origin of the ultrasound probe. The field of view depends on the articulatory phenomena examined. For example, in the case of coronal segments, it is preferable that the field capture activity towards the mid and front sections of the tongue. In the case of velar segments, it is preferable that the field be more retracted to capture more activity towards the back of the tongue. Establishing the field of view is crucial because if it is not properly established, there may be key articulations missed in the range chosen for observation. It is important to highlight the fact that the field of view must be considered before purchasing a machine and transducer since this cannot always be changed.

In this paper, I have implemented a functionality for users to specify the range or subsection of the original Field of View. To distinguish the internal capability of the app from the field of view, I refer to this as the Analysis Fan View (hereafter AFV). This refers to the sections from the original tongue contours that the user chooses to focus on. For this purpose, the angle origin in the AFV is created based on the tongue contours uploaded in the app, which does not directly correspond with the angle origin from the surface of the probe used in the data collection. This is developed in more detail in Section 3.3.2.

Articulatory landmarks are relevant because they can be used to define the areas in the tongue to be analyzed. Tongue ultrasound imaging poses challenges in relation to defining articulatory landmarks. Since the tongue moves as a whole unit and there are no hard-defined sections, as there are in passive articulators (e.g., teeth, alveolar ridge, soft palate), landmark definitions are necessary to accurately interpret the dynamics of the tongue (c.f. Kier & Smith, 1985; Stone & Murano, 2007). In ultrasound research, landmarks can be acoustic and/or articulatory. In the case of acoustic landmarks (cf. Dawson, Tiede, & Whalen, 2016; Lawson & Stuart-Smith, 2019; Markó, Bartók, Csapó, Deme, & Gráczi, 2019; Mizoguchi, Tiede, & Whalen, 2019; Zharkova, 2013), acoustic cues are the bases for defining moments during the articulatory process, for example, the acoustic time of the mid-point of a vowel (Mielke et al., 2017) or the maximum constriction of a consonant located at the mid-point of its duration. On the other hand, articulatory landmarks are defined by the gestural behaviour of the segment. For example, the constriction location in low front vowels is located at the still frame showing the most retracted tongue contour on the tongue dorsum (as in Decker & Nycz, 2012), and the maximum constriction of a velar stop is located at the frame showing the most raised tongue body during stop closure (as in Davidson, 2006). These two approaches for landmark definition are not mutually exclusive. It is common practice in the field to use mixed approaches in which both acoustic and articulatory cues are considered.

The articulatory moment(s), or the time frame, is another important parameter and it refers to the dynamic windows for analysis, for example, analyzing transitions between a vowel and the maximum constriction of the following consonant. In static approaches, the time window is just one screenshot at a given moment, e.g., maximum constriction of a consonant or the onset of a vowel. The definition of these articulatory landmarks strongly depends on the phonological phenomena observed and the questions to be answered. In the case of dynamic studies, the purpose is to measure gestural patterns from point A to point B of a given sequence.

The last stage in the ultrasound workflow is the analysis stage. This includes the type of measurement and the statistical approaches for analysis—qualitative, quantitative, or a combination of both. For example, in static approaches, studies have used Principal Component Analysis (Stone, Goldstein, & Zhang, 1997) and Smoothing Splines to compare tongue contours and measure differences between segments (Davidson, 2006; Decker & Nycz, 2012), and dynamic studies have looked at tongue displacement (Gonzalez, 2015) and velocities (Strycharczuk & Scobbie, 2015).

There are five tabs in this section (See Figure 3). First is the Home tab, which shows a static logo and general information. The second is the Documentation tab and it offers general information on the different parts of the app. The third Load tab is where the user loads the data for analysis. The app only reads csv and tab-separated files. The data has to be previously wrangled from the specific source software so it can be read in. The fourth one is the Data tab. It shows the data imported in a table giving information on the number of speakers, segments, repetitions, and frames. The fifth tab is Manage Data. Different from the Data tab, in the Manage Data tab users can edit the data within the app. Here, the user has three options: delete, modify, and compare. In the delete option, users can delete one or more frames, repetitions, and speakers. The second option is the modify option, which can be used to relabel speaker names, segments, or repetition labels. The compare option makes comparisons of tongue contours of the selected rows. This is relevant when some data is being considered for deletion; having the option to visualize it beforehand is very important.

The visualization section has three parts: Data inspection, Landmarks, and In context. The first one is the Data inspection tab. Here users can explore all the data imported by allowing extensive interactions with all contours using the widgets available. This section as well as all tabs that visualize data have the structure as shown in Figure 3. The main controls on the plotted data are located on the left pane. This includes speakers, repetitions, and frames, as well as an option to save the current plot. The main plot is located on the top-right section. All the changes applied on the left controls are automatically updated on this main plot. Here, the visualization is done in individial speakers, that is, no speakers can be compared in the same graphics. One important feature in this section is that the app enables users to manually set anchor points in the visualization. Anchor points are fixed points in the figure which become the origin points from which polar-like lines are drawn. This allows for exploring areas that show more articulatory behaviour, which is the basis of the visualization and analysis approach of the app. After setting the anchor point, a second point is created by the user to create a line. The algorithm then automatically calculates distances based on the intersections between tongue contours and the line created. The app gives xy coordinates for each intersection. It also gives the angles to the left and to the right, which can be used to inspect articulatory advancement or retraction of specific segments based on angle differences (Proctor, 2009). The overlaid features are located on the bottom right. These are extra options to inspect the location of the point of origin and the option to manipulate the behaviour of other features such as lines, points, and labels.

The next visualization tab is Landmarks. This section allows for creating labels for articulatory landmarks to be analyzed in the data. These landmarks are not automatically created but instead they are manually established by the users. It can range from one single landmark to multiple landmarks. Users also have the option to modify the labels of the landmarks once these have been created. Landmark definition is not required for the visualization functionality of the app. However, they are required, as well as their assignation, for the dynamic analysis functionality.

The next tab is the In Context tab. This section assigns contours to the pre-established articulatory landmarks. This is a tagging capability of the app. Similar to the Data inspection tab, users can create lines with anchor points. One difference is that only one token with its corresponding repetition can be visualized, this is, no multiple repetitions are allowed in the same plot. The purpose is to establish the context in which each contour is located, allowing the user to see the previous and following contour(s). In the case of consonants, this allows for specification of the frame that contains the maximum constriction, which is defined as the frame where the constriction is held and then returns to a lower/resting position. Users can define the number of frames in context to be visualized.

The Graphics tab gives extensive manipulation options to users. Users can edit the graphics of tongue contours, the palate trace, and the text on the images. For tongue contours and the palate trace, options include line colour, line thickness, alpha value, the smoothness level, and the line type. The last option is to modify the text in the plots. It includes font type, font size, and colour of the axis labels, axis ticks, and legend. The settings modified in this section automatically apply to plots in the tabs Data Inspection, Landmarks, and In Context.

In relation to the smoothness parameter, this is only used within the app for visualization purposes. The smoothness is therefore used for the front end to smooth tongue contours in the plots. This does not affect the calculations of the intersections as explained in Section 3.3. For the visualization, users can change the smoothness method of contours in the Graphics section. The methods implemented are gam and loess, as they are available in the ggplot2 (Wickham, 2016) R package.

The Gridlines tab is a foundational section in the app (this section is expanded in Section 3.3). This section is used to establish all gridlines in the data. This can be done on one or multiple segments at a time. The first step is to define the location of the origin point. The first option is the manual option where the user defines where the best location is by clicking on the image. The other two options are for automatic location of the origin point, which is described in more detail in Section 3.3.1. The first option is the Wide setting. It defines the origin point considering the most anterior initial point and the most posterior last point of contours. On the other hand, the Narrow option considers the most posterior initial point and the most anterior last point. The following step after establishing the origin point is the definition of the analysis fan view, which establishes the range of comparison across contours. There are four options. The first three are the same as in the origin point, Manual, Narrow, and Wide. The extra option is the Angle option. It allows the user to define the analysis fan view based on angle degrees for the angle aperture. Once the field-view is established, the user defines the number of gridlines of the fan. The two options are by number and by angle. If chosen by numbers, the user selects the number of gridlines. If by angle, the user decides to choose the location of the grid lines by angle increments.

The last tab is Analysis. In this section, the user has the opportunity to examine the data established in the previous sections. It is based on the intersections calculated in the Gridlines tab and with the landmarks created in the Landmarks tab and tagged in the In Context tab. Following the same layout structure as in the previous sections, the main controls are on the left and the main plot is at the top-right. The main difference is that there is an extra visualization at the bottom right that shows the dynamic differences between the contours selected. The four options are displacement, distance, velocity, and acceleration, which are explained in Section 3.3.4.

The definition of the gridlines origin location can be done manually or automatically. The first option allows the user to locate the origin point manually by clicking on the desired location within the plot. The second is the automatic option. In this case there are two further options, selection of either a narrow window or a wide window. In the Wide option, the x value of the midpoint is located between the extreme points: the most advanced point and the most retracted point of tongue contours, as shown in Figure 5. For the Narrow option, the x value is calculated between the most retracted first point and the most advanced last point of tongue contours.

The y value is calculated by first extracting the highest y value of all contours and the lowest y value of all contours, as shown as the highest and lowest points in Figure 6. Then the final y value is calculated by subtracting the highest y value from the lowest y value.

After the gridlines origin has been created, the next step is to define the Analysis Fan View (AFV), which establishes how much of each tongue contour is included in the analysis for comparison. A wider AFV can capture more articulatory activity at different sections of the tongue. However, if it is too wide, it risks having sections where not all contours are comparable due to tokens which are missing intersections in specific areas. On the other hand, a narrow AFV can be used to isolate areas of interest and analysis. The risk is that if looking at a very narrow AFV, the analysis may lose important articulatory activity that is outside the range. The trade-off between the two therefore has to be kept in mind by the researcher. The other sections of the app, especially Data Inspection and In Context, allow users to inspect areas of interest before deciding the AFV for analysis.

Similar to the gridlines origin, the AFV can be established manually or automatically. For the manual option, the user can choose the anterior and posterior lines manually by clicking on the plot or specifying the aperture angle for each of them. In the case of automatic options, it can be narrow or wide. These two depend on the angle aperture (See Figure 7 for reference). In the case of the wide option, the left line is located at the intersection point with the widest angle. The right line is located at the intersection point with the narrowest angle. The main strength of this option is that is captures all contours across their length.

The narrow option creates an AFV that captures only the sections that are within the common area. There are two stages to calculate the second point for each line. For the left line, the algorithm calculates the angle between the origin point and the last point of each contour. It first starts with the widest angle and for each iteration it checks whether the projected line intersects with all contours. If the projected line does not intersect with all contours, the following angle is selected. The process stops when the line intersects with all contours in the data. The same process is applied to find the right line, but in this case, it starts from the narrowest angle and iterates until finding a line that intersects with all tongue contours. The result is then an angle aperture which captures a section that is common in all the data. As shown in the triangle in Figure 7, there is a common area which captures all contours that stay within the AFV. The areas at the left and the right outside the common area are not considered in the analysis.

When the AFV is established, the next step is the number of gridlines, which is decided based on either a selected number of gridlines or angle increments. In the first option, the user defines a specific number of gridlines and the algorithm divides the angle aperture of the AFV by the number of lines specified. For the angle option, the user defines the angle step increment. Then the lines are added by the angle step increment, starting from the right line and adding new lines until reaching the maximum angle within the field view aperture. In both cases, the result is a fan-like view of gridlines superimposed on all tongue contours. This allows extraction of tongue contours based on intersections between traces and gridlines. Figure 8 shows the resulting gridlines with their corresponding angles in a sample data.

The following step is the intersection points. After all gridlines have been established, all intersections are calculated for all contours. The result is a new data frame with the same number of tongue contours but narrowed down to the intersection points within the AFV. This new data is the baseline for the analysis within the app. The app can analyze dynamic patterns in four ways: displacement, distance, velocity, and acceleration, which are key when examining tongue motion. In this context, tongue motion is defined as the change of a tongue section within the mouth in respect to time (how fast the tongue is moving). One parameter is displacement, which is the length of the path traveled by the tongue section from one landmark to another, based on a specific gridline. It is important to point out that this displacement is a relative measure from one tongue contour to another. Different from EMA techniques, which track tongue flesh points, the displacement calculated here can only measure the relative movement from point A to point B in a given gridline. It is represented in Figure 9 and calculated using Formula 1 where d_LMB is the distance from the origin point to the intersection with a tongue contour in the second articulatory landmark and d_LMA is the distance from the origin point to the intersection with a tongue contour in the first articulatory landmark.

Formula 1: Displacement formula.

It first calculates the distances from the origin point to the intersections of the first landmark contours at a given gridline. For example, the displacement of the tongue from the midpoint of a vowel to the maximum constriction of a following consonant on the 5^th gridline. The displacement can therefore be positive, negative, or zero. Displacement in this framework is defined as the movement of the tongue in specific locations, which are determined by the gridlines. For each line, the displacement in mm or pixels is the distance from the first tongue contour of comparison (Intersection A in Figure 9) to the next articulatory moment (Intersection B in Figure 9) to capture more fined-tuned articulatory patterns.

The second dynamic analysis is distance. It calculates all the space the tongue has covered throughout its trajectory from the first landmark to the second landmark. Figure 10 depicts this difference. On the left, there are three tongue intersections and they ascend in the gridline from the first intersection to the third, both landmarks. On the right, there are four tongue intersections. One difference is that from the first to the second, there is a downward movement, then it ascends to the third and finally to the fourth. As shown here, if we consider only landmark displacement, they have the same value. However, the distances for each token differ, since the example at the right has a transition contour intersection that adds more distance to the trajectory. This difference between displacement and distance is important when considering multiple contours in the trajectory between articulatory landmarks. This is also implemented in the app.

Velocity, the third dynamic analysis, measures the speed of a section of the tongue in a specified gridline (how fast the displacement is in relation to time). In Strycharczuk and Scobbie (2015), tongue contour velocity is measured based on upward or downward movements along lines placed on a fan-like shape. This measurement is also implemented here and is calculated following Formula 2, where t represents time calculated by multiplying the number of frames between the first and second landmark by the frame rate of the ultrasound images (e.g., 30 fps are 0.033 seconds per frame-to-frame transition).

Formula 2: Velocity formula.

The last metric, acceleration, measures changes in the velocity of that section, with respect to time, using Formula 3.

Formula 3: Acceleration formula.

Both velocity and acceleration are based on tongue displacement: Velocity measures the rate of change of tongue displacement and acceleration measures the rate of change of the velocity. One important aspect of interpreting positive and negative values is that it depends on the measurement analyzed. In the case of displacement, distance, and velocity, negative values represent negative motion in relation to the baseline of comparison, which in this case is the first articulatory landmark. This reflects the directionality in the specified gridline. In the case of acceleration, it depends on the velocity. Since acceleration measures changes in velocity, negative acceleration takes place when there is slowing motion from one frame to another. In these terms, the first initial acceleration, from the first landmark contour to the next frame, the acceleration is positive. Then the following values can be positive or negative, depending on whether velocity slows down in frame sequences. This allows measuring a very important aspect of tongue articulations not observed at the spatial level but on a spatial-temporal dimension.