PLAViMoP and Eye Tracking: A Method to Integrate 2D Gaze Data Within a C3D File of a Point-Light Display

The Point-Light Display (PLD) paradigm, introduced by (Johansson, 1973), is a method used to study the perception of biological motion (BM) by capturing the kinematic of key joints using illuminated points. This played a decisive role in revealing our innate sensitivity to biological motion (Simion et al., 2008; Vallortigara et al., 2005), which develops further with age, and is widely used in neuroscience and psychology to investigate perceptual and action recognition processes. Today, advances in technology allow PLDs to be created through algorithms (Cutting, 1978) or real motion capture (Manera et al., 2016) or mixed, capturing and modifying real movements (Mahmood et al., 2019), providing flexibility for experimental research. There are also new AI-based approaches for extracting coordinates from conventional video (such as SmartDetector; Bidet-Ildei, BenAhmed et al., 2024). Since 2018, we have been working with several colleagues on facilitating the use of PLD for more precise investigations of human sensitivity to BM (Bidet-Ildei, BenAhmed et al., 2024; Bidet-Ildei, Francisco et al., 2022; Decatoire et al., 2018). Indeed, numerous researchers have consistently investigated the perception of BM for more than five decades. The quest for the understanding of biological movement seems to be a remarkably deep source of inquiry. This depth is partly due to the sensitivity of BM, which flirts with the innate and the acquired. There will be a marked precocity (Bidet-Ildei et al., 2014; Craighero et al., 2016) that can become more refined with development into adulthood (Ross et al., 2012). This particularity allowed for the investigation of numerous cognitive abilities at different developmental stages, notably recognition and discrimination, as well as emotion (Atkinson et al., 2004; Bachmann et al., 2018; Venesvirta et al., 2016 for a review), identity (Beardsworth & Buckner, 1981, Coste et al., 2021, Sevdalis & Keller, 2011), sex (Pollick et al., 2005: Runeson & Frykholm, 1983) and language (Beauprez et al., 2020; Boulenger et al., 2009). Additionally, it has contributed to research on learning including motor learning (Francisco et al., 2022; Horn et al., 2002); mathematical cognition (Badets et al., 2012; Bidet-Ildei, Vilain et al., 2024) and memory (Villatte et al., 2022) as well as pathological conditions (Bidet-Ildei, Deborde et al., 2022; Francisco et al., 2023; Izawa et al., 2012; Mazza et al., 2010). Moreover, PLD is also of interest in other domains, such as computer science and engineering (for a review, see Yousefi & Loo, 2019).

To date, there are already numerous tools available to assist researchers wanting to use kinematics to investigate the understanding of BM by a participant. In this regard, several databases (Bidet-Ildei, Francisco et al., 2022; Ma et al., 2006; Manera et al., 2010; Okruszek & Chrustowicz, 2020; Shipley & Brumberg, 2004; Vanrie & Verfaillie, 2004) and new technologies (Bidet-Ildei, BenAhmed et al., 2024; Decatoire et al., 2018; Piwek et al., 2016; Shi et al., 2018; Thomas & Seiffert, 2010) have been developed to easily produce their own stimuli. Yet, although current techniques allow to analyse the movement of the eye by means of its relative speed in relation to an element of the video (Koerfer et al., 2024; Souto et al., 2023), these techniques do not allow to specify the absolute position of the eye in relation to each element of a PLD. Eye-tracking tools record a participant’s eye movements, enabling the analysis of eye kinematics and the determination of gaze direction and positions at a given moment or over a period of time. Traditionally, eye-tracking has been used to analyse eye movements on conventional screens with 2D stimuli. To have a precise idea of gaze direction or gaze location various methodologies have already been proposed in the literature (see for a review, see Sundstedt & Garro, 2022). The most common technique is the gaze plot, which consists of superimposing circles on what is seen to indicate fixation points. Fixation maps (Kasprowski & Harężlak, 2017; Rahman et al., 2020) utilize circles whose radius is proportional to the time spent during each fixation, these circles being numbered to indicate the order of fixations in the scene. It is also possible, and even common, to use the notion of area of interest (AOI). The AOI can be defined either by the experimenter (King et al., 2019), who checks the number of times gaze enters and leaves the area, or by an algorithm that calculates this area a posteriori by grouping fixations (clustering).

At this stage, many visualisation tools or techniques tools are thus available for using gaze trackers. However, until now, all of these have been designed in the context of a static image or natural video. As a result, they are not suitable for observing kinematics via PLDs, despite having a specific original format (C3D stands for “Coordinate 3D” and refers to a standard file format used for storing 3D motion capture data.) that could be exploited. For instance, when watching a PLD depicting a walking action, if gaze is positioned on the thigh, i.e. between hip and knee when the two legs cross each other, it would be very difficult to say which leg was being observed. In the same vein, when using AOI, given a stimulus animated by the movement of the action observed, it would be very complicated to obtain moving AOI capable of following the different segments unless a frame-by-frame labelling is employed. However, PLDs can be viewed as a sequence of images with no context, but where points of interest that have already been labelled (R_hip; R_knee; R_ankle; etc.) are tracked over time.

To address these issues, while taking advantage of modern PLD acquisition techniques (i.e., 3D motion capture), we propose a new method that directly reconstructs gaze position within the file containing the PLD coordinates (c3d file). This direct reconstruction in the initial kinematic files used as stimuli, which contains the PLD, offers the advantage of keeping all data in a single file (including the possibility of integrating the gaze kinematics of n-participants for the same observed stimulus), thus facilitating direct comparison between gaze and joint positions. Overall, we believe that our new method offers a neat tool for continuous psychophysical studies (Burge & Bonnen, 2025) that seek to understand the evolution of eye behaviour in response to the task demand (ex. moment of recognition).

Method

EyeTracking and Setup

Participants were comfortably seated in a dark room facing a computer screen (ACER predator, 1920×1080, 27 inch, 240Hz) positioned 57cm away from them (see Figure 1). At that viewing distance, a change in gaze of 1 degree corresponds to a displacement of 1 cm on the computer screen. To restrain head movements, participants had a chin rest and a padded forehead rest with gaze naturally oriented toward the center of screen. Horizontal and vertical positions of the right eye were recorded using an infrared video-based eye tracker (Desktop Eyelink 1000 system; SR Research). The output from the eye tracker was calibrated at the beginning of the experiment by recording the raw eye positions as participants fixated a grid composed of 9 known locations. The output of the eye tracker was fed into a data acquisition system (Keithley ADwin Real Time, Tektronix) and was recorded with a custom software (DOCoMETRe, 2025) at 1000 Hz. Importantly, we ensured that the initiation/ending of eye data collection and display of PLDs were synchronous.

Click to enlarge

Figure 1

Top View of the Experimental Setup

Note. Participants sat in front of the screen with their head movements restrained by a headrest positioned 57 cm away from the screen.

Video Stimuli Preparation

Starting from a c3d file, we first created a new “planar” c3d file using MATLAB (MathWorks) in conjunction with the Biomechanical Toolkit (Barre & Armand, 2014). This was achieved by projecting each point light onto a camera plane based on the camera’s azimuth and elevation settings (see Figure 2). Consequently, coordinates along the orthogonal axis in the camera plane were always set to zero, as illustrated in Figure 2d. These new c3d files were then resized and centered to ensure that the point-light display in each file fit within the same bounding box (e.g., width: 1888.6 mm, height: 2016.6 mm). Finally, 2D video files (frame rate: 30 Hz, resolution: 1920 × 1080 pixels) were generated for each stimulus and displayed on the screen.

Click to enlarge

Figure 2

Example of a PLD Depicting a Human Performing a Pedaling Movement

Note. (a) The original PLD in the motion capture reference frame from the front view. (b) The right-side view and its corresponding projection in a camera reference frame with azimuth set to 45° and elevation to 20°. (c) The three-quarter front view, which was displayed on the screen for participants. (d) The right-side view demonstrating that the new c3d file is planar. (e) It provides a perspective illustration of the pedaling action from a three-quarter view, accompanied by a kinogram of the lower limbs for better visualization.

Inter-Referential Consistency Test (ICT)

To prevent potential issues arising from the conversion of centimeter-based eye coordinates in the EyeTracker’s screen reference frame to millimeters-based ones in the c3d reference frame, an ICT stimulus (Figure 3) was created. The ICT stimulus allows to check referential changes (screen to .c3d file) after conventional eye-tracker calibration. This stimulus was specifically designed to validate the spatial transformation and ensure consistency between coordinate systems. The ICT stimulus consisted in a single point of light moving within the previously described bounding box, starting at the center of the frame and remaining there for five seconds. Additionally, two extra points of light were placed at the bottom-left and top-right corners of the bounding box to allow for manual localization of the video frames displayed on the screen (Figure 3). In this way (the ICT stimulus is initially coded in c3d format and then converted to video) we can find out exactly how much space on the screen is occupied by the PLD videos. Each participant completed this trial once before the experimental session, in which they viewed more complex PLDs.

Click to enlarge

Figure 3

Description of the Calibration Stimulus With the Calibration Trajectory and Bounding Box Limits

Note. Only the white point was visible to participants during the experiment; the trajectory and additional point lights are included here for illustrative purposes only.

Gaze Data Conversion Process

As exposed above, the EyeTracker was calibrated to provide gaze coordinates in centimetres within the computer screen frame of reference, whose origin (0;0) corresponds to the centre of the screen. Here, the positive direction of the X-axis was to the right and that of the Y-axis was to the top. To determine which point(s) of the PLD, participants were looking at during the recognition task, it is mandatory that we employ a common frame of reference for the gaze data (2D) and the PLD (3D), meaning that they rely on a similar origin and units. We opted to convert the gaze coordinates in millimetres so as to match the c3d file reference frame. Below, we present a flow chart that describes the successive stages to achieve that switch in reference frame.

Click to enlarge

Figure 4

Flow Chart of the Conversion Process for the Eye Data

Note. For each participant, X and Y offsets and gain factors were identified by the optimisation process through the calibration stimulus. These conversion parameters were applied to all trials of the participant.

To ensure a good matching between eye data and the PLD, the participant was first asked to follow the ICT stimulus point-light (implicit smooth pursuit instruction) (Figure 3) and the eye’s movement of each participant was recorded. Using the size of the screen as well as its resolution, it was easy to convert the gaze coordinates in centimetres provided by the eyetracker to pixel coordinates ($k_x^1$ and $k_y^1$ ratio). Then, by using the two additional point-lights in the corners of the ICT stimulus, we computed the ratio between screen sizes and c3d distances ($k_x^2$ and $k_y^2$ ratio). To justify the need of these two points in the corners, one needs to understand that the projection of a full-screen video keeps its proportions: on a rectangular screen a square video will touch the top/bottom edges, but not the right/left ones. A simple localisation tool developed with AutoIt (2025) was used to get the mouse coordinates when the two additional point-lights were clicked by the experimenter and then compute $k_x^2$ and $k_y^2$ ratio. At this point, the eye coordinates are known in the c3d reference frame and expressed in millimetres. Even though great care was taken in calibrating the eye tracker, the eye trajectory obtained in the ICT c3d file after conversion still differed from the target the participants were supposed to follow (see Figure 5a). An optimisation algorithm has been written to adjust the eye tracker measurements. The optimisation variables are the x and y offsets of the trajectory ($x_1$ and $x_2$) as well as the x and y scale factor ($x_3$ and $x_4$). Only bounds constraints (lb and ub) have been set to [-400 -400 0.5 0.5] and [400 400 1.5 1.5] respectively. The following cost function (f) was used to minimize the gap between the eye trajectory and the target one:

$$\begin{array}{ll}\hfill & f=\frac{1}{2}{\displaystyle \sum _{i=1}^n}\left[{(e_x-t_x)}^2+{(e_y-t_y)}^2\right]\hfill \end{array}$$

With:

$$\begin{array}{ll}\hfill & \left\{\begin{array}{cc}{e}_x& =\left(k_x^2{k}_x^1{e}_x^c-x_1\right)x_3,\\ e_y& =\left(k_y^2{k}_y^1{e}_y^c-x_2\right)x_4\end{array}\right.\hfill \end{array}$$

Where $e_x$ and $e_y$ are respectively the optimized $x$- and $y$-coordinates of the eye in millimetres in the C3D reference frame, $t_x$ and $t_y$ are the $x$- and $y$-coordinates of the target to follow in millimetres in the C3D reference frame, $e_x^c$ and $e_y^c$ are the $x$- and $y$-coordinates of the eye in centimetres expressed in the screen reference frame, and $n$ is the number of frames of the C3D file.

The problem was solved using the fmincon function of MATLAB R2021b with the sqp algorithm and ${\mathbf{x}}_0=[0011]$ as the initial guess. Figure 5b shows the adjustments of the eye trajectory after the optimisation process. Finally, portions of eye signals containing blinks were removed using the rate of change of pupil diameter, with a threshold set to $5\text{cm/s}$.

Click to enlarge

Figure 5

Comparison Between the Eye and Target Trajectory, a) Before, b) After the Optimisation Process

These four parameters (x and y offsets of the trajectory and the x and y scale factors) obtained with this ICT stimulus should be applied to each trial for a subject throughout the same session. In this way, the kinematics of the eye can be visualised along with the movement seen by the participant (Figure 6).

Although we had to use a specific device described in the “Eye tracking and configuration” section, we are providing the scientific community with a tutorial in the following section explaining how to add the eye trajectory of a subject watching a PLD video directly from the native recordings of the EyeTrackers (edf format).

Please note that the data set and code specific to our initial experiment (with DOCoMETRe) or related to the tutorial can be found by following the OSF link in this article.

Tutorial

This section details the step-by-step process for adding the gaze movement into an observed c3d file, starting from the records of edf files by EyeLink (all necessary files are available for download at the OSF link).

Click to enlarge

Figure 6

Planar Projection of the Kinogram of the Jumping Movement, With the Participant’s Eye (Red Tail) Reintroduced Into the C3D File

Note. This file is the original point-light display file that was used to generate the video stimulus, corresponding to the projection of each point light in the camera plane, depending on the camera’s azimuth and elevation parameters (i.e. the coordinates along the orthogonal axis in the camera plane are always equal to zero). The trajectories of the left and right ankles are displayed to help with recognising the movement.

The prerequisites are:

To facilitate this step, below we provide the procedure and the functions needed to create the stimulus videos and the ICT stimulus.

First, gather all the c3d files that will be used during the experiment into a single folder (called for example “Original C3D”) and create two other folders called for example: “Projected C3D” and “Experimental C3D and videos.”

For each one (c3d files), identify the 2D viewing angle (camera azimuth and elevation) at which you want to generate the stimuli, then use the “Project_C3D” function for each file, which will create a projected C3D in the camera view. Then place these files into the folder “Projected C3D.”

Finally, call the function “Generate_ICT_and_Stimulus_Videos_and_c3d.” You will need to provide the projected C3D folder, the backup folder for newly created files (for example, “Experimental C3D and videos”), the BTK library folder and the following information (see Figure 7 below):

Click to enlarge

Figure 7

Input Screen for Generating ICT, Stimulus Videos and C3D

Margins represent the minimum distance (in mm) between the edges of the image and the PLD dots. By adjusting the “Point’s velocity of the ICT file,” you can manage how fast (or slow) the target point moves during the inter-referential consistency test.

Optionally, you can request the creation of the ICT stimulus and experimental c3d and MPEG4 files with the two additional points added in order to verify the spatial concordance between the experimental stimuli and the ICT stimulus (same bounding box). The function outputs the bounding box limits in the Matlab command prompt and creates the video associated with each c3d file, as well as a c3d file and the ICT stimulus video, if requested (see Figure 8 below).

Click to enlarge

Figure 8

Bounding Box Limits in the Matlab Command Prompt

At this stage, you have a Pos.txt file containing the coordinates (in pixels) of the two additional point-lights of the ICT stimulus, two edf files (one for the ICT stimulus called afterwards ICT.edf and the other for the observed stimulus called Stim.edf) and two c3d files corresponding to ICT stimulus and to the observed stimulus. For the rest of this tutorial, we assume that all these files are located in the folder called File_Path afterwards.

With Matlab, prepare the Input variable (type struct) required to call the function AddEye_From_EDF.m as follow:

Once the Inputs variable is created, just call AddEye_From_EDF(Inputs) in the command Window of Matlab (see Figure 9 below).

Click to enlarge

Figure 9

Prepare the Input Variable (Type struct) Required to Call the Function AddEye_From_EDF.m

Note that if you do not specify the Inputs variable, the function will ask for files with uigetfile dialog controls, thus just call AddEye_From_EDF() in the command window of Matlab.

The following input dialog box will be displayed and you will have to fill in all fields with your appropriate settings (here are the default settings corresponding to the dataset provided in this tutorial; see Figure 10).

Click to enlarge

Figure 10

Input Box Display to Call AddEye_From_EDF() in the Command Window of Matlab

After clicking on the OK button, two new c3d files will be created with the suffix “_With eye:” one for the ICT file and one for the observed PLD. A marker represented the movement of the eye has been added into these files (see Figure 11).

Click to enlarge

Figure 11

(Left) ICT C3D File With Eye Added in Yellow and Target Marker in Purple, (Right) Observed Stimulus With Eye Added in Yellow

Discussion

For the past five decades, researchers have been interested in and have deepened their knowledge of the perception of biological movement. By investigating the reason for this great sensitivity to the biological movement of human (Bidet-Ildei et al., 2014; Craighero et al., 2016) or non-human animals (Blake, 1993; Lorenzi et al., 2024; Regolin et al., 2000), the scientific community has continued to propose new questions. However, to date, the inability to locate where gaze is directed at during observation represents an obstacle for deeper understanding of biological movement perception, a problem exacerbated by the lack of tools to define areas of interest (AOI) that move. As mentioned in the introduction, the PLD is a sequence of images with multiple labelled AOIs. The second challenge is that the PLD consists of a cloud of points whose meaning emerges only once the participant interprets it based on their own perception. For example, the kinematics (the PLD) of walking in orthogonal projection (not in perspective) and in the transverse plane of the screen (i.e. walking in the direction of the observer) could just be easily perceived as walking forwards or backwards. These particularities make the selection of AOIs more problematic. Using walking kinematics as an example, walking can be interpreted as either a forward movement or a backward movement, thus constituting an ambiguous stimulus. Participants can adopt either interpretation depending on perceptual cues. When the stimuli is perceived as walking forward, attention is probably focused on the lower limbs. In contrast, when the stimuli is perceived walking backward, attention may shift toward more salient features such as trunk or head movements. This example illustrates the inherent difficulty in accurately predicting areas of visual interest (AOI) in perceptual tasks. Here, we propose an initial approach to address this challenge by accounting for the specific nature of PLD stimuli. Our method involves directly reconstructing eye kinematics — represented by a new point — within the PLD’s own kinematics using the stimulus points. It is important to note that this approach was developed based on data from an EyeLink eye tracker and a final output file of digital data. However, the process requires only a file containing the 2D [x,y] coordinates (in centimeters) of the eye on the screen. Additionally, while the present study records the kinematics of a single eye, the same method could be applied to binocular recordings by integrating the coordinate table of the second eye in the same manner. Before reconstructing eye movements within the stimulus, we propose, as a precaution, a specific ICT procedure consisting of a smooth tracking of a single dot (close to a smooth-tracking calibration task, one approach provides a more natural and efficient calibration, while maintaining good accuracy (Hassoumi et al., 2019; Li et al., 2025). The primary purpose of this ICT is to counteract potential participant-specific drift in tracking, which differs from the standard calibration using a fixation grid (Hooge et al., 2024). Indeed, calibration by eye smooth pursuit felt less intrusive and more pleasant for participants, in the sense that it allows calibration without explicit cooperation. Moreover it is functionally relevant because it is based on the correlation between eye movement and target movement (Pfeuffer et al., 2013). However, the real motivation for ICT was to ensure the correct integration of the homographic coordinate in the stimulus coordinate file itself. While optimizing the calibration is not mandatory, even a small offset can become problematic during the conversion and integration of PLD coordinates and frame rate. Therefore, an optimization step could be implemented to correct any offset, ensuring that this correction is applied to all subsequent conversions and eye movement reconstructions for the participant.

The present proposal for eye reconstruction in the PLD stimuli, like we introduce above, takes advantage of the particularity of the stimuli. Indeed, a PLD is initially a c3d file. This kind of file contains the 3D coordinates of each point in addition to their individual labels. It should be noted that each point is identifiable and locatable over time, thus each point can be considered as its own AOI. The added new points, corresponding to X and Y coordinates of gaze, can then be directly compared to the other points at each moment. Moreover, the methodology exposed in this paper which allows to insert the eye kinematic data into the c3d file, will be affiliated to a new plug-in of PLAViMoP software (Decatoire et al., 2018; PLAViMoP, 2025). Importantly, on the PLAViMoP platform (Bidet-Ildei, Francisco et al., 2022), we provide the algorithm that allows the eye motion reconstruction within the PLD stimulus. In the future, we plan to develop a more user-friendly interface enabling to characterize ocular behaviour such a fixation or saccade, and draw confidence ellipses around a given point using sliders to select thresholds. Indeed, in its current version, you need first to manually identify fixation/saccade in order to visualise it as a new point in the stimulus. In the PLAViMoP tool, it would then be sufficient to colour the two points to visualise the eye, for example blue for fixation and red for saccade, as a single point changing colour. In addition to the visualisation aspect, it would also be very easy to measure the distance of the “fixation” point and the “saccade” point from the other markers.

Finally, our approach enables further investigation into what captures the eye’s attention during action observation, depending on the nature of the task. This issue is important to subsequently explore additional aspects such as the salience features of BM and moments of recognition. One promising approach is to compile a dataset of eye movements across observers in order to identify distinguishing features of gaze in biological motion, in a manner analogous to approaches in face perception. For instance, Caldara et al. (2010) showed that people from different cultural backgrounds may have different fixation strategies (e.g., looking at the eyes or the nose) when faced with limited visual conditions. This suggests that eye movement patterns can reflect different perceptual strategies when processing ambiguous stimuli. Moreover, in the context of biological movement, the question of action observation in motor learning (Rizzolatti et al., 2021; Vogt & Thomaschke, 2007), particularly in sports, arises (D’Innocenzo et al., 2016; Horn et al., 2002) and understanding where the gaze is directed during observation could provide new and interesting insights. Notably in a recent study (Francisco et al., 2022), we demonstrated that action observation combined with physical training could guide the observer’s attentional focus, suggesting that certain regions or points of the PLD might be more informative than others. However, a key missing element is the ability to determine what actually captured the observer’s gaze to support this conclusion. It is possible that the perceived importance of certain parameters is influenced by different kinematic cues, making gaze localization a valuable area of investigation. In the same way, identifying the observer’s gaze location could help optimize action observation training for the rehabilitation of central and peripheral motor disorders (Ryan et al., 2021; Sarasso et al., 2015).

In summary, the proposed gaze localization methodology allows for the study of gaze behavior during perception, offering insights into the mechanisms involved in processing biological motion.

Conclusion

To conclude, although the analysis of eye movements is believed to enhance the relevance of PLDs for both theoretical and applied research, there is, to date, a lack of methods that align eye movements with PLD. The technique we propose here, which combines 2D eye-tracking data with the PLD’s c3d file within the same reference frame, provides a solution to this gap. This approach opens new avenues for understanding biological movement perception and offers potential applications in fields such as health, sports, and occupational studies.