Facial expressions in mice reveal latent cognitive variables and their neural correlates

a, Decision variables (for example, consecutive failures) evolve after each lick outcome. Therefore, to predict these variables from movement PCs, we aligned the PCs to lick events using a 200 ms window. b, Decoding accuracy of various decision variables from movement PCs was estimated using a 200 ms sliding window (75% overlap) at different time lags relative to the lick event (example session shown). c, Peak decoding accuracy (grey dots) for each decision variable occurred between 100 ms and 350 ms after the lick. Error bars represent the median and m.a.d. across sessions (N = 10 sessions, 8 sessions from distinct animals, two from the same animal). d, This schematic describes the method used to partial out the linear relationship between latent variables. This approach allowed us to decompose DV1 (for example, consecutive failures) into the sum of two time series: one proportional to DV0 (for example, action outcomes) and another orthogonal (uncorrelated) to DV0, which we denote as unique DV1 (for example, unique consecutive failures). The unique consecutive failures residual (pink) is orthogonal to the action outcomes (black). Subsequently, the same procedure generated the unique negative value residual, orthogonal to both action outcomes and unique consecutive failures. The three resulting orthogonal time series (action outcomes, unique consecutive failures, and unique negative value) were then fit using movement PCs.

a, Changes in lick rate across 4 example bouts. b, Correlation (Pearson’s coefficient) between lick rate and the two decision variables (median across 10 sessions with 25th and 75th percentiles; whiskers represent minimum and maximum values). A small negative correlation exists between lick rate and the decision variables, suggesting that mice tend to lick slightly faster during reward consumption and slow down towards the end of a lick bout. c, Decoding accuracy (cross-validated R²) for action outcome, unique aspects of the two decision variables, and arbitrary signals in each region of interest (median across 10 sessions with 25th and 75th percentiles; whiskers represent minimum and maximum values). The movement PCs are used as predictors in multivariate regression models to predict the action outcome and the decision variables with lick rate variance removed (partialed out). The variance in action outcome, consecutive failures and negative value that is not explained by lick rate remains highly decodable. This suggests that the relationship between facial expressions and latent variables is not solely explained by licking behavior. d, Decoding of bout number from facial movement PCs in an example session. The actual bout number is shown as a thick light orange line, and the decoded projection is shown as a thin dark orange line. e, Decoding accuracy (cross-validated R²) across all sessions (N = 10, orange dots). Boxplots show the median (center line), 25th and 75th percentiles (box edges), and minimum and maximum values (whiskers).

a, Mice (N = 5 for a total of 20 behavioral sessions) were presented with single tones (150 ms) of varying frequencies (low: 9.9, 12, and 13 kHz; high: 15, 16.3, and 20 kHz) and, after a delay period (500 ms after stimulus offset), reported their perceived frequency (high or low) by licking one of two water spouts (left or right) to receive a water reward if the response was correct. Simultaneously videos were recorded at 60 fps. b, Task schematics and time intervals analyzed. Video analysis focused on three periods, color-coded in the schematic: “pre-stimulus” (before tone onset; 1 s), “pre-response” (including stimulus presentation and part of the delay period; 500 ms), and “response” (around the time of the lick response; 200 ms). c, Representative frame from the co-registered video (generated by combining the 20 sessions), along with the first four eigenfaces. Error bars represent the median and m.a.d. across mice (N = 5). d, Average motion energy across task periods. Motion energy (relative to the video average) is shown for each analysis period. Low motion is observed during the pre-stimulus period (top), increasing during stimulus presentation (middle), and peaking during the response (bottom), reflecting movement associated with licking. e, Decoding accuracy of task variables from facial movement PCs. Decoding accuracy from facial movement PCs for previous trial outcome (rewarded/unrewarded) and trial number. Each dot represents the average decoding accuracy across sessions for a single mouse; error bars indicate the median and m.a.d. across mice. The task variables, especially the slow latent variable ‘Trial number’, can be decoded with relatively high accuracy from facial movement PCs. f, Weighted masks (that is, facial representation of the decoded task variables in panel e) for the three different time intervals. The expression of the task variables on the face is highly consistent across the different time intervals, suggesting that this representation is independent of the animal’s overall movement.

a, Eight facial landmarks (red) were manually identified on average frames from each video. An affine transformation determined by using these landmarks and MATLAB’s fitgeotrans function, co-registered frames from each video to a reference frame (orange, bottom). This transformation facilitated comparisons and averaging of weighted masks (Fig. 2), video concatenation (Fig. 3), and definition of an average facial silhouette (Fig. 2). b, Improvement in pairwise 2D cross-correlation between average video frames before and after co-registration (N = 10 videos; error bars represent the median and m.a.d. across mice). c, Example traces of the first five principal components (PCs) derived from motion energy analysis of three co-registered videos. Singular value decomposition (SVD) was applied to the merged video data. d, t-SNE visualization of the first 100 PCs of the merged video. Points are color-coded by video identity, revealing substantial overlap between videos.

a, Weighted masks for a single example session (top) of each wild-type mouse (N = 3) and averaged across all sessions (N = 24, bottom) from all mice during the laser OFF condition. b, Same as in panel (a) but for a single example session (top) of each VGAT (N = 5) and averaged across all sessions (N = 24, bottom) from all mice during the laser OFF condition. c, Inter-animal similarity of facial expression of decision variable (Action outcome: OUT; Unique consecutive failure: CF; Unique negative value: VAL; Arbitrary signal: ARB). Colors represent the normalized 2D cross-correlation at zero lag between the mean weighted masks of two mice (for each mouse the mean weighted mask was the average across sessions, N = 8 mice, 6 ± 2.7 sessions per mouse). d, Mean weighted mask similarity for each mouse and decision variable. Each gray dot represents the average pairwise correlation of the mean weighted mask of a mouse for a given decision variable with the mean weighted masks of all the other mice for the same decision variable. Each gray diamond represents the average pairwise correlation of the mean weighted mask of a mouse for a given decision variable with the mean weighted masks of the same mouse for all the other decision variables. Color errorbars represent median and m.a.d across mice (N = 10, 8 sessions from distinct animals, two from the same animal). e, Distribution of average pairwise 2D correlations at zero lag for electrophysiology (data in Figs. 2–4) and downsampled optogenetics (data in Fig. 5) datasets. Pairwise correlations of facial expression of decision variables were calculated for all sessions in the electrophysiology dataset (black dots; median and median absolute deviation (m.a.d.) indicated). For comparison, 10 sessions were randomly sampled from the optogenetics dataset multiple times, and average correlations were calculated for each sample (gray dots). The sampling procedure was repeated twenty times, each represented as a row of gray dots. Note that correlations are overall smaller than the ones estimated using averages across sessions (rather than single ones) as in (d). f, Within- and across-mice correlation of decision variable facial expression. Each point represents the average correlation of a single behavioral session with all other sessions, either from the same mouse (within) or different mice (across). Errorbars represent mean and s.d. across sessions per mouse (N = 8 mice, 6 ± 3 sessions per mouse).

a, We recorded with Neuropixels probes in multiple regions of the frontal cortex. Schematic target location of the neuropixels probe insertion. Vertical insertions were performed within a 1 mm diameter craniotomy centered around +2.5 mm anterior and +1.5 mm lateral from Bregma. b, An example of histology with the electrode track. We painted the probe with a red, fluorescent dye to recover the probe’s location post-hoc. c, To decode the instantaneous value of multiple decision variables (pink & blue traces, right), we used regression models taking as predictors the activity of simultaneously recorded neurons in each brain region (black traces, left, example activity from M2). The model predictions (the weighted sums of neural activity, black trace right) overlap with the decision variables. d, Neural vs. facial movement PC decoding latencies for different decision variables (N = 10 sessions). Each session contributes three points (gray, pink, blue), one per decision variable. Points above the identity line indicate later facial movement representation relative to neural representation. Black cross: median ± m.a.d. across all points. e, Predicting facial movements from neural activity in M2, OFC, and OC. GLMs were trained to predict facial movement PCs using a 50 ms non-overlapping sliding window of lagged neural activity from M2, OFC, and OC. Facial movement PCs were derived from concatenated videos to enable cross-session comparisons. f, Relationship between decoding accuracy and the time of peak decoding accuracy for facial movements. Peak times (median of cross-validated R² across sessions, N = 10) are shown for the 25 PCs of facial movement with the largest variance, as a function of decoding accuracy from neurons in M2, OFC, and OC. Each dot represents one facial movement PC. Negative values, particularly in M2 (M2 = −0.05 ± 0.06, p = 0.002; OFC = −0.05 ± 0.10, p = 0.194; OC = 0.05 ± 0.09, p = 0.0194; median ± m.a.d, N = 10 sessions, Wilcoxon signed rank test, Holm-Bonferroni corrected), indicate that neural activity preceding the facial movement is most predictive.

a, Laser-induced changes in facial movement patterns. 2D masks show the difference in average facial motion (calculated from movement PCs) between laser ON and OFF conditions, across mice and at different lick numbers (decision time points). b, Variability of laser-induced facial motion changes. The variance of the difference in motion energy between laser ON and OFF conditions (normalized by the variance in laser OFF) is shown as a function of lick number for inactivated (green, N = 5 mice) and control (black, N = 3 mice) groups. Thick lines represent group means. c, Decoding accuracy after removing the effect of lick rate (top & middle): Comparison of laser ON vs. laser OFF conditions. Dots below the unity line indicate that representations of decision variables derived from facial movement PCs were decoded less accurately during laser ON than laser OFF. Difference in decoding accuracy (bottom): Laser ON minus laser OFF (mean across sessions for each mouse in the inactivated and control groups). Individual mice are indicated by color. d, Same as in (c) but for decoding latency. Dots above the unity line indicate that representations of decision variables derived from facial movement PCs were decoded later during laser ON than laser OFF. Partial silencing of M2 reduced the accuracy and increased the latency with which facial movement PCs predicted decision variables, even after controlling for lick rate. This suggests that the latent variable represented in facial movements is not simply a consequence of the relationship between M2 activity and lick rate.