Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translation

Abstract

Sign Language Translation (SLT) is a challenging task that requires bridging the modality gap between visual and linguistic information while capturing subtle variations in hand shapes and movements. To address these challenges, we introduce BeyondGloss, a novel gloss-free SLT framework that leverages the spatio-temporal reasoning capabilities of Video Large Language Models (VideoLLMs). Since existing VideoLLMs struggle to model long videos in detail, we propose a novel approach to generate fine-grained, temporally-aware textual descriptions of hand motion. A contrastive alignment module aligns these descriptions with video features during pre-training, encouraging the model to focus on hand-centric temporal dynamics and distinguish signs more effectively. To further enrich hand-specific representations, we distill fine-grained features from HaMeR. Additionally, we apply a contrastive loss between sign video representations and target language embeddings to reduce the modality gap in pre-training. BeyondGloss achieves state-of-the-art performance on the Phoenix14T and CSL-Daily benchmarks, demonstrating the effectiveness of the proposed framework. We will release the code upon acceptance of the paper.

1 Introduction

Sign languages use hand and body movement alongside facial expressions, mouth gestures and the complex use of space for communication [Rastgoo et al.(2022)Rastgoo, Kiani, Escalera, Athitsos, and Sabokrou, Stokoe(2005)]. Sign language translation (SLT) aims to convert these visual elements into spoken sentences, enabling communication with non-signers. However, due to the inherent complexities of natural signing and the fundamental difference to spoken languages, SLT faces challenges in accurately recognizing the components of sign and bridging the gap between the visual and textual representations [Duarte(2019), Zhao et al.(2022)Zhao, Qi, Zhou, Duan, Zhou, and Li]. SLT approaches typically fall into two categories: gloss-based and gloss-free. Gloss-based methods [Chen et al.(2022a)Chen, Wei, Sun, Wu, and Lin, Zhang et al.(2023a)Zhang, Müller, and Sennrich, Chen et al.(2022b)Chen, Zuo, Wei, Wu, Liu, and Mak] rely on intermediate representations, where sign videos are first translated into a sequence of glosses, written representations of signs using corresponding words in a spoken language, before generating spoken sentences. In contrast, gloss-free methods directly map videos to text without an explicit gloss representation [Camgoz et al.(2020)Camgoz, Koller, Hadfield, and Bowden, Voskou et al.(2021)Voskou, Panousis, Kosmopoulos, Metaxas, and Chatzis, Chen et al.(2024b)Chen, Zhou, Li, Wan, Lei, Jiang, Lu, and Zhao]. While gloss-based approaches benefit from structured linguistic information, gloss-free methods bypass the need for gloss annotations, making them more flexible but often more challenging. The high cost and effort required for data collection and meticulous gloss annotation have led to increased interest in gloss-free SLT [Wong et al.()Wong, Camgöz, and Bowden, Gong et al.(2024)Gong, Foo, He, Rahmani, and Liu, Liang et al.(2024)Liang, Huang, Xu, Tang, Ye, Zhang, Chen, Yu, and Xu].

Figure 1: Generated hand motion description from our Sign Video Descriptor.

The increasing use of Large Language Models (LLMs) has had a significant impact on this area [Achiam et al.(2024)Achiam, Adler, Agarwal, Ahmad, Akkaya, Aleman, Almeida, Altenschmidt, Altman, Anadkat, and others.]. As a result, the latest gloss-free models utilise the remarkable language prediction capabilities of LLMs by converting sign language videos into text-like representations that LLMs can process effectively [Chen et al.(2024b)Chen, Zhou, Li, Wan, Lei, Jiang, Lu, and Zhao, Wong et al.()Wong, Camgöz, and Bowden, Zhou et al.(2023)Zhou, Chen, Clapés, Wan, Liang, Escalera, Lei, and Zhang]. When LLMs are applied directly to visual features, they become mere transformers with pre-trained weights due to the large shift between the textual representations on which they were trained and the new visual inputs. To bridge this gap, recent studies have introduced a pre-training phase before the SLT task, which aims to align the visual and textual representations [Wong et al.()Wong, Camgöz, and Bowden, Zhou et al.(2023)Zhou, Chen, Clapés, Wan, Liang, Escalera, Lei, and Zhang]. Given the central role of hand shape and motion in conveying meaning in sign language, our framework explores the benefit of emphasizing hand-centric features. We achieve this by applying feature distillation from Hand Mesh Recovery (HaMeR) [Pavlakos et al.(2024)Pavlakos, Shan, Radosavovic, Kanazawa, Fouhey, and Malik], which guides the model to focus more on hand poses and orientations in individual video frames. We then align the video features with a textual description of the entire video, generated using VideoLLMs, which we employ in the context of SLT. These descriptions provide high-level semantic guidance that is more directly aligned with the visual input than conventional spoken language targets. Finally, to enhance performance in the downstream SLT task, we adopt an encoder-decoder architecture based on language models. To further improve the alignment between the encoder’s output embeddings and the spoken language targets, we incorporate a contrastive learning strategy inspired by the CLIP framework [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al.], projecting both video and text features into a shared embedding space.

Our primary contributions can be summarized as: (i) Our proposed framework, BeyondGloss, achieves state-of-the-art results in gloss-free sign language translation on two widely used benchmark datasets. (ii) We introduce a novel method for generating hand-centric textual descriptions of sign language videos using VideoLLMs, offering a new form of semantic supervision aligned with visual content. (iii) We are the first to leverage HaMeR’s transformer-based hand representations in the SLT task. Through feature distillation, we guide the visual encoder to focus on hand pose and orientation, demonstrating the positive impact of this strategy on translation quality.

The rest of this paper is organised as follows: In Section 2, we review the literature in sign language translation. In Section 3, we outline the proposed BeyondGloss approach. Section 4 presents quantitative and qualitative evaluation, whilst Section 5 concludes.

2 Related Works

Gloss-free Sign Language Translation. Gloss-based SLT methods achieve strong performance by using annotated glosses as intermediate supervision [Zhang et al.(2023a)Zhang, Müller, and Sennrich, Chen et al.(2022b)Chen, Zuo, Wei, Wu, Liu, and Mak]. However, glosses are labor-intensive to create and not available for all languages. Gloss-free SLT avoids this by directly translating sign videos into spoken sentences using only video-text pairs. While more scalable, it faces a larger modality gap and typically performs worse than gloss-based methods. To address this gap, CSGCR [Zhao et al.(2022)Zhao, Qi, Zhou, Duan, Zhou, and Li] predicts possible language words, generates sentences based on these predictions, and selects the most appropriate sentence using cosine similarity with sign language images. Recently, GFSLT-VLP [Zhou et al.(2023)Zhou, Chen, Clapés, Wan, Liang, Escalera, Lei, and Zhang] introduced a pre-training method that aligns sign language images with spoken sentences, converting them into text-like features familiar to LLMs. Leveraging this advancement, subsequent gloss-free models focus on generating more comprehensible features for LLMs. FLa-LLM [Chen et al.(2024b)Chen, Zhou, Li, Wan, Lei, Jiang, Lu, and Zhao] introduces a two-step approach: initially training on visual information from sign language images using a lightweight translation model and fine-tuning the LLM for SLT. Sign2GPT [Wong et al.()Wong, Camgöz, and Bowden] pre-trains a sign encoder by aligning visual features with prototypes under the supervision of pseudo-glosses, words from spoken sentences via Parts-Of-Speech (POS) tagging, and then utilizes it for SLT. SignLLM [Gong et al.(2024)Gong, Foo, He, Rahmani, and Liu] designs a vector-quantization technique to convert sign videos into discrete sign tokens. Lastly, LLaVA-SLT [Liang et al.(2024)Liang, Huang, Xu, Tang, Ye, Zhang, Chen, Yu, and Xu] integrates linguistic pretraining, visual contrastive learning, and visual-language tuning within a Large Multimodal Model to effectively bridge the gap between sign language videos and textual translation.
Video Large Language Models. Early VideoLLMs like Video-LLaMA [Zhang et al.(2023b)Zhang, Li, and Bing] and Video-ChatGPT [Maaz et al.(2024)Maaz, Rasheed, Khan, and Khan] extended large language models to the video domain by incorporating temporal and multimodal context. Initially limited to short clips and basic temporal understanding, recent models have advanced by combining large-scale video-language data with autoregressive decoding [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin, Graves(2013)]. These systems integrate spatiotemporal encoders with LLMs (e.g. LLaMA, GPT) to handle longer sequences and support open-ended video understanding, including tasks like action recognition and captioning. [Touvron et al.(2023)Touvron, Lavril, Izacard, Martinet, Lachaux, Lacroix, Rozière, Goyal, Hambro, Azhar, et al., Brown et al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, McCandlish, Radford, Sutskever, and Amodei]. Although sign language falls under video understanding, it poses unique challenges due to its reliance on fine-grained, frame-by-frame details, especially in hand shapes, movements, and facial expressions. Current VideoLLMs [Zhang et al.(2023b)Zhang, Li, and Bing, Maaz et al.(2024)Maaz, Rasheed, Khan, and Khan, Li et al.(2023)Li, He, Wang, Li, Wang, Luo, Wang, Wang, and Qiao, Lin et al.(2024)Lin, Ye, Zhu, Cui, Ning, Jin, and Yuan] often lose this crucial information due to frame sampling or summarization, and processing all frames at full resolution is computationally costly [Weng et al.(2024)Weng, Han, He, Chang, and Zhuang]. To address this, we propose an approach that leverages VideoLLMs for generating detailed hand-centric descriptions suited for sign language understanding.
Vision-Language Pre-training. Existing visual-language pre-training (VLP) models fall into two types: single-stream models, which fuse visual and textual inputs in one encoder using self-attention [Kim et al.(2021)Kim, Son, and Kim, Li et al.(2019)Li, Yatskar, Yin, Hsieh, and Chang, Kamath et al.(2021)Kamath, Singh, LeCun, Synnaeve, Misra, and Carion], and dual-stream models, which use separate encoders for each modality and align them via contrastive learning or cross-attention [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al., Jia et al.(2021)Jia, Yang, Xia, Chen, Parekh, Pham, Le, Sung, Li, and Duerig, Alayrac et al.(2022)Alayrac, Donahue, Luc, Miech, Barr, Hasson, Lenc, Mensch, Millican, Reynolds, et al.]. These pre-trained models have demonstrated remarkable performance in downstream tasks [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al., Jia et al.(2021)Jia, Yang, Xia, Chen, Parekh, Pham, Le, Sung, Li, and Duerig]. GFSLT-VLP [Zhou et al.(2023)Zhou, Chen, Clapés, Wan, Liang, Escalera, Lei, and Zhang] is the first study to utilize CLIP [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al.], a pioneering approach in VLP, to tackle the modality gap in gloss-free SLT. Like CLIP, GFSLT-VLP adopts a dualstream structure, pre-training the alignment between sign videos and spoken sentences before performing the translation.
HaMeR. The typical input to SLT is either RGB video [Camgoz et al.(2018)Camgoz, Hadfield, Koller, Ney, and Bowden, Liang et al.(2024)Liang, Huang, Xu, Tang, Ye, Zhang, Chen, Yu, and Xu] or skeletal based approach [Gan et al.(2021)Gan, Yin, Jiang, Xie, and Lu, Jiang et al.(2021)Jiang, Sun, Wang, Bai, Li, and Fu]. But recent approaches such as HaMer [Pavlakos et al.(2024)Pavlakos, Shan, Radosavovic, Kanazawa, Fouhey, and Malik], provide potential alternatives that SLT can be built on. HaMeR (Hand Mesh Recovery) is a fully transformer-based framework for 3D hand reconstruction from monocular images, offering enhanced robustness and accuracy by scaling both model capacity and training data [Pavlakos et al.(2024)Pavlakos, Shan, Radosavovic, Kanazawa, Fouhey, and Malik]. It utilises a large Vision Transformer and integrates multiple datasets with 2D and 3D hand annotations, achieving state-of-the-art results on standard 3D hand pose benchmarks. Given the central role of hand shapes and movements in sign language communication, HaMeR has recently been adopted for sign language understanding tasks. Notably, [Low et al.(2025)Low, Walsh, Sincan, and Bowden] was the first to apply HaMeR for sign language segmentation. Inspired by this, our approach similarly leverages HaMeR to extract detailed hand features from video frames, aiding in more accurate sign interpretation.

3 Methodology

This section presents our proposed BeyondGloss framework. We first describe how high-quality hand features are extracted via distillation in Sec. 3.1, followed by a detailed explanation of our novel Sign Video Descriptor for generating hand motion descriptions in Sec. 3.2. Finally, Sec. 3.3 outlines the overall architecture for SLT.

3.1 Distillation

Distillation [Heo et al.(2019)Heo, Lee, Yun, and Choi, Zagoruyko and Komodakis(2017), Romero et al.(2014)Romero, Ballas, Kahou, Chassang, Gatta, and Bengio] is a learning technique where a smaller model (student) is trained to replicate the behavior of a larger, well-trained model (teacher). In feature distillation [Romero et al.(2014)Romero, Ballas, Kahou, Chassang, Gatta, and Bengio, Hinton et al.(2015)Hinton, Vinyals, and Dean], instead of only imitating final predictions, the student learns to match the intermediate feature representations of the teacher. This helps the student model learn richer internal patterns, improving performance and generalization with fewer parameters. In our framework, we apply feature distillation to emphasize hand modeling in sign language videos. Specifically, we align the visual encoder’s features with those from the HaMeR transformer head to enhance sensitivity to hand shapes and orientations. To ensure that broader spatial information is not lost, we combine the hand-focused features with the original ones, maintaining a balance between detailed hand information and overall scene context.

3.2 Sign Language Video Description

To encourage our framework to focus on hands and their movements, key components of sign language, we aim to align sign video representations with hand-centric descriptions extracted from the videos. However, due to the complexity of hand motions and the inability of off-the-shelf VideoLLMs to capture fine-grained temporal details (as well as memory constraints) [Weng et al.(2024)Weng, Han, He, Chang, and Zhuang], we cannot directly use VideoLLMs by simply feeding them full sign videos. To address these challenges, we propose a novel sign video descriptor that leverages the strengths of both VideoLLMs and LLMs, as illustrated in Fig. 2. An example of a generated description is provided in Fig 1. As shown in Fig. 1, we begin by segmenting each video into non-overlapping clips of 16 frames. This length is chosen as a balance between temporal resolution and semantic content: segments shorter than 16 frames may lack sufficient motion or variation to be meaningfully described, while longer segments often lead VideoLLMs to produce less accurate or overly generalized descriptions. For each segment, we prompt the VideoLLM to describe hand movements, shapes, and trajectories. After obtaining a separate description for each segment, we merge them into a single sequence and pass them to a LLM that refines the content by removing redundant or irrelevant information. This final description preserves the temporal order and captures the most important aspects of hand motion across the entire video. See Appendix for more details and examples.

3.3 Architecture

Figure 2: Overview of our approach. a) Our main framework is divided into two main phases: pre-training(dashed line) and fine-tuning(solid line). b) The overview of proposed sign video descriptor in our framework.

Visual Encoder. Our video-based sign language translation framework is built around an image encoder that extracts spatial features $F$ from input video frames $V=\{v_{0},v_{1},\dots,v_{T^{*}}\}$, where $T^{*}$ represents the total number of frames. These features, with a dimensionality of $D^{*}$, are structured as $F_{video}\in\mathbb{R}^{T^{*}\times D^{*}}$. We use and fine-tune Dino V2 [Oquab et al.(2024)Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, Fernandez, Haziza, Massa, El-Nouby, Assran, Ballas, Galuba, Howes, Huang, Li, Misra, Rabbat, Sharma, Synnaeve, Xu, Jegou, Mairal, Labatut, Joulin, and Bojanowski] (ViT-S/14 variant), a self-supervised Vision Transformer, as our image encoder. The class token’s output serves as a feature vector, which is transformed linearly and batch-normalized to obtain $F_{video}$.
Feature Distillation. In parallel to the visual encoder, sign videos are input to the HaMeR model to extract hand shape features. Specifically, we utilize the transformer head section of HaMeR, which was originally used to regress hand and camera parameters. In our work, we repurposed the transformer head to extract hand pose and global orientation parameters, as they were highly effective hand shape descriptors for our task. For each frame, the hand pose features are represented as a tensor of $H\in\mathbb{R}^{2\times 15\times 3\times 3}$, while the global orientation is parameterized as $G\in\mathbb{R}^{2\times 3\times 3}$. For simplicity and computational efficiency, we flatten these parameters, resulting in a 1D feature vector of $H^{\prime}\in\mathbb{R}^{270}$ for the hand pose and, $G^{\prime}\in\mathbb{R}^{18}$ for the global orientation. The flattened features are then fused to form $F_{HaMeR}\in\mathbb{R}^{T^{*}\times 288}$ for each input sign video, where $T^{*}$ represents the total number of frames. To guide visual encoder to extract hand feature, we need to map $F_{video}$ to $F_{HaMeR}$ and then combine these mapped features to the original features in order not to lose other important features captured by visual encoder:

$$F^{*}_{\text{video}}=\texttt{mapper}_{1}(F_{\text{video}} )\in\mathbb{R}^{T^{*}\times 288},\quad F^{\prime}_{\text{video}}=\texttt{mapper}_{2}(F^{*}_{\text{video}})\in\mathbb{R}^{T^{*}\times D}.$$ (1)

$$\hat{F}_{\text{video}} =\texttt{concat}(F^{\prime}_{\text{video}},F_{\text{video}})\in\mathbb{R}^{T^{*}\times 2D}.$$ (2)

For hand features distillation to the visual encoder, a simple $L_{2}$ loss can suffice. The distillation loss between $F^{*}_{\text{video}}$ and $F_{\text{HaMeR}}$ can be represented by:

$$\mathcal{L}_{\text{distill}}=\frac{1}{H\times W}\sum_{i}^{H}\sum_{j}^{W}\left\|F^{*}_{j,t}-F_{j,t}\right\|_{2}.$$ (3)

where $H$ and $W$ are the number of frames, $T^{*}$, and HaMeR feature length, 288, respectively. The feature $\hat{F}_{\text{video}}$ is then passed into the temporal encoder for further processing.
Temporal Encoder. We use features $\hat{F}_{video}$ as input to our temporal encoder to learn spatio-temporal sign representations. Since sign language sequences often contain hundreds of frames, we design a spatio-temporal transformer inspired by prior SLT approaches, incorporating two key modifications for efficiency and effectiveness. We integrate local self-attention with a window size of seven, a proven technique for SLT [Yin et al.(2023)Yin, Zhong, Tang, Jin, Jin, and Zhao]. Combined with downsampling, this extends the model’s temporal receptive field while reducing redundancy, ensuring compatibility with the LLM encoder. The resulting features are represented as $Z_{video}\in\mathbb{R}^{T\times D}$, where $T$ is the downsampled sequence length and $D$ is the new feature dimension. To effectively model both temporal and spatial information, we use Rotary Position Embedding (RoPE) [Su et al.(2024)Su, Ahmed, Lu, Pan, Bo, and Liu] for positional encoding.
Video-Description Alignment. To focus on hand-centric temporal dynamics and distinguish signs more effectively, after the temporal encoder, we introduce a contrastive alignment between video and description representations. Rather than relying on a computationally expensive VideoLLM at inference time or discarding temporal information, we project the encoded video features into the description space and then combine them with the original video features to preserve both global and temporal context. As illustrated in Fig. 2, each sign video is processed by a video description module, which generates a textual description. This description is then passed through a text encoder to obtain token-level features, $\hat{L}\in\mathbb{R}^{N\times K}$, where $N$ is the number of tokens and $K$ is the embedding dimension. Simultaneously, the video features $Z_{video}$ are mapped to the same feature space using a two-stage projection:

$$Z^{*}_{\text{video}}=\texttt{mapper}_{3}(Z_{\text{video}} )\in\mathbb{R}^{T\times D^{\prime}},\quad Z^{\prime}_{\text{video}}=\texttt{mapper}_{4}(Z^{*}_{\text{video}})\in\mathbb{R}^{T\times D}.$$ (4)

$$\hat{Z}_{\text{video}} =\texttt{concat}(Z^{\prime}_{\text{video}},Z_{\text{video}})\in\mathbb{R}^{T\times D^{\prime\prime}}.$$ (5)

The resulting feature $\hat{Z}_{\text{video}}$ are forwarded to subsequent network components. To align video and description features for contrastive learning, we apply a linear projection if $D^{\prime\prime}\neq k$ , mapping both to a common dimension $\hat{D}$. We then use average pooling across the temporal and token dimensions to obtain global representations $\tilde{Z}\in\mathbb{R}^{\hat{D}}$ for the video and $\tilde{L}\in\mathbb{R}^{\hat{D}}$ for the description. The contrastive loss is computed as:

$$\mathcal{L}_{\text{desc}}=-\frac{1}{2\times B}\Big{(}\sum^{B}_{j=1}log\frac{\exp(\texttt{sim}(\tilde{Z}_{j},\tilde{L}_{j})/\tau_{1})}{\sum^{B}_{k=1}\exp(\texttt{sim}(\hat{Z}_{j},\tilde{L}_{k})/\tau_{1})}+\sum^{B}_{j=1}log\frac{\exp(\texttt{sim}(\tilde{L}_{j},\tilde{Z}_{j})/\tau_{1})}{\sum^{B}_{k=1}\exp(\texttt{sim}(\tilde{L}_{j},\tilde{Z}_{k})/\tau_{1})}\Big{)}.$$ (6)

where $\texttt{sim}(x,y)$ is the cosine similarity, and $\tau_{1}$ is a learnable temperature parameter.
Video-Target Alignment. To improve the effectiveness of converting video features into output text, we employ an encoder-decoder-based LLM. The previously constructed feature representation $\hat{Z}_{video}$ is provided as input to the encoder, producing the transformed video features $Y\in\mathbb{R}^{T\times K}$, which are used for subsequent alignment. To address the modality gap between video and language, we adopt a contrastive learning strategy inspired by CLIP [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al.]. This approach aligns the video features with their corresponding spoken sentence representations, encouraging similarity between matched pairs while distinguishing them from unrelated ones. Given a target sentence $TS$ associated with a video, we encode it into textual features $M\in\mathbb{R}^{\overline{N}\times K}$ using a frozen text encoder, where $\overline{N}$ is the number of tokens and $K$ is the embedding dimension. Average pooling is applied to both $Y_{video}\in\mathbb{R}^{T\times K}$ and $M$ across temporal and token dimensions, respectively, resulting in global video and sentence features: $\tilde{{Y}}\in\mathbb{R}^{K}$ and $\tilde{M}\in\mathbb{R}^{K}$. We apply a symmetric contrastive loss as follows:

$$\mathcal{L}_{\text{align}}=-\frac{1}{2\times B}\Big{(}\sum^{B}_{j=1}log\frac{\exp(\texttt{sim}(\tilde{Y}_{j},\tilde{M}_{j})/\tau_{2})}{\sum^{B}_{k=1}\exp(\texttt{sim}(\hat{Y}_{j},\tilde{M}_{k})/\tau_{2})}+\sum^{B}_{j=1}log\frac{\exp(\texttt{sim}(\tilde{M}_{j},\tilde{Y}_{j})/\tau_{2})}{\sum^{B}_{k=1}\exp(\texttt{sim}(\tilde{M}_{j},\tilde{Y}_{k})/\tau_{2})}\Big{)}.$$ (7)

where $\texttt{sim}(x,y)$ is the cosine similarity, and $\tau_{2}$ is a learnable temperature parameter.
The final pre-training loss is:

$$\mathcal{L}_{\text{pre-train}}=\lambda_{\text{align}}\times\mathcal{L}_{\text{align}}+\lambda_{\text{desc}}\times\mathcal{L}_{\text{desc}}+\lambda_{\text{distill}}\times\mathcal{L}_{\text{distill}}$$ (8)

where $\lambda_{\text{align}}$, $\lambda_{\text{desc}}$ and $\lambda_{\text{distill}}$ control the balance between these three losses. See the Appendix for a comprehensive evaluation conducted to determine the optimal weighting in the pre-training loss.
Sign Language Translation As illustrated in Fig. 2, during the fine-tuning stage, the feature distillation, video-description alignment, and video-target alignment components are removed. The only module not initialized from the pre-training stage is the LLM decoder, which is attached to the pre-trained LLM encoder.

Given a sign language video $V_{i}$, the decoder aims to generate the corresponding spoken sentence $\widehat{TS}_{i}=(\widehat{TS}_{i,1},\dots,\widehat{TS}_{i,T^{\prime}})$, where $T^{\prime}$ denotes the number of tokens in the target spoken sentence. The decoder operates in an autoregressive manner, it begins with the special <bos> token and predicts tokens one by one until the <eos> token is reached. The model is trained by minimizing the cross-entropy loss between the predicted tokens $\widehat{TS}_{i,j}$ and the ground truth tokens $TS_{i,j}$ as follows:

$$\mathcal{L}_{SLT_{i}}=-\sum_{j=1}^{T^{\prime}}\log\hskip 2.0ptp(\widehat{TS}_{i,j}\mid TS_{i,1:j-1},V_{i}).$$ (9)

4 Experiments

4.1 Datasets and Evaluation Metrics

Datasets. We conduct experiments on the Phoenix2014T [Camgoz et al.(2018)Camgoz, Hadfield, Koller, Ney, and Bowden] (or Phoenix14T) and CSL-Daily [Zhou et al.(2021)Zhou, Zhou, Qi, Pu, and Li] datasets for SLT, evaluating performance on their development and test sets.Phoen-ix-2014T is a German sign language dataset containing 2,887 unique German words, with 7,096 training samples, 519 validation samples, and 642 test samples. CSL-Daily is a Chinese sign language dataset with a vocabulary of 2,343 Chinese words, comprising 18,401 training samples, 1,077 validation samples, and 1,176 test samples.
Evaluation Metrics. We use BLEU [Papineni et al.(2002)Papineni, Roukos, Ward, and Zhu] and ROUGE-L [Lin(2004)] to evaluate translation quality. BLEU-n measures n-gram precision, and we report BLEU-1 to BLEU-4 scores. ROUGE-L (or ROUGE) evaluates the F1 score based on the longest common subsequence between predicted and reference translations. For simplicity, we denote BLEU-1 to BLEU-4 as B1, B2, B3, and B4, and ROUGE as R.

4.2 Implementation details

Model. To address memory constraints, we apply LoRA [Hu et al.(2021)Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, and Chen] adapters to the top three layers of the visual encoder. The adapters are configured with a rank of 4, a dropout rate of 0.1, scaling factor 4.0, and are applied to both self-attention and feed-forward weights. The temporal encoder is a four-layer transformer with a hidden dimension of 512, 8 attention heads and an intermediate size of 2048, with temporal downsampling applied after the second layer. We adopt the mBART-large-cc25 model [Liu et al.(2020)Liu, Gu, Goyal, Li, Edunov, Ghazvininejad, Lewis, and Zettlemoyer], which consists of 12 layers in both the encoder and decoder, as the backbone of our LLM-based encoder-decoder architecture. LoRA [Hu et al.(2021)Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, and Chen] is applied to the LLM encoder and decoder modules, targeting $q_{\text{proj}}$ and $v_{\text{proj}}$ layers with a rank of 16, a scaling factor (LoRA alpha) of 32 and a dropout rate of 0.1. All mappers in Fig. 2 are trainable and consist of a linear layer followed by GELU activation function. Training is optimized with the XFormers [Lefaudeux et al.(2022)Lefaudeux, Massa, Liskovich, Xiong, Caggiano, Naren, Xu, Hu, Tintore, Zhang, Labatut, Haziza, Wehrstedt, Reizenstein, and Sizov] library with Flash Attention [Dao(2023)] for improved efficiency. In pre-training, we use mBART-large-50 [Liu et al.(2020)Liu, Gu, Goyal, Li, Edunov, Ghazvininejad, Lewis, and Zettlemoyer] as the text encoder for video-description alignment due to its broader multilingual coverage and robustness to noisy, diverse descriptions. For video-target alignment, we choose mBART-large-cc25 [Liu et al.(2020)Liu, Gu, Goyal, Li, Edunov, Ghazvininejad, Lewis, and Zettlemoyer] as the text encoder as it offers more stable embeddings for clean, well-structured target sentences in high-resource languages. We use ShareGPT4Video-8B  [Chen et al.(2024a)Chen, Wei, Li, Dong, Zhang, Zang, Chen, Duan, Lin, Tang, Yuan, Qiao, Lin, Zhao, and Wang] as the video LLM in the pre-training stage to generate detailed and semantically rich hand-focused descriptions from sign language video segments, where each segment consists of 16 frames. We also use GPT-4o-mini API [OpenAI(2023)] to generate a single coherent description, maintaining the temporal order of events and removing irrelevant or unnecessary information.
Implementation. The model is pre-trained, for 100 epochs, and fine-tuned, for 200 epochs, with a batch size of 16 on one A100 GPU. The input sequences are first resized into $256\times 256$, and then randomly/centrally cropped into $224\times 224$ during training/inference. We use the AdamW [Loshchilov and Hutter(2019)] optimizer with a learning rate of $3\times 10^{-4}$ and weight decay of $0.001$.

Table 1: Experimental results on Phoenix14T and CSL-Daily datasets. The best results for gloss-free models are highlighted in bold, while the second-best results are underlined.

Method	Phoenix14T					CSL-Daily
	B-1	B-2	B-3	B-4	R	B-1	B-2	B-3	B-4	R
Gloss-based
MMTLB [Chen et al.(2022a)Chen, Wei, Sun, Wu, and Lin]	53.97	41.75	33.84	28.39	52.65	53.31	40.41	30.87	23.92	53.25
SLTUNET [Zhang et al.(2023a)Zhang, Müller, and Sennrich]	52.92	41.76	33.99	28.47	52.11	54.98	41.44	31.84	25.01	54.08
TS-SLT [Chen et al.(2022b)Chen, Zuo, Wei, Wu, Liu, and Mak]	54.90	42.43	34.46	28.95	53.48	55.44	42.59	32.87	25.79	55.72
Gloss-free
NSLT+Luong [Luong et al.(2015)Luong, Pham, and Manning]	29.86	17.52	11.96	9.00	30.70	34.16	19.57	11.84	7.56	34.54
CSGCR [Zhao et al.(2022)Zhao, Qi, Zhou, Duan, Zhou, and Li]	36.71	25.40	18.86	15.18	38.85	–	–	–	–	–
GFSLT-VLP [Zhou et al.(2023)Zhou, Chen, Clapés, Wan, Liang, Escalera, Lei, and Zhang]	43.71	33.18	26.11	21.44	42.49	39.37	24.93	16.26	11.00	36.44
Sign2GPT [Wong et al.()Wong, Camgöz, and Bowden]	49.54	35.96	28.83	22.52	48.90	41.75	28.73	20.60	15.40	42.36
SignCL [Ye et al.()Ye, Wang, Jiao, Liang, and Xiong]	49.76	36.85	29.97	22.74	49.07	47.47	32.53	22.62	16.16	48.92
FLa-LLM [Chen et al.(2024b)Chen, Zhou, Li, Wan, Lei, Jiang, Lu, and Zhao]	46.29	35.33	28.03	23.09	45.27	37.13	25.12	18.38	14.20	37.25
SignLLM [Gong et al.(2024)Gong, Foo, He, Rahmani, and Liu]	45.21	34.78	28.05	23.40	44.49	39.55	28.13	20.07	15.75	39.91
LLaVA-SLT [Liang et al.(2024)Liang, Huang, Xu, Tang, Ye, Zhang, Chen, Yu, and Xu]	51.20	37.51	29.39	23.43	50.44	52.15	36.24	26.47	20.42	51.26
BeyondGloss	52.38	38.57	30.74	25.49	52.89	53.12	38.63	27.82	21.53	53.46

4.3 Results

Results on Phoenix14T and CSL-Daily. Tab. 1 compares gloss-based, and gloss-free methods on the Phoenix14T and CSL-Daily test datasets. Our method outperforms other gloss-free SLT approaches in terms of BLEU-4 and ROUGE.
Qualitative Results. Tab. 6 shows translation examples from Phoenix14T. We compare the reference translations with outputs from GFSLT-VLP [Zhou et al.(2023)Zhou, Chen, Clapés, Wan, Liang, Escalera, Lei, and Zhang], the only other publicly available gloss-free model, and our method. While GFSLT-VLP often captures only a few correct words or produces unrelated sentences, our method generates more accurate translations. These qualitative examples demonstrate the effectiveness of our approach. See appendix for more comparison.

Table 2: Qualitative results of Phoenix14T. Green means totally same as the reference.

Reference:	am freitag scheint abseits der nebelgebiete haufig die sonne
	(On Friday the sun often shines outside of the foggy areas)
GFSLT-VLP:	am freitag sobald der nebel weg ist sonnenschein.
	(On Friday as soon as the fog is gone there will be sunshine)
BeyondGloss:	am freitag scheint abseits der nebelgebiete haufig die sonne.
	(On Friday the sun often shines outside of the foggy areas)

4.4 Ablation studies

Analysis of Pre-Training Components Contributions. We conduct experiments to evaluate the impact of the core components of our proposed method. Tab. 7 reports the performance obtained by progressively incorporating each component. As shown in the first row, omitting pre-training and alignment between sign videos and output texts results in significantly lower downstream performance, highlighting the importance of pre-training for SLT. The second row demonstrates that video-target alignment contributes substantially to performance improvement. Rows three and four further show the positive effect of our proposed video-description alignment. In rows four and five, feature distillation also contributes to modest improvements in performance. Finally, combining all components together (row five) yields the best overall performance, achieving state-of-the-art results.
Evaluating Different Visual Encoders. We conduct an additional ablation study to examine the impact of different visual encoders on final performance. As shown in Tab. 5, using DINO V2 [Oquab et al.(2024)Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, Fernandez, Haziza, Massa, El-Nouby, Assran, Ballas, Galuba, Howes, Huang, Li, Misra, Rabbat, Sharma, Synnaeve, Xu, Jegou, Mairal, Labatut, Joulin, and Bojanowski], a transformer-based visual encoder, yields better results compared to ResNet-18 [He et al.(2016)He, Zhang, Ren, and Sun], a convolutional encoder. The best performance is achieved when applying LoRA to the last three layers of DINO V2, leading to our state-of-the-art results.
Evaluating Different LLMs. As shown in Tab. 5, we assess the impact of using various LLMs within our framework. Both MT5-Base [Xue et al.(2021)Xue, Constant, Roberts, Kale, Al-Rfou, Siddhant, Barua, and Raffel] and MBART (24-layer) [Liu et al.(2020)Liu, Gu, Goyal, Li, Edunov, Ghazvininejad, Lewis, and Zettlemoyer], which have a comparable number of parameters, yield strong results. MBART slightly outperforms MT5-Base, likely due to its stronger multilingual prior knowledge. These findings highlight the generalizability of our approach. Additionally, we evaluate a 6-layer version of MBART, similar to the configuration used in GFSLT-VLP [Zhou et al.(2023)Zhou, Chen, Clapés, Wan, Liang, Escalera, Lei, and Zhang], which achieves moderate performance. It is worth mentioning that all the tests in the ablation studies are conducted on the Pheonix14T validation set.

Table 3: Ablation study on key elements in our framework.

	Feature Distillation	Video-Description Alignment	Video-Target Alignment	B-4	R
(1)	–	–	–	16.31	36.33
(2)	–	–	✓	21.87	44.30
(3)	–	✓	✓	23.47	47.17
(4)	✓	✓	–	22.47	44.87
(5)	✓	✓	✓	25.68	53.85

Table 4: Ablation study on visual encoder.

Visual Encoder	Trainable	B-4	R
ResNet-18[He et al.(2016)He, Zhang, Ren, and Sun]	✓	18.97	40.17
Dino V2[Oquab et al.(2024)Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, Fernandez, Haziza, Massa, El-Nouby, Assran, Ballas, Galuba, Howes, Huang, Li, Misra, Rabbat, Sharma, Synnaeve, Xu, Jegou, Mairal, Labatut, Joulin, and Bojanowski]	$\times$	20.43	42.75
Dino V2[Oquab et al.(2024)Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, Fernandez, Haziza, Massa, El-Nouby, Assran, Ballas, Galuba, Howes, Huang, Li, Misra, Rabbat, Sharma, Synnaeve, Xu, Jegou, Mairal, Labatut, Joulin, and Bojanowski]	3 last layers fine-tuned by LoRA	25.68	53.85

Table 5: Ablation study on LLM.

Model	Num of Layers/Num of parameters	B-4	R
MBart [Liu et al.(2020)Liu, Gu, Goyal, Li, Edunov, Ghazvininejad, Lewis, and Zettlemoyer]	6 / ~115M	22.50	45.70
MT5-Base [Xue et al.(2021)Xue, Constant, Roberts, Kale, Al-Rfou, Siddhant, Barua, and Raffel]	24 / ~580M	24.18	49.87
MBart [Liu et al.(2020)Liu, Gu, Goyal, Li, Edunov, Ghazvininejad, Lewis, and Zettlemoyer]	24 / ~600M	25.68	53.85

5 Conclusion

In this work, we introduced BeyondGloss, a novel gloss-free Sign Language Translation framework that emphasizes hand-centric modeling and leverages the capabilities of VideoLLMs. By generating fine-grained, temporally-aware textual descriptions of hand motion and aligning them with sign video features through contrastive learning, our approach effectively bridges the modality gap between visual and linguistic representations. Additionally, the integration of detailed hand features distilled from HaMeR further enriches the sign video representations. Extensive experiments on Phoenix14T and CSL-Daily demonstrate that BeyondGloss achieves state-of-the-art performance, underscoring the importance of structured hand modeling and multimodal alignment in advancing gloss-free SLT.

6 Acknowledgement

This work was supported by the SNSF project ‘SMILE II’ (CRSII5 193686), the Innosuisse IICT Flagship (PFFS-21-47), EPSRC grant APP24554 (SignGPT-EP/Z535370/1) and through funding from Google.org via the AI for Global Goals scheme. This work reflects only the author’s views and the funders are not responsible for any use that may be made of the information it contains.

7 Sign Video Descriptor

7.1 Video Segmentor Algorithm

Algorithm 1 Video Segmentation into 16-Frame Segments

0: Video file $V$

0: List of 16-frame segments $S$

1: Open video $V$ and get total frames $N$

2: Initialize empty list $S\leftarrow[\,]$

3: $segment\leftarrow[\,]$ {Current segment buffer}

4: for $i\leftarrow 1$ to $N$ do

5:  $frame\leftarrow$ read next frame from $V$

6:  Append $frame$ to $segment$

7:  if $\text{length}(segment)==16$ then

8:   Append $segment$ to $S$

9:   Reset $segment\leftarrow[\,]$ {Start new segment}

10:  end if

11: end for

12: if $\text{length}(segment)>0$ then

13:  Append remaining $segment$ to $S$ {Handle leftover frames}

14: end if

15: return  $S$

7.2 More Examples

As shown in Fig. 3, the rectangle around the sequence of frames and its corresponding description in the final video description share the same color.

Figure 3: More examples of video description generated by the proposed Sign Video Descriptor in Phoenix-2014T and CSL-Daily.

8 Qualitative Results

Table 6: Qualitative results of Phoenix14T. Green means totally same as the reference.

Reference:	am tag wechseln sonne und wolken einander ab teilweise ist es auch langere zeit sonnig.
	(During the day sun and clouds alternate partly it is sunny for a long time)
GFSLT-VLP:	am tag wechseln sonne und wolken einander ab es bilden sich langere zeit viel sonnenschein.
	( During the day, sun and clouds alternate There is a lot of sunshine for a long time )
BeyondGloss:	am tag wechseln sonne und wolken einander ab zeigt sich die auch langere zeit sonnig.
	(During the day, sun and clouds alternate, and the weather remains sunny for longer periods.)
Reference:	am tag nur hier und da einige sonnige momente vor allem an den alpen.
	(During the day only here and there some sunny moments, especially in the Alps)
GFSLT-VLP:	morgen zeigt sich mal die sonne wenn dann vor allem an den alpen.
	(Tomorrow the sun will show up , especially in the Alps)
BeyondGloss:	und morgen scheint und da einige die sonnige vor allem an den alpen.
	(Tomorrow will shine and there will be some sunny weather, especially in the Alps.)
Reference:	am freitag scheint abseits der nebelgebiete haufig die sonne
	(On Friday the sun often shines outside of the foggy areas)
GFSLT-VLP:	am freitag sobald der nebel weg ist sonnenschein.
	(On Friday as soon as the fog is gone there will be sunshine)
BeyondGloss:	am freitag scheint abseits der nebelgebiete haufig die sonne.
	(On Friday the sun often shines outside of the foggy areas)

9 Ablation studies

9.1 Impact of Loss Weighting in Pre-training for SLT

As shown in Tab. 7, we have conducted a comprehensive evaluation to determine the optimal weighting in the pre-training loss.

Table 7: Impact of Loss Weighting in Pre-training for SLT.

	$\lambda_{\text{distill}}$	$\lambda_{\text{desc}}$	$\lambda_{\text{align}}$	B-4	R
(1)	0.0	0.0	0.0	16.41	36.33
(2)	0.0	0.0	1.0	21.87	44.30
(3)	0.0	1.0	1.0	22.13	45.14
(4)	0.0	0.5	1.0	23.47	47.17
(5)	1.0	1.0	1.0	22.85	46.32
(6)	0.5	1.0	1.0	24.56	50.21
(7)	0.3	0.5	1.0	25.68	53.85