Metrics

  • Force Profile Accuracy (FPAcc): Measures how closely the predicted force profile \( \hat{x}_f \) matches the ground truth \( x_f \) using Mean Squared Error (MSE). $$ \text{FPAcc} = \text{MSE}(\hat{x}_f, x_f) $$
  • Force Direction Accuracy (FDAcc): Evaluates directional alignment between predicted impulse \( \hat{J}(T) \) and ground truth impulse \( J(T) \) using cosine similarity. $$ \text{FDAcc} = \frac{\hat{J}(T) \cdot J(T)}{||\hat{J}(T)|| \cdot ||J(T)||} $$
  • Modifier Similarity (ModSim): Cosine similarity between the predicted modifier \( \hat{w}_m \) and the ground truth \( w_m \) using SBERT embeddings \( E \). $$ \text{ModSim} = \frac{E(\hat{w}_m) \cdot E(w_m)}{||E(\hat{w}_m)|| \cdot ||E(w_m)||} $$
  • Direction Similarity (DirSim): Cosine similarity between predicted direction words \( \hat{w}_d \) and ground truth \( w_d \). $$ \text{DirSim} = \frac{E(\hat{w}_d) \cdot E(w_d)}{||E(\hat{w}_d)|| \cdot ||E(w_d)||} $$
  • Full Phrase Similarity (PhraseSim): Average of \( \text{ModSim} \) and \( \text{DirSim} \). $$ \text{PhraseSim} = \frac{1}{2} \left(\text{ModSim} + \text{DirSim} \right) $$

Baseline Models

  • $\text{SVM/KNN}$: Uses SVM to classify force signals into phrases and KNN to predict force profiles from phrases.
  • $\text{DMLP}_B$: Direct MLP mapping between force profiles and binary phrase vectors without a shared latent space.
  • $\text{DMLP}_G$: Direct MLP mapping between force profiles and GloVe-embedded phrases without a shared latent space.
  • $\text{DAE}_B$ (Ours): Dual autoencoder with binary phrase vectors.
  • $\text{DAE}G$ (Ours): Dual autoencoder with GloVe embeddings.

Experiments

To evaluate the effectiveness of our cross-modality embedding framework, we conducted three main experiments:

  1. In-Distribution Evaluation: The dataset was randomly split into 90% training and 10% testing sets. Models were trained and tested on the same distribution of force–phrase pairs and averaged across 30 independent seeds to reduce variability in the results.
  2. Out-of-Distribution Modifiers: To test generalization, we systematically held out one modifier word (e.g., “slowly”) from the training set while keeping it in the test set. This experiment measured how well the models handled unseen adverbial variations in force descriptions.
  3. Out-of-Distribution Directions: Similar to the modifier experiment, we excluded a single direction word (e.g., “up”) from training and evaluated the model’s ability to generalize to new directional inputs. This tested whether the models could infer force trajectories for previously unseen movement directions.

Overall Results

Our framework outperforms baseline models in both force-to-language and language-to-force translation, demonstrating the effectiveness of a shared latent space.

In-Distribution Performances
(a)
Out-of-Distribution Modifier Performances
(b)
Out-of-Distribution Direction Performances
(c)
Performance comparison across models for: (a) in-distribution testing, (b) out-of-distribution modifiers, and (c) out-of-distribution directions. Higher values indicate better performance, except for FPAcc where lower values are better.

Analysis

Our results showed that the dual autoencoder (DAE) models significantly outperformed baseline approaches in both force-to-language and language-to-force translation. The DAE models achieved 20-30% better performance than simpler SVM and MLP-based baselines, demonstrating that a shared latent space effectively captures the bidirectional relationship between physical force and linguistic descriptions.

Additionally, the models successfully generalized to unseen modifiers and directions, particularly when using GloVe embeddings, which improved force profile reconstruction. However, we observed a trade-off between force reconstruction accuracy and linguistic precision: the GloVe-based model (\( \text{DAE}_G \)) was better at estimating realistic force trajectories, while the binary phrase vector model (\( \text{DAE}_B \)) provided more precise textual descriptions.

These findings confirm that integrating force and language into a unified representation enhances natural human-robot communication, with different representation choices offering advantages depending on whether force accuracy or linguistic fidelity is prioritized.

Additional Results

Translation Examples

Example predictions from both $\text{DAE}_B$ and $\text{DAE}_G$ variants for corresponding force profile—phrase pairs. We generated 10 different 90-10 training-test splits, ensuring that the union of all test sets covers the full dataset. Each data point was randomly assigned to one of 10 bins, with each bin serving as a distinct test set in one of the splits. The model was trained on the remaining 9 bins and evaluated on the held-out test bin. The examples below show model predictions for cases where the corresponding force-phrase pair was not explicitly seen during training.

Phrase-To-Force Trials

Corresponding force profile and phrase pairs collected from the Phrase-To-Force procedure.

User ID Trial # Phrase Text Modifier Direction Trial Card
0 0 down (None) down View
0 1 forward (None) forward View
0 2 up (None) up View
0 3 backward (None) backward View
0 4 left (None) left View
0 5 right (None) right View
0 6 below sharply sharply down View
0 7 sharply left and downward sharply down-left View
0 8 leftward and behind (None) backward-left View
0 9 up and to the right immediately immediately right-up View
0 10 to the left and above significantly significantly left-up View
0 11 sharply behind and right sharply backward-right View
0 12 slowly leftward slowly left View
0 13 right and backward (None) backward-right View
0 14 immediately to the right immediately right View
0 15 immediately ahead and up immediately forward-up View
0 16 ahead smoothly smoothly forward View
0 17 to the right and forward (None) forward-right View
0 18 leftward and to the front (None) forward-left View
0 19 backward and leftward (None) backward-left View
0 20 to the front and down harshly harshly down-forward View
0 21 harshly to the front and right harshly forward-right View
0 22 softly below and right softly down-right View
0 23 forward lightly lightly forward View
0 24 to the front and to the left sharply sharply forward-left View
0 25 sharply downward sharply down View
0 26 lightly backward lightly backward View
0 27 downward and right (None) down-right View
0 28 to the left and backward (None) backward-left View
0 29 ahead softly softly forward View
0 30 down and left immediately immediately down-left View
0 31 forward and below (None) down-forward View
0 32 lightly backward lightly backward View
0 33 left sharply sharply left View
0 34 above and ahead (None) forward-up View
0 35 left and to the back (None) backward-left View
0 36 ahead slowly slowly forward View
0 37 downward and left (None) down-left View
0 38 immediately forward and up immediately forward-up View
0 39 left and forward greatly greatly forward-left View
0 40 slowly above slowly up View
0 41 downward and behind (None) backward-down View
1 0 left (None) left View
1 1 forward (None) forward View
1 2 up (None) up View
1 3 right (None) right View
1 4 down (None) down View
1 5 backward (None) backward View
1 6 to the front and downward (None) down-forward View
1 7 to the back quickly quickly backward View
1 8 smoothly right smoothly right View
1 9 quickly rightward and downward quickly down-right View
1 10 to the front and left slightly slightly forward-left View
1 11 below significantly significantly down View
1 12 ahead softly softly forward View
1 13 quickly above quickly up View
1 14 above and rightward greatly greatly right-up View
1 15 rightward and up (None) right-up View
1 16 forward softly softly forward View
1 17 greatly forward greatly forward View
1 18 below and to the front (None) down-forward View
1 19 to the back and up (None) backward-up View
1 20 lightly up and to the left lightly left-up View
1 21 slowly below slowly down View
1 22 above slightly slightly up View
1 23 smoothly behind smoothly backward View
1 24 slightly behind slightly backward View
1 25 slowly down and forward slowly down-forward View
1 26 quickly to the right and to the front quickly forward-right View
1 27 immediately to the left and backward immediately backward-left View
1 28 to the right and upward significantly significantly right-up View
1 29 rightward and ahead (None) forward-right View
1 30 below and to the left (None) down-left View
1 31 upward and to the back (None) backward-up View
1 32 significantly downward and to the right significantly down-right View
1 33 to the front and to the left quickly quickly forward-left View
1 34 below and forward (None) down-forward View
1 35 downward and behind (None) backward-down View
1 36 rightward and below (None) down-right View
1 37 upward and to the right (None) right-up View
1 38 backward and below (None) backward-down View
1 39 to the back and left quickly quickly backward-left View
1 40 forward and leftward sharply sharply forward-left View
1 41 above smoothly smoothly up View
2 0 down (None) down View
2 1 forward (None) forward View
2 2 right (None) right View
2 3 up (None) up View
2 4 backward (None) backward View
2 5 left (None) left View
2 6 slowly ahead and upward slowly forward-up View
2 7 softly upward softly up View
2 8 below and leftward (None) down-left View
2 9 significantly down significantly down View
2 10 to the back immediately immediately backward View
2 11 ahead sharply sharply forward View
2 12 leftward and backward immediately immediately backward-left View
2 13 greatly backward and downward greatly backward-down View
2 14 quickly to the left quickly left View
2 15 to the front and upward (None) forward-up View
2 16 to the back and upward (None) backward-up View
2 17 backward and down slowly slowly backward-down View
2 18 left and backward (None) backward-left View
2 19 to the right and down (None) down-right View
2 20 left and downward (None) down-left View
2 21 slowly downward and to the back slowly backward-down View
2 22 softly backward softly backward View
2 23 below and to the right (None) down-right View
2 24 downward and ahead (None) down-forward View
2 25 smoothly left and behind smoothly backward-left View
2 26 upward and leftward slowly slowly left-up View
2 27 softly behind softly backward View
2 28 backward and right quickly quickly backward-right View
2 29 upward slowly slowly up View
2 30 downward gradually gradually down View
2 31 to the back and to the left (None) backward-left View
2 32 to the back and rightward slowly slowly backward-right View
2 33 to the front and downward (None) down-forward View
2 34 lightly down and forward lightly down-forward View
2 35 slightly to the back slightly backward View
2 36 softly above and backward softly backward-up View
2 37 left and down (None) down-left View
2 38 to the left significantly significantly left View
2 39 behind and above immediately immediately backward-up View
2 40 behind and above (None) backward-up View
2 41 above gradually gradually up View
3 0 down (None) down View
3 1 up (None) up View
3 2 forward (None) forward View
3 3 backward (None) backward View
3 4 left (None) left View
3 5 right (None) right View
3 6 harshly backward and leftward harshly backward-left View
3 7 leftward and below (None) down-left View
3 8 down and left harshly harshly down-left View
3 9 right and upward softly softly right-up View
3 10 significantly to the back significantly backward View
3 11 slightly to the front slightly forward View
3 12 quickly down and backward quickly backward-down View
3 13 forward sharply sharply forward View
3 14 down and to the right (None) down-right View
3 15 slowly to the back slowly backward View
3 16 above and right quickly quickly right-up View
3 17 downward and to the left (None) down-left View
3 18 gradually behind and left gradually backward-left View
3 19 to the left lightly lightly left View
3 20 behind and left (None) backward-left View
3 21 leftward and below lightly lightly down-left View
3 22 leftward smoothly smoothly left View
3 23 rightward and to the back (None) backward-right View
3 24 up and to the back immediately immediately backward-up View
3 25 upward and to the right (None) right-up View
3 26 gradually rightward and below gradually down-right View
3 27 slowly down and rightward slowly down-right View
3 28 slowly up slowly up View
3 29 down quickly quickly down View
3 30 behind and downward (None) backward-down View
3 31 below and behind (None) backward-down View
3 32 leftward and below (None) down-left View
3 33 slowly to the front slowly forward View
3 34 slightly above slightly up View
3 35 above slightly slightly up View
3 36 ahead and below (None) down-forward View
3 37 down and forward slowly slowly down-forward View
3 38 to the front softly softly forward View
3 39 right and above (None) right-up View
3 40 softly up and to the back softly backward-up View
3 41 downward and backward (None) backward-down View
4 0 down (None) down View
4 1 forward (None) forward View
4 2 left (None) left View
4 3 backward (None) backward View
4 4 right (None) right View
4 5 up (None) up View
4 6 greatly ahead and leftward greatly forward-left View
4 7 right harshly harshly right View
4 8 forward and rightward (None) forward-right View
4 9 slightly right slightly right View
4 10 rightward and down (None) down-right View
4 11 backward significantly significantly backward View
4 12 quickly up quickly up View
4 13 up greatly greatly up View
4 14 below and backward immediately immediately backward-down View
4 15 quickly up and to the right quickly right-up View
4 16 quickly behind and to the right quickly backward-right View
4 17 to the back and below (None) backward-down View
4 18 gradually up gradually up View
4 19 lightly left lightly left View
4 20 behind and up greatly greatly backward-up View
4 21 downward and to the front (None) down-forward View
4 22 significantly ahead significantly forward View
4 23 backward immediately immediately backward View
4 24 leftward and to the back (None) backward-left View
4 25 above and leftward gradually gradually left-up View
4 26 to the left and upward (None) left-up View
4 27 to the front and leftward (None) forward-left View
4 28 upward and to the front (None) forward-up View
4 29 leftward and forward (None) forward-left View
4 30 gradually to the back and down gradually backward-down View
4 31 quickly leftward quickly left View
4 32 quickly ahead and left quickly forward-left View
4 33 immediately backward and right immediately backward-right View
4 34 ahead and right (None) forward-right View
4 35 ahead and leftward gradually gradually forward-left View
4 36 right and backward (None) backward-right View
4 37 upward and to the back (None) backward-up View
4 38 to the right immediately immediately right View
4 39 to the right and backward softly softly backward-right View
4 40 to the back and to the right greatly greatly backward-right View
4 41 ahead harshly harshly forward View
5 0 left (None) left View
5 1 backward (None) backward View
5 2 up (None) up View
5 3 right (None) right View
5 4 forward (None) forward View
5 5 down (None) down View
5 6 upward and leftward (None) left-up View
5 7 upward and left slightly slightly left-up View
5 8 to the front and upward (None) forward-up View
5 9 gradually to the front and above gradually forward-up View
5 10 ahead and right (None) forward-right View
5 11 to the front greatly greatly forward View
5 12 downward and right gradually gradually down-right View
5 13 up and to the back (None) backward-up View
5 14 slightly to the back and upward slightly backward-up View
5 15 to the back and leftward (None) backward-left View
5 16 smoothly left smoothly left View
5 17 downward and ahead (None) down-forward View
5 18 leftward and forward (None) forward-left View
5 19 greatly behind greatly backward View
5 20 sharply to the front sharply forward View
5 21 up softly softly up View
5 22 to the back and to the left (None) backward-left View
5 23 ahead and right (None) forward-right View
5 24 harshly ahead harshly forward View
5 25 backward immediately immediately backward View
5 26 downward and left gradually gradually down-left View
5 27 backward greatly greatly backward View
5 28 harshly up harshly up View
5 29 significantly backward and below significantly backward-down View
5 30 greatly down and to the front greatly down-forward View
5 31 to the left lightly lightly left View
5 32 ahead and downward (None) down-forward View
5 33 upward and behind sharply sharply backward-up View
5 34 sharply rightward sharply right View
5 35 downward and forward (None) down-forward View
5 36 behind quickly quickly backward View
5 37 smoothly leftward and behind smoothly backward-left View
5 38 forward and upward quickly quickly forward-up View
5 39 softly below and left softly down-left View
5 40 above and to the left (None) left-up View
5 41 leftward and below lightly lightly down-left View
6 0 left (None) left View
6 1 right (None) right View
6 2 forward (None) forward View
6 3 up (None) up View
6 4 backward (None) backward View
6 5 down (None) down View
6 6 sharply upward sharply up View
6 7 to the right and behind (None) backward-right View
6 8 up and backward greatly greatly backward-up View
6 9 greatly below greatly down View
6 10 ahead and downward (None) down-forward View
6 11 downward and backward (None) backward-down View
6 12 right and downward (None) down-right View
6 13 to the left and down gradually gradually down-left View
6 14 ahead lightly lightly forward View
6 15 above and behind (None) backward-up View
6 16 softly rightward and to the back softly backward-right View
6 17 above quickly quickly up View
6 18 behind and above (None) backward-up View
6 19 to the back harshly harshly backward View
6 20 rightward and up (None) right-up View
6 21 quickly to the back quickly backward View
6 22 to the left and above harshly harshly left-up View
6 23 softly right and down softly down-right View
6 24 rightward and backward slowly slowly backward-right View
6 25 lightly down and left lightly down-left View
6 26 slowly to the right and to the front slowly forward-right View
6 27 significantly ahead and up significantly forward-up View
6 28 to the left and to the back smoothly smoothly backward-left View
6 29 slightly left slightly left View
6 30 to the front and right (None) forward-right View
6 31 ahead immediately immediately forward View
6 32 downward and left quickly quickly down-left View
6 33 left smoothly smoothly left View
6 34 to the front and to the right (None) forward-right View
6 35 leftward and up (None) left-up View
6 36 slightly to the back slightly backward View
6 37 greatly rightward greatly right View
6 38 upward and right (None) right-up View
6 39 slightly above and backward slightly backward-up View
6 40 to the left and ahead (None) forward-left View
6 41 above immediately immediately up View
7 0 right (None) right View
7 1 down (None) down View
7 2 forward (None) forward View
7 3 backward (None) backward View
7 4 left (None) left View
7 5 up (None) up View
7 6 harshly to the left harshly left View
7 7 backward and above (None) backward-up View
7 8 to the right slightly slightly right View
7 9 smoothly to the back smoothly backward View
7 10 below and forward (None) down-forward View
7 11 gradually below gradually down View
7 12 downward slightly slightly down View
7 13 backward and rightward (None) backward-right View
7 14 down and right (None) down-right View
7 15 ahead and down lightly lightly down-forward View
7 16 sharply rightward and ahead sharply forward-right View
7 17 backward quickly quickly backward View
7 18 downward lightly lightly down View
7 19 above and to the right gradually gradually right-up View
7 20 to the back and upward (None) backward-up View
7 21 above and leftward (None) left-up View
7 22 up and to the left sharply sharply left-up View
7 23 quickly left and forward quickly forward-left View
7 24 harshly downward harshly down View
7 25 ahead and downward (None) down-forward View
7 26 up harshly harshly up View
7 27 gradually to the left and backward gradually backward-left View
7 28 lightly leftward lightly left View
7 29 up and rightward smoothly smoothly right-up View
7 30 immediately leftward immediately left View
7 31 to the right and forward (None) forward-right View
7 32 down and behind (None) backward-down View
7 33 forward sharply sharply forward View
7 34 forward and to the right quickly quickly forward-right View
7 35 softly to the front and leftward softly forward-left View
7 36 upward and left (None) left-up View
7 37 gradually downward and rightward gradually down-right View
7 38 left and forward (None) forward-left View
7 39 smoothly forward and down smoothly down-forward View
7 40 rightward and down (None) down-right View
7 41 to the right and ahead slowly slowly forward-right View
8 0 up (None) up View
8 1 forward (None) forward View
8 2 left (None) left View
8 3 backward (None) backward View
8 4 down (None) down View
8 5 right (None) right View
8 6 downward smoothly smoothly down View
8 7 upward and ahead (None) forward-up View
8 8 right lightly lightly right View
8 9 rightward and ahead (None) forward-right View
8 10 harshly down and to the left harshly down-left View
8 11 backward and below (None) backward-down View
8 12 right softly softly right View
8 13 right and downward (None) down-right View
8 14 smoothly to the left smoothly left View
8 15 to the right immediately immediately right View
8 16 below and to the back slowly slowly backward-down View
8 17 leftward and downward (None) down-left View
8 18 downward and rightward (None) down-right View
8 19 greatly downward and rightward greatly down-right View
8 20 leftward and behind (None) backward-left View
8 21 quickly forward quickly forward View
8 22 below and to the left (None) down-left View
8 23 slowly below and right slowly down-right View
8 24 to the right immediately immediately right View
8 25 above and to the front slowly slowly forward-up View
8 26 below and to the left (None) down-left View
8 27 ahead slightly slightly forward View
8 28 ahead and to the right (None) forward-right View
8 29 left and down slightly slightly down-left View
8 30 significantly to the back significantly backward View
8 31 downward and ahead (None) down-forward View
8 32 forward and to the right sharply sharply forward-right View
8 33 quickly above quickly up View
8 34 left and backward significantly significantly backward-left View
8 35 right and upward (None) right-up View
8 36 harshly up and rightward harshly right-up View
8 37 immediately to the front and right immediately forward-right View
8 38 smoothly ahead smoothly forward View
8 39 sharply below sharply down View
8 40 immediately to the left and up immediately left-up View
8 41 upward and to the front gradually gradually forward-up View
9 0 left (None) left View
9 1 right (None) right View
9 2 down (None) down View
9 3 backward (None) backward View
9 4 forward (None) forward View
9 5 up (None) up View
9 6 leftward and to the back (None) backward-left View
9 7 greatly rightward greatly right View
9 8 to the front and rightward (None) forward-right View
9 9 behind and down (None) backward-down View
9 10 upward quickly quickly up View
9 11 upward and ahead quickly quickly forward-up View
9 12 to the back and above gradually gradually backward-up View
9 13 harshly below harshly down View
9 14 up and backward (None) backward-up View
9 15 forward and up (None) forward-up View
9 16 right and up (None) right-up View
9 17 behind gradually gradually backward View
9 18 above gradually gradually up View
9 19 leftward and up (None) left-up View
9 20 harshly right harshly right View
9 21 down and to the front (None) down-forward View
9 22 gradually below gradually down View
9 23 to the right and up (None) right-up View
9 24 immediately to the right and down immediately down-right View
9 25 right and forward (None) forward-right View
9 26 significantly to the left and to the front significantly forward-left View
9 27 slowly below and ahead slowly down-forward View
9 28 down and right (None) down-right View
9 29 to the left sharply sharply left View
9 30 right smoothly smoothly right View
9 31 slightly to the front and rightward slightly forward-right View
9 32 above and rightward (None) right-up View
9 33 smoothly downward smoothly down View
9 34 significantly to the right and down significantly down-right View
9 35 up and forward greatly greatly forward-up View
9 36 right and downward slightly slightly down-right View
9 37 smoothly below and forward smoothly down-forward View
9 38 backward and leftward immediately immediately backward-left View
9 39 up slightly slightly up View
9 40 smoothly up smoothly up View
9 41 leftward and to the front immediately immediately forward-left View

Out-of-Distribution Modifiers

The model successfully generalizes to unseen modifiers, with GloVe embeddings improving force profile reconstruction.

Modifier Result
Modifier Result
Modifier Result
Modifier Result
Modifier Result
Modifier Result
Modifier Result
Modifier Result
Modifier Result
Modifier Result
Modifier Result
Modifier Result

Out-of-Distribution Directions

Our approach effectively handles unseen directional inputs, capturing force-language relationships even for novel movement directions.

Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result
Direction Result