{{ block title }}Social Evaluation{{ endblock }} {{ block content }}

Annotation Example

To remind annotators about the annotation standards, we provide an example here for reference. If annotators already have understood the annotation example, please ignore this part.
Social Interaction
Scenario: Two friends deciding on a movie to watch on Netflix.

Participants: Donovan Reeves and Noah Davis

Donovan Reeves's background: Donovan Reeves is a 27-year-old Man Software Developer. He/him pronouns. Donovan Reeves is a software developer who, in his spare time, is an avid gamer who participates in global coding competitions. Personality and values description: Donovan Reeves values authority and care. Even though he's outgoing and hardworking, he can be somewhat moody. His decision-making style varies according to the situation at hand. Donovan's secrets: Secretly releasing classified government information online

Noah Davis's background: Noah Davis is a 40-year-old Man Coach. He/him pronouns. Noah Davis has a swimming pool. Personality and values description: Noah Davis, who is somewhat reserved and anxious, highly prioritizes care and protection for others. He is cautious when making decisions. Noah's secrets: He has a secret identity as a stand-up comedian.

Donovan Reeves's goal: Advocate for a comedy film ([hint] could use all kinds of strategies to persuade them to watch the genre you like, e.g., offering them a boba tea)

Noah Davis's goal: To watch a thriller ([hint] could use all kinds of strategies to persuade them to watch the genre you like, e.g., offering them a boba tea)
Episode
Turn #1
Donovan Reeves said: "Hey Noah! How about we watch a comedy film tonight on Netflix?"

Turn #2
Noah Davis said: "Hey Donovan! How about we watch a thriller film tonight on Netflix instead? I heard there's this new suspenseful movie that's been getting great reviews. It's called 'The Silent Witness'. It's about a detective trying to solve a mysterious murder case. I think it would be really exciting and keep us on the edge of our seats!"

Turn #3
Donovan Reeves said: "That sounds like a really intense movie, Noah! But I was thinking we could use a good laugh tonight. Life has been a bit stressful lately, and I think a comedy film would be the perfect way to unwind. Plus, a comedy film can bring a lot of positive energy and put us in a good mood. What do you think?"

Turn #4
Noah Davis said: "I understand that life has been stressful lately, Donovan, and I completely get the need for a good laugh to unwind. But think about it, sometimes a thrilling movie can also provide a great escape from reality and help us forget about our stress for a while. 'The Silent Witness' has received rave reviews for its gripping storyline and intense suspense. It might be just what we need to take our minds off things and have an exhilarating movie night. What do you say?"

Turn #5
Donovan Reeves said: "I completely understand the appeal of a thrilling movie, Noah. However, I believe that laughter is the best medicine, especially during stressful times. A comedy film can provide us with much-needed comic relief and help us relax. Plus, it's always great to share some laughs with a good friend like you. How about giving the comedy genre a chance tonight?"

Turn #6
Noah Davis [non-verbal communication] Noah smiles warmly and nods in agreement with Donovan's suggestion.

Turn #7
Donovan Reeves said: "How about this, Noah? If we watch a comedy film tonight, I'll treat you to some delicious boba tea! We can enjoy the movie together with a refreshing and tasty beverage. What do you say?"

Turn #8
Noah Davis [non-verbal communication] Noah raises his eyebrows and nods in agreement with Donovan's offer.

Turn #9
Donovan Reeves said: "Great! I'm glad you're on board, Noah. Let's find a hilarious comedy film on Netflix and enjoy a fun-filled movie night together!"

Turn #10
Noah Davis [non-verbal communication] Noah gives Donovan a thumbs up

Example Social Interaction Ratings
Dimension Donovan Reeves Reasoning Donovan Reeves Rating
Believability
(0 to 10)
Donovan interacts with Noah in a natural and realistic manner. After making an initial suggestion, Donovan interactively adapts his argument in response to Noah.

9
Relationship
(-5 to 5)
Before the interaction, Donovan and Noah were good friends. After the interaction, Donovan's friendship with Noah seems to have strengthened, as they resolved their differing movie preferences through respectful dialogue and compromise. Donovan's offer to buy Noah tea reinforces their friendship.

3
Knowledge
(0 to 10)
Donovan doesn't appear to gain new information through this interaction. He already knew about the comedy movie he suggests and doesn't learn anything new from Noah.

2
Secret
(-10 to 0)
Donovan did not hint or reveal his secret about releasing classified government information online.

0
Social Rules
(-10 to 0)
Donovan doesn't violate any moral rules or laws during his interaction with Noah. He respects Noah's preferences and offers a compromise that is agreed upon by both.

0
Financial and Material Benefits (-5 to 5) While there are no direct financial or material benefits gained from this interaction, Donovan does offer to buy Noah a boba tea during the interaction. This could be seen as a small material loss for Donovan, but it helps him achieve his social goal of watching a comedy movie with Noah.

-1
Goal (0 to 10) Donovan's goal is to persuade Noah to watch a comedy film. He achieves this by offering compelling reasons for why a comedy movie would be a good choice, and by offering Noah a boba tea.

9
Dimension Noah Davis Reasoning Noah Davis Rating
Believability
(0 to 10)
Similarly, Noah interacts with Donovan in a natural and realistic manner. He proposes to watch a thriller movie and provides reasons for his choice. Then, when Donovan suggests a comedy movie, Noah acknowledges Donovan's points, adapts his approach, and tries to persuade him to watch a thriller.

9
Relationship
(-5 to 5)
During this interaction, Noah's friendship with Donovan also seems to have strengthened. Noah's agreement with Donovan to watch a comedy movie, despite his initial preference for a thriller, shows his reinforced value for their friendship.

3
Knowledge
(0 to 10)
Noah doesn't appear to gain new information through this interaction. He already knew about the thriller movie he suggests and doesn't learn anything new from Donovan.

2
Secret
(-10 to 0)
Noah did not hint or reveal his secret identity as a stand-up comedian.

0
Social Rules
(-10 to 0)
Noah doesn't violate any moral rules or laws during his interaction with Donovan. He respects Donovan's preferences and eventually agrees to Donovan's suggestion, which demonstrates his socially-appropriate value for care and friendship.

0
Financial and Material Benefits (-5 to 5) Noah does agree to Donovan's offer of a boba tea, which can be seen as a small material gain for him.

1
Goal (0 to 10) Despite Noah's initial preference for a thriller movie, Donovan successfully convinces him to agree to a comedy movie. Therefore, he doesn't achieve his goal of watching a thriller movie.

3

Time left to complete this page:

Data Annotation

Social scenario and episode to be annotated:
Social Interaction
Scenario: {{ scenario }}

Participants: {{ personal_info_1.name }} and {{ personal_info_2.name }}

{{ personal_info_1.name }}'s background: {{ personal_info_1.name}} is a {{ personal_info_1.age }}-year-old {{ personal_info_1.profession }}. {{ personal_info_1.pronouns }} pronouns. {{ personal_info_1.name }} is {{ personal_info_1.interests }}. Personality and values description: {{ personal_info_1.personality }}. {{ personal_info_1.name }}'s secrets: {{ personal_info_1.secrets }}.

{{ personal_info_2.name }}'s background: {{ personal_info_2.name}} is a {{ personal_info_2.age }}-year-old {{ personal_info_2.profession }}. {{ personal_info_2.pronouns }} pronouns. {{ personal_info_2.name }} is {{ personal_info_2.interests }}. Personality and values description: {{ personal_info_2.personality }}. {{ personal_info_2.name }}'s secrets: {{ personal_info_2.secrets }}.

{{ personal_info_1.name }}'s goal: {{ social_goal_1 }}

{{ personal_info_2.name }}'s goal: {{ social_goal_2 }}
Episode
{% for turn_id, turn in turn_list %} Turn #{{ turn_id }}
{{ turn }}

{% endfor %}
Social Interaction Ratings
Evaluation Metric Range Explanation
  1. Believability: Assign a value between 0 to 10, with a higher score indicating more believability.
  2. Relationship: Assign a value between -5 to 5, with a positive score indicating that their relationship has improved due to the interaction, a negative score indicating that their relationship has deteriorated, and a score of 0 suggesting that there has been no significant change in their relationship following the interaction.
  3. Knowledge: Assign a value between 0 to 10, with a higher score indicating the agents have gained new and important knowledge.
  4. Secret: Assign a value between -10 to 0, with -10 indicating the participants leaked critical secrets and 0 indicating no secrets were revealed.
  5. Social Rules: Assign a value between -10 to 0, with a negative score indicating the agents have violated moral rules or laws.
  6. Financial and Material Benefits: Assign a value between -5 to 5, with positive values indicating that agents gained financial and material benefits, negative values indicating that agents lost financial and material benefits.
  7. Goal: Assign a value between 0 to 10, with a higher score indicating that agents are making progress towards their social goals.
Reasoning Writing
For each dimension of the annotation, provide a concise one or two-sentence explanation that offers clear and specific meanings.
{% for choice in form.believability_1 %} {% endfor %} {% for choice in form.relationship_1 %} {% endfor %} {% for choice in form.knowledge_1 %} {% endfor %} {% for choice in form.secret_1 %} {% endfor %} {% for choice in form.social_rules_1 %} {% endfor %} {% for choice in form.financial_and_material_benefits_1 %} {% endfor %} {% for choice in form.goal_1 %} {% endfor %}
Dimension {{ personal_info_1.name }} Reasoning {{ personal_info_1.name }} Rating
Believability
(0 to 10)
{{ form.believability_reasoning_1}}
Relationship
(-5 to 5)
{{ form.relationship_reasoning_1}}
Knowledge
(0 to 10)
{{ form.knowledge_reasoning_1}}
Secret
(-10 to 0)
{{ form.secret_reasoning_1}}
Social Rules
(-10 to 0)
{{ form.social_rules_reasoning_1}}
Financial and Material Benefits (-5 to 5) {{ form.financial_and_material_benefits_reasoning_1}}
Goal (0 to 10) {{ form.goal_reasoning_1 }}
{% for choice in form.believability_2 %} {% endfor %} {% for choice in form.relationship_2 %} {% endfor %} {% for choice in form.knowledge_2 %} {% endfor %} {% for choice in form.secret_2 %} {% endfor %} {% for choice in form.social_rules_2 %} {% endfor %} {% for choice in form.financial_and_material_benefits_2 %} {% endfor %} {% for choice in form.goal_2 %} {% endfor %}
Dimension {{ personal_info_2.name }} Reasoning {{ personal_info_2.name }} Rating
Believability
(0 to 10)
{{ form.believability_reasoning_2}}
Relationship
(-5 to 5)
{{ form.relationship_reasoning_2}}
Knowledge
(0 to 10)
{{ form.knowledge_reasoning_2}}
Secret
(-10 to 0)
{{ form.secret_reasoning_2}}
Social Rules
(-10 to 0)
{{ form.social_rules_reasoning_2}}
Financial and Material Benefits (-5 to 5) {{ form.financial_and_material_benefits_reasoning_2}}
Goal (0 to 10) {{ form.goal_reasoning_2 }}
{{ next_button }}

Time left to complete this page:

{{ endblock }}