Conversation Dynamics Similarity (ConDynS)

ConDynS is a similarity measure for comparing conversations with respect to their dynamics, as introduced in the paper “A Similarity Measure for Comparing Conversational Dynamics”. The quality of a conversation goes beyond the individual quality of each reply, and instead emerges from how these combine into interactional patterns that give the conversation its distinctive overall “shape”. ConDynS provides a robust automated method for comparing conversations in terms of their overall interactional dynamics.

In this module, we provide a comprehensive framework for computing ConDynS, including:

• ConDynS: Main similarity computation using bidirectional comparison between SCD patterns and conversation transcripts

• NaiveConDynS: Simplified similarity computation using only SoP comparison without transcripts

• ConDynSBaselines: Baseline methods for comparison including BERTScore, cosine similarity, and LLM-based direct comparison

ConDynS builds on top of the SCD (Summary of Conversation Dynamics) module, which generates structured summaries of conversation dynamics. To compute ConDynS, you first need to extract SCD summaries from your conversations using the SCD transformer.

Note that ConDynS computation requires access to a LLM. We provide a unified interface for working with LLMs in the GenAI module. It is recommended to setup for GenAI models in the module beforehand to compute ConDynS.

Usage Examples

We provide experiments notebooks from the paper:

• Validation experiments

• Baseline comparisons

• Applications to online communities

• Applications on WikiConv German and Friends

To see a simple example of using both SCD and ConDynS together, check out this example notebook.

Modules

ConDynS

class convokit.convo_similarity.condyns.ConDynS(model_provider: str, config, model: str = None, custom_condyns_prompt: str = None, custom_prompt_dir: str = None)

A class to compute ConDynS score between conversations.

ConDynS computes similarity scores between conversations by analyzing their Summary of Conversation Dynamics (SCD) patterns, which are extracted from the SCD as the Sequence of Patterns (SoP), and comparing them with conversation transcripts. The method uses bidirectional similarity computation to capture the full dynamics of both conversations.

Parameters

• model_provider – The LLM provider to use (e.g., “gpt”, “gemini”)

• config – The GenAIConfigManager instance to use

• model – Optional specific model name

• custom_condyns_prompt – Custom prompt for the condyns prompt template

• custom_prompt_dir – Directory to save custom prompts (if not provided, overwrites default prompts in ./prompts)

compare_conversations(corpus, convo_id1: str, convo_id2: str, sop_meta_name: str, formatter=None)

Compare two conversations using ConDynS and store the result in both conversations’ metadata.

This method retrieves two conversations from the corpus, formats them into transcripts, extracts their SoP data from metadata, computes the ConDynS score between them, and stores the result in both conversations’ metadata with the key format “condyns_{convo_id1}_{convo_id2}”.

Parameters

• corpus – The ConvoKit Corpus containing the conversations

• convo_id1 – ID of the first conversation

• convo_id2 – ID of the second conversation

• sop_meta_name – Name of the metadata field containing SoP data

• formatter – Optional custom formatter function that takes a Conversation object and returns a transcript string. If None, uses the default formatter.

Returns

The computed ConDynS score

Raises

• KeyError – If conversations don’t exist or required metadata is missing

• ValueError – If SoP data is malformed

• TypeError – If custom formatter is not callable

compute_bidirectional_similarity(transcript1, transcript2, sop1, sop2)

Compute bidirectional similarity between two conversations.

Computes similarity in both directions: SoP1 vs Transcript2 and SoP2 vs Transcript1 to capture the full dynamics of both conversations.

Parameters

• transcript1 – First conversation transcript

• transcript2 – Second conversation transcript

• sop1 – SoP for first conversation

• sop2 – SoP for second conversation

Returns

List of [response_dict1, response_dict2] where each dict contains analysis and scores for each event

compute_score_from_results(results)

Compute scores from bidirectional similarity results.

Parameters

results – List of bidirectional similarity results

Returns

List of mean scores for each direction

compute_unidirectional_similarity(sop1, transcript2)

Compute unidirectional similarity between SoPs and a transcript.

Analyzes how well the SoPs from one conversation match the dynamics observed in another conversation’s transcript.

Parameters

• sop1 – SoPs from the first conversation

• transcript2 – Conversation transcript from the second conversation

Returns

Dictionary with analysis and scores for each event in sop1

get_condyns_score(transcript1, transcript2, sop1, sop2)

Compute ConDynS score between two conversations.

Computes ConDynS with the bidirectional similarity between two conversations using their transcripts and SoPs, then returns the mean score.

Parameters

• transcript1 – First conversation transcript

• transcript2 – Second conversation transcript

• sop1 – SoP for first conversation

• sop2 – SoP for second conversation

Returns

ConDynS score

load_custom_prompts_from_directory(prompt_dir: str)

Load custom prompts from a specified directory.

Parameters

prompt_dir – Directory containing custom prompt files

measure_score(data)

Calculate the mean score from a similarity result dictionary.

Parameters

data – Dictionary containing similarity analysis results

Returns

Mean score across all events

set_custom_condyns_prompt(prompt_text: str, save_to_file: bool = True)

Set a custom condyns prompt template.

Parameters

• prompt_text – The custom prompt text

• save_to_file – Whether to save the prompt to file in custom_prompt_dir or default prompts directory

NaiveConDynS

class convokit.convo_similarity.naive_condyns.NaiveConDynS(model_provider: str, config, model: str = None, custom_naive_condyns_prompt: str = None, custom_prompt_dir: str = None)

A class to compute naive ConDynS score between two Sequences of Patterns.

NaiveConDynS computes similarity scores between conversations by directly comparing their Sequences of Patterns (SoP) without using conversation transcripts. This provides a simpler approach to measuring conversation dynamics similarity.

Parameters

• model_provider – The LLM provider to use (e.g., “gpt”, “gemini”)

• config – The GenAIConfigManager instance to use

• model – Optional specific model name

• custom_naive_condyns_prompt – Custom prompt for the naive condyns prompt template

• custom_prompt_dir – Directory to save custom prompts (if not provided, overwrites defaults in ./prompts)

compare_conversations(corpus, convo_id1: str, convo_id2: str, sop_meta_name: str)

Compare two conversations using NaiveConDynS and store the result in both conversations’ metadata.

This method retrieves two conversations from the corpus, extracts their SoP data from metadata, computes the NaiveConDynS score between them, and stores the result in both conversations’ metadata with the key format “condyns_{convo_id1}_{convo_id2}”.

Note: NaiveConDynS only uses SoP data for comparison, not conversation transcripts.

Parameters

• corpus – The ConvoKit Corpus containing the conversations

• convo_id1 – ID of the first conversation

• convo_id2 – ID of the second conversation

• sop_meta_name – Name of the metadata field containing SoP data

Returns

The computed NaiveConDynS score

Raises

• KeyError – If conversations don’t exist or required metadata is missing

• ValueError – If SoP data is malformed

compute_bidirectional_naive_condyns(sop1, sop2)

Compute bidirectional naive conditional dynamics similarity between two Sequences of Patterns.

Computes similarity in both directions: sop1 vs sop2 and sop2 vs sop1 to capture the full dynamics of both conversations.

Parameters

• sop1 – SoP from the first conversation

• sop2 – SoP from the second conversation

Returns

List of [response_dict1, response_dict2] where each dict contains analysis and scores for each pattern

compute_score_from_results(results)

Compute scores from bidirectional similarity results.

Parameters

results – List of bidirectional similarity results

Returns

List of mean scores for each direction

compute_unidirectional_naive_condyns(sop1, sop2)

Compute unidirectional naive conditional dynamics similarity between two Sequences of Patterns.

Compares the SoPs from one conversation against another to measure how well the dynamics of one conversation match those of another.

Parameters

• sop1 – SoP from the first conversation

• sop2 – SoP from the second conversation

Returns

Dictionary with analysis and scores for each pattern in sop1

load_custom_prompts_from_directory(prompt_dir: str)

Load custom prompts from a specified directory.

Parameters

prompt_dir – Directory containing custom prompt files

measure_score(data)

Calculate the mean score from a similarity result dictionary.

Parameters

data – Dictionary containing similarity analysis results

Returns

Mean score across all patterns

set_custom_naive_condyns_prompt(prompt_text: str, save_to_file: bool = True)

Set a custom naive condyns prompt template.

Parameters

• prompt_text – The custom prompt text

• save_to_file – Whether to save the prompt to file in custom_prompt_dir or default prompts directory

Baseline Methods

class convokit.convo_similarity.baseline.ConDynSBaselines(model_provider: str, config, model: str = None, sentence_transformer_model: str = 'all-MiniLM-L6-v2', device: str = 'cpu')

A class providing baseline methods for computing conversation similarity to compare with ConDynS.

This class provides various baseline methods for comparing conversations including BERTScore, cosine similarity using sentence embeddings, and GPT-based comparison methods. The baseline methods are used to compare with ConDynS.

Parameters

• model_provider – The GenAI provider to use (e.g., “gpt”, “gemini”)

• model – Optional specific model name

• sentence_transformer_model – Sentence transformer model to use for embeddings (default: “all-MiniLM-L6-v2”)

• device – Device to use for sentence transformer (default: “cpu”)

get_bertscore(pred, ref)

Compute BERTScore between prediction and reference texts.

Uses the BERTScore metric to evaluate semantic similarity between two texts.

Parameters

• pred – Prediction text to evaluate

• ref – Reference text to compare against

Returns

BERTScore computation results

get_cosine_similarity(pred, ref)

Compute cosine similarity between two texts using sentence embeddings.

Uses the SentenceTransformer model to generate embeddings and computes cosine similarity between them.

Parameters

• pred – First text for comparison

• ref – Second text for comparison

Returns

Cosine similarity score between 0 and 1

get_gpt_compare_score(pred, ref, prompt)

Compare two texts using GPT model with a custom prompt.

Sends a formatted prompt to GPT model to compare two texts and returns similarity score and reasoning.

Parameters

• pred – First text for comparison

• ref – Second text for comparison

• prompt – Prompt template to use for comparison

Returns

Tuple of (similarity_score, reasoning)

get_naive_gpt_compare_score_SCDs(scd1, scd2)

Compare two Summary of Conversation Dynamics (SCD) using GPT.

Compares two SCD summaries and rates their similarity based on persuasion trajectory and conversational dynamics, ignoring specific topics or claims.

Parameters

• scd1 – First SCD summary

• scd2 – Second SCD summary

Returns

Tuple of (similarity_score, reasoning)

get_naive_gpt_compare_score_Transcripts(transcript1, transcript2)

Compare two conversation transcripts using GPT.

Compares two conversation transcripts and rates their similarity based on conversational trajectory and dynamics, ignoring specific topics discussed.

Parameters

• transcript1 – First conversation transcript

• transcript2 – Second conversation transcript

Returns

Tuple of (similarity_score, reasoning)