Think Aloud on Cultural Intelligence
In this project, I used a combination of quantitative and qualitative analysis, with think-aloud protocol as the key method, to study the effectiveness of the instrument Cultural Intelligence Scale(CQS). The research also helped identify potential opportunities of improvement for the cultural intelligence(CQ) development of CMU students.
Context
The Core Competencies Initiative(CCI) project of Carnegie Mellon University aims to study a set of essential capabilities important across disciplines and professions. For now, there are four core areas identified and Intercultural & Global Learning + DEI is one of them. The CCI project group found an instrument called Cultural Intelligence Scale(CQS) developed by Earley, Ang, and Van Dyne in 2015, which has been proved effective across cultures. Therefore, they considered using this instrument to keep track of the Cultural Intelligence(CQ) development of CMU students and improve CMU’s effort on their CQ development. However, though it has been proven effective elsewhere, the project group wanted to make sure that the instrument was effective for CMU students. When I part-timed as a project assistant from January to May 2023 for CCI, I was assigned the task of analyzing the pilot assessment data and understanding the effectiveness of the instrument by conducting think-aloud interviews.
Pilot Assessment Data Analysis: Students May be Confused by “Marriage System” & “Nonverbal Behavior”
Before I worked as the CCI project assistant, there was a pilot assessment conducted with 228 undergraduate CMU students from Dietrich College and Mellon College of Science. I was handed over the data and did in-depth analysis on the individuals of the 10% highest and lowest CQ scores. I also analyzed the mean CQ scores of each item of CQS.
Through the quantitative analysis on the pilot assessment, it was found that the items “I know the marriage systems of other cultures.” and “ know the rules for expressing nonverbal behaviors in other cultures.” have relatively low mean CQ scores, while “I enjoy interacting with people from different cultures.” have the highest CQ score for both colleges. Meanwhile, the mean CQ score distribution showed that while it largely fitted normal distribution, people tended to rate themselves high. Therefore, the data revealed that there might be issues with the CQS when it was applied on CMU students.
30 Think-Aloud Interviews: 4 Insights for Improving The CQS & Encouraging CQ Development of CMU Students
Think-Aloud Interviews
I conducted 30 one-hour think-aloud interviews with students across different levels(undergraduate, graduate and PhD students) and disciplines (from School of Fine Art, School of Engineering, School of Computer Science, Tepper School of Business). During the think-aloud interview, I prepared them for the think-aloud by explaining what it meant to think aloud and showing them an example of think aloud. They were then asked to complete the CQS first and think aloud on each of the questions. Follow-up questions regarding stand-out questions, scale, general perception and other aspects were asked at the end.
Interview Data Analysis
Code The Data: To conduct qualitative analysis on the think-aloud data, I set up a series of categories that were mentioned frequently in the interview and the part of the transcript fitted into the category was coded accordingly.
Rich-experienced vs. Limited-experienced: Participants with more intercultural experiences were compared with those with less experiences. The classification of rich-experienced and limited-experienced participants was based on self-reported data obtained through the interview.
Comparisons Across Levels & Disciplines: Participants across different levels and disciplines were compared.
Analysis on Some Categories: Some codes were analyzed in depth to better understand opportunities for improvement that can be influenced by CMU.
Insights
01: Concrete experiences situated in cross-cultural settings and interactions with people from different cultural backgrounds contribute to development of cultural intelligence.
A. Rich-experienced participants tend to have higher average CQ scores than limited-experienced participants. 2/3 of the randomly selected rich-experienced students have average CQ scores higher than 2/3 of the selected limited-experienced students.
B. Rich-experienced students tend to answer questions by reflecting on past experiences, while limited-experienced students rely more on prediction. 2/3 of the rich-experienced students have highest frequency of mention on “Family & Ethnicity”, while 2/3 of the limited-experienced students have highest frequency on “Predicted Scenarios”.
C. Richer experiences do not necessarily lead to higher CQ scores. For example, despite reflecting on similar experiences on a question, PE03 interprets that the experience has provided him with ample knowledge so he rates himself 5, while PCS27 thinks it only gives him a high-leveled understanding and thus rates himself 3.
02: The current CQS can lead to confusion and ineffective distinction between individuals of different cultural intelligence.
A. Many participants find the Q15 “I am confident that I can get accustomed to the shopping conditions in a different culture” confusing. 7/30 participants (23%) point out that Q15 appears confusing to them.
B. Some participants find Q3 “I am conscious of the cultural knowledge I apply to cross-cultural interactions” confusing. 6/30 participants (20%) point out Q3 is unclear.
C. Too much room for interpretation may lead to ineffective at distinguishing different CQ levels. For example, while PE03 and PCS27 reflect on similar experiences, the different interpretations of the knowledge they obtain from the interaction lead them to rate themselves differently on a question.
03: A diverse CMU community and in-depth study in different cultural knowledge contribute to the development of cultural intelligence of the students.
A. The diverse community helps students learn more about different cultures and how to adjust themselves in different scenarios. 20/30 participants (66.7%) have input in “School-Indirect” and most of them talk about how they are able to learn from people of different cultures by talking and listening to them when they are in CMU.
B. In-depth research and classes help students get more knowledge and develop their cultural intelligence. The average CQ score of art students is higher than science students, and one probable reason is their higher score at Q9 “I know the arts and crafts of other cultures.” For example, an art student rates herself high for Q9 because “I do a lot of research into like different cultural artists…”
04: The activity of thinking aloud and reflecting on past intercultural experiences helps students get more understanding about intercultural interactions and what to do in future.
14/30 (46.7%) participants explicitly state that the survey and probably the think-aloud activity help them to recall their experiences and think about the topics they have never considered before.
Reflection
The key takeaway I had from this project is the method of combining quantitative and qualitative analysis in research, especially using quantitative analysis to direct qualitative research. The analysis of the pilot assessment data revealed some problems of the instrument that we could look into, which helped frame the following qualitative research. The think-aloud interviews then revealed the reasons behind the issues identified in the early stages.
Besides the learning on the research techniques, I was very inspired by the insights from this research on Cultural Intelligence development. It was impactful that many students found the diverse classmates and friend groups around them helped them learn more about other cultures and how to interact with people from different cultural background. This insight has become an important insight that informed student recruitment of CMU. It is such a pleasure to see that the result of this research I conducted may bring positive impact to the diversity, inclusion and Cultural Intelligence development of CMU.