Evaluation Method Deep Dive, Tips for Practice, Steps, Reflections on Practice
"Sometimes, we have to break down to break through." - Gina Greenlee
Great evaluators know that findings are never black or white. At the best of times, a program, project, intervention, or policy is likely a blend of wins and fails, wisdom and wishful thinking, and good intentions and oversights. That's because the people who created or experienced the initiative being evaluated are human. Humans are a hot mess! Even when properly hydrated, we contradict ourselves, hold multiple conflicting truths, and tell ourselves all kinds of stories that change from one moment to the next. It's a lot to keep track of!
Often, evaluators try to constrain the human factor. Like squeezing into a girdle, evaluators squish people into ill-fitting scales and boxes. But, collage - the break down to break through technique - lets it all out. Phew! Collage helps us take a deep breath, tap into multisensory knowledge, and embrace the mess. This is an asset in evaluation when we want to understand the nuance, complexity, and multi-dimensional nature of a project or experience. This blog post explores:
Plus, we sprinkle in hot tips from our practice along the way. Let's dive in!
How does collage work as a data collection tool?
Like all arts-based methods in evaluation, the art-making activity elicits qualitative data that can be analyzed and used to understand project impacts and inform recommendations. In this case, an individual or group creates a collage by assembling and reassembling specific materials in response to a prompt, which helps to answer the evaluation question.
Then, the individual or group unpacks their collage and the collage process in an interview or focus group with the evaluator.
Check out this timelapse to see me make a 30-minute collage in 40 seconds.
Always starting with the evaluation question, in our practice Jennica and I:
Generate a prompt for participants to collage toward or in response to - this should help us answer the evaluation question, but is not the evaluation question (e.g., a key concept, central experience, or tension that needs unpacking)
Facilitate or assign a collage activity for individuals or groups to play with pre-selected or their own curated materials (this could be a synchronous or asynchronous collage session)
Unpack the collage activity and/or finished piece (depending on the evaluation question and activity's purpose) with a qualitative debrief (e.g., a focus group or individual interviews)
Analyze the qualitative data that resulted from the conversation (not the collage)
Use the findings to inform the next steps (e.g., decision-making, further reflection, recommendations)
Before trying any activities with clients or participants, I ALWAYS practice the prompt, activity, and debrief. If it doesn’t work with Jennica or my mom, it won’t work with my clients!
When is collage a good idea in evaluation?
Collage embraces messiness, layering and clustering, and transformation or subversion. As a data collection strategy, it works best when the evaluation questions are also concerned with multi-layered, messy, contradictory, or evolving perspectives. When a project or initiative is complex, collage can be a helpful tool to embrace - rather than fight - the wisdom of mess.
For example, collage is great when:
The evaluation aims to render the invisible visible or articulate something previously unspoken (e.g., social or group norms, power dynamics, gut wisdom or intuition, underlying values or assumptions)
The evaluation requires a deeper understanding of nuanced relationships and (dis)connections (e.g., a project looking at interpersonal dynamics, relationships between people and their environments/contexts, and/or shifting dynamics over time or across spaces)
The project, program, or initiative being evaluated has a rocky past or trajectory that needs to be grappled with (e.g., if relationships need mending, we need to build a bridge between where we’ve been and where we’re going, or we need to un-make or un-do before fixing and doing over)
The wisdom required to answer the evaluation question sits in the body, not the head (e.g., multi-sensory wisdom needs to be evoked through multi-sensory, cathartic, and playful activities rather than questions alone)
There is a need to (re)imagine what is or what could be (e.g., when planning for an unknown future, attempting to move from where you are to where you want to be, and/or dreaming about something that does not exist yet)
Hot tip: Jennica and I used collage in our annual strategic planning meeting to imagine the next year of our consulting practice. This process helped us articulate important business priorities that we had never talked about because we didn't have the language.
How do I use collage in an evaluation?
Starting from your evaluation question, identify a foundational component of your evaluation that you need to unpack or grapple with (e.g., a buzzword or undefined concept central to the initiative being evaluated; a complex human experience related to the evaluation; values, beliefs, mis/understandings about the project being evaluated; and/or expected outcomes or impacts).
Craft a prompt for participants to collage toward or in response to that helps you answer your evaluation question - but is not your evaluation question. Hot Tip: Use a strengths-based question to help mitigate the risk of unwanted emotions bubbling up during the activity or qualitative debrief. And, consider using a prompt that helps folks tap into embodied or multi-sensory wisdom so that the form and function of the collage activity match.
Consider the supplies you need and how to provide them.
The evaluator can provide collage materials so everyone starts with a uniform package (e.g., if it is important to highlight similarities/differences in interpretation from a similar starting point)
The participants can bring their own materials so everyone's bundles reflect their interpretation of the prompt and/or spaces they have access to (e.g., if it is important to also delve into reflections about materials used, where they came from, and why they became part of the collage or got left on the table)
The evaluator and participants can collect materials together as a precursor to the collage activity (e.g., when looking at place-based experiences or as part of a broader mapping or exploration of space)
Decide on a format (e.g., synchronous or asynchronous, in-person or online) and set a time limit for the collage to make the experience manageable, not overwhelming.
Hot Tip: The time limit should convey that this is not meant as a polished work of art, an assignment that will be graded, or anything else that might cause stress. For example, I often limit time to 30-60 minutes, depending on the context.
Debrief the collages (and choice of materials) with participants either individually through an interview (if you want to dive deep into one person’s insights or account for vulnerability/possible power dynamics in a group) or in a focus group (if you want to compare and contrast interpretations, build consensus, or explore diversity in experiences). In the debrief, explore the process (e.g., what it felt like to do the activity) and the product (e.g., what the collage means).
Analyze your qualitative data summary or transcript from the debrief as you would with any other qualitative data. Resist the urge to analyze the art itself or make assumptions about what the finished pieces mean. Art does not speak for itself!
Lefkowich, 2023. Annual Strategic Planning Collage. For example, here is a collage I made during my annual strategic planning meeting for AND implementation (because we practice what we preach). The prompt was to create a collage representing the ideal next year for our company.
From looking at my collage, can you answer the following questions?
Why did I use certain materials (or leave them on the table)?
What does my collage say about my vision for the best possible year ahead for AND implementation?
Am I using the words and images
Literally
For lack of a better choice (e.g., a stand-in for what I really mean)?
Subversively, ironically, or playfully (e.g., in opposition to the meaning in the source materials)?
Who is the audience for this piece? (Often, evaluators think they are the audience because they created the prompt and facilitated the activity. But, evaluators may not be granted access to the many layers of meaning infused into the piece).
Decide how and by whom the collages will be used. At AND, Jennica and I always give the collages back to participants and create options for public dissemination so they can individually opt in or out.
When I do share collages broadly, it is helpful to consider the format. For instance, collages do not photograph well. As you can see from the photo and video above, flat snapshots miss the three-dimensional and multi-textural elements. So, photos are used sparingly or as a second choice if a gallery tour is not possible. I also include artist statements or short captions written by the participant to capture the essence of their piece so others do not misinterpret it.
What are ethical considerations to keep in mind?
Ethics are essential to the success of all good arts-based methods. Here are some helpful questions when planning a collage exercise in evaluation.
How will consent work for participation? (e.g., What process will you follow to get consent from participants? What happens if people change their minds about participation?)
How will you be thoughtful about potential risks? (e.g., Because collage is a multi-sensory experience, what happens if it surfaces unexpected, unpleasant, or unwanted sensations for participants? How will you ensure their emotional safety?)
How will consent work for sharing collages in a group debrief or in any final deliverables? (e.g., Because someone can only consent to sharing their collage after making it, how will you plan for this? If you provided existing texts/images for the exercise, is there a copyright or license issue to be aware of?)
Who owns the collage and the resulting qualitative data? (e.g., Will there be limits on how collages can be used after/by whom? Are there existing data ownership and sovereignty protocols or policies for this group or community? )
What existing materials are appropriate for you to engage with in this way? (e.g., Are any materials you selected sacred or inappropriate to mark up, cut, or distort? How will you be mindful of representation in the source materials so folks can see themselves and their experiences in the text/images provided or gathered? How will you navigate any "oops, ouch" moments should someone come across a derogatory or harmful representation in a magazine, article, pamphlet, or report included in the supplies?)
If you are gathering texts from a specific place to represent an environment or context, do you need permission to access or remove items from a particular place?
If you are instructing participants to gather their own materials, do you need any additional instructions or precautions to ensure that they don't cause harm to others (e.g., I always include the instruction not to pick or take anything living or from a living creature. So, they can take a fallen leaf or petals, but they cannot pick anything off a tree or plant).
What are common mistakes to avoid when using collage in evaluation?
Despite its seemingly playful and simplistic form, collage is one of the more difficult art techniques I have used in evaluation. And, I have seen many evaluators (and researchers) make big mistakes when facilitating. Here are a few stumbling points Jennica and I have observed and how we mitigate potential mistakes.
Collage takes time, careful prep work, and materials While not all arts-based methods require more time and resources, collage does. The evaluator needs to budget time and money for thoughtful planning. This includes consulting or partnering with a collage artist (if they do not have familiarity with this art form), creating a thoughtful prompt and designing an interview/focus group guide that attends to the nuance and complexity of both (a) the art piece/process and (b) the evaluation themes and questions, gathering materials or planning for participants to gather materials (and coming up with a back-up plan in case this process fails), ensuring that the activity is inclusive and accessible for folks with diverse needs and realities, and identifying a consensual and meaningful way to use any data and art pieces that come from this activity.
Hot Tip: Jennica and I persistently root through our neighbours' recycling bins for magazines and
newspapers, search our Buy Nothing neighbourhood Facebook groups for helpful scraps, and hoard scissors and glue sticks. Everything we find throughout the year goes in a bin under my bed, so we are always collage-ready. This helps to (a) ensure that we aren't scrambling at the last minute to find materials, (b) keep costs low because we have a surplus of found materials, and (c) keep our environmental footprint reasonable since we are upcycling.
Collage isn't always appropriate or wanted The magic of collage lies in its ability to help folks deconstruct and reconstruct. Breaking down to build up can be a powerful and meaningful activity for some people, in some contexts, and for some topics. Sometimes, this can be a disrespectful, damaging, or awkward invitiation. Consider the materials. When is it appropriate to rip, scrunch, crumble, layer, stack, cut, glue, cover, or disregard the source materials? Or in relation to a particular experience or topic? Or with others (potentially)? A strong culture of care must accompany any data collection - especially in a facilitated activity where the goal is to transform to make meaning. Not everything can and should be transformed. And, in some cases, the evaluator and participants may not have the authority or consent to do so (e.g., cultural, religious, or sacred texts, family records or photo albums, original documents). When in doubt, don't.
Collage evokes BIG FEELS Some evaluators make the mistake of thinking that because collage is something they did as children, it is easy, light, silly, and fun. While it can be all those things, it is also a powerful tool for tapping into multi-sensory wisdom, contradiction, and unspoken or unarticulated truths. This makes the range of emotions that can surface during collage vast and predictably unpredictable. For example, I often ask about the best parts of an experience in a collage prompt. Nevertheless, most participants will grapple with the worst parts to make sense of the best. Negative representations show up in the collage and follow-up interviews because this art form invites people to compare/contrast, show movement between states or experiences, and transform or layer good and bad elements. Even when I ask folks not to delve into sensitive, negative, or hard topics, they still do. So, I prepare for the activity and interview by anticipating that some folks will do some emotional heavy lifting - even if I hope the experience is cathartic and sparks joy. Once again, a culture of care is critical. If the evaluation topic is already sensitive or requires significant vulnerability or risk to discuss, collage is probably not a good choice.
The facilitator's skills will make or break the whole process Like any qualitative or arts-based technique, the skills of the person or people facilitating the art-making activity and qualitative debrief - not the quality of the art - will determine the success of this process. Unfortunately, evaluation as a field does not have a brilliant reputation for nurturing or valuing qualitative or creative competencies on par with quantitative skills. Many evaluators stumble into arts-based methods and assume they are easy, fluffy, or right for a first-time facilitator. Collage requires advanced facilitation skills, a deep sense of buy-in for the technique, and familiarity with the subject of the guiding prompt (i.e., subject-matter expertise). Anything less does the art form and folks participating in the activity a disservice.
A capitalist or efficiency mindset & the use of AI in the process Collage is an analog process. It requires human-to-human interactions. The magic lies in everything that will never make it into the evaluation report - the rhythm and sounds of a group of people cutting and scrunching, the deep breaths people take when they have the time and spaciousness to think, the laughter that bubbles up when someone spills glitter, the gentle hum of side conversations and whispers as folks peak at and compare with each other's half-assembled pieces, and the touch and texture of a story built-up and/or dismantled rather than recalled and recited. Collage (and I would argue any evaluation method) will always fail when the success metric is bang for buck - or when the evaluator wants to capture as much data as possible in as little time and with as few resources as possible.
Because the nuance happens off the page (literally), using an AI data analysis platform to mine transcripts for themes, quotes, and nuggets of wisdom is also not a wise choice. This is an embodied technique that requires a whole feeling person - not just a thinking brain - to make sense of what is said, unsaid, and left on the table with the scraps.
Learn More
Collage is one of many exciting techniques growing in popularity in the arts-based methods category. That's why it is one of the four tools we teach in our beginner Arts-Based Methods for Evaluators course. No artistic experience is required to join the class. Our next cohort starts in January 2025 and runs for six weeks. Learn more.
We offer organizational workshops on arts-based techniques. Ranging from 90 minutes to a full-day deep dive, we teach hands-on skills to folks who have any amount of experience and some or no artistic skill. Book now!
Subscribe to our blog to have new content delivered right to you!
#collage #artsbasedmethods #artsbased #creative #qualitative #qualitativemethods #evaluation #skillbuilding #professionaldevelopment #trysomethingnew #evaluation #programevaluation #research #methodology #learning #creativepractice #creativeconsulting #tipsandtricks #tipsforpractice #methods