SWOT AI Assistant (SAIA)

Important

As of September 3, 2024, four (4) groups of students have expressed interest in this project. Consequently, it is highly likely that the project will be assigned to one of these groups.

Overview

A SWOT analysis is a strategic planning tool used to assess the Strengths, Weaknesses, Opportunities, and Threats of a project, business, or academic program. It helps in understanding internal capabilities and external influences, facilitating informed decision-making.

This project is conducted in collaboration with the Office of Program Evaluation and the Teaching and Learning Support Service (TLSS). As part of the University of Ottawa’s Institutional Quality Assurance Process (IQAP), all programs are evaluated every eight years to ensure the highest standards and quality are maintained. SWOT analyses provide a comprehensive assessment of all aspects of a program and involve various stakeholders, including students, alumni, employers, professors, and support personnel.

Summarizing findings from numerous participants can be laborious. To address this, the project will leverage generative AI for efficient summarization.

The proposed application consists of two primary modules: data collection and data summarization. Data collection will be conducted via a responsive web application, while generative AI will handle data summarization. This project is well suited for a team of two students but can also be split into two separate projects.

Requirements

Users

We anticipate the need for three types of users:

1. The system administrator who is able to configure the system and perform maintenance tasks.
2. SWOT administrator. A user who is able to create a SWOT and generate reports.
3. Participants. The participants answer the survey.

Data Collection

• The application should support multiple languages, particularly English and French for testing, with a simple mechanism on each page to switch languages.
• It should facilitate conducting multiple SWOT analyses, each with customizable questions.
• The interface must be accessible on both mobile and desktop devices.
• Compliance with university data protection regulations is crucial, especially when handling sensitive information.
• In order to create a SWOT, a user must have an account. This allows them to modify their SWOT analyses and create reports.
• Participants do not need to have an account. Instead, they are using a flexible mechanism that uses their Email address in order to obtain a code, similar to that of medium.com, to input and edit their responses.

Design of questions (AI Assistant)

We are aiming to leverage contextual information and generative AI in order to assist the SWOT administrator generating questions.

• There should be a mechanism to upload sample questions.
• Likewise, provide a mechanism to upload background information on the project to be evaluated: the program description, syllabi, the previous self-study, etc.
• Provides a mechanism to generate questions (AI generated questions).
• Offers a critical analysis of questions.
• Ability to add, remove, and edit questions.
• Mechanisms to import/export questionnaires.

Data Summarization

We are aiming to leverage generative AI in order to summarize the data collected.

• Collect inputs for each SWOT category and use AI to generate consolidated summaries. Similarly, the system needs to summarize the strategies for improvement that have been proposed by the users.
• Design prompts to guide AI in producing relevant and concise summaries. These should be carefully designed to circumvent hallucinations. When summarizing the results, the system is expected to produce quote. It is important that these quotes are reproduced verbatim from the input; not invented.
• Ensure the summaries focus on key points from each SWOT category.
• Validate the prototype with a small user group to refine processes and iterate based on feedback.
• Accommodate multilingual responses and produce a report that includes main findings, participant counts, and response-backed claims.
• Provide the model with background information on the academic program for contextually appropriate summarizations.
• The prototype should mirror the current process and produce an output according to a fixed template.
• The AI API should be isolatable so it can be updated or replaced as technology evolves or as project needs change.
• As a first step, look at the existing solutions.
• Carefully design a benchmark and measures of success for the application.

Challenges in the Project: Addressing Hallucinations

This application is designed to support the central administration in analyzing data for cyclical program reviews. Ensuring the accuracy of the generated summaries is crucial. Specifically, the application must avoid fabricating information, including citations. Strategies such as careful prompt engineering and retrieval-augmented generation (RAG) are likely components of the solution, although other approaches may also be considered. Here are two links that could be helpful.

• A Ranking & Evaluation Framework For LLM Hallucinations
• PaperQA uses RAG to doing question and answering from PDFs or text files.

Advanced Features

• Enable real-time updates or collaborative input options to enhance the analysis process, providing immediate summaries during data collection sessions.
• Ensure interoperability between the data collection and summarization modules, defining a flexible data format to support multiple analytical parts, such as initial analysis, mitigation strategies, and open feedback.
• Allow for interactive querying of the data and report, enabling users to request specific information or request rephrasing of the report to exclude certain terms.
• An option to identify and mask confidential information (such as names).
• Produce additional visualizations: clusters, word clouds, etc.
• Generating a summary of all the steps of the project for auditory purposes.
• Avoid jailbreaking.

Innovation

To the best of our knowledge, commercial tools employing generative artificial intelligence for facilitating SWOT analysis are presently nonexistent. Consequently, this project holds potential for further development beyond its initial scope.