Strategic Plan Evaluation and Tracking Initialtives

Media coverage and sentiment analysis are frequently used to evaluate communication effectiveness and are strategic assets that contribute to credibility, reach, resource mobilization, social impact, and overall organizational effectiveness. They are vital tools for achieving mission-driven goals and creating positive change. CSM's new strategic plan implementation evaluation for 2025-2030 includes media coverage and output as one of the KPIs. Additionally, positive and widespread media attention acts as a powerful motivator, validator, and resource amplifier that directly benefits the doctors who are central to the organization's mission. It fosters a more engaged, proud, and stable workforce, ultimately enabling the organization to better serve its intended beneficiaries. The request consists of three main components: 1) the media coverage (by country, state, and city), 2) the number of unique media sources that cover CSM stories (by country, state, and city), 3) sentiment analysis for all the news, and 4) categorization of the media outputs into 5 categories: research, innovation, education, health policy, and indigenous. A fine-tuned local large language model was deployed to support sentiment analysis and categorization.

Alumni Engagement is a critical Key Performance Indicator (KPI) in the evaluation of the CSM strategic plan implementation, particularly within the 'support people and community' area. We depend on the main campus to furnish this data for analysis and visualization. Due to data governance challenges and limitations, we refer to it as “contactable alumni engagement data,” given that the university employs a universal system for disseminating events and opportunities, whereas research institutes or departments may utilize distinct, independent systems. As this is exclusively for KPI tracking, the visualization aspect is straightforward: a stacked year-versus-engagement chart, and a comparison between CSM and other faculties. I also conducted the behavior analysis to explore the relations among activities. 

The award project more like a data governance project rather than a data analytics project. At CSM, an award specialist facilitates award applications for faculty, staff, and students and maintains records of awarded individuals. Historical award data, dating back to 2008, is stored, encompassing all awards granted to CSM professionals. Two primary challenges exist in this project. Firstly, while the data exhibits some inconsistencies in each row, it is not extensively disorganized compared to other projects. A solution is required to standardize the data. Secondly, the dataset currently contains only award titles, recipients, and years, which is insufficient. Assembling attributes for thousands of unique awards manually would be time-consuming, thus necessitating an automated approach. To address the first challenge, an award mapping file was utilized. RE and LLM were employed to identify and group the awards by the name. The specialist determined the formal names, which were then used to replace informal titles. For the second challenge, an LLM agent was created to research awards online and analyze their descriptions. Based on these analyses, the model categorized awards by scope (national, international, local), categories (education, research, mentorship, clinical excellence, etc.), internal distinctions, and career achievement levels (early, mid, senior, or open).

Following the resolution of these challenges, a further issue emerged: the development of a pipeline for the award specialist to input high-quality data and ensure data governance. Currently, a pipeline is being developed with the award specialist to promote efficiency and consistency.

The strategic plan for the education area aims to delineate scholarly activities and contributions to medical education research. This endeavor presents significant challenges. Our tenure-track and clinical researchers have produced approximately 27,000 distinct publications since 2020. A comprehensive review of all publications to determine their relevance to medical education research is impractical. Reliance on author-provided keywords is also deemed unreliable, as research may impact medical learner training (e.g., virtual simulation environments) without explicitly incorporating education-related terminology. Similarly, the topic assignment by the research intelligence database does not consistently reflect the true rationale or implications of the publication. Furthermore, defining medical education research poses another difficulty. In a broad sense, all medical research projects, including but not limited to clinical trials, possess some degree of educational value.

The initial approach to address this data request involves training a deep learning model with a selection of pre-identified medical education research. This concept has been deemed viable and valuable. However, medical education research encompasses diverse aspects such as curriculum, teaching/learning methodologies, evaluation and assessment, learner characteristics development, faculty development, professional development, and education technology implementation. A comprehensive analysis of all these aspects would necessitate substantial time investment for data preparation and classification.

This task is currently in progress.

The evaluation of return on seed grants and support provided to researchers is of significant interest to numerous departments and research institutes. Such an assessment extends beyond the quantification of subsequent grant acquisitions and is pivotal for strategic planning, program enhancement, and the cultivation of a culture characterized by innovation and entrepreneurship. An initial framework for Return on Investment (ROI) evaluation in the medical research context entails the examination of three primary domains: 1) publication metrics and impact, 2) attainment of subsequent, larger grants, and 3) societal benefits. This framework will be expanded to include two additional components: the development of novel research tools, methodologies, or facilities, and the creation of research opportunities for students.

The execution of this evaluation presents considerable challenges, primarily due to data fragmentation and the complexity of data integration. The comprehensive compilation of all investments is complicated by the multifaceted nature of support provided, which extends beyond seed grants to encompass facility access and resource allocation. Furthermore, a robust ROI analysis necessitates the incorporation of both quantitative data and narrative accounts to facilitate effective communication, assessment, promotion, and transparency. 

This task is currently in progress.