Remote 2025 Summer Internship: Psychometrics and Data Science (PhD)

POSITION OVERVIEW

NBME invites applications for multiple full-time remote internship positions for the Summer of 2025. Over an 8-week period, interns will have the opportunity to collaborate with NBME staff and interact with fellow graduate students as they complete a research project. The expected deliverable is an internal research presentation. Specific projects for the summer of 2025 will be discussed with applicants as part of the interview process. Examples of recent summer internship projects are listed below.

APPLICATION PROCESS
? Applications that include a cover letter outlining relevant experience and specific research programs or topics of interest will be given priority. When writing your cover letter, please highlight your specific role on projects or when working with faculty. It can be difficult to infer this information from short CV entries. Please help us learn more about your skills and interests!
? The deadline for applications is Friday, January 31, 2025
? Applicants will be informed of final decisions by Friday, February 28, 2025.

Diversity, Equity, and Inclusion Statement

At NBME, we continue to innovate and improve how we fulfill the evolving needs of the healthcare community. This commitment starts and ends with the people at NBME. By recruiting and empowering talented individuals from various disciplines and backgrounds, which includes professionals with diverse life experiences, abilities, and perspectives, NBME can take a well-informed, robust approach to advancing medical education and assessment for years to come. We also continue to focus on ensuring that our DEI work is impactful and ingrained in everything we do, including with our staff, workplace culture, products and services, the Philadelphia community and the broader medical education landscape.

RESPONSIBILITIES

Operational Research

This research supports NBME's current portfolio of assessments. Utilizing already defined capabilities and focusing on established constructs, this foundational programmatic research improves our existing measurement capabilities or gathers validity evidence for our assessments.

Project Examples:
? Modeling Item Revisit Behavior: The Hierarchical Speed Accuracy Revisit Model
? Using Process Data to Improve Examinee Ability Estimation Precision
? Evaluation of a Virtual Clinical Reasoning Performance Assessment
? The Utility of Nonfunctioning Distractors for Credentialing Tests
? Modeling the Conditions That Influence Response Time and Performance
? Examining USMLE Step 1 Feature Usage: A Response Process Validity StudyThe Association between USMLE Step 2 CK Scores and Patient Mortality: A Validity Study

Innovation Research

This research expands NBME's capabilities to provide more insight into our assessments and examinee behavior. For example, NBME now has a dedicated natural language processing (NLP) team that has its own line of research and supports various other initiatives in Operational Research and Innovation Research. Other research in the capabilities category includes building user score reports or creating stand-alone web-based applications, such as with Shiny.

Project Examples:
? Automated Short Answer Question Scoring with Adversarial Training
? Automated distractor generation for clinical MCQs
? Automatic Item Generation for Medical Assessment with Large Language Models: Challenges of Accuracy and Novelty Coupl
? Using NLP to Explore Item Language Patterns as Part of Test Bias and Test Validity
? Clinical Reasoning Scoring and AnalysisAutomatic Enemy Item Detection Using Natural Language Processing: Latent Semantic Analysis

QUALIFICATIONS
? Must be eligible to work June 2-July 25, 2025. We are unable to shift the timeline of the internship program.
? Active enrollment in a doctoral program in measurement, statistics, computer science, cognitive science, medical education, or related field.
? Completion of at least one year of your PhD program by Summer 2025.
? Experience or coursework in one or more of the following: statistics, measurement theory, natural language processing, data science, data science, research design, IRT, and test development.
? Working knowledge of statistical software (e.g., R, Python, C#).
? Interns will be assigned to one or more mentors and must be able to work independently.
? Must be authorized to work in the US for any employer. If selected, F-1 holders will need to apply for Curricular Practical Training authorization through their school's international student office and have a social security number for payroll purposes.
? Must not have any competing employment throughout the duration of the internship.
? Must be available between NBME core hours (10am-3pm Eastern Time).

About NBME:

NBME offers a versatile selection of high-quality assessments and educational services for students, professionals, educators, regulators and institutions dedicated to the evolving needs of medical education and health care. To ensure our assessments meet the highest standards of quality, stay relevant and align to the current curriculum in medical schools and training programs, we rely on a wide network of collaborators. These include the volunteers who help develop our exam questions, the committees and panels who represent various groups within the medical education community, external researchers and health profession organizations.

We are committed to meeting the needs of educators and learners globally with assessment products and expert services such as NBME?Subject Examinations, Customized Assessment Services, Self-Assessments, the International Foundations of Medicine?Program and Item Writing Workshops. Together with the Federation of State Medical Boards, NBME develops and manages the United States Medical Licensing Examination? which measures the ability to apply knowledge and skills that form the basis of safe and effective patient care. Our Competency-based Assessment unit is focused on new methods as well as the optimization of assessment in the workplace and education.

As a result of leadership in ongoing research, innovative measurement practices and the exploration of forward-thinking assessment modalities and improvements, NBME advances assessment science. Our grant and funding opportunities further support this dedication to medical education and assessment science. We help develop the next generation of assessment professionals through our Summer Psychometric Internship Program. Through the Stemmler Fund, Strategic Educators Enhancement Fund and Latin America Grants Program, researchers and educators can continue to improve the assessment of health care professionals around the world.

NBME views diversity, equity and inclusion (DEI) as foundational and enduring to our strategy and vision. We continue to focus on ensuring that our DEI work is impactful and ingrained in everything we do, including with our staff, culture, products and services, the Philadelphia community and the broader medical education landscape. Our commitment manifests in our hiring and staff development, recruitment for committees, grants programs, design and review of our assessments, and involvement in our local and national communities.

Learn more about NBME at NBME.org.

The NBME offers competitive salaries, excellent benefits, and a rewarding work environment. Excellent Benefits include: Healthcare, Dental, Prescription, and Vision plans; 401(k) w/match, Tuition Reimbursement Plan, Commuter Benefit: Public Transit or Parking options. Remote Friendly Workplace.

NBME is an equal opportunity employer as defined by the EEOC.

The compensation we are offering for this position is $45 per hour, amounting to a total stipend of $12,600 for full-time interns (35 hours per week). Additionally, all interns are eligible to receive up to $1,000 to support their attendance at a conference. This is not conditional on presenting at the conference.

Los Angeles