AI Safety & Resilience Founding Opportunities
A founder’s first organization idea is almost never their best idea. This collection can be used to broaden your scope and serve as a launchpad for ideation, and for understanding how rich the AI safety opportunity space is. Whether you are looking for for-profit venture ideas or high-impact research projects to start, there is a massive amount of critical work waiting to be done.
We have compiled a comprehensive longlist of founding opportunities, RFPs, and project ideas ranging from technical alignment and evaluations to biosecurity and hardware. This resource is designed to help (potential) AI safety & resilience founders map out the specific niches where their unique expertise can be leveraged for AI safety.
Catalyze is a non-profit accelerating the most ambitious founders and experts from pre-idea to scaling crucial AI safety & resilience organizations.
Get in touch with us if you are (potentially) interested in launching something in AI safety.
Mixed Resources
- Concrete Projects to Prepare for Superintelligence (Forethought, 2026)
- Technical AI Safety Research RFP (Coefficient Giving)
- Coefficient Giving Substack (Coefficient Giving) - Staff sharing current priorities.
- Ten AI Safety Projects I'd Like People to Work On (Julian Hazell, Coefficient Giving, 2025)
- MIT AI Risk Repository & Mitigations database (MIT, 2025)
- For-Profit AI Safety Ideas (Apart Research, 2024)
- Project Ideas for Making Transformative AI Go Well (Lukas Finnveden, EA Forum, 2024)
- Atlas: AI Resilience Gap Map (Atlas Computing, 2024)
- Future of Life Institute: AI-Driven Power Concentration RFP (FLI, 2024)
- Foresight Institute: Secure AI Focus Area (Foresight Institute, 2024)
- Legible AI Safety Problems That Don't Get Enough Attention (Inchpin, 2024)
- You Actually Can Just Go Build Defensive Tech (Julian Hazell, 2024)
- AI Assurance Technology Landscape (Risk and Reward Report) (AIAT, 2024)
- Jaan Tallinn's Philanthropic Priorities (Jaan Tallinn / SFF, 2022)
AI Safety Ecosystem-Building
- The Case for AI Safety Capacity Building (Asya Bergal, Coefficient Giving, 2026)
- Field-Building and Capacity RFP (Coefficient Giving)
- Probably Good Projects for the AI Safety Ecosystem (LessWrong, 2022)
- AI Safety Field-Building Projects I'd Like to See (LessWrong, 2022)
AI x Biosecurity - Societal Resilience
- Scaling Pathogen Detection with Metagenomics (Institute for Progress, 2025)
- List of Lists of Concrete Biosecurity Project Ideas (EA Forum, 2022)
- Concrete Biosecurity Projects (Some Could Be Big) (EA Forum, 2021)
AI x Cybersecurity
- The Great Refactor (Institute for Progress, 2025)
- Operation Patchlight (Institute for Progress, 2025)
- A Sprint Toward Security Level 5 (Institute for Progress, 2025)
- Atlas: Cybersecurity Pivot List (Atlas Computing, 2024)
AI Governance & Hardware
- AI Preparedness: Robust Policy Options for Europe (Centre for Future Generations, 2026)
- AI governance and policy RFP (Coefficient Giving)
- Faster AI Diffusion through Hardware-based Verification (Institute for Progress, 2025)
- Verifying International Agreements on AI: Six Layers of Verification (RAND, 2025)
- AI Governance to Avoid Extinction (MIRI, 2025)
- Request for Proposals: Hardware-Enabled Mechanisms (HEMs) (Longview Philanthropy, 2024)
- Survival and Flourishing Fund's FlexHEG RFP & FlexHEG Interim Report (2024)
- Verification Methods for AI Agreements (MIRI, 2024)
- Open Problems in Technical AI Governance (Bucknall, Reuel et al., 2024)
AI Alignment
- Mapping the Brain for Alignment (Institute for Progress, 2025)
- Preventing AI Sleeper Agents (Institute for Progress, 2025)
- BlueDot Impact: AI Alignment Project Ideas (BlueDot Impact, 2023)
- Some For-Profit AI Alignment Org Ideas (LessWrong, 2023)
Evaluations
- Anthropic: Third-Party Model Evaluations Initiative (Anthropic, 2024)
- SafeBench Competition Benchmark Guidance (ML Safety, 2024)
- 100+ Concrete Projects and Open Problems in Evals (LessWrong, 2023)
- Evals Projects I'd Like to See (EA Forum, 2022)