The form evaluates the falsifiability of a single claim in support of a hypothesis that is likely to be taken to be confirmatory (e.g., “our study suggests…” or “we found evidence that…”; Höfler et al., 2022). It is not suitable for inconclusive (e.g., “our study revealed unclear results…”) and exploratory claims (e.g., “we propose the new hypothesis that…”; Höfler et al., 2022). While flexibility in the research process opens the door to the discovery of novelty in exploratory research, flexibility in confirmatory research must be entirely constrained. A confirmatory claim can be typically extracted from the abstract, discussion, or conclusion of a paper. In the planning phase of a study, FAF can guide the choice of rigorous research methods, so that once the study is conducted, the claims made would pass the FAF items (except possibly Domain 4, see below).

The form is to be filled out once for every claim in a paper and can be used several times for different claims. For composite claims (e.g., “Therapy A is effective, but Therapy B is not”), it is advisable to evaluate each part separately if their implications for theory building or intervention are not the same (Bender & Lange, 2001). FAF covers original, quantitative research; it does not apply to meta-analyses, re-analyses, or other study types. Additionally, it does not evaluate the hypothesis’s substantive quality and relevance.

Within these limits, FAF aims to uncover as many issues as possible, using only a minimum set of straightforward questions. It returns a list of potential issues, which are assumed to be highly context-dependent. The form does not judge these issues but encourages authors and reviewers to address them — ideally before publication. In this sense, FAF is meant to be normative. Crucially, it does not constitute a measurement scale or diagnostic instrument requiring high inter-rater reliability, but a structured qualitative audit framework for identifying ambiguities or undisclosed flexibilities that could compromise falsifiability.

Each item contains a broad question paired with an example to illustrate its scope. For example, Item 1.1 asks “Are there any ambiguities about the meaning of the hypothesis or the conditions under which it is claimed to hold?” and then explains “The formulation of a hypothesis must eliminate flexible interpretations. A common instance for flexibility is ambiguity about whether the hypothesis concerns a causal or associational relation. Besides, the hypothesis has to include the conditions under which it holds: the population, materials, stimuli, design procedures and outcomes used.” A broad question, together with the example, is intended to prompt more specific considerations that may affect how the question is answered. A potential issue then is raised by endorsing a concern, or by highlighting that information is ambiguous or missing, which is to be described in a free text field. In doing so, we expect a paper or its supplementary material to report anything in the research process where flexibility in procedures could undermine falsifiability. We also expect the reporting to be verifiable, wherefore the form ends by evaluating if these procedures have been carried out as reported through a final domain on the transparency measures used (Lakens, 2019; Lakens & Mesquida, 2024).

Although the items on the wording of a hypothesis, data processing, data analysis (Domains 1 – 3) and transparency measures (Domain 5) are designed to elicit a fairly objective assessment, it is not a concern if some users of the form identify more potential issues than others. FAF aims to identify as many of them as possible, not to assert that they are all critical. Domain 4, “Alternative explanations for the claim, not addressed so far” includes a single free-text item in which any alternative explanations can be listed. This considers the well-known fact that there are always several possibilities for why a hypothesis may be wrong, and a single study can never test all of them (e.g., the effect does not exist, the measurement instrument is not valid; Meehl, 1967; Rakover, 2003). We therefore expect that this item will only remain blank if a paper presents a set of studies that, together, rigorously test all plausible alternative assumptions. Finally, the last domain “Transparency measures” asks in detail about preregistration and registered reports, specifically whether they have been timestamped before data access. Other transparency measures are listed in checklist format: the use of open data, open materials, open analysis (code), a reproducibility check, the 21-word solution, or “anything else”. For each transparency measure, it is asked whether it was implemented “in a way that is insufficient to assess falsifiability” and, if so, a free text explanation of why it is insufficient is requested.

Table 1 provides a full list of each domain’s items, i.e., items in FAF (Version 1.0).

After completing the form, FAF returns the following results: The extracted claim with paper information (including title and year of publication), a domain-wise listing of “Identified issues of the claim which might have impaired falsifiability and require consideration” and a summary of “transparency measures used”, which might include identified flexibilities, e.g., in the preregistration, that remained undisclosed.

FAF was implemented as a Google Sheet (FAF Research Group, 2025). It consists of the sheets “Background” (including “Instructions”), “Form” and “Results”. Clicking the form link creates a private copy, ensuring that the data is only stored in one’s private Google account. The results sheet can be stored in another spreadsheet or PDF format. (Höfler, Kräplin, Elsherif et al., 2025).

MH had the initial idea in late 2023 to create a form that would shed light on how well a scientific claim could have turned out differently. He drafted a concept and initial items and preregistered the form on the Open Science Framework (Höfler, 2023). AK piloted the items by reviewing a couple of papers. MH presented the idea at the SIPS (Society for the Improvement of Psychological Science) conference in June 2023, in person and online. Fourteen colleagues joined the project at that point, and a first draft of the FAF was agreed upon by consensus in October 2024, after several online meetings. FAF was then implemented on Google Sheets by MS, who then joined the project. The ten authors of this paper remained involved in the project ever since.

After completing the draft, we decided to collect feedback on the form from a wider group of experts. Doing so, we aimed to gauge how the form is received by experts and, above all, to identify any major inaccuracies and incomprehensibilities. The feedback form included the following items:

We decided to evaluate these questions with a collegial feedback assessment. This anonymous format was chosen to encourage open and honest responses while protecting participant confidentiality. As a result, no identifying information about the respondents was collected. For the development of the initial version of FAF, we considered it sufficient that all participants were recognized as experts, as each was personally invited by a member of the FAF project. These invitations were made independently to ensure at least a minimal degree of diversity in perspectives. The FAF members were instructed to select experts based on predefined criteria: “You consider somebody to be an expert in meta-science or methods (based on their publications or teaching).” For group invitations, the criteria included: “Open science initiatives, meta-science groups, teams working to improve scientific practices and methods, and journal editors.” In addition, members were explicitly instructed not to invite anyone who had previously been involved in the project.

Feedback was provided anonymously and no data beyond the responses to the survey items were collected or stored. However, the final item gave participants the option to leave their email address if they wished to discuss their input further. By doing so, they voluntarily waived their anonymity to ensure their concerns could be properly addressed. Invitations to participate were sent out between 3 and 31 December 2024, and feedback was collected until 31 January 2025. Eighty-five colleagues from psychology, 27 colleagues from other disciplines, and 13 groups were invited by email. For data protection reasons and since each member sent out invitations, we did not create a central database of invitees. Therefore, some overlap in the invitations may have occurred and the number of unique recipients may be slightly lower.

Before discussing FAF in detail, some examples of its use are given in Table 2.

FAF is a qualitative tool. It identifies potential issues without asserting that a real issue has been brought up. This reflects that the FAF items address topics whose impact on falsifiability may depend on context and subjective judgement. For example, the “meaning of a hypothesis” may be unclear to one researcher but not to another, or clarity may be achieved by explanation in response to completing the form (e.g., an unpublished claim on a hypothesis can be revised to achieve clarity). Potential issues flagged by FAF may be resolved through additional explanation provided in response to filling out the form. While we believe that FAF meets its goal if it stimulates the impetus to address such instances, it is natural to ask whether its qualitative approach could be extended towards a quantitative scoring of a claim's trustworthiness.

The quantitative counterpart to falsifiability is the severity of a test, the probability that a false hypothesis turns out to be false (Lakens, 2019; Mayo, 2018). This quantity appeals for its clarity of interpretation, but it is extremely difficult to calculate in practice, because it depends on numerous factors whose impact is difficult to determine, including those covered by FAF. Such factors involve bias in analysis — for example, due to unconsidered correlations between observations (see Item 3.2: “Among all the assumptions in the statistical test or model, are there any that are clearly inaccurate and favour the claimed hypothesis?”). However, bias is highly context-dependent, and quantifying its extent requires a profound understanding of the mechanisms that produced a particular dataset, like the true magnitude of unconsidered correlations or features of measurement, selection, and unaccounted-for confounders (Greenland, 2005).

Nevertheless, anchoring it in the extremely simple cases in which computing the severity of a test is feasible helps to get an idea on its wider range in more complex, realistic scenarios. Mayo (2018) has elaborated much on severity calculations under the unrealistic assumptions of the absence of any bias (Greenland, 2005) and questionable research practices such as p-hacking and HARKing. In this case, and if a claim on a hypothesis is based on a single (frequentist) statistical test, severity is as follows:

If the claim relies on multiple tests — as addressed in FAF Item 3.4 — and the decision rule requires that at least one out of k tests be passed (Item 3.3), severity decreases. If the claim is in favor of an effect, severity ranges between (1 − k·α), assuming independent tests, and (1 − α), assuming completely correlated tests. If the claim is against the effect, severity ranges between (1 − k·β) and (1 − β). Conversely, if the decision rule requires that all tests be passed, severity is higher. In that case, severity in favour of the effect ranges between (1 − α)^k and (1 − α), depending on the degree of dependence between tests. The analogous range for claims against the effect is between (1 − β)^k and (1 − β).

Questionable practices reduce severity, and the more ambiguities and flexibilities remain undisclosed (i.e., the more potential issues FAF identifies), the more room there is for severity to fall below the above values (Lakens, 2019). In cases of extensive fishing through flexible hypothesis formulation, data processing or analysis, severity can even approach 0 (Head et al., 2015; Simmons et al., 2011). Such a claim is essentially unfalsifiable. When a potential issue is identified by endorsing a FAF item — for example, unclear wording of the hypothesis (Item 1.2) or a lack of preregistration or insufficient detail in it (domain on transparency measures) — we do not know which questionable research practices, if any, have been employed, or to what extent. These unknowns determine the degree to which severity is reduced.

While these considerations apply only to specific situations and yield broad severity ranges, we strongly encourage mathematical elaborations of severity in fairly more complex scenarios, particularly regarding the impact of issues addressed by specific FAF items.

A key strength of FAF is its foundation in Popper’s principle of falsifiability, the most widely accepted scientific standard for hypothesis testing. FAF’s qualitative approach highlights potential issues without making direct judgments about the validity of a claim. This reduces the risk of arbitrary or overly context-dependent evaluations. Furthermore, since falsifiability is a universal principle of scientific inquiry, FAF provides wide usability, allowing researchers to apply it without adjustments in different contexts and across disciplines. Its areas of usage include manuscript preparation, peer review, and post-publication evaluation. While many existing approaches are restricted to post hoc evaluation of compliance with certain reporting standards or methodological benchmarks, FAF, when used yet at the planning stage, promotes better research design from the outset and encourages researchers to formulate hypotheses that can be meaningfully tested and potentially refuted. Since FAF was developed through a consensus among the authors and refined via a collaborative feedback process involving 19 experts from diverse backgrounds, major weaknesses and inaccuracies in the form should have been identified. The result aims to strike a balance: while it does not identify every possible issue, it captures many and remains concise and accessible to a broad range of researchers.

Despite these strengths, FAF has limitations. It is specifically designed for original, quantitative research and does not apply to meta-analyses, re-analyses, or other study types. Assessing falsifiability in such research requires other approaches, though elements of FAF may still be useful. For example, in meta-analysis, assessing reproducibility is a major challenge (Maassen et al., 2020), yet flexibilities in the research process must still be disclosed (e.g., predefining the hypothesis, data processing, and analysis before accessing the data). To illustrate FAF items, we used examples from the behavioural sciences because they should be familiar to everyone. However, we would like to emphasise that the same items apply to data-intensive research such as neuroimaging or ecological momentary assessment. These fields are not exempt from falsifiability standards simply because they pose greater practical challenges.

As a second limitation, the effectiveness of FAF depends on researchers’ willingness to engage critically with the identified issues. While the form encourages users to document and reflect on potential ambiguities, there is no mechanism to enforce proper resolutions of these concerns. FAF includes a section on transparency measures, but this only addresses whether the reported procedures could be verified, not whether they have actually been verified.

Finally, while the expert feedback process contributed valuable insights, the relatively small number of responses (n = 19) means that the revisions to the form were based on a limited set of perspectives. Therefore, the version presented in this paper represents only the initial release — Version 1.0 — of FAF. We invite the broader scientific community to contribute to its further development. To facilitate this, we have assigned it a Creative Commons Attribution license (CC BY) and created a GitHub repository (Höfler, Kräplin, Elsherif et al., 2025), which will allow for the collection of community feedback that can guide future refinements. Further systematic evaluation should aim to achieve broader validation, including large-scale testing across different disciplines. Additionally, interrater agreement should be examined to identify which items — and in which scientific contexts — researchers are most likely to disagree. Such insights could inform improvements to FAF’s utility and generalizability.

FAF is beneficial because it avoids the heuristic approach to complexity and bias of multi-criteria assessments by focusing on a single, fundamental principle — falsifiability. This makes it broadly applicable and theoretically grounded, while also encouraging more rigid and less biased research practices. Possible future extensions include training artificial intelligence to automatically extract the relevant information from papers. AI-based large-scale meta-science could analyse thousands of publications to identify not only research fields with particularly high numbers of “not even wrong” hypotheses (Scheel, 2022), but also steps in the research process with particularly frequently undisclosed flexibilities. These insights could then inform the development of targeted research tools and teaching approaches to close the flexibilities. Ultimately, this could encourage the risk-taking that researchers need for conducting studies capable of providing robust evidence.