Open ScienceREFSQ 2025

Open Science is a movement that seeks to make scientific research more transparent, rigorous, accessible, and reproducible. This increases the accountability of those contributing the research, but also their credit for opening up scientific work. Open science aims to make research more open to participation, review/refutation, improvement, and (re-)use for the community to benefit through making research material available and reusable.

Like REFSQ’24, the Open Science Track of REFSQ’25 encourages and supports authors in making their research and artifacts more accessible, reproducible, and verifiable. Upon submission of the camera-ready versions of the papers, the Open Science Co-Chairs assess the state of Open Science of submissions to REFSQ’25.

In addition to continuing and expanding the existing Open Science Policy, which all authors are encouraged to adhere to, REFSQ’25 and the Open Research Knowledge Graph (ORKG) organize the first Open Science Competition with two challenges to contribute to the promotion of Open Science in Requirements Engineering.

All authors are invited to take up these challenges. Fame, honor, ORKG Awards, and prize money await you.

REFSQ’25 and the Open Research Knowledge Graph (ORKG) organize the first Open Science Competition to encourage and support authors to promote Open Science in Requirements Engineering. This competition aligns with the broader movement of Open Science and the Open Science Policy of REFSQ, which aims to make scientific research more accessible, reproducible, and verifiable. This competition is an opportunity for the research community to showcase its commitment to Open Science and its promotion in Requirements Engineering by making scientific knowledge FAIR - Findable, Accessible, Interoperable, and Reusable. It is a call to action for researchers to adopt practices that will shape the future of scientific publishing and communication.

The best submission in each challenge receives an ORKG award, including prize money.

The Open Science Co-Chairs will involve other reviewers, i.a., the Program Committee Co-Chairs and experts from the ORKG team to determine the winners.

In addition to instructions and materials (see below in the respective challenge), the Open Science Co-Chair plan to organize free webinars similar to the RE’24 tutorial on Open Science that help authors successfully participate in the challenges.

For further information, questions, or help regarding the Open Science Competition, please contact Oliver Karras (oliver.karras@tib.eu) and Julian Frattini (julian.frattini@bth.se)

Challenges

Challenge 1: Annotate your paper with SciKGTeX to describe its research contribution.

The accepted paper that is best annotated with SciKGTeX will be awarded the Best ORKG Annotation Award, including prize money of 100€.

SciKGTeX is a LaTeX package that allows authors to semantically annotated the scientific contribution of their research in the LaTeX document at the time of writing. These annotations are embedded into the PDF’s metadata in a structured XMP format, which can be harvested by search engines and knowledge graphs. This process does not only simplify the contribution to research knowledge graphs but also promotes the practice of semantic representations in scientific communication and thus Open Science.

You can find the LaTeX package and more information at the following links:

• GitHub Repository of the LaTeX package with detailed documentation and examples
• Guiding materials for using SciKGTeX from the RE‘24 tutorial on Open Science
  • Important slide deck: 1. Slides –> “4. RE24 Tutorial - Using SciKGTeX.pdf”
  • Example project: 2. Exercise Materials –> 2.1 SciKGTeX Materials –> SciKGTeX_Solution

Task

The first challenge of the Open Science Competition asks authors to annotate their papers using the five default annotations of SciKGTeX, as well as the additional annotation for the replication package, code repository, and dataset, if applicable. Comprehensive information on how to use the annotations can be found in the materials provided above.

Remark: Experiments have shown that annotating a paper with SciKGTeX takes an average of just 7.5 minutes, with a range of between 5 - 10 minutes.

Default annotations:

\researchproblem{“Your annotated text describing the research problem”}
\objective{“Your annotated text describing the research objective(s)”}
\method{“Your annotated text describing the research method(s)”}
\result{“Your annotated text describing the research result(s)”}
\conclusion{“Your annotated text describing the research conclusion(s)”}

Additional Open Science Competition annotations:

\contribution{replication package}{“Your URL to the replication package”}
\contribution{code repository}{“Your URL to the code repository”}
\contribution{dataset}{“Your URL to the dataset”}

Evaluation criteria

Submissions will be evaluated by the following criteria:

• Correctness of the annotated information (in respect to the manuscript)
• Completeness of the annotations (bonus points for relevant, additional annotations)
• Conciseness of the information in the annotations

Participation

For participation in Challenge 1, authors must submit their annotated paper in EasyChair and indicate whether their paper is annotated with SciKGTeX during the submission process - we will have a specific optional field in the EasyChair submission form.

Challenge 2: Enrich your paper with an ORKG comparison to provide an overview of the state of the art for your particular research domain, question, or problem.

The accepted paper that is enriched with the best ORKG comparison will be awarded the Best ORKG Comparison Award, including prize money of 200€.

The ORKG is a ready-to-use and sustainably operated platform with infrastructure and services that aims to open scientific knowledge and improve its FAIRness - Findability, Accessibility, Interoperability, and Reusability. Researchers can use the ORKG to systematically describe and organize research papers and their contributions. ORKG Comparisons are a specific feature of the ORKG that allows researchers to organize research contributions in a tabular view, enabling easy comparison and filtering along different properties. This kind of representation facilitates a more efficient review of the state-of-the-art for specific research domains, questions, or problems, thereby enhancing the rigor and accountability of scientific inquiries. However, ORKG comparisons are not just auxiliary tools. They are recognized as scholarly outputs in their own right, complete with Digital Object Identifiers (DOIs) for easy citation. This feature underscores the ORKG’s potential to enhance the visibility and credibility of research contributions.

You can find more information on how to accompany your paper with an ORKG Comparison at the following links:

• ORKG help center: Accompany your paper with an ORKG Comparison
• Examples of ORKG Comparisons with corresponding publications from the RE‘24 tutorial on Open Science
  • Important slide deck: 1. Slides: “5. RE24 Tutorial - Using the ORKG.pdf”
• Guiding materials for using the ORKG, including the creation and publication of an ORKG Comparison, from the RE‘24 tutorial on Open Science
  • Important slide deck: 1. Slides -> “5. RE24 Tutorial - Using the ORKG.pdf”
• Instruction and demonstration videos on how to create an ORKG Comparison and features of the ORKG interface

Task

The second challenge of the Open Science Competition asks authors to create, publish, and cite an ORKG Comparison in their paper. The ORKG comparison must provide an overview on the state of the art for the particular research domain, question, or problem of the corresponding paper. A comparison is particularly applicable when you systematically compare studies, software/tools, or approaches. Comprehensive information on how to create, publish, and cite an ORKG Comparison can be found in the materials provided above.

Evaluation criteria

Submissions will be evaluated by the following criteria:

• Expressiveness of (semantic) attributes by which elements are compared
• Level-of-detail and complexity of the attributes
• Exhaustiveness of the compared elements

Participation

For participation in challenge 2, authors must submit their paper that cites their published ORKG Comparison, in EasyChair and provide the link to their published ORKG Comparison during the submission process - we will have a specific optional field in the EasyChair submission form.

When authors publish their ORKG Comparison, they must assign a DOI to make it persistent and citable and associate the ORKG Comparison with the REFSQ‘25 conference by selecting “31st International Working Conference on Requirement Engineering: Foundation for Software Quality (REFSQ’25)” from the corresponding drop-down menu, compare the screenshot below.

We expect authors to include an explicit Data Availability Statement at the end of their paper (similar to acknowledgments). Specifically, authors should provide details about any material disclosed alongside their submission such as data, or code, and other relevant material. Otherwise, the authors should justify the reasons why disclosure is not possible (e.g., due to IP agreements). While including a data availability statement does not affect the review process, it certainly increases the credibility of results.

We strongly encourage authors to read and apply the following best practices to contribute to Open Science. Follow the Open Science - Artefact Management Guideline and consider the recommendations below:

Before submission:

• Adequately document the material associated with your submission to allow for, to the best extent possible, replicability, reproducibility, and reusability.
• Prepare a README.md file that lists precisely what is being shared and how to use and re-use the research material.
• Introduce a running example that helps reviewers and the community validate and reuse the shared material.
• If you are sharing a code, think thoroughly about the requirements for running the code and how to facilitate this process (e.g., use virtual environments such as Docker where possible).
• Annotate your paper with SciKGTeX to describe its research contribution and thus participate in the first challenge of the Open Science Competition.
• Enrich your publication with an ORKG Comparison, if applicable, e.g., to provide an overview of the state of the art for your particular research domain, question, or problem and thus participate in the second challenge of the Open Science Competition.

Upon submission:

• Establish a good intent by making data, code, and evaluation material available on a persistent repository with a public link, e.g., archiving your replication package on Zenodo, which generates a permanent DOI for your material.
• Indicate in your EasyChair submission if you annotated your paper with SciKGTeX and provide the link to your published ORKG comparison if you created one.

Upon acceptance:

• Attribute an appropriate open licence to the shared material and make sure you share this licence together with the material under the same folder in a LICENSE.md file.
• Upload your annotated paper to the ORKG in less than 90 seconds to increase its visibility: Youtube - Import of SciKGTeX annotated PDF into the ORKG.

Further reading:

• Guiding materials for using SciKGTeX and ORKG Comparisons:

  O. Karras, A. Ferrari, D. Fucci, D. Dell’Anna: Supplementary Materials of the Tutorial: “Promotion of Open Science in Requirements Engineering - Leveraging the Open Research Knowledge Graph for FAIR Scientific Information”, Zenodo, 2024. Link

• General introduction into Open Science in Software Engineering and generally recommended practices:

  D. Mendez, D. Graziotin, S. Wagner, H. Seibold: Open Science in Software Engineering, In: M. Felderer, G.-H. Travassos (eds.) Contemporary Empirical Methods in Software Engineering, Springer, 2020. Link

• Open Science - Artefact Management Guideline:

  J. Frattini, L. Montgomery, D. Fucci, M. Unterkalmsteiner, D. Mendez, J. Fischbach: Requirements Quality Research Artifacts: Recovery, Analysis, and Management Guideline, Journal of Systems and Software, 2024. Link