The SPLC challenge track aims at providing the software product-line (SPL) community with a set of case studies that tackle relevant SPL problems and challenge the state of the art. The challenge track was established with SPLC 2018 in Gothenburg, initiated by Sarah Nadi, Timo Kehrer, and Thorsten Berger. The challenge is organized in two phases every year. The first phase is the call for cases. Submitted cases will be reviewed by the challenge co-chairs to ensure that all required information is clearly described. Accepted cases will be part of the official conference proceedings. Authors of the accepted cases must attend the challenge track and participate in the discussion. The second phase is a call for solutions to both newly accepted and previously accepted cases. Submitted case solutions will be peer-reviewed by the challenge program committee. Accepted solutions will also be part of the official conference proceedings. Authors of accepted solutions must present their papers during the conference. 

This year, we are placing a special emphasis in our call for cases on the need for actionable benchmarks and competition, akin to those prevalent in the machine learning and SAT communities, and as developed on platforms like Kaggle. We believe that introducing such benchmarks and competition will significantly enhance the ability to compare state-of-the-art solutions and new contributions in the field of software product-line engineering. This kind of proposal should not only foster a spirit of innovation and collaboration but also provides a concrete, measurable framework for assessing the effectiveness and efficiency of new methodologies or implementations. We encourage submissions that offer well-defined, challenging, and replicable benchmarks, thereby setting the stage for a more lively and results-oriented challenge track. 

Submission Guidelines

The idea of the challenge track is to provide participants with a set of case studies that tackle relevant SPL problems and challenge the state of the art. The challenge track happens in two phases.

In the first phase, there will be a call for cases. The challenge co-chairs will review the submitted cases to ensure that all required information is clearly described. Accepted cases will be part of the official conference proceedings. Authors of the accepted cases must attend the challenge track and participate in the discussion.

In the second phase, there will be a call for solutions to the accepted cases and cases from previous years. Submitted case solutions will be peer-reviewed by the challenge program committee. Accepted solutions will also be part of the official conference proceedings. Authors of accepted solutions must present their papers during the conference.

You can check the challenges from previous years on the official website of the challenge track: https://variability-challenges.github.io/

Proposed challenges should be new and should be kept simple, as too complex ones may discourage people from trying to propose solutions.

A case description can be a particular dataset with specific questions:

• What is your dataset, and how was it obtained?
• What is the size of the dataset?
• How can this data be accessed?
• Do you provide tools to process the data?

Provide at least one concrete question you want participants to answer. You can provide multiple questions. For each question, provide the criteria for evaluating an answer/solution.

A case description can also be a call for a solution to a specific problem in a given context:

• What is the concrete problem you want participants to solve?
• How can a solution be evaluated? The following are some suggestions on how to specify evaluation criteria:
  • Concrete evaluation metrics (e.g., precision, recall, accuracy etc. depending on the problem)
  • Concrete test cases participants can evaluate their solution against (e.g., provide inputs and expected outputs and participants are expected to provide a solution that gets from one point to the other)
  • A list of systems to evaluate their solution against (e.g., a list of C systems that have numerous nested #ifdefs, because your problem only makes sense in the context of higher-order variability)
  • A reference implementation to compare against, according to particular metrics

In both case description (either a dataset or a solution), there are some requirements to fulfill:

• The description should contain the URL of a public repository or artifact page containing all the instructions needed to initiate the case study.
• The description should describe the requirements for the solution, or a requirement specification document should be published in the repository. The requirements’ description should help researchers who want to build the solution of or answer to the specific questions. 
• Optional: Case authors may include a list of 5 senior PhD students or junior researchers (postdocs or junior professors) who have the expertise required to evaluate submitted solutions. The case authors themselves may be part of this list. The challenge track co-chairs will consider this list when creating the SPLC challenge program committee.

The challenge track co-chairs will select a small but representative set of case studies to be used for the challenge. After the description of accepted cases is finalized, a call for solutions will be released. Please note that case authors cannot submit a solution to their own case study.

List of Topics

• software product line
• configurable systems
• adaptive systems
• variability modelling 
• performance prediction
• variability reasoning 
• benchmark
• fault localization
• software evolution 
• variability testing
• model checking 
• automated reasoning
• constraint programming 
• reverse engineering variability
• machine learning 

Committees

Program Committee

The list will perhaps evolve:

• Mohammad Reza Mousavi, UK
• Son Nguyen, Vietnam
• Wesley K. G. Assunção, USA
• Xhevahire Ternava, France
• Eduardo Figueiredo, Brazil
• Jacob Krüger, The Netherlands
• Javad Ghofrani, Germany
• Michael Lienhardt, France
• Shiyi Wei, USA

Organizing committee

• Mathieu Acher (University of Rennes, France)
• Thomas Thuem (University of Ulm, Germany) 

Publication

Accepted cases as well as accepted solutions will be part of the official proceedings of SPLC 2024

The SPLC proceedings will be published in the ACM Digital Library, please refer to the general instructions for formatting your proposals (e.g., in LaTeX). 

Case study proposals have a minimum length of 2 pages and a maximum length of 6 pages, including all references and figures.

Case solution proposal have a maximum of 6 pages including all text, figures, etc, plus 1 additional page containing only references may be included.

Venue

The conference will be held in Luxembourg 

Contact

All questions about submissions should be emailed to mathieu.acher@irisa.fr and thomas.thuem@uni-ulm.de