Automated and Efficient Test-Generation for Grid-Based Multiagent Systems

TY - JOUR

T1 - Automated and Efficient Test-Generation for Grid-Based Multiagent Systems

AU - Entekhabi, Sina

AU - Mostowski, Wojciech

AU - Mousavi, Mohammadreza

PY - 2023/8

Y1 - 2023/8

N2 - Automatic generation of random test inputs is an approach that can alleviate the challenges of manual test case design.However, random test cases may be ineffective in fault detection and increase testing cost, especially in systems where test execution is resource-and time-consuming. To remedy this, the domain knowledge of test engineers can be exploited to select potentially effective test cases. To this end,test selection constraints suggested by domain experts can be utilized either for filtering randomly generated test inputs or for direct generation of inputs using constraint solvers. In this paper, we propose a domain specific language (DSL) for formalizing locality-based test selection constraints of autonomous agents and discuss theimpact of test selection filters, specified in our DSL, on randomly generated test cases. We study and comparethe performance of filtering and constraint solving approaches in generating selective test casesfor different test scenario parameters and discuss the role of these parameters in test generation performance.Through our study, we provide criteria for suitability of the random data filtering approach versusthe constraint solving one under the varying size and complexity of our testing problem.We formulate the corresponding research questions and answer them by designing and conductingexperiments using QuickCheck for random test data generation with filteringand Z3 for constraint solving. Our observations and statistical analysis indicate that applying filters cansignificantly improve test efficiency of randomly generated test cases. Furthermore, we observe thattest scenario parameters affect the performance of the filtering and constraint solving approaches differently. In particular, our results indicate that the two approaches have complementary strengths: random generation and filtering works best for large agent numbers and long paths, while its performance degrades in the larger grid sizes and more strict constraints. On the contrary, constraint solving has a robust performance for large grid sizes and strict constraints, while its performance degrades with more agents and long paths.

AB - Automatic generation of random test inputs is an approach that can alleviate the challenges of manual test case design.However, random test cases may be ineffective in fault detection and increase testing cost, especially in systems where test execution is resource-and time-consuming. To remedy this, the domain knowledge of test engineers can be exploited to select potentially effective test cases. To this end,test selection constraints suggested by domain experts can be utilized either for filtering randomly generated test inputs or for direct generation of inputs using constraint solvers. In this paper, we propose a domain specific language (DSL) for formalizing locality-based test selection constraints of autonomous agents and discuss theimpact of test selection filters, specified in our DSL, on randomly generated test cases. We study and comparethe performance of filtering and constraint solving approaches in generating selective test casesfor different test scenario parameters and discuss the role of these parameters in test generation performance.Through our study, we provide criteria for suitability of the random data filtering approach versusthe constraint solving one under the varying size and complexity of our testing problem.We formulate the corresponding research questions and answer them by designing and conductingexperiments using QuickCheck for random test data generation with filteringand Z3 for constraint solving. Our observations and statistical analysis indicate that applying filters cansignificantly improve test efficiency of randomly generated test cases. Furthermore, we observe thattest scenario parameters affect the performance of the filtering and constraint solving approaches differently. In particular, our results indicate that the two approaches have complementary strengths: random generation and filtering works best for large agent numbers and long paths, while its performance degrades in the larger grid sizes and more strict constraints. On the contrary, constraint solving has a robust performance for large grid sizes and strict constraints, while its performance degrades with more agents and long paths.

M3 - Article

SN - 1049-331X

JO - ACM TRANSACTIONS ON SOFTWARE ENGINEERING AND METHODOLOGY

JF - ACM TRANSACTIONS ON SOFTWARE ENGINEERING AND METHODOLOGY

ER -