安卓智能终端自动化测试技术综述

doi:10.3778/j.issn.1673-9418.2304032

摘要/Abstract

摘要： 随着新一代移动通信技术和芯片的发展，智能移动终端用户规模不断增加。为了快速抢占市场，开发商缩短了智能终端的开发周期，这对应用系统的可靠性、稳定性等提出了更高的要求，而自动化测试技术是保障这些智能终端高可靠、强稳定运行的重要手段。结合目前主流智能终端的架构特点和组件特征，分别探讨了安卓系统的黑盒测试技术和白盒测试技术。在黑盒测试方面，比较分析了最新的用户界面测试和模糊测试技术以及工具使用情况，评价了它们在保证应用系统可靠性和稳定性方面的效果。在白盒测试方面，总结了自动生成测试用例技术、动静态的污点分析技术、第三方库检测技术和权限检测技术。随着人工智能大模型等新兴技术不断涌现，越来越多的智能终端设备开始搭载各种深度学习模型，这些模型的不透明性使得内部决策过程难以解释和理解，从而黑盒测试在评估模型可靠性和稳定性过程中越发重要。自动化测试正在面临从传统规则基础下的测试向更加智能的机器学习驱动的测试转变。未来将人工智能大模型等新兴技术引入到现有的智能终端测试实践中，成为解决这一问题的必然趋势。

关键词: 智能终端, 安卓系统, 软件测试, 基于搜索的测试生成

Abstract: With the development of new generation mobile communication technology and chips, the number of intelligent mobile terminal users is increasing. In order to quickly seize the market, developers have shortened the development cycle of intelligent terminals, which raises higher requirements for the reliability and stability of application systems. Automation testing technology is an important means to ensure the high reliability and strong stability of these intelligent terminals. This paper discusses the black box testing technology and white box testing technology of Android system respectively, combined with the architectural characteristics and component features of mainstream intelligent terminals. In terms of black box testing, this paper compares and analyzes the latest UI testing and fuzz testing technology and tool usage, and evaluates their effects in ensuring the reliability and stability of application systems. In terms of white box testing, this paper summarizes the technology of automatically generating test cases, dynamic and static taint analysis technology, third-party library detection technology, and permission detection technology. Finally, with the emergence of emerging technologies such as AI models, more and more intelligent terminal devices are starting to carry various deep learning models. The opacity of these models makes the internal decision-making process difficult to explain and understand, so the black box testing is increasingly important in evaluating model reliability and stability. Automation testing is undergoing a transformation from traditional rule-based testing to more intelligent machine learning-driven testing. In the future, it is necessary to introduce emerging technologies such as AI models into existing intelligent terminal testing practices, which has become a necessary trend to solve this problem.

Key words: intelligent terminal, Android system, software testing, search-based test case generation

曹捷, 黄翰, 雷丰强, 刘方青. 安卓智能终端自动化测试技术综述[J]. 计算机科学与探索, 2024, 18(1): 1-23.

CAO Jie, HUANG Han, LEI Fengqiang, LIU Fangqing. Overview of Android Intelligent Terminal Automation Testing Technology[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(1): 1-23.

参考文献

[1] The China academy of information and communications technology (CAICT)[EB/OL]. (2023-02-17) [2023-04-10]. http://www.caict.ac.cn/.
[2] 郑炜, 唐辉, 陈翔, 等. 安卓移动应用兼容性测试综述[J]. 计算机研究与发展, 2022, 59(6): 1370-1387.
ZHENG W, TANG H, CHEN X, et al. State-of-the-art survey of compatibility test for Android mobile application[J]. Journal of Computer Research and Development, 2022, 59(6): 1370-1387.
[3] MIUI 14 a new lease of life[EB/OL]. [2023-04-10]. https://home.miui.com/.
[4] Flyme official home page[EB/OL]. [2023-04-10]. https://www. flyme.com/.
[5] BRANT C D, YAVUZ T. A study on the testing of android security patches[C]//Proceedings of the 2022 IEEE Conference on Communications and Network Security, Austin, Oct 3-5, 2022. Piscataway: IEEE, 2022: 217-225.
[6] SRIVASTAVA P R. Estimation of software testing effort: an intelligent approach[C]//Proceedings of the 20th International Symposium on Software Reliability Engineering, 2009.
[7] BOSE J. A state-of-the-art analysis of Android malware detection methods[C]//Proceedings of the 2022 6th International Conference on Trends in Electronics and Informatics, Tirunelveli, Apr 28-30, 2022. Piscataway: IEEE, 2022: 851-855.
[8] Pinduoduo App malware detailed by cybersecurity researchers[EB/OL]. [2023-04-09]. https://www.theedgemarkets.com/node/660899.
[9] PAN M, XU T, PEI Y, et al. GUI-guided test script repair for mobile apps[J]. IEEE Transactions on Software Engineering, 2022, 48(3): 910-929.
[10] DA Z, TAO Q, CHEN L. Research on automated testing framework for multi-platform mobile applications[C]//Proceedings of the 2016 4th International Conference on Cloud Computing and Intelligence Systems, Beijing, Aug 17-19, 2016. Piscataway: IEEE, 2016.
[11] YU S, FANG C, FENG Y, et al. LIRAT: layout and image recognition driving automated mobile testing of cross-platform[C]//Proceedings of the 2019 34th IEEE/ACM International Conference on Automated Software Engineering, Diego, Nov 11-15, 2019. Piscataway: IEEE, 2019: 1066-1069.
[12] MOHAMED I, PATEL D. Android vs iOS security: a comparative study[C]//Proceedings of the 2015 12th International Conference on Information Technology-New Generations, Vegas, Apr 13-15, 2015. Piscataway: IEEE, 2015: 725-730.
[13] KAZMI Z, FELGUERA T, VILA J A, et al. TASAM-towards the smart devices app-stores applications security management related best practices[C]//Proceedings of the 2012 International Conference on New Technologies, Istanbul, May 7-10, 2012. Piscataway: IEEE, 2012: 1-5.
[14] BENENSON Z, GASSMANN F, REINFELDER L. Android and iOS users’ differences concerning security and privacy[C]//Proceedings of the 2013 ACM SIGCHI Conference on Human Factors in Computing Systems, Paris, Apr 27-May 2, 2013. New York: ACM, 2013: 817-822.
[15] WANG H, LIU Z, LIANG J, et al. Beyond google play: a large-scale comparative study of Chinese Android app markets[C]//Proceedings of the Internet Measurement Conference 2018, Boston, Oct 31-Nov 2, 2018. New York: ACM, 2018: 293-307.
[16] HarmonyOS development_HarmonyOS ecology_HarmonyOS OS application device development-HarmonyOS official website[EB/OL]. [2023-04-10]. https://www.harmonyos.com/cn/develop.
[17] Tutorials Archive-Fuchsia OS Chinese community[EB/OL]. [2023-04-10]. https://fuchsia-china.com/toturial/.
[18] Official Website for Android Application Developers. Provides Android SDK tools and API documentation[EB/OL]. [2023-04-10]. https://developer.android.google.cn/.
[19] OH H S, KIM B J, CHOI H K, et al. Evaluation of Android Dalvik virtual machine[C]//Proceedings of the 10th International Workshop on Java Technologies for Real-Time and Embedded Systems, Copenhagen, Oct 24-26, 2012. New York: ACM, 2012: 115-124.
[20] FIKRI A, PRESEKAL A, SARI R F, et al. Performance comparison of Dalvik and ART on different Android-based mobile devices[C]//Proceedings of the 2018 IEEE International Seminar on Research of Information Technology & Intelligent Systems, Yogyakarta, Nov 20-22, 2018. Piscataway: IEEE, 2019.
[21] ALéGROTH E, GAO Z, OLIVEIRA R, et al. Conceptualization and evaluation of component-based testing unified with visual GUI testing: an empirical study[C]//Proceedings of the 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, Graz, Apr 13-17, 2015. Piscataway: IEEE, 2015: 1-10.
[22] Appium: automation for apps[EB/OL]. [2023-04-10]. http://appium.io/docs/cn/about-appium/intro/.
[23] UI Automator is an interface testing framework for system-wide and cross-application functional interface testing across multiple installed applications[EB/OL]. [2023-04-10]. https://developer. Android.google.cn/training/testing/ui-automator.
[24] COPPOLA R, ARDITO L, TORCHIANO M. Fragility of layout-based and visual GUI test scripts: an assessment study on a hybrid mobile application[C]//Proceedings of the 10th ACM SIGSOFT International Workshop on Automating Test Case Design, Selection, and Evaluation, Tallinn, Aug 26-27, 2019. New York: ACM, 2019: 28-34.
[25] Airtest Project Docs[EB/OL]. [2023-04-10]. https://airtest. doc.io.netease.com/.
[26] LI X, ZHOU D, ZHANG L, et al. Human-like UI automation through automatic exploration[C]//Proceedings of the 2020 2nd International Conference on Big Data and Artificial Intelligence, Johannesburg, Apr 28-30, 2020. New York: ACM, 2020: 47-53.
[27] UI/application exerciser Monkey[EB/OL]. [2023-04-10]. https://developer.Android.google.cn/studio/test/monkey.
[28] MACHIRY A, TAHILIANI R, NAIK M. Dynodroid: an input generation system for Android apps[C]//Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, Saint Petersburg, Aug 18-26, 2013. New York: ACM, 2013: 224-234.
[29] LI Y, YANG Z, GUO Y, et al. DroidBot: a lightweight UI-guided test input generator for Android[C]//Proceedings of the 2017 IEEE/ACM 39th International Conference on Software Engineering Companion, Buenos, May 20-28, 2017. Piscataway: IEEE, 2017: 23-26.
[30] JAMROZIK K, ZELLER A. DroidMate: a robust and extensible test generator for Android[C]//Proceedings of the 2016 International Conference on Mobile Software Engineering and Systems, Austin, May 14-22, 2016. New York: ACM, 2016: 293-294.
[31] LI Y, YANG Z, GUO Y, et al. Humanoid: a deep learning-based approach to automated black-box Android app testing[C]//Proceedings of the 2019 34th IEEE/ACM International Conference on Automated Software Engineering, San Diego, Nov 11-15, 2019. Piscataway: IEEE, 2019: 1070-1073.
[32] 叶佳, 葛红军, 曹春, 等. 规则驱动的Android应用DFS测试技术[J]. 计算机科学, 2018, 45(9): 99-103.
YE J, GE H J, CAO C, et al. Rule-driven DFS testing technolohy for Aandroid application[J]. Computer Science, 2018, 45(9): 99-103.
[33] ADAMO D, KHAN M K, KOPPULA S, et al. Reinforcement learning for Android GUI testing[C]//Proceedings of the 9th ACM SIGSOFT International Workshop on Automating Test Case Design, Selection, and Evaluation, Lake Buena Vista, Nov 5, 2018. New York: ACM, 2018: 2-8.
[34] HU G, ZHU L, YANG J. AppFlow: using machine learning to synthesize robust, reusable UI tests[C]//Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Lake Buena Vista, Nov 4-9, 2018. New York: ACM, 2018: 269-282.
[35] BORGES JR N P. Data flow oriented UI testing: exploiting data flows and UI elements to test Android applications[C]//Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, Santa Barbara, Jul 10-14, 2017. New York: ACM, 2017: 432-435.
[36] ARTHO C, MA L. Classification of randomly generated test cases[C]//Proceedings of the 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering, Osaka, Mar 14-18, 2016. Piscataway: IEEE, 2016, 2: 29-32.
[37] CHEN T Y, HUANG D H, KUO F C. Adaptive random testing by balancing[C]//Proceedings of the 2nd International Workshop on Random Testing: Co-located with the 22nd IEEE/ACM International Conference on Automated Software Engineering, Atlanta, Nov 6, 2007. New York: ACM, 2007: 2-9.
[38] CSALLNER C, SMARAGDAKIS Y. JCrasher: an automatic robustness tester for Java[J]. Software: Practice and Experience, 2004, 34(11): 1025-1050.
[39] PACHECO C, LAHIRI S K, ERNST M D, et al. Feedback-directed random test generation[C]//Proceedings of the 29th International Conference on Software Engineering, Minneapolis, May 20-26, 2007. Piscataway: IEEE, 2007: 75-84.
[40] YATOH K, SAKAMOTO K, ISHIKAWA F, et al. Feedback-controlled random test generation[C]//Proceedings of the 2015 International Symposium on Software Testing and Analysis, Baltimore, Jul 13-17, 2015. New York: ACM, 2015: 316-326.
[41] CADAR C, SEN K. Symbolic execution for software testing: three decades later[J]. Communications of the ACM, 2013, 56(2): 82-90.
[42] MIRZAEI N, MALEK S, P?S?REANU C S, et al. Testing Android apps through symbolic execution[J]. ACM SIGSOFT Software Engineering Notes, 2012, 37(6): 1-5.
[43] LUO L, ZENG Q, CAO C, et al. System service call-oriented symbolic execution of Android framework with applications to vulnerability discovery and exploit generation[C]//Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, New York, Jun 19-23, 2017. New York: ACM, 2017: 225-238.
[44] GAO X, TAN S H, DONG Z, et al. Android testing via synthetic symbolic execution[C]//Proceedings of the 33rd ACM/ IEEE International Conference on Automated Software Engineering, Montpellier, Sep 3-7, 2018. New York: ACM, 2018: 419-429.
[45] HARMAN M, JONES B F. Search-based software engineering[J]. Information and software Technology, 2001, 43(14): 833-839.
[46] YANG X S. Engineering optimization: an introduction with metaheuristic applications[M]. Hoboken: John Wiley & Sons, 2010.
[47] PANICHELLA A, CAMPOS J, FRASER G. EvoSuite at the SBST 2020 tool competition[C]//Proceedings of the 42nd IEEE/ACM International Conference on Software Engineering Workshops, Seoul, Jun 27-Jul 19, 2020. New York: ACM, 2020: 549-552.
[48] FRASER G, ARCURI A. Whole test suite generation[J]. IEEE Transactions on Software Engineering, 2013, 39(2): 276-291.
[49] ZHU Z, JIAO L. Improving search-based software testing by constraint-based genetic operators[C]//Proceedings of the Genetic and Evolutionary Computation Conference, Prague, Jul 13-17, 2019. New York: ACM, 2019: 1435-1442.
[50] CRUZ-SALINAS A F, PERDOMO J G. Self-adaptation of genetic operators through genetic programming techniques[C]//Proceedings of the Genetic and Evolutionary Computation Conference, Berlin, Jul 15-19, 2017. New York: ACM, 2017: 913-920.
[51] DOERR B, LE H P, MAKHMARA R, et al. Fast genetic algorithms[C]//Proceedings of the Genetic and Evolutionary Computation Conference, Berlin, Jul 15-19, 2017. New York: ACM, 2017: 777-784.
[52] KIM J, MOON B R. A genetic algorithm for linear ordering problem using an approximate fitness evaluation[C]//Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, Vancouver, Jul 12-16, 2014. New York: ACM, 2014: 1461-1462.
[53] 杨瑞, 陈振宇, 张智轶, 等. 一种基于扩展有限状态机的自动化测试用例生成方法[J]. 中国科学: 信息科学, 2014, 44(5): 588-609.
YANG R, CHEN Z Y, ZHANG Z Y, et al. A new approach of automated test case generation on extended finite state machine[J]. Science in China: Information Sciences, 2014, 44(5): 588-609.
[54] WANG S, ALI S, YUE T, et al. UPMOA: an improved search algorithm to support user-preference multi-objective optimization[C]//Proceedings of the 2015 IEEE 26th International Symposium on Software Reliability Engineering, Gaithersbury, Nov 2-5, 2015. Piscataway: IEEE, 2015: 393-404.
[55] XU X, ZHU Z, JIAO L. An adaptive fitness function based on branch hardness for search based testing[C]//Proceedings of the Genetic and Evolutionary Computation Conference, Berlin, Jul 15-19, 2017. New York: ACM, 2017: 1335-1342.
[56] XU X, JIAO L, ZHU Z. A dynamic fitness function for search based software testing[C]//Proceedings of the Genetic and Evolutionary Computation Conference Companion, Kyoto, Jul 15-19, 2018. New York: ACM, 2018: 320-321.
[57] BRAIONE P, DENARO G, MATTAVELLI A, et al. Combining symbolic execution and search-based testing for programs with complex heap inputs[C]//Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, Santa Barbara, Jul 10-14, 2017. New York: ACM, 2017: 90-101.
[58] KIM J, KWON M, YOO S. Generating test input with deep reinforcement learning[C]//Proceedings of the 11th International Workshop on Search-Based Software Testing, Gothenburg, May 28-29. 2018. New York: ACM, 2018: 51-58.
[59] AYARI K, BOUKTIF S, ANTONIOL G. Automatic mutation test input data generation via ant colony[C]//Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation, London, Jul 7-11, 2007. New York: ACM, 2007: 1074-1081.
[60] JI F. Research on software testing automation mechanism based on improved genetic algorithms[J]. Information Technology, 2019 (10): 88-93.
[61] 张大林, 张哲玮, 王楠, 等. AutoUnit: 基于主动学习和预测引导的测试自动生成[J]. 计算机科学, 2022, 49(11): 39-48.
ZHANG D L, ZHANG Z W, WANG L, et al. AutoUnit: automatic test generation based on active learning and prediction guidance[J]. Computer Science, 2022, 49(11): 39-48.
[62] PADURARU C, MELEMCIUC M C, STEFANESCU A. A distributed implementation using apache spark of a genetic algorithm applied to test data generation[C]//Proceedings of the Genetic and Evolutionary Computation Conference, Berlin, Jul 15-19, 2017. New York: ACM, 2017: 1857-1863.
[63] FERGUSON R, KOREL B. The chaining approach for software test data generation[J]. ACM Transactions on Software Engineering and Methodology, 1996, 5(1): 63-86.
[64] KOREL B, AL-YAMI A M. Assertion-oriented automated test data generation[C]//Proceedings of the IEEE 18th International Conference on Software Engineering, Berlin, Mar 25-30, 1996. Piscataway: IEEE, 1996: 71-80.
[65] ENCK W, GILBERT P, HAN S, et al. TaintDroid: an information-flow tracking system for realtime privacy monitoring on smartphones[J]. ACM Transactions on Computer Systems, 2014, 32(2): 1-29.
[66] XU R, SA?DI H, ANDERSON R. Aurasium: practical policy enforcement for Android applications[C]//Proceedings of the 21st USENIX Security Symposium, Bellevue, Aug 8-10, 2012. Berkeley: USENIX, 2012: 539-552.
[67] TAM K, KHAN S J, FATTORI A, et al. CopperDroid: automatic reconstruction of Android malware behaviors[C]//Proceedings of the 2015 Network and Distributed System Security Symposium, San Diego, Feb 8-11, 2015. Reston: Internet Society, 2015: 1-15.
[68] ARZT S, RASTHOFER S, FRITZ C, et al. FlowDroid: precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for Android apps[J]. ACM SIGPLAN Notices, 2014, 49(6): 259-269.
[69] VALLéE-RAI R, CO P, GAGNON E, et al. Soot: a Java bytecode optimization framework[C]//Proceedings of the 1999 Conference of the Centre for Advanced Studies on Collaborative Research, Mississauga, Nov 8-11, 1999.
[70] 王蕾, 周卿, 何冬杰, 等. 面向Android应用隐私泄露检测的多源污点分析技术[J]. 软件学报, 2019, 30(2): 211-230.
WANG L, ZHOU Q, HE D J, et al. Multi-source taint analysis technique for privacy leak detection of Android apps[J]. Journal of Software, 2019, 30(2): 211-230.
[71] WEI F, ROY S, OU X. Amandroid: a precise and general inter-component data flow analysis framework for security vetting of Android apps[J]. ACM Transactions on Privacy and Security, 2018, 21(3): 1-32.
[72] Gordon M I, Kim D, Perkins J H, et al. Information flow analysis of Android applications in droidsafe[C]//Proceedings of the 2015 Network and Distributed System Security Symposium, San Diego, Feb 8-11, 2015. Reston: Internet Society, 2015: 110.
[73] CHEN K, LIU P, ZHANG Y. Achieving accuracy and scalability simultaneously in detecting application clones on Android markets[C]//Proceedings of the 36th International Conference on Software Engineering, Hyderabad, May 31-Jun 7, 2014. New York: ACM, 2014: 175-186.
[74]LIN J, LIU B, SADEH N, et al. Modeling users’ mobile app privacy preferences: restoring usability in a sea of permission settings[C]//Proceedings of the 10th Symposium on Usable Privacy and Security, Menlo Park, Jul 9-11, 2014. Berkeley: USENIX, 2014: 199-212.
[75] ZHOU W, ZHOU Y, JIANG X, et al. Detecting repackaged smartphone applications in third-party Android marketplaces[C]//Proceedings of the 2nd ACM Conference on Data and Application Security and Privacy, San Antonio, Feb 7-9, 2012. New York: ACM, 2012: 317-326.
[76] LIU B, LIU B, JIN H, et al. Efficient privilege de-escalation for ad libraries in mobile apps[C]//Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services, Florence, May 18-22, 2015. New York: ACM, 2015: 89-103.
[77] ISHIO T, KULA R G, KANDA T, et al. Software ingredients: detection of third-party component reuse in Java software release[C]//Proceedings of the 13th International Conference on Mining Software Repositories, Austin, May 14-22, 2016. New York: ACM, 2016: 339-350.
[78] BACKES M, BUGIEL S, DERR E. Reliable third-party library detection in Android and its security applications[C]//Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Oct 24-28, 2016. New York: ACM, 2016: 356-367.
[79] 黄思荣, 陶非凡, 张源, 等. LibSeeker: 参数自整定的安卓应用第三方库检测方法[J]. 小型微型计算机系统, 2019, 40(2): 332-340.
HUANG S R, TAO F F, ZHANG Y, et al. LibSeeker: detecting Android third-party libraries using parameter auto-tuning[J]. Journal of Chinese Computer Systems, 2019, 40(2): 332-340.
[80] XIAO J, CHEN S, HE Q, et al. An Android application risk evaluation framework based on minimum permission set identification[J]. Journal of Systems and Software, 2020, 163: 110533.
[81] 徐尉, 吴敬征. Android权限滥用检测系统[J]. 计算机系统应用, 2016, 25(10): 39-46.
XU W, WU J Z. Abused detection system of Android[J]. Computer Systems & Applications, 2016, 25(10): 39-46.
[82] LIU Z, XIA X, LO D, et al. Automatic, highly accurate app permission recommendation[J]. Automated Software Engineering, 2019, 26: 241-274.
[83] YU L, LUO X, QIAN C, et al. Enhancing the description-to-behavior fidelity in Android apps with privacy policy[J]. IEEE Transactions on Software Engineering, 2017, 44(9): 834-854.
[84] LIU F, HUANG H, SU J, et al. Manifold-inspired search-based algorithm for automated test case generation[J]. IEEE Transactions on Emerging Topics in Computing, 2022, 10(2): 1075-1090.
[85] 黄翰, 曹捷, 叶垒, 等. 交互式测试用例的自动生成方法: CN112506766A[P]. 广州: 华南理工大学, 2021.
HUANG H, CAO J, YE L, et al. Automatic generation of interactive test cases: CN112506766A[P]. Guangzhou: South China University of Technology, 2021.
[86] CAO J, HUANG H, LIU F. Android unit test case generation based on the strategy of multi-dimensional coverage[C]//Proceedings of the 2021 IEEE 7th International Conference on Cloud Computing and Intelligent Systems, Xi’an, Nov 7-8, 2021. Piscataway: IEEE, 2021: 114-121.
[87] ZHAO W X, ZHOU K, LI J, et al. A survey of large language models[J]. arXiv:2303.18223, 2023.
[88] MATHUR A, PRADHAN S, SONI P, et al. Automated test case generation using T5 and GPT-3[C]//Proceedings of the 2023 9th International Conference on Advanced Computing and Communication Systems, Coimbatore, Mar 17-18, 2023. Piscataway: IEEE, 2023, 1: 1986-1992.
[89] ZIMMERMANN D, KOZIOLEK A. Automating GUI-based software testing with GPT-3[C]//Proceedings of the 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops, Dublin, Apr 16-20, 2023. Piscataway: IEEE, 2023: 62-65.
[90] PRENNER J A, BABII H, ROBBES R. Can OpenAI??s codex fix bugs? An evaluation on QuixBugs[C]//Proceedings of the 3rd International Workshop on Automated Program Repair, Pittsburgh, May 19, 2022. New York: ACM, 2022: 69-75.
[91] DRAIN D, CLEMENT C B, SERRATO G, et al. DeepDebug: fixing python bugs using stack traces, backtranslation, and code skeletons[J]. arXiv:2105.09352, 2021.
[92] LUTELLIER T, PHAM H V, PANG L, et al. CoCoNuT: combining context-aware neural translation models using ensemble for program repair[C]//Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Jul 18-22, 2020. New York: ACM, 2020: 101-114.
[93] JIANG N, LUTELLIER T, TAN L. CURE: code-aware neural machine translation for automatic program repair[C]//Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering, Madrid, May 22-30, 2021. Piscataway: IEEE, 2021: 1161-1173.