The ultimate objective for the project is to have AI write clear and concise end-to-end (E2E) test specifications in Cypress.IO.
Unlike standard PHP and JavaScript, which AI has become fairly proficient at when writing new code, it turns out writing Cypress test specifications appears to be a new challenge for AI. It is likely due to the fact that things like Cypress specifications are built on a myriad of higher level JavaScript layers, frameworks to be more precise, that are combined in novel ways. The AI has far less training with these models as few companies write E2E test cases for their applications and of those that do few publish online. These extra layers of logic also rely on some knowledge of the application output specifically the exact HTML structure that is rendered by the underlying SaaS application.
The theory we are testing in this research paper is that some AI models will be more proficient than others at writing complex Cypress.IO test specification. Our initial testing that led to this paper was the discovery that AI is straight up awful at writing Cypress test specifications. Even after multiple prompt revisions, AGENTS.md precursor rules to assist in writing better scripts, AI still struggles. We have yet to build an AI prompt stack that generates specifications that come anywhere close to a final proper specification.
This research is meant to find the best LLM as a foundation on which to build these prompt stacks.