MaintainCoder is a code generation framework focused on maintainability, combining software design principles with multi-agent collaboration. It outperforms existing methods on dynamic benchmarks from our proposed MaintainBench.
 |
| Overview of MaintainCoder.MaintainCoder follows a two-stage pipeline: a framework module builds maintainable architecture from requirements, and a generation module produces code aligned with both design and intent. |
 |
| The systematic construction of MaintainBench. MaintainBench systematically evolves curated problems via structured requirement changes, co-evolves code and tests, and ensures quality through testing and expert review. |
Clone the repository and install dependencies:
git clone https://github.com/IAAR-Shanghai/MaintainCoder.git
cd MaintainCoder
pip install -r requirements.txtThe main functionality is accessible through the main.py script which accepts multiple parameters:
python main.py --method <method> --dataset_name <dataset> --evaluate <true/false> [additional options]-
--method: Generation method to usegenerate_raw_agent: Use agent-based model to generate codegenerate_raw_llm: Use LLM to generate code with specific prompt typesgenerate_dyn: Generate code that adapts to changing requirementsskip: Skip generation phase
-
--prompt_type: Type of prompting (for LLM generation)direct: Direct promptingCOT: Chain-of-thought promptingselfplanning: Self-planning approach
-
--dataset_name: Dataset to use- Options:
apps_dyn,codecontests_dyn,humaneval_dyn,mbpp_dyn,xcodeeval_dyn
- Options:
-
--evaluate: Whether to evaluate the generated code (true/false) -
--api_key: API key for the LLM service (optional) -
--base_url: Base URL for the API (optional) -
--model_name: Model name to use (optional)
The code evaluates several aspects of the generated solutions:
- Pass5 Score: Measures how many generated solutions pass all test cases across 5 iterations
- Maintainability Index (MI): A software metric that indicates how maintainable the code is
- Cyclomatic Complexity (CC): A measure of code complexity
- AST Similarity: Measures structural similarity between original and modified code
- Code Change Metrics:
- Ratio: Percentage of code changed
- Absolute: Absolute amount of code changed
Generate code using MaintainCoder on HumanEval dataset:
python main.py --method generate_raw_agent --dataset_name humaneval_dyn --evaluate false --model_name gpt-4o-miniGenerate code using LLM with direct prompting on HumanEval dataset:
python main.py --method generate_raw_llm --prompt_type direct --dataset_name humaneval_dyn --evaluate false --model_name gpt-4o-miniGenerate code adaptations for changing requirements:
python main.py --method generate_dyn --dataset_name mbpp_dyn --evaluate false --model_name gpt-4o-miniEvaluate previously generated code without generating new code:
python main.py --method skip --prompt_type direct --dataset_name codecontests_dyn --evaluate true --model_name gpt-4o-miniThis project benefits from the following open-source projects:
For any questions or feedback, please reach out to us via GitHub Issues.