Skip to content

Add auto-review admonishments and instructions to try and improve generated code - #5103

Merged
adamnovak merged 13 commits into
masterfrom
auto-review-skills
Nov 13, 2026
Merged

adamnovak merged 13 commits into
masterfrom
auto-review-skills

Conversation

@adamnovak

Copy link
Copy Markdown
Member

Changelog Entry

To be copied to the draft changelog by merger:

  • Users working on vg with Anthropic Claude should now have an automated code review process, based on that from the UCSC Genome Browser.

Description

This adds some skill files adapted from those shared with me by Maximilian Haeussler from the Genome Browser group, which they are using for their own automated post-merge code review.

I sent the robot to try and do some example reviews and compare them against actual reviews of those PRs, and to then try and tweak the instructions to improve the automated reviews. "Just tell system to work better" is not ranked very high on the list of validated optimization techniques, so it's not clear if that approach will have helped. If you are getting reviews from this you do not like, please double down on this dubious technique and keep admonishing it to improve.

Automated Commit and others added 7 commits November 10, 2026 16:36
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
…build

Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
An unattended session no longer waits on a permission prompt. The agent
can send unreviewed code by prefixing the command with VG_SEND_UNREVIEWED=1.

Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
@adamnovak

Copy link
Copy Markdown
Member Author

This seems to be OK at finding potential bugs (among a bunch of non-bugs for cases that implicitly aren't allowed, which would be useful if we were trying to program that defensively). It doesn't really seem to think about design or catch bad design; it really really likes tests and comprehensiveness and error handling. It might be missing a "design lens".

Automated Commit and others added 6 commits November 12, 2026 13:01
The lens draws on vg review comments from the last two months, Real
Python's checklist for AI-generated code, and screened web research.

Tests may share code when each test still reads as plain statements.

Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
@adamnovak

Copy link
Copy Markdown
Member Author

I asked for a notion of design review, using the Semble vector database to try and enable the review process to detect duplicate ideas across the codebase without trying to load the whole project up into a language model context.

I have no idea whether this works well, since the circular evaluations used during generation can't really count, but we will find out.

@adamnovak

Copy link
Copy Markdown
Member Author

There are some new generated TODOs in the surjector that came from comments from the test reviews. I think they indicate places where we might want reassuring comments more than places where we really need behavior to change, but I asked for them to be added so they could maybe eventually be answered.

@adamnovak
adamnovak merged commit aa1adc2 into master Nov 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant