It isn’t easy to score how good an AI-coded solution is. Of the possible metrics, code complexity might not be as meaningful as long as the AI understands the code, as would “maintainability,” as that’s based on human limitations; the AI can refactor on the fly. Test coverage is a good metric as it measures how well the AI-generated test suite covers the code.

Other subjective measures of the generated application are difficult to quantify. For example, UX (User Experience) and how well the problem was solved are more difficult to measure and automate. Still, a combination of all these metrics may serve as a good proxy for choosing the best candidates.

I will use radon, a Python tool that extracts various metrics from source code, and PyTest to pull other statistics:

• Raw statistics:
  • Logical lines of code (LLOC) - Every LLOC contains exactly one statement and is a good proxy for how “chatty” the code is. Chatty code is typically more complicated.
• Halstead metrics:
  • Difficulty - another measure of complexity.
  • Time to program (sec) - a size metric.
  • Delivered bugs - estimated bug rate.
• Cyclomatic Complexity:
  • Average complexity - measures path complexity in the code.
• Test coverage percentage - using PyTest coverage reporting.

Lastly, I will add a 1-5 score of the UX based on my interaction with the application. This score will include a sense of how well it addresses the given problem and how easy it is to use generally.

I will only use three of these metrics to compare solutions: UX score, time to program, and average complexity, but I will include the other metrics to double-check how the various statistics relate and compare. This approach will favor the best UX and tie-break using the other two size and complexity metrics.

The Good, The Bad…

Key Points

• How do the models compare (time to generate, variability, prompting, and overall quality)
• From the general to the specific - updating the problem to improve overall quality.
• Further areas to explore: performance, change management, AI-coding vs. Human-coding, mixing models.

The Problem

Version 1 of the problem asks for a basic four-function calculator as a command line REPL (Read-Eval-Print-Loop)

create a cli calculator that runs in a REPL loop and handles basic arithmetic. 
Ensure that operations follow the precedence rules for arithmetic, and handle
negative number literals properly, keeping in mind that consecutive minus signs
can precede a unary expression. 
For example: --3 is equal to 3. 
The negative sign should have higher precedence when preceding a unary
expression. 
The basic arithmetic operations (+,-,/,*) when used to take two arguments should
treat multiplication and division with higher priority than addition and 
subtraction. When precedent is equal, operations are handled in sequence.

The problem stated above was originally given without all the details about precedence, but using that description caused all the models to generate code that badly fumbled the arithmetic. So, I needed to add more context to improve the generated applications.

Tip: the more context you can provide to the AI models, the better the solutions will be—sometimes in surprising ways. However, a follow-up to this tip would be to add only as much context as needed. Don’t overdo it.

Problem One Results

GPT-4.1

The experiments are ordered as shown in the sort statement, with UX descending, and ascending by time-to-code, and complexity. This will be consistent throughout the following examples, leaving the top 2 experiments as the best candidates.

The sixth and second runs gave the best results, but there were some bad examples in experiments three and four - where the code did not recognize parentheses. This will cause me to add some more context to the problem later on (see Tip 1)

Speed-wise, GPT-4o is the fastest model, followed by GPT-4.1 and Claude 3.7. Another interesting thing to note is that LLOC and complexity will typically be low in the higher-ranked solutions, even though I’m not explicitly sorting by these features.

Example session:

GPT-4o and Claude 3.7 produced similar results on this problem, so I will skip ahead to the revised problem, adding in the mention of parentheses handling, and also adding a requirement for a ‘help’ command.

Problem Update 1

I added this context to the problem:

Allow expressions to be inside parentheses.
Provide a help command that explains how to use the calculator. 

GPT-4.1

GPT-4o

Claude 3.7

All the models produced good solutions to this revised problem, with some surprises.

Note the variability in LLOC and difficulty. The metrics related to complexity and size are more or less scattered depending on the model. Of these, Claude 3.7 shows the most variability. Surprisingly, Claude produced both the best and the worst solution in this run.

I need to add more data to confirm my intuition that Claude 3.7 tends to produce more complex and diverse solutions than GPT-4.1 and GPT-4o, yet I’ve noticed this in other projects. It can produce very high-quality solutions, but it’s hard to predict what you’ll get. Adding more context often improves the result, but adding more cases is a little painful, as Claude is chatty and slower than GPT-4.1 or GPT-4o.

I don’t want to pick on Claude 3.7. I think I may also need to provide better (or different) agent prompting to my framework tuned specifically to Claude 3.7—that may help as well.

With all this said, Claude produced the best version, which included command line history, even though I didn’t ask for that. So, was this “creativity” at work or something else? I’ll leave that discussion for another day.

Example session:

Final Version

In the final iteration, I added:

the calculator should support command line history via up/down arrow.

Overall, for this exercise, GPT-4.1 turned out to be the best option, as it shows more consistent results and runs faster than Claude 3.7.

GPT-4.1

Example session:

Further Areas to Explore

This experimental framework has been a useful and fun tool for exploring AI code generation.

A key takeaway is the value of adding context to the initial prompt used for code generation. This technique proves useful across all the models.

Choosing metrics to evaluate solutions is still an art form. In many cases, I think the way AI models generate code is based on the human examples they have been trained on. So, for now, some of these metrics are still useful for automated evaluation—especially ones related to complexity and code coverage. In the future who knows?

We still need a “human in the loop” to evaluate the user experience, but perhaps a mixture of automated evaluations can help us narrow down the candidates for review.

Other topics:

• Change management - how to minimize the impact of new requirements.
• Performance - how to scale the generation of code.
• Evaluation - more ideas are needed.
• Computer Language
  • Are AI’s limited by human-designed languages?
  • Could they do better with languages designed by AI’s themselves?
  • What metrics would apply to evaluating these?

I’ll explore more on these topics in future posts.

Especially that last one 😀.

I’ve also become fascinated by how these tools will change the way programmers think about code and its organization — how many rules will be thrown completely out the window and how, oddly, the new rules will harken back to the early days of programming before Google and the Internet.

One of these ideas is that we may no longer need so many dependencies in our code bases. Eventually, all LLMs will be able to generate all of the code needed and, more importantly, only the code needed to solve any given problem.

Why do we have so many dependencies in our projects? In a nutshell, programmers are lazy. Alright, maybe it’s just because we are human beings with limited time and memory. We solve related problems and lump code into libraries to save time through reuse; cognitively, we can stay focused on the main problems we are trying to solve.

Why do LLMs need so many dependencies in their generated projects? Perhaps because they learned to code from humans? “Cargo culting” all of that code has brought certain old problems into the equation, such as security exploits and buggy dependencies.

We can ask our LLMs to code from “first principles.”

To learn more and explore some of these ideas while also having a meaty example to later test on the open source and commercial tools I mentioned earlier, I decided to use a powerful code generation model, Claude 3.7, to generate an “agentic” code generator from first principles.

The journey has revealed some insights and taken on a life of its own.

Hey Claude, make this…

Using a Claude Desktop project, I generated a basic code generation framework using these prompts:

Write a python program which implements an agentic flow from basic
principles, using the openai api and the GPT-4o model.
There will be two agents:
1. Architect - an agent which takes a problem description and 
   converts it into a plan for developing a software solution.
2. The second agent will be a software engineer that can write python code
   which implements the system outlined by the architect agent.

we need to add a third agent to this framework that will be able to compile
and test the code developed by the software engineer. It will take the
implementation from the software engineer agent, and return a test_report
showing whether the code is compiling, and whether the tests are passing,
along with suggestions how the code might be fixed if it fails to compile
or run properly.
The report may itself be a json object with a boolean 'passed' key,
along with 'results' key to suggested fixes if passed == False, or test results
if passed = True.
we can adjust the software engineer agent to look for test_report,
along with the software engineer specification, and its own last-generated
implementation. If the test_report shows passed == False, it can use the
suggestions from the test agent to attempt to fix the implementation.
We can then put this interaction in a loop that breaks once the passed == True
is returned from the test agent, or after a maximum number of attempts.

The resulting code worked, but the testing agent did not actually run tests; rather, it generated test cases and then attempted to predict whether each test case passed. The results were passed back and integrated into future passes. So, although it hallucinated its ability to interpret the code, it did create the overall flow correctly.

So, I needed to create a real software testing agent, for which I used the following prompts to generate:

Create a python class that can take a python program as text and
perform the following:
1. Determine the python packages needed by the program.
2. Create a Dockerfile that loads the required packages along
   with the pytest framework required to run tests against the program.

but our code, implementation.py, and its test test_implementation.py
need to be copied over to app dir in the Dockerfile, and the pytest
command can be run with app as the curent dir, so tests will run.
Make the changes needed for this to be done in the dockerfile

let's change this code as follows:
1. replace the sample_code with a check for a src_dir on the command line,
   from which python files are loaded, and checked for dependencies so that
   a requirements.txt file can be created.
2. the src_dir will be used in the Docker file to copy python files into
   the WORKDIR

minor change to the requirements file generation needed to exclude package
names matching the application files loaded from the source directory.

Finally, Claude returns code for a testing framework:

Putting all the pieces together

Putting the code together into a project, I ended up adding a few enhancements, including the ability to choose other model providers. I used Claude 3.7 to analyze the combined code and asked it to create a DOT-language (Graphviz) representation of the code’s flowchart below using the following prompt:

Examine the code and create a flowchart of its operation using the DOT language.
(code attached)

As was my goal, the generated agent code has just standard packages as dependencies (shown below). Although I initially had Claude create the ability to choose different model providers, I eventually chose LiteLLM to make it easier for me to compare output from other providers without having to re-generate this part.

The current project can be found at CodeAgents on GitHub

Taking it out for a test drive

I have included notable examples of two projects that I generated using this framework in the GitHub repository above: a scientific calculator as CLI and a weather API using FastAPI. I found these project types useful for testing across different models and providers. They are non-trivial enough to be interesting and small enough to run frequent passes with different providers. In my next post on this topic, I’ll present some results and conclusions about these comparisons.

Coming next

For now, I can say the promise of “hands-free” code generation of production-quality code is still a way off, although I can see no reason why the quality won’t continue to improve. I think we need to set our expectations accordingly. One thing is clear: These tools will change the way we develop software - mostly for the better.