Generative AI has many applications, including code generation. I usually program in Python, so I tested five code generators to see how they performed on an HTML file search operation. I restricted myself to browser-based generators. I did not test all such generators, but the results show an interesting range in quality and completeness.
The Task: Extracting the Title from an HTML File
I do a lot of text manipulation using Python. For example, the text source for my book Dancing with Qubits, Second Edition, combines HTML and LaTeX. I process the HTML and pull out the LaTeX in PRE blocks that have a special class designator. I run these through pdflatex to create small PDF files and then use ImageMagick to generate JPG images. I then insert IMG tags with links to the images into the final HTML file.
I also have special SPAN tags with HTML math in the content areas and LaTeX markup in an attribute. This combination allowed Packt Publishing and me to create an eBook, a PDF for printing the book, and a color-highlighted PDF for purchasers of the other two forms. Sometimes, I think it took me longer to write the processing software than the book itself.
Over several years, I wrote all the code and used an earlier version to generate my Dancing with Python book. I use certain idioms repeatedly, especially to parse HTML files. The workhorse Python package I use is beautifulsoup4. It has methods to read a file or string containing HTML and then search or navigate the document tree.
Since I had not used a generative AI code generator before except for regular expressions, I thought a fun initial task would be asking several of them to write code to read an HTML file and print the TITLE tag’s contents. There are more code generators than I have shown, and I have only used browser-based prompt and response services.
Many AI code generators have extensions for Integrated Development Environments (IDEs), such as Visual Studio Code. I will examine some of these in a future Analyst Insight.
The Prompt
The prompt I used with each code generation service is:
Write me Python code that parses an HTML file and prints the title.
I will score the code this way:
-
- 0 points total if the code is not Python. I stop looking at the code.
- 5 points if it reads a file, 1 point if it uses a with statement, and 1 point if it specifies an encoding.
- 5 points if it checks that the HTML file exists.
- If it doesn’t read a file, 1 point if it reads a string.
- 5 points if it finds the TITLE tag and contents.
- 1 point if it checks that the title exists
- 2 points if it prints the title.
The code should be in Python, read an HTML file using modern Python syntax, understand that the title is in an HTML tag, extract the tag’s contents, and print the result. The maximum score is 20.
The input:
The output:
The score: 8 = 1 (reads string) + 5 (finds title) + 2 (prints title)
The main issue with this code is that I asked for code to read a file, and it reads a string instead.
The input:
The output:
The score: 13 = 6 (reads file using with) + 5 (finds title) + 2 (prints title)
This code did not get the points for checking that the file exists, using file encoding, or checking that the title existed.
Microsoft Copilot
The input:
The output:
The score: 14 = 6 (reads file using with) + 5 (finds title) + 1 (checks title exists) + 2 (prints title)
Very nice touch at the end noting that I should change the file name in the code! The code should have checked that the file existed.
The input:
The output:
The score: 15 = 7 (reads file using with and encoding) + 5 (finds title) + 1 (checks that the title exists) + 2 (prints title)
This is great code, except it doesn’t check if the file exists. The extra factoring out of the function to find the title is very nice, as is the instruction for installing the beautifulsoup4 package.
The input:
The output:
The score: 19 = 5 (check if file exists) + 6 (uses file and reads it using with) + 5 (finds title) + 1 (checks that the file exists) + 2 (prints title)
This is the most complete code with the top score, only losing a point by not specifying a file encoding. It has extensive error checking, something I would require in production quality code.
Key Takeaway: Generative AI code generators can give developers a head start in developing applications. Human intervention may be required to ensure the code checks for errors and uses modern coding practices and libraries.
I was impressed at first with the code these five services generated. They appeared to use the standard Python package to perform the requested action. On closer comparison, one did not follow the prompt well, and most others had limited to no error checking. Do not use generated code without carefully inspecting what it does and how. Though my example did not have security considerations, you should carefully examine the code to guarantee it does not open holes that hackers could exploit.
Disclosure: The Futurum Group is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.
Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of The Futurum Group as a whole.
Other Insights from The Futurum Group:
AI in Context: UXL to Be an Open-Source Alternative to NVIDIA’s CUDA?
Quantum in Context: A Qubit Primer
AI in Context: Remarks on the NVIDIA GTC 2024 GenAI and Ethics Panel
Author Information
Dr. Bob Sutor has been a technical leader and executive in the IT industry for over 40 years. Bob’s industry role is to advance quantum and AI technologies by building strong business, partner, technical, and educational ecosystems. The singular goal is to evolve quantum and AI to help solve some of the critical computational problems facing society today. Bob is widely quoted in the press, delivers conference keynotes, and works with industry analysts and investors to accelerate understanding and adoption of quantum technologies. Bob is the Vice President and Practice Lead for Emerging Technologies at The Futurum Group. He helps clients understand sophisticated technologies in order to make the best use of them for success in their organizations and industries. He is also an Adjunct Professor in the Department of Computer Science and Engineering at the University at Buffalo, New York, USA. More than two decades of Bob’s career were spent in IBM Research in New York. During his time there, he worked on or led efforts in symbolic mathematical computation, optimization, AI, blockchain, and quantum computing. He was also an executive on the software side of the IBM business in areas including middleware, software on Linux, mobile, open source, and emerging industry standards. He was the Vice President of Corporate Development and, later, Chief Quantum Advocate, at Infleqtion, a quantum computing and quantum sensing company based in Boulder, Colorado USA. Bob is a theoretical mathematician by training, has a Ph.D. from Princeton University, and an undergraduate degree from Harvard College.
He’s the author of a book about quantum computing called Dancing with Qubits, which was published in 2019, with the Second Edition released in March 2024. He is also the author of the 2021 book Dancing with Python, an introduction to Python coding for classical and quantum computing. Areas in which he’s worked: quantum computing, AI, blockchain, mathematics and mathematical software, Linux, open source, standards management, product management and marketing, computer algebra, and web standards.
