Analyst: Dr. Bob Sutor
Publication Date: August 29, 2024
Original Document #: AIESBS202408
Key Points
- In July, the US Defense Advanced Research Projects Agency (DARPA) started the Quantum Benchmarking Initiative (QBI).
- On September 3, DARPA will host a meeting for US and international experts who believe they and their partners can build such sufficiently powerful systems. DARPA will explain its goals and engagement rules and answer participants’ questions.
- However, is there any point in having quantum benchmarks right now? Are the systems too small to extrapolate their performance to the powerful machines we will eventually need?
Quantum Computing Benchmarks – Too Soon?
Analyst Take: Quantum systems today vary in almost every way you can imagine, in the number of qubits, the fidelity of operations, qubit modality, cost per qubit, energy usage, cooling required, physical footprint, and manufacturability. We’re not even sure which quantum computing applications will show the most value or which specific algorithms we will use at that time.
The DARPA QBI
Rather than only developing tests to prove that this quantum processing unit (QPU) is faster than that on algorithms or small applications, the QBI asks a profound question:
“[Is it] possible to actually build an industrially useful quantum
computer much faster than conventional predictions[?]”
Indeed, they want to know if it will be possible to build such a system in the first place. Why are we bothering with all the other quantum software and hardware development if we can’t?
Benchmarks
The computing industry has long used benchmarks to measure performance. Vendors use them to show their new chips and systems are better than their old ones and often issue press releases when their results significantly outperform those of competitors. We have benchmarks for PC performance and classical algorithms, and researchers are developing new ones for machine learning. Will AI become a part of benchmarks?
Quantum Benchmarks
Today’s quantum benchmarks can be challenging to understand if you do not have a technical knowledge of qubits, modalities, quantum gates and operations, and connection topologies. While we have the so-called quantum application benchmarks, they are not yet the use cases that companies or consumers will recognize or deploy. Doing better on a benchmark can be as much a case of improving your software and development tool chain as it is of switching hardware.
Should We Bother Now?
Suppose you are in 1903 and looking at the airplane designed by the Wright brothers. Someone suggests you develop benchmarks for a plane that can fly from New York to London at supersonic speed. Might that have been a tad early? Could you meaningfully extrapolate from the day’s technology to the eventual requirements? The QBI is asking the same kind of question for quantum computing. There must be some value in starting now.
The full report is available via subscription to the Quantum IQ service from Futurum Intelligence – click here for inquiry and access.
Disclosure: The Futurum Group is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.
Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of The Futurum Group as a whole.
Other insights from The Futurum Group:
Quantum in Context: A Qubit Primer
Enterprising Insights: Episode 33 – Salesforce Announces an LLM Benchmark for CRM
Quantum in Context: The Case for On-Premises Quantum Computers
Author Information
Dr. Bob Sutor has been a technical leader and executive in the IT industry for over 40 years. Bob’s industry role is to advance quantum and AI technologies by building strong business, partner, technical, and educational ecosystems. The singular goal is to evolve quantum and AI to help solve some of the critical computational problems facing society today. Bob is widely quoted in the press, delivers conference keynotes, and works with industry analysts and investors to accelerate understanding and adoption of quantum technologies. Bob is the Vice President and Practice Lead for Emerging Technologies at The Futurum Group. He helps clients understand sophisticated technologies in order to make the best use of them for success in their organizations and industries. He is also an Adjunct Professor in the Department of Computer Science and Engineering at the University at Buffalo, New York, USA. More than two decades of Bob’s career were spent in IBM Research in New York. During his time there, he worked on or led efforts in symbolic mathematical computation, optimization, AI, blockchain, and quantum computing. He was also an executive on the software side of the IBM business in areas including middleware, software on Linux, mobile, open source, and emerging industry standards. He was the Vice President of Corporate Development and, later, Chief Quantum Advocate, at Infleqtion, a quantum computing and quantum sensing company based in Boulder, Colorado USA. Bob is a theoretical mathematician by training, has a Ph.D. from Princeton University, and an undergraduate degree from Harvard College.
He’s the author of a book about quantum computing called Dancing with Qubits, which was published in 2019, with the Second Edition released in March 2024. He is also the author of the 2021 book Dancing with Python, an introduction to Python coding for classical and quantum computing. Areas in which he’s worked: quantum computing, AI, blockchain, mathematics and mathematical software, Linux, open source, standards management, product management and marketing, computer algebra, and web standards.
