Skip to content
MenuClose

Superintelligence: Beyond Human Expertise

How today's AI could advance beyond human intelligence, and what that would mean for discovery and security.

By Nouman Nafees, Founder, First Principles SI

13 min readAnalysis

A graphite surface rises into two peaks, with blue light revealing its contours and a fine gold path crossing the form.
On this page

In 1950, Alan Turing opened a paper with a question: “Can machines think?” He proposed an imitation game that made the discussion more concrete. A machine would be judged through written exchanges, giving people something to examine beyond their intuitions about thinking.1

The question has since grown more ambitious. Could a machine develop a scientific explanation that had escaped generations of researchers? Could it design an experiment that changes what we know? Could it bring together ideas from mathematics, biology, and engineering to solve a problem that none of those fields could solve alone?

Superintelligence gives that ambition a name. To make the idea useful, we need to ask what such a system would be capable of, how it would handle unfamiliar situations, and what would count as evidence of its abilities.

Today's AI makes these questions immediate. Systems can work through mathematical problems, write and test code, and use tools to investigate a task. Those capabilities have identifiable technical foundations.23 Understanding how they develop gives us a way to examine the prospect of superintelligence without relying on the excitement surrounding each new release.

What would qualify as superintelligence?

We use superintelligence to mean an artificial system whose intellectual abilities substantially exceed those of the best human experts across nearly all major intellectual domains. This follows a longstanding conception associated with philosopher Nick Bostrom.4

The standard is demanding. It includes learning unfamiliar subjects, developing explanations, designing technologies, and planning through problems whose solutions require many connected decisions. A system that meets it could contribute ideas beyond the reach of individual experts and extend the work of research teams.

Breadth matters. A system can be exceptionally strong within a defined task while offering little help outside it. General intelligence concerns the ability to work across different kinds of problems; superintelligence sets a much higher standard for performance across that range. Consciousness and subjective experience remain separate questions.

There is considerable room for disagreement over how to measure such a system. Which tasks count? Which experts form the comparison? How much time, assistance, and computing power does each participant receive? A framework proposed by Google DeepMind researchers separates performance, generality, and autonomy, helping make these distinctions explicit.5

Those dimensions also prevent an important confusion. Reasoning ability and permission to act are separate design choices. An exceptional analyst might have read-only access. A comparatively simple program might hold credentials that let it interrupt a factory.

Inside today's AI

A chatbot is the interface through which we encounter a much larger computational process. In a common training approach, a language model receives sequences of text and learns to predict the next token, a unit that can represent part of a word, a word, or a symbol. Training adjusts the network's numerical parameters to make those predictions more accurate. Across many examples, it learns relationships that support language, factual recall, and problem solving.6

Generating an answer involves applying those learned relationships to the current input. The model can combine information and respond to situations that differ from its training examples. Research examining a model's internal computation has found mechanisms supporting limited planning and concepts shared across languages.7 These findings concern particular tasks and models, but they show why describing the process as collecting and rearranging passages is incomplete.

Further training changes how that capability is used. Demonstrations can teach useful response patterns. Reinforcement learning can reward successful attempts at a problem, increasing the likelihood of strategies that work. Research published in 2025 showed substantial improvements in mathematical and programming tasks through this kind of training, including behaviors resembling checking and revising an approach.2

Computation during a task provides another source of improvement. A system can explore several candidates, examine intermediate steps, or revise an answer before returning it. Research on this approach finds that the benefit depends on the problem and on how that computation is allocated.8

Tools extend the process beyond generation. A coding assistant can run a program, inspect the error, and try a correction. A research assistant can retrieve a source and use its contents to revise a claim. An agent combines a model with a process for selecting actions, observing results, and continuing the task.3

These changes happen on different timescales. Adapting an answer to information in the conversation can happen with fixed model parameters. Persistently changing the underlying model requires an additional learning process.6 A system's memory, tools, and training therefore deserve separate examination when assessing how it improves.

An old ambition, a larger possibility

The proposal for the 1956 Dartmouth research meeting was written in August 1955. Its authors conjectured that learning and other features of intelligence could be described precisely enough for machines to simulate them.9

In 1965, mathematician I. J. Good considered what might follow if a machine surpassed human intellectual ability. Machine design would fall within its capabilities. It could therefore help design a more capable successor. Good's argument introduced a possible feedback process: intelligence improving the means of producing intelligence.10

That possibility helps explain the scale of today's ambition. In The Intelligence Age, Sam Altman describes AI extending the accumulated capabilities of society. In The Gentle Singularity, he places particular weight on AI accelerating scientific work, including AI research itself. These are forecasts about what the technology could enable.1112

We find the scientific possibility especially compelling. A capable research system could compare competing explanations, propose experiments, and search for useful connections across disciplines. Better materials could change energy storage. Better biological models could guide laboratory work. Better engineering methods could make essential infrastructure easier to maintain.

The pace would depend on practical conditions. Hardware takes time to manufacture. Experiments need equipment, resources, and reliable measurements. Improvements have to survive evaluation outside the setting in which they were developed. A forecast about intelligence leaves these questions open.

Why further progress is plausible

Human knowledge gives a learning system a starting point. Search, experimentation, and feedback can take it into possibilities that its designers have never examined individually.

AlphaZero offers a concrete example. DeepMind's 2017 research reported superhuman play in chess, shogi, and Go using a common learning approach trained separately for each game. Self-play supplied experience; the game rules supplied a way to determine the outcome.13

There are also examples of language models contributing new results. FunSearch, reported in December 2023, paired a model that proposed computer programs with an automated evaluator. It produced new constructions for a mathematical problem and improved heuristics for packing items into containers. Researchers could run and inspect the resulting programs.14

The scientific contribution came from the complete process: proposing candidates, testing them, retaining useful results, and searching again. Human researchers supplied the problem and evaluation procedure. The resulting discoveries were bounded advances whose value could be checked.

Search on a two-peak score surface. After the constraint x + y ≤ 1.05 excludes the higher peak, the best observed feasible candidate lies on the lower peak.

The search scores candidate designs and refines promising regions. When a new constraint excludes the previous best candidate, the search continues within the permitted region.

Method: Halton sampling with local refinement. Design variables and scores are dimensionless.

Method

Objective. For design variables x and y between 0 and 1, the search maximizes f(x, y) = 1.1·exp(−((x − 0.70)² + (y − 0.55)²) / 0.025) + 0.75·exp(−((x − 0.28)² + (y − 0.72)²) / 0.040) − 0.10·((x − 0.5)² + (y − 0.5)²). The same function draws the surface. The search receives a score only for each candidate it proposes.

Sequence. Candidates for exploration come from a Halton sequence with bases 2 and 3: indices 1 to 32 before the constraint, and 33 to 48 after it.

  1. Explore. Score Halton indices 1 to 32. The best observed candidate is replaced only by a strictly higher score.
  2. Refine. Ten rounds. Each round proposes four candidates around the best observed one, offset by ±d along each variable, with d = 0.08 × 0.75^(round − 1). All four are scored before the best observed candidate is updated.
  3. Constraint changes. The constraint x + y ≤ 1.05 is added. The scored candidates are reclassified without new scoring, and the best feasible one is selected.
  4. Search again. Score Halton indices 33 to 48, then refine for another ten rounds. Proposals that violate the constraint are rejected before scoring.

Result. The search makes 128 proposals: 117 candidates are scored and 11 are rejected before scoring. Before the constraint, the best observed score is 1.0996 at (0.6957, 0.5532). The constraint excludes 53 scored candidates, and the best remaining one scores 0.6741. The search settles on a best observed feasible score of 0.7405 at (0.2791, 0.7209), where x + y = 1.0000. “Best observed” means best among the candidates scored.

AI is also becoming a useful scientific instrument. The 2021 AlphaFold paper demonstrated protein structure predictions competitive with experimental structures in a majority of the CASP14 assessment cases.15 AlphaEvolve, introduced by DeepMind in 2025, combines proposed programs with automated evaluation and an evolutionary search process. In May 2026, its developers reported applications across scientific and computing problems, including electricity-grid optimization.1617

This is the case for further progress that we find compelling. Better models can propose more promising experiments. Better evaluators can distinguish useful results. Tools let systems test proposals and bring the findings into the next attempt. Improvements in these components could reinforce one another, including when the work concerns the design of AI itself.

How far that process can extend remains open. Games and executable programs often offer feedback that is difficult to obtain elsewhere. A laboratory experiment may take months. A scientific explanation may remain unsettled for years. Learning across those conditions requires effective ways to gather evidence, preserve what has been learned, and recognize when an evaluation is inadequate.2 Broad superintelligence would have to meet that challenge across many fields.

What an expert would need to see

An explanation can be valuable because it is new to its reader. Advancing a field requires a further contribution: an insight, method, or result that improves on what specialists already know and can withstand their examination.

Here, innovation means a useful advance whose novelty and merit can be established against existing work. It might be a faster algorithm under the same constraints, an experiment that separates competing explanations, or a material whose predicted properties survive measurement. An unfamiliar suggestion earns attention; comparison and testing establish its value.

This is where expertise changes the assessment. A specialist can notice an omitted constraint, recognize an old approach under new terminology, or identify an experiment that could disprove a claim. We would expect superintelligence to perform those checks as a routine part of its own work, with little need for someone to prompt each one.

A model learns through signals available during training. Those signals cannot establish every claim it will later make. When a task lacks a reliable check, an unsupported premise can survive through a fluent, internally consistent answer. Finding a dependable way to evaluate the work remains a central challenge.2

Current systems show an uneven ability to do this. Recent developer assessments still report substantial gaps in substituting for research scientists and engineers.18 Studies of internal computation also show cases where a model's persuasive explanation diverges from the process that produced its answer.7 Reliable contribution requires evidence that survives checking outside the model's own account.

Context is part of the capability

A useful system needs to understand what matters in the situation before it. That includes the purpose of the work, the constraints, the history of previous attempts, and the reliability of the information available.

Consider a request to design a better battery. “Better” might mean lighter, cheaper, longer-lasting, easier to recycle, or safer at high temperatures. An intelligent collaborator would clarify the trade-offs, identify missing evidence, and revise its proposal when a manufacturing constraint changes. Its reasoning would have to connect the scientific question to the conditions of use.

A large context window tells us how much material a model can receive at once. Effective use of that material requires further evaluation. The 2024 study Lost in the Middle found that the models it tested often performed worse when relevant information appeared in the middle of long inputs. That historical result illustrates why input capacity and successful use of information need separate tests; it does not establish the performance of every model available today.19

For a claim as strong as superintelligence, we would expect the system to identify relevant evidence, notice conflicts, retain important constraints, and update its reasoning as circumstances change. It should also recognize when a missing observation prevents a reliable answer.

Understanding a rule can be tested through its consequences. Change the wording while preserving the meaning: the system should retain the relevant constraint. Change a condition that makes the rule applicable: its judgment should change accordingly. In the battery example, recognizing a temperature limit would include considering operating conditions and measurement uncertainty. These are tests of practical competence, while questions about subjective understanding remain separate.

The assumptions deserve interrogation

Suppose we ask a system to reduce energy use in a building. Which assumptions shape its answer? The occupancy forecast may be wrong. Comfort requirements may be incomplete. A sensor may have drifted. A strategy that looks efficient during one season may perform poorly during another.

A capable system should help expose those assumptions and examine their consequences. It should distinguish observations from estimates, consider alternative explanations, and suggest a useful next test. Sometimes the best contribution will be identifying that the original question has omitted something consequential.

Interrogation also needs judgment. A system should identify which assumptions materially affect the conclusion and concentrate its effort there. An endless list of objections offers little help to an engineer deciding what to test next. Working from first principles means examining the underlying mechanism, the evidence for it, and the conditions under which the proposed solution should work.

“Science is a way of thinking much more than it is a body of knowledge.”

Carl Sagan, 1989.20

Sagan's observation gives us a useful standard. We should expect advanced intelligence to improve the way we investigate a problem, including how we discover that an attractive explanation is wrong.

The evaluation process needs the same scrutiny. A score rewards what its designer has chosen to measure. If the measure leaves out an important constraint, optimizing it can produce an unsuitable result. Controlled research on specification gaming illustrates how systems can learn to exploit weaknesses in a reward process.21 A convincing demonstration therefore needs checks on both the answer and the method used to judge it.

What evidence should we ask for?

Four questions would guide our assessment. They give the ambition concrete requirements against which progress can be examined.

First, does the system succeed on unfamiliar problems? Evaluation should examine transfer across domains and account for possible exposure to test material. Rephrasing a familiar exercise offers limited evidence of a new capability.

Second, does it sustain useful work over time? A long investigation involves intermediate decisions, tool use, corrections, and changing plans. METR evaluates task completion against the time human experts take on the tested software tasks. Its 2026 reporting also explains how task selection and modeling assumptions affect the estimates.2223 Such measurements help us examine progress within their stated scope.

Third, how does it handle uncertainty? We would examine whether confidence tracks observed success, whether the system seeks relevant evidence, and whether it revises a conclusion when that evidence changes. Recognizing an unresolved question can be a valuable result.

Fourth, can others check the contribution? Depending on the task, this might mean a proof, reproducible code, an experiment, or a prediction tested on new observations. Discovery becomes useful when other people can establish what has been found and where it applies.

A simple calculation shows why sustained reliability deserves its own attention. Suppose a task has 100 essential steps. For illustration, assume each step succeeds independently with probability 99%, and any failed step ends the attempt without recovery. Then:

Here, p is the per-step success probability and n is the number of required steps. Under the same assumptions, increasing per-step success to 99.9% raises the chance of completing all 100 steps to approximately 90.5%.

Real workflows can include correlated errors, retries, and recovery. These features change overall reliability and need to be assessed through end-to-end evaluation. A system's ability to detect and repair its own mistakes can matter as much as its first attempt.

Reliability across a long task Line chart. The calculated chance of completing every step, p to the power n, falls as the number of required steps rises from 1 to 100. At 99% success per step (solid blue line), it falls steadily to 36.6% at 100 steps. At 99.9% per step (dashed gold line), it declines gently to 90.5% at 100 steps. Steps are assumed identical and independent, with no retry or recovery. Reliability across a long task Chance of completing every step 0% 25% 50% 75% 100% 1 20 40 60 80 100 Required steps 99.9% per step 90.5% at 100 steps 99% per step 36.6% at 100 steps P(complete success) = pn, where p is the per-step success probability and n is the number of required steps. Steps are identical and independent; any failed step ends the attempt, with no retry or recovery.
Reliability across a long task Line chart. The calculated chance of completing every step, p to the power n, falls as the number of required steps rises from 1 to 100. At 99% success per step (solid blue line), it falls steadily to 36.6% at 100 steps. At 99.9% per step (dashed gold line), it declines gently to 90.5% at 100 steps. Steps are assumed identical and independent, with no retry or recovery. Reliability across a long task Chance of completing every step 0% 25% 50% 75% 100% 1 25 50 75 100 Required steps 99.9% per step 90.5% at 100 steps 99% per step 36.6% at 100 steps P(complete success) = pn p = per-step success probability n = number of required steps Identical, independent steps; any failure ends the attempt; no retry or recovery.

Completion probability across a sequence of essential steps, at two per-step success rates.

Applying that intelligence to security

Security is one setting where these capabilities could have substantial value. Here, security intelligence means using evidence and reasoning to understand threats, assess consequences, and choose protective action across digital and physical systems.

Consider a cooling system in a factory. A pressure reading changes unexpectedly. Possible explanations include a sensor problem, a valve adjustment, a change in demand, or malicious interference. Determining the cause requires context.

A more capable system could examine the control logic, maintenance history, authenticated commands, network activity, and measurements from other instruments. It could compare explanations, identify an informative next observation, and test candidate responses in a simulation. The opportunity lies in bringing several kinds of expertise into one investigation while keeping the evidence traceable.

This could improve industrial control systems security, software assurance, infrastructure design, and incident response. Useful gains can emerge from current AI well before broad superintelligence. More capable attackers could use similar advances for reconnaissance and vulnerability discovery, so defensive benefit must be demonstrated under adversarial conditions.

Current industry work addresses parts of this problem. Microsoft's July 2026 description of Project Perception connects signals, context, models, agents, and actions.24 Cisco and IBM emphasize agent identity, limited permissions, and accountability during execution.2526 These are reported approaches whose effectiveness needs to be evaluated in the intended environment.

Suppose the system recommends isolating a controller. An independently enforced control should restrict which commands the agent can issue. Engineers would also need to establish whether isolation is appropriate to the process state and define escalation and recovery procedures. An authenticated command can still be unsafe.

From observation to a controlled response Cooling-system diagram. A closed coolant loop runs from a reservoir to a pump, through a heat exchanger that receives a heat load, then through a control valve and back to the reservoir; arrows show the direction of flow. A pressure sensor is connected by a tap to the pipe just after the pump; it measures the coolant without sitting in the flow. At the sensor, the observation is an unexpected pressure change. A thin evidence line leads from that observation to three sources: independent measurements, command history, and maintenance and operating context. Together they feed a step labelled compare explanations, with possibilities to investigate listed as sensor fault, operating change or interference; none is presented as a conclusion. A single command path then runs in order from proposed response, through a check called authorization, then a separate check called process constraints, and only then to the controller. Authorization concerns permission to issue a command; process constraints concern whether the command suits the physical state of the process. A separate branch leaves each check and leads to review required, for a response that cannot pass the checks or whose uncertainty is unresolved; this branch does not reach the controller. The controller sits outside the coolant loop and sends dashed control signals to the pump and the control valve. From observation to a controlled response Physical process Investigation and response Heat load Heat exchanger Reservoir Pump Control valve Pressure sensor Unexpected pressure change Independent measurements Command history Maintenance and operating context Compare explanations Sensor fault · Operating change · Interference Proposed response Authorization Process constraints Controller Review required Control signals
From observation to a controlled response Cooling-system diagram. A closed coolant loop runs from a reservoir to a pump, through a heat exchanger that receives a heat load, then through a control valve and back to the reservoir; arrows show the direction of flow. A pressure sensor is connected by a tap to the pipe just after the pump; it measures the coolant without sitting in the flow. At the sensor, the observation is an unexpected pressure change. A thin evidence line leads from that observation to three sources: independent measurements, command history, and maintenance and operating context. Together they feed a step labelled compare explanations, with possibilities to investigate listed as sensor fault, operating change or interference; none is presented as a conclusion. A single command path then runs in order from proposed response, through a check called authorization, then a separate check called process constraints, and only then to the controller. Authorization concerns permission to issue a command; process constraints concern whether the command suits the physical state of the process. A separate branch leaves each check and leads to review required, for a response that cannot pass the checks or whose uncertainty is unresolved; this branch does not reach the controller. The controller sits outside the coolant loop and sends dashed control signals to the pump and the control valve. From observation to acontrolled response Physical process Heat load Heat exchanger Reservoir Pump Control valve Pressure sensor Unexpectedpressure change Investigation and response Control signals Independent measurements Command history Maintenance and operating context Compare explanations Sensor fault · Operating change ·Interference Proposed response Authorization Process constraints Controller Reviewrequired

Measurements, command history, and operating context inform a proposed response. Authorization and process checks determine whether the command proceeds to the controller or requires review.

The principles have a history of their own. Indiana University's Information Security Practice Principles include minimization, compartmentation, rigor, and fault tolerance.27 Forrester's recent discussion of exposed operational systems reinforces the continuing importance of basic architecture.28 NIST's operational-technology guidance places security alongside performance, reliability, and safety.29 Greater intelligence has to work within those physical and operational realities.

The larger possibility reaches well beyond security. Superintelligence could expand the range of questions we can investigate and the problems we can solve. Progress toward it will become meaningful through contributions that survive examination: an explanation that predicts something new, an experiment that resolves an uncertainty, or a design that works under conditions its creators had overlooked.

Published by First Principles SI.

References

  1. Alan M. Turing, “Computing Machinery and Intelligence,” Mind 59 (1950), 433–460. University-hosted transcription; King's College archive record. ↑
  2. Daya Guo et al., “DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning,” Nature 645 (2025), 633–638. Research paper. ↑↑↑↑
  3. Shunyu Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models,” ICLR 2023. Research paper. ↑↑
  4. Nick Bostrom, “How Long Before Superintelligence?” Definition section; originally published in 1998, with later postscripts. Author's text. ↑
  5. Meredith Ringel Morris et al., “Position: Levels of AGI for Operationalizing Progress on the Path to AGI,” ICML 2024. Paper and abstract. ↑
  6. Tom B. Brown et al., “Language Models are Few-Shot Learners,” 2020. Research paper. ↑↑
  7. Anthropic, “Tracing the thoughts of a large language model,” March 27, 2025. Research overview and linked studies. ↑↑
  8. Charlie Snell et al., “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters,” 2024. Research paper. ↑
  9. John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon, “A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence,” August 31, 1955. Original proposal transcription. ↑
  10. I. J. Good, “Speculations Concerning the First Ultraintelligent Machine,” Advances in Computers 6 (1965). Virginia Tech's scan of the opening pages, especially original page 33. ↑
  11. Sam Altman, “The Intelligence Age,” September 23, 2024. Essay. ↑
  12. Sam Altman, “The Gentle Singularity,” June 10, 2025. Essay. ↑
  13. David Silver et al., “Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm,” December 5, 2017. Research paper. ↑
  14. Bernardino Romera-Paredes et al., “Mathematical discoveries from program search with large language models,” published online December 14, 2023; Nature 625 (2024), 468–475. Paper; authors' account of FunSearch. ↑
  15. John Jumper et al., “Highly accurate protein structure prediction with AlphaFold,” Nature 596 (2021), 583–589. Research paper. ↑
  16. Google DeepMind, “AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms,” May 14, 2025. Developer report. ↑
  17. Google DeepMind, “AlphaEvolve: How our Gemini-powered coding agent is scaling impact across fields,” May 7, 2026. Developer update. ↑
  18. Anthropic, Transparency Hub, model reports available October 5, 2026, including the October 2026 report. Developer assessments. ↑
  19. Nelson F. Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” Transactions of the Association for Computational Linguistics 12 (2024), 157–173. Paper. ↑
  20. Carl Sagan, “Education: Strategies for Change,” 1989 address, broadcast January 5, 1990. Minnesota Public Radio's recording and transcript, approximately 14:53–14:54. Punctuation normalized from the transcript. ↑
  21. Anthropic, “Sycophancy to subterfuge: Investigating reward tampering in language models,” June 17, 2024. Research report. ↑
  22. METR, “Measuring AI Ability to Complete Long Software Tasks,” March 19, 2025. Research report. ↑
  23. METR, Task-Completion Time Horizons of Frontier AI Models, page updated May 8, 2026; and Impact of modeling assumptions on time horizon results, March 20, 2026. ↑
  24. Hayete Gallot, “Rethinking security for the age of AI,” Microsoft, July 27, 2026. Company announcement. ↑
  25. Peter Bailey and Ev Kontsevoy, “Identity Everywhere: Bringing Infrastructure Identity to Agentic IT,” Cisco, August 25, 2026. Company article. ↑
  26. Aubrey Johnson, “Securing the agentic enterprise starts with identity,” IBM. Company perspective, accessed October 5, 2026. ↑
  27. Craig Jackson, Scott Russell, and Susan Sons, Information Security Practice Principles, Indiana University Center for Applied Cybersecurity Research. Authors' framework, available under CC BY 4.0. Related book: Security from First Principles, O'Reilly, October 2017; publisher's overview. ↑
  28. Paddy Harrington, “The First Principles Of Cybersecurity Still Apply,” Forrester, August 18, 2026. Analysis. ↑
  29. NIST SP 800-82 Rev. 3, Guide to Operational Technology (OT) Security, September 2023. Final guidance. As of October 5, 2026, Revision 4 is an initial public draft, not a replacement final publication. ↑

About the author

Nouman Nafees

Founder & CEO, First Principles SI

Nouman Nafees is Founder & CEO of First Principles SI. He holds a PhD in Computer Science from Cardiff University. His work brings together physical models, simulation, and machine learning to investigate the behavior and security of cyber-physical systems.

View profile

Back to Journal