Interface-bound agents #2: Defining “raw intelligence” out of existence

Epistemic status: Exploring the idea! Don’t take this as a trusted intellectual output. Probably wrong on some specific mathematical/definitional points, I am new to control theory. Not sure if it is wrong in substance in any way.

In yesterday’s post, I started exploring the idea of “narrowness-of-interface”¹ being a limiting factor for the harm an AI agent can do. I broke down the system into:

• Sensors (transducers): convert physical quantities into signals (the input interface)

• Input channel: receives information from the world at rate R_in bits/second

• Processing: arbitrarily powerful (can compute any function of its inputs)

• Output channel: emits actions into the world at rate R_out bits/second

• Actuators: convert signals back into physical effects (the output interface)

And mapped this onto standard ideas in Control theory, such as Controllability (the ability to push the environment into any state via control inputs), and Observability (the ability to infer any state of the environment from sensor inputs). I then gestured at the idea that it’s not straightforward to achieve Controllability, which is the condition that lets a superintelligent AI “do anything” from its interface.

Today’s post is a straightforward continuation of this, exploring the idea of privilege escalation that was introduced in that post. Having established that “the interface” (i.e. sensors, the input + output channels, actuators) are important in some way, we can propose the “recursive privilege escalation hypothesis”:

An arbitrarily intelligent agent with a narrow interface will generally be able to widen its interface, and then use this increased capability to widen its interface more, and so on.²

This (per my definition) is one of the two ways a narrow-interface AI could achieve big deliberate effects. With the other being just using its existing thin interface to achieve the effect (e.g. the “design a super-virus and email it to a synthesis lab” case). This design-a-super-virus thing is NOT covered under the recursive privilege escalation hypothesis.

Widening each aspect of the interface

By enumeration, the four ways an agent would widen its interface are:

• Improving sensors

• Improving the input channel

• Improving the output channel

• Improving actuators

Sensors and the input channel

Improving the input channel

There’s a bit of a fuzzy relationship between improving the input channel (extracting a higher rate of information from sensors) and improving processing, and similarly for output/actuators. E.g. if the AI (through being super smart) works out some correlation in the noise of the channel, it could subtract this off, and thus extract more information. To clarify the boundary, I am counting this under improving the input channel, because this corresponds to the Shannon capacity theorem understanding of an information channel:

every communication channel has a capacity C (in bits/second) determined by its physical properties, bandwidth and noise. It is possible to approach C bits/second through using better encodings, but not exceed it

Example: if the AI manages to extract audio from a vision channel by looking at reflected vibrations, that falls under improving the input channel.

Putting this into the control theory formalism³:

Sensor readings are y(t) = Cx(t), where x(t) is the actual environment state, C is called the observation matrix (representing the physical sensors in some way), and y(t) are the readings of those sensors. So for a camera, y(t) would be the pixel values. Note that this is not an estimate of the world state, it’s literally just the sensor readings, which may contain “hidden information” (as in the audio-from-vibrations case).

There is a new concept that needs to be added, which is the state estimator x̂(t)⁴. This takes the sequence of raw sensor readings (combined with the known control outputs, and “the dynamics of the system” A) and produces an estimate of the actual world state:

x̂(t+1) = Ax̂(t) + L[y(t) - Cx̂(t)] + Bu(t)

[x̂(t) is the estimated state; L is the “observer gain”; y(t) are the actual sensor readings; Cx̂(t) is what the sensors would read if the estimate were correct; B and u(t) are from the output side: the actuator matrix and control outputs]

This matrix L does the job of extracting signal from noise in the sensors. “Improving the input channel” corresponds to working out a more optimal choice of L⁵. In a complex AI system, this would be a complicated non-linear function, where it may be possible to come up with a really good one to extract way more information than a human can. This is an idea in classic AI-in-a-box-is-dangerous arguments, e.g. that the AI could pick up stray radio frequency signals to gather information⁶.

Improving the sensors

I claim that improving the sensors requires physically modifying the system to add/upgrade sensors. There are some sensor-improvement-adjacent ideas, like finding a way to sample at a higher frequency, but to me this falls under “improving the input channel”. In fact, I will define improving sensors to require a physical change to the system.

An agent could improve its sensors by:

• If embodied, building/buying new sensors and connecting them up. E.g. setting up cameras throughout the entire factory it operates in.

• Convincing a human to do the above on its behalf (including via this being useful to the human).

• Hacking existing sensors is an edge case, arguably it should go under “input channel” but I think it does fit more naturally here

Actuators and the output channel

As with observability and controllability, there is an exact dual relationship here which simplifies things. Just-before-publishing note: Actually I think just saying “exact dual relationship” leaves quite a few questions open.

Improving the output channel

Recall that the world state evolves as x(t+1) = Ax(t) + Bu(t), where B represents “the actuators”, and u(t) the control output. There is a dual to L, which is called the controller gain K. This determines what signal to send: u(t) = -Kx̂(t). The actual effect on the state is then:

x(t+1) = Ax(t) - BKx̂(t)

“Improving the output channel” corresponds to optimising K. Note: The last post talked about the “reachable set”, and the concept of a “fully controllable” system being one where all possible states are reachable. In a linear system (to my understanding), improving K can’t actually increase the reachable set, just allow you to get to any state more efficiently. But in a non-linear system it can.

Improving the actuators

Again, I define this as physically upgrading the actuators, e.g. adding another robot to the fleet.

Conclusion

The astute readers among you may be thinking “where did processing go?”. The input side seems to have subsumed a lot of processing under “state estimation”, including potential clever-tricks like extracting audio from visual inputs. The output side directly takes this estimated state and converts it into a control output via u(t) = -Kx̂(t). The problem of cleverly choosing outputs to achieve any goal then reduces to the concept of the reachable set.

I wasn’t expecting this, but my best interpretation is to say: Yes we have defined processing out of existence as an independent concept, and rolled it into “state estimation” + “cleverly choosing outputs”. This allows us to neatly separate “the interface” as the physical sensors and actuators. The boundary of the sensors are their readings, y(t). The boundary of the actuators are their control inputs, u(t). The system can’t “know” anything beyond that with certainty, it can only try to infer it.

To be continued, probably tomorrow. I claim this is more good news for the “narrowness-of-interface” idea having legs. E.g. a few more steps of argument will probably let us say concretely that the accuracy of state estimation is physically bound by the sensor interface.

1. I’ve decided to rebrand thin -> narrow, because escalating narrow -> wide sounds more normal than thin -> thick.

2. To get to AI doom you just have to add ‘and then it would use that to do something bad’, but here it is framed neutrally, it’s just a question of capability.

3. Again assuming linearity, mainly as a way to make the algebra clean and understandable.

4. Wikipedia article, also called the state observer or Luenberger observer.

5. Remember we’re making linear assumptions to make the algebra easy, which means our “extract audio from visual” idea doesn’t neatly apply here. But an analogous nonlinear framework could incorporate this.

6. See e.g. Another problem with AI confinement: ordinary CPUs can work as radio transmitters, though this is actually about the output side