GPT as a basic understanding processor

I've been playing around with GPT3 for 6 months now. The more I use it, the more I think the world has been fundamentally changed.

GPT3 will be marked as the beginning of AGI. I don't say this lightly. It is not AGI, but it has penetrated the realm of understanding previously reserved for humans. It is a crude, imperfect tool. But in it I see genius.

This is one of my favorite tricks, getting ChatGPT to make poems of unlikely things. This would be beyond by ability and patience.

Now ChatGPT has many many many drawbacks. It is not AGI. But in the research community, I've seen enough discussions about how to mitigate, hack its drawbacks, and now that the starting gun has been fired, I expect a mad rush to build. I expect many of the issues to be resolved. I expect janky, half working AI for the next decade. But by the end of the decade, I expect most people will believe AGI has been achieved, even if it a weird non-human form of intelligence, even if we are not very sure it it is really conscious or just pretending.

Basically the uncanny valley will disappear. And what will it matter if it is alive or not alive.. as long as it helps us do things that are useful to us.

What are the identified ChatGPT drawbacks and how will they be addressed?

1. Slow - humans are very sensitive to latency in terms of how they evaluate the "liveness" of something, so this will have to be fixed. Solutions probably include much smaller models running closer to the edge, but in any case, scaling is something the tech industry has many decades of experience in, solutions will be found.
2. Expensive - many more interesting use cases will be unlocked with ubiquity, and I expect many different optimizations will be necessary to get costs down sufficiently. I do expect it to happen, my best guess being 5 years for most early adopters to have widespread access to low cost ubiquity.
3. Doesn't remember - Context window currently is 4 pages. It will get bigger. But there are also techniques to use the context window as kind of active memory, with a vector store of embeddings acting as storage. I see parallels to the RAM upgrades of the x86 era here. I expect this to expand but there to always be some tradeoffs vs latency in using longer context
4. Non-factual/hallucinates/lies - I believe this is partially solved. Key points are that humans need to decide in prompt first that you want factual answers (as opposed to imaginative) and then introspecting to access external data prior to attempting the answer ("Will I be able to provide a better answer if I look up an external source? Which source?"). It will get better.
5. Doesn't calculate, use Google etc - I've already seen plenty of tools being plugged into langchain, from calculator, to browser, to SQL databases. GPT3 can be told it has access to a database, it can create SQL queries, we can send the responses back, etc. It can make decisions on when to use such tools on its own.
6. Doesn't plan/introspect/reason - Chain-of-thought prompting, internal monologue, can often be achieved by using "let's think step by step". A great example from langchain. The below is achievable on GPT3 right now. This will happen automatically for every interaction in the future without additional prompting. This probably the longest lead time item. There will be many edge cases that we will have to work through this decade.

7. Doesn't feel/emotionless/without personality - this is perhaps the easiest to fix. GPTs can be fine tuned to a single personality, or fed enough of context that they remain as one personality rather than an average. Character.ai has some pretty nuts content here. Attached is a forced feminization bot 4chan users designed at Character.ai. Character's reddit boards are all aflutter about how they're trying to limit porny content.

The above really means that the combination of the current capabilities of GPT3, just scaled by a 100,000k expansion in processing power over the next decade should go far. This is even before any algorithmic innovations, Chincilla scaling, or more interesting techniques like using larger models to fine tune smaller models for specific use cases.

There will be no denying that we've achieved AGI by 2030, and if you trace back the moment stuff got real, I do believe GPT3 will mark the start of the final push.

We will see how solid this thesis is when GPT4 is introduced this year.

And just for you benefit, here's what character.ai's webpage looks like just now, on a Saturday night.