ChatGPT and Beyond: How did we get here and where are we heading?

It's not everyday that we see a new product make the news, globally, on major news outlets, within a week of its being announced. But that is what has happened in the few months since the release of ChatGPT.

The impact, however, was not just pure delight with the technological progress of humanity, as, for example, with the release of the first pictures from the James Webb telescope last summer. This time it was a shock, accompanied by a large amount of concern, angst, and fear.

But it's not just the general public expressing those emotions. Even if you have been paying attention to the developments in the field of AI, specifically since the rise of deep learning methods in the early 2010s, there is still a similar sentiment. If you have been in the field of NLP and even knee-deep working on technology developed in just the last 4-5 years, you could have seen this coming. Just not as fast nor exactly how it actually happened.

There is a good story about how we got here. A story we started talking about at Lengoo a few years ago and we called it HALOS® , short for Human Augmented Language Operating System. And if you are curious about this, and how the last decade or so of AI research has led to this moment in history, then keep reading. In this series of three articles, we will walk through what has happened in the field of AI over the last few decades, how we got to where we are today specifically with language technology, and what the implications of this are  to our future work environment, which is already starting to look very different from how it was just months ago.

So first: Let's walk through history and build up to today.

History - How did we get here?

AI - The Last Invention

Civilization has historically been accompanied by groups of people systemizing the methods for the acquisition and application of knowledge. This is collectively viewed as science and technology.

Formalizing these methods has been the task of the fields of logic, mathematics, and, more recently, computer science.

These fields are concerned with the topics of thinking, computation, and, consequently,  intelligence. In this series, we focus on the definition of intelligence concerned with problem solving and the various steps towards intelligence: knowledge acquisition, understanding, application, and learning.

From that perspective, one could view AI as the fundamental problem to solve. Being that if you solve or automate intelligence itself, you solve all the problems it can solve.

“Artificial intelligence is the last invention that humanity will ever need to make." - Nick Bostrom

But similar to how our dreams of machines that can eliminate the need for all physical human labor has not been fully realized, AI and the automation of human thought has been just as, if not more, elusive. In both cases a lot of progress has been made, but the goal post keeps moving.

An Elusive Endeavour

The most interesting thing about AI is not its difficulty, which is, of course, substantial, but that our understanding of what intelligence is constantly changes as we progress.

"The real problem is not whether machines think but whether [humans] do." - B.F. Skinner

While there is a lot of intelligence or mental work that was automated or outsourced earlier in history, it wasn't until we started to mechanically automate arithmetic and computation that our obsession with how far this can go accelerated. Dreams of building machines that determine the truthfulness of logical or mathematical statements followed. These dreams were shattered but in their demise, a new hope arose, that of building machines that could execute a solution to any problem given a correct algorithm.

Over the following decades, the field of computer science matured, and general purpose computers started to get more scalable and more usable  until they became mass produced household objects. They started to be increasingly used for more use cases, in everything that can be formulated into an algorithmic computation. And once they went online, the cycle of increased digitalization, improved algorithms, the need for more computation, improved performance and so on, accelerated.

Software is now used to completely automate or support almost all aspects of white collar jobs, and personal computers are an essential tool for that work. Even more recently, smartphones which are even more personal personal computers have become another important part of the toolbox.

AI moves forward

In parallel to the effort to automate all computable work, efforts to build AI were leveraging these newly built computing machines to compute newly found equations and algorithms that encapsulate different models of intelligence. Along the way, solutions were found to problems that we thought required more "intelligence", such as chess. And today, a chess engine running on a personal computer is better than any grandmaster.

Other games took much longer to crack, such as more recently with the game Go.

Ironically, things that we humans do casually in day-to-day life and thought were easy turned out to be quite hard for machines: such as telling cats and dogs apart, speaking a language, and doing tasks that require coordinating multiple simpler skills such as driving.

The main lesson of thirty-five years of AI research is that the hard problems are easy and the easy problems are hard. - Steven Pinker

One common thing among all of these problems is that they come quite intuitively to us and are based on experience.

For example, the picture below can refer to multiple things depending on your knowledge. Someone completely unfamiliar with musical instruments might even see it as a kind of hammer, but most of us would recognise that it's a guitar. And with a bit more depth of knowledge you would know that it's an electric guitar. Someone into electric guitars you would say it's a Telecaster style guitar. An a connoisseur would say it's a 1952 American Fender Telecaster.

As you can see this knowledge is a bit intertwined, with the visual, the label, and the definition having not so simple boundaries to define. This is partially what our brains do: build complex associations between inputs from our different senses that can then trigger other associations. And why wouldn't our brains do that, after all it's a complex network of interconnected cells, just like the rest of our body is.

This idea has been fundamental to the recent breakthroughs in solving many of the problems mentioned above. The idea of a neural network.

Breakthrough Progress

The concept and algorithms of neural networks have been around for a long time. And as with AI, they have gone through the AI winters and survived.

One key idea that started to dominate the field was using an end-to-end learning approach for all parts of the machine learning system, including its input features. This is in contrast to approaches such as feature engineering an input and applying machine learning algorithms on it to get the desired output. This was key because especially for tasks intuitive to us, to which we have not been able to manually write algorithms to solve or concretely define the inputs, we can leave it up to the machine to find the useful representation to solve the particular task at hand.

This meant that all we needed to do was find neural network architectures, which now basically meant  just any graph of connected, differentiable mathematical operations, that can generalize to different sets of problems we were applying it to.

But it wasn't until the last decade that we have finally amassed enough computational power, along with a more detailed digitized version of our world, that larger implementations of these algorithms (deep neural networks, hence “deep learning”) started to finally pay off. So much so that very quickly they started to become part of our everyday lives, and their power is being triggered everyday by our mere fingertips.

Wrap up

Now that we have covered a bit of AI history, its importance, and how the current paradigm of deep learning came to be, in the next article we will focus more closely on the language AI part and in particular how we got to large language models, what they are, and what they can and can’t do.

Till next time!

‍