“Learning” in the age of LLMs

Large language models are excellent information retrieval agents. While their reasoning ability still isn’t on par with human capability, there’s no question that language models are orders of magnitude more efficient at extracting highly relevant information from large corpora of data.

What does this mean for human learning, especially in specialized, domain-specific tasks? Let’s look at an example using computer programming, in a context-specific domain.

This code outputs the array “[1]”, but it isn’t immediately obvious why. An experienced Python programmer will immediately be able to tell that the program is behaving as expected, but a novice may need to build some understanding. Here is how the context-building process may go for a beginner, in Google searches:

• “Python variables of object affected by other members of same class”
• “Python default arguments changing between executions”
• “Reference vs value types python”
• “Are default arguments stored as variables”
• “How are default arguments stored in python classes”
• “How to read all attributes of a class python”
• “Are default arguments function variables python”
• “Read default arguments from function object”

Time taken: ~30 minutes

This may be a bit verbose, but for a new programmer trying to gain context on a programming context, it’s the best way to gain broad domain knowledge. The downside, however, is that it takes forever. The more context-heavy the task, the longer the context-building process takes to truly understand what you’re doing. This was a simple example, but in domains like biology, computer engineering, physics, literature, etc., a lot of context contains cross-dependencies (often forcing you to learn two or more topics at once to gain understanding, regardless of how many you’re interested in).

Now, let’s take a look at solving the above problem, using an LLM (specifically GPT-4-Turbo).

Prompt:

Why does this python code output “[1]”, when the “data” attribute of e2 has not been modified?

class Example:
    def __init__(self, data=[]):
        self.data = data

e1 = Example()
e2 = Example()

e1.data.append(1)
print(e2.data)

Result:

Obviously, GPT-4 was able to answer and explain the behaviour correctly, but let’s pay attention to the remarkable information retrieval that it had to perform to arrive at the answer, and compare it with our imaginary student’s debugging process.

Both the student and the LLM identified that:

• Lists in python are reference types, and modifications persist between different aliases.
• Default arguments in python are mutable.
• Default arguments are associated function objects, function closures use the same list reference.

Moreover, the LLM also identified that:

• Methods are class attributes and persist between instances.
• Variables within function closures do not persist between instances.

Time taken: <30 seconds

There is no comparison to be made in terms of speed. Using the LLM to perform the context gathering task when debugging is much faster. When trying to iterate quickly and ship, these gains compound rapidly. I’ve personally seen my development workflow completely transformed by LLMs like GPT.

But what about learning from first principles? Is the former method of context gathering more effective? More memorable in the long term? What topics are worth diving deep into, and which ones should just be delegated to an LLM?

tbh, I have no idea.

It’s important to know, however, that LLMs can be wrong, catastrophically wrong at times. Specialized knowledge, learned from first principles, is the most effective way to see through the times when LLMs are wrong. Traditional methods of learning are still important, but, as we just showed below, they can be highly augmented by LLMs.

This is also widely different across domains. Simply due to the distribution of data that they’re trained on, LLMs still suck at reasoning in certain domains like advanced scientific search, ethical issues, geospatial reasoning, highly technical engineering problems, etc.