Who has shaped your thinking?

Independent Research and Analysis Institutions for Compute & Cognition.

These institutions shape parts of the synthesis and conclusions in Substrate Signal that evolve around compute & cognition.

[1] This one is controversial because it does not always agree well with other indicators. It suggests strongly that there is some overfitting on regular benchmarks happening. Reddit does not like this benchmark because it does not agree with what they are seeing everywhere else. To me, it shows that they continue to expose the jaggedness of AI and some benchmaxxing. So far i am not aware with an issue of their methods, let me know if you see something suspicious with their methods.

People.

Most the people i listen to, i knew already from before ChatGPT. Most of them I do not follow on X or other socials but I do listen to their long-form talks, lectures or discussions. Most of what they say in interviews, during lectures and at talks resonates well, even if some of their social content sometimes spirales into controversy, which is why i do rarely consume that kind of content.

Of course, there’s thousands of people in the space, but if i had to pick just three, here’s the people i resonate with most and why.

• François Chollet.
• Andrej Karpathy.
• Lev Selector

François Chollet, because i liked building neural nets with keras, and he built keras. When i heard this, i had to listenen to what he had to say. He knows down to the last literal bit what he is talking about. The thing is, he talks in a language that, even after my PhD, i have to pay close attention to follow him. Most of the stuff he says, is in a context, that is heavy to process. He’s still a great communicator, it is just, the topic itself is vastly more complex than people realize.

Andrej Karpathy, not for his X posts, but for his university lectures and great talks. I remember watching one of his lectures way back in 2018. He too, has a really deep understanding of how current systems work and its implications. Sometimes it seems he forgets his reach and his words arrive at a much wider audience than he intends which leads to all sorts of misunderstandings around his “claims”. As someone who has a similar background i can mostly understand where the controversies are coming from, usually because people couldn’t follow the context.

Lev Selector because he provides condensed weekly updates in the least hype-way possible. He’s not famous, he just keeps to make really good weekly updates that are worth watching.

I don’t watch as much content from Andrew Ng or Yan LeCun but i do not think they are wrong either, it’s just that i don’t have infinite time and can check in on everyone. When one of their takes gets a lot of attention i will have probably still heard about it. The same goes for people that are mostly looking for investors, or money, or, who haven’t talked about AI before ChatGPT. As with everything there’s always an exception.

One of these expections is Jensen Huang, with his particular good interview where he talks with Lex Friedman. It went viral for something he said about AGI, but I do not even mean the part where he jokingly said we achieved AGI (under a very narrow condition) but everything he said after that. https://www.youtube.com/watch?v=vif8NQcjVf0

Literature.

When you want to get into machine cognition, i.e. deep learning yourself.

Be ware that it’s basically one or two years on top of a computer science degree. This is no bedtime reading.

I recommend to start from the maths side, i strongly recommend it.

You must get a handle on linear algebra and calculus to be able to tackle this subject. It’s nothing like any design pattern you have seen before. Neural networks are not programmed in the typical sense. They are pure math. You’re dealing with functions of functions of multiplications and derivatives in arbitrary dimensions. You need to be able to think in a way that allows this to make sense.

If you think you know a bit of pgoramming, or even a lot of programming, it is not going to help you much here. It’s a deeper rabbit hole than you expect.

You are doing yourself a great disservice if you dive straight into deep learning. If you are unable to explain what is a derivative/chain rule, residual error, entropy, variance / bias, tensor, loss, model, feature, label, loglikelyhood, gradient in context of data science then these deep learning books will do very little, because they compress the foundation a lot. As someone who has mentored multiple data scientists to the point of deep learning your curriculum should be:

• Basic Computer Science, Information Theory
• Basic Statistics & Linear Algebra
• Statistical Learning
• Neural Networks
• Deep Learning

The more you know from the layer before, the easier the layer thereafter will be. Whatever layer you’re at, this should be your absolute goto to deepen first!

That being said. The greatest resources i used during my studies:

• Stanford records most of their lectures for public use. As I said, look for the lectures with Karpathy when you’re mathmatically ready, they are genuinly great. Yes it will take hours to sit through these lectures but this is arguably already the most compressed from of learning this material. It’s literally the shortest route you can take. Learning it by yourself will take even longer. You can waste hours on a single concept he explains perfectly in 10 minutes of the lectures.
• 3Blue1Brown’s Channel: - whatch everything deep leanring and linear algebra related especially the vidoes on neural networks, CNNs, Transfomer are great.
• Lastly i really enjoyed the interactive online book Neural Networks and Deep Learning by M. Nielsen whichs sadly is no longer online but you can find the work that his interactive book builds on here: https://www.deeplearningbook.org/ by Goodfellow et al which is unfortunately less accessible but still considered the standard.

Clickbait Sources.

To look beneath the headlines, where do you find headlines in the first place? One source for clickbait headlines are:

• https://reddit.com/r/singularity

Or r/technology, etc. They mostly differ in the discourse below the posts. Another source is

• https://news.ycombinator.com/

Looking at HackerNews and you will find the occasional update here. The discussions here seem sometimes to be a bit more informed, but ultimately it is still an online forum everyone can and will participate in.

More Resources.

You might stumble upon hundreds of different resources when looking for a specific thing but these are some renowned ones.

• HuggingFace: Models and Datasets Find pretrained models and publicly available datasets for your task.
• A previous website called paperswithcode.com, which was a great resource, is now absorbed into HuggingFace at https://huggingface.co/papers/trending to watch for upcoming and trending research publications in any machine learning related task or field.
• GitHub: A decade ago there was a trend where any github starting with awesome-XY was usually an awesome collection of resources about XY. Unfortunately, with the rise of AI-generated content, everyone now can quickly compile an “awesome” repository, so don’t be fooled. Many projects that gain a lot of traction today often have no after-thought, are incomplete or already dead projects, but github is still the standard for sharing open source, so gems do still get shared. It just became harder to find them.