Disclaimer: Don’t take my word because I’m not an AI engineer but a developer for fun. This article aims to provide information about the “disaster” happening because of something known as Devin. This title is also not a clickbait, this tweet was actually posted by Cognition Labs; click here.
The Disaster 📉☠️
So the day before yesterday(12th March 2024) there was an announcement that shocked the world that has been going pretty viral about Devin, the first AI software engineer.
Let me be honest for a moment, this caused a disaster on Tech Twitter. (you know it if you are active on the platform)
Cognition Labs(its creator) describe Devin as “a tireless, skilled teammate, equally ready to build alongside you or independently complete tasks for you to review.”
Sounds promising?
He’s pretty good and by pretty good I mean, he’s as good as being able to pass actual engineering interviews.
Just imagine you’re conducting the interview, you’re the interviewer, and then this AI named Devin joins in and wants to do the interview.
Can take on a Side Hustle? 😱
According to the announcement he can also do real jobs on Upwork.
This is some crazy stuff.
Congnition Labs 💸
I don’t know about you but I’d never heard of that company before this announcement, they seem like a super new thing.
It seems like they just kind of locked themselves in an underground thing for years just to launch the step in AI into the market.
Well, to be honest, there’s pretty much some information here: https://www.cognition-labs.com/blog
Talking about numbers, they are well funded, including a $21 million Series A led by the Founders Fund.
Devin: Performance & Comparison with Other Models
*Devin was evaluated on a random 25% subset of the dataset. Devin was unassisted, whereas all other models were assisted (meaning the model was told exactly which files needed to be edited).Source: https://www.cognition-labs.com/blog
This image shows the impressive performance of Devin on the Benchmark.
The SWE BENCH Benchmark evaluates large language models on real-world issues from GitHub repositories.
The model is given a random open-source code base and tasked with solving issues. The model’s task is to generate fixes for problems in the code that pass unit tests.
If all unit tests run successfully, the model gets a score of one. If the model’s code fails any unit test, it receives a score of zero.
Devin can solve about three times as many problems unassisted as any other model.
Devin correctly resolves 13.86%* of the issues end-to-end, far exceeding the previous state-of-the-art of 1.96%. Even when given the exact files to edit, the best previous models can only resolve 4.80% of issues.
While 13.86% might seem low, it marks significant progress compared to previous models.
Things You Need to Know 👇
This thing can:
✅ Learn how to use unfamiliar technologies.
✅ Build and deploy apps end to end.
✅ Autonomously find and fix bugs in codebases.
✅ Train and fine-tune its own AI models.
✅ Address bugs and feature requests in open-source repositories.
✅ Contribute to mature production repositories.
Final Thoughts 🤔
We, humans, have the power to do anything but something which holds this capacity back is our slow upload(learning) speed.
This is a fact for sure that an AI cannot truly replicate human intelligence.
Human brain is the most complex kilo in the universe, connected by trillions of connections.
The final question still remains: Will this completely replace us?
Drop your insights on this topic.
According to me – Haha, not even close.
Yes, it’ll replace bad programmers with good or exceptional programmers or simply, programmers who leverage the use of Artificial Intelligence to efficiently complete work.
By bad, I mean programmers with less than 80% knowledge of a core language/framework and not constantly learning new things.
They also said, “With Devin, engineers can focus on more interesting problems and engineering teams can strive for more ambitious goals.” That clearly says that they are not replacing the good developers, just providing a way to the top for them; to build something better.
These insights are greatly influenced by @mehulmpt. (Founder, CEO @ codedamn | I watched his video on YouTube)
