Where good ideas come from (for software engineers)
Stop waiting for a eureka moment. Innovation is a process, not magic. Learn to optimize your environment, tinker, and collaborate to solve complex problems.
I used to think solving problems came from locking myself in a room and staring at code until I had an idea. I was wrong.
There isn’t a magical “eureka moment” that solves a complex problem. Instead, we could create a productivity system to solve these problems, deterministically.
History shows that innovation is rarely a flash of individual brilliance. It’s a process that relies on specific environmental conditions. We can optimize for that environment
I broke down Steven Johnson’s book "Where Good Ideas Come From” and distilled the actionable insights from it. I decided to keep a lot of examples from the book to see how the ideas apply in other areas than software engineering.
Let’s expand your adjacent possible! (Keep reading and this last point will make sense 😉)
In this post, you’ll learn
Why the “lone genius” is a myth and how great ideas actually happen.
How to tinker with existing tools to build the future.
How to build on top of platforms to accelerate your output.
The importance of liquid networks and sharing your work early.
The Adjacent Possible: Tinker more
Innovation isn’t a huge leap into the unknown. There are no research papers without a bibliography; nobody invents something completely new. They always cite others’ work. Innovation is more about opening specific doors that are available from the room you are currently standing in. For example, simple molecules had to combine into cells before the door to complex organisms could open. You cannot jump from a single-celled organism to a human in one step.
Great engineers are tinkerers. They don’t invent sci-fi technology from scratch. They put together existing APIs and libraries to solve new problems. Even with AI, researchers did not invent the transformer in a vacuum. They tinkered with existing text and image generators like GANs until they created a new spare part. That new part allowed companies to build ChatGPT and Claude.
I think this concept relieves a lot of pressure from all of us. We don’t have to be some genius inventing science fiction tech. Also, if there is some technology that looks like science fiction for us, it means we have to keep learning and expand our adjacent possible to be closer to the state of the art.
There is a trap in trying to skip steps in the adjacent possible. Charles Babbage designed a programmable computer in the 1830s using steam and gears. It was a brilliant idea that failed completely. The vacuum tubes were not in his adjacent possible. He was missing that spare part to combine everything into a computer. In contrast, YouTube succeeded in 2005 not because the idea was new, but because the ecosystem of Flash, broadband, and cheap cameras was finally ready.
The actionable advice here is to focus on first-order combinations. You should look at the tools currently in your stack and ask what you can build today. Do not try to be ahead of your time. Try to be exactly on the edge of the present possibilities.
Also, this is a reminder to avoid comparing your level 1 with someone’s level 100; their adjacent possible is outside your league.
Liquid Networks: Collaborate
The book uses gas, liquid, and solid states to explain collaboration:
Gas is too unstable, like a group of people talking different languages and unable to communicate.
Solid is too rigid, like a big corporate hierarchy.
Liquid allows for a lot of connections with enough plasticity to change.
Kevin Dunbar conducted a study to see where scientific breakthroughs actually happened. He found they rarely occurred when a scientist was looking through a microscope alone in the lab. The breakthroughs happened at the weekly lab meeting. This was a liquid network where scientists discussed.
You can apply this by treating code review as your conference table. Don’t review PRs as a chore. View them as a collision point for ideas. This also applies when you review an architecture proposal or any other document. Participate in internal chats or open discussions. Your ideas alone aren’t valuable, and you’ll make progress in the idea by soundboarding it against different people.
Serendipity: Productive Distraction
Your brain neurons also have to function as a liquid network. During REM sleep, neurons fire in a chaotic fashion, which allows the brain to make surprising connections between unrelated concepts.
The mathematician Henri Poincaré famously solved complex problems while stepping onto a bus or walking by the sea. I’m sure you’ve solved more than a bug when commuting home or while in the shower. Engaging in mindless tasks removes the brain’s focus and allows it to explore random associations.
Serendipity is often misunderstood as luck. It isn’t just a random event; it requires an anchor. You need a problem or question you have been holding in your mind. Then, your brain will try to associate future observations with the anchor.
We need to reframe how we view the web and social media. We often view the internet as an enemy of focus. The book argues that the architecture of the web, specifically hyperlinks, makes it the greatest engine for serendipity in history. Your new mantra should be to “look everything up”. It isn’t wasted time as long as you keep looking at what you started looking at (an anchor). The goal is to follow the thread of curiosity rather than getting distracted by algorithmic feeds about unrelated topics.
Diversify your inputs: Don’t just read documentation for your team or company projects. Read blogs about other companies solving other problems. Read about other engineering disciplines and their way of working. And always have an anchor about “how can I improve my team, my services, my codebase?”
Platforms: Building on the Reef
Coral reefs create the physical structure that supports millions of other species. This concept of ecosystem translates directly to technology. Good platforms are generative. This means they create value that their inventors did not predict. For example, the creators of GPS technology designed it for tracking submarines. They never imagined it would be used for playing Pokémon GO or finding the nearest pasta restaurant.
We work within a system of stacked platforms. HTTP supports HTML, which supports the Web, which supports YouTube. You don’t need to understand the OSI layer to watch a YouTube video of Mr Beast doing something crazy.
Twitter is a strong example from the book. They built their API first. Early users then invented the @reply, the hashtag, and the search function. No company PM invented these core features. They were added to the “official web application” after users started using them.
My lesson here is to stop reinventing the wheel. Don’t try to do something 0.00001% better because you can’t compete with tech giants anyway. Build on top of existing stacks like AWS or open source frameworks. For example, some entrepreneurs were successful at fine-tuning a checkpoint of an AI model created by a tech giant. But none of them created their own model trained from scratch, unless they are VC-funded startups.
You should also strive to be a platform yourself. Write code that others can extend and consume through APIs, and listen to the feedback you receive.
Other Mechanics of Evolution (Slow Hunch, Error, Exaptation)
There is a myth that great ideas arrive in a sudden “Eureka” moment. The reality is the “slow hunch”. Darwin had the theory of natural selection in his notebooks years before he fully realized it. Tim Berners-Lee started the web as a personal project called “Enquire” in 1980 to connect academic work, and took a full decade of tinkering and slow maturation before it launched as the World Wide Web. Keep a journal to let ideas mature over the years, and write all of them down.
Error plays a critical role in evolution and innovation. What looks like noise is often a signal. Penzias and Wilson thought their telescope had static noise, but it was actually Cosmic Background Radiation from the Big Bang. In software, an example would be that when there’s some recurring bug, instead of patching it with a hacky workaround, look at what that’s telling you about your architecture and your process.
Exaptation is the process of repurposing an existing trait for a new function. For example, Gutenberg “exapted” a wine press to create the printing press. This is why having diverse interests is important. Have hobbies. Being a musician or a writer can make you a better software engineer because you bring skills from those worlds to your day-to-day work as a software engineer.
#6 What I Liked and Disliked
What I liked:
✅ I appreciated the technology examples used in the book. Even though it was written in 2010 and misses the rise of social media and AI, the examples are interesting and relatable. It is refreshing compared to books that draw only from research papers.
✅ I also liked the optimism regarding the internet. The book reframes the web not as a distraction, but as the ultimate engine for innovation if used correctly. That’s something I battled myself because I want to focus on doing good, deep work, but I also write this newsletter and on social media, which could be considered a distraction. Now, checking social media or answering comments is just a way to form a liquid network and to increase serendipity.
✅ Another strong point was killing the “lone genius” myth. It relieves the pressure of needing to come up with a eureka moment or needing to be brilliant in isolation. Instead, I just need to keep learning.
What I disliked:
❌ The density of some examples can be overwhelming. The book is packed with historical details ranging from the 1400s to 2010. If you are looking for quick takeaways, this post covers them, and you don’t need to read the full book (I already included a lot of examples from the book, which is not my usual post).
❌ The era it was written in can be harder to relate to. While I think the same concepts apply, some sections would benefit from a disclaimer saying that scrolling TikTok or AI-generated videos for two hours isn’t serendipity, it’s just slop.
#7 Who should read?
Who is this book for:
✅ Senior engineers who need to move beyond writing code to designing systems and teams. The more talking and designing you do in your day job, the better this book is for you.
✅ Indie hackers and entrepreneurs who need to understand how to leverage existing platforms and spare parts to build products cheaply.
Who is this book NOT for (or maybe who needs to read it the most to expand their adjacent possible 😉):
❌ For junior software engineers, who should focus on becoming independent, not so much on ideas. If you have already read so far, focus on learning the fundamentals, they are spare pieces to expand your adjacent possible.
❌ For anyone who dislikes the historical examples and just wants the technique. Read book reviews regarding the concepts if you won’t find value in learning about how people like Gutenberg, Darwin, or Tim Berners-Lee did their best work.
Conclusion
Innovation is not about being smarter than everyone else. It is about placing yourself in an environment that encourages connectivity, repurposing, and slow evolution.
We can prime our brains for innovation and design a system for idea connection
Keep learning!
P.S. If you’re interested in the book, here’s a link
🗞️ Other articles people like
P.S. This may interest you:
Are you in doubt whether the paid version of the newsletter is for you? Discover the benefits here
Could you take one minute to answer a quick, anonymous survey to make me improve this newsletter? Take the survey here
Are you a brand looking to advertise to engaged engineers and leaders? Book your slot now
Give a like ❤️ to this post if you found it useful, and share it with a friend to get referral rewards
The fancy images are likely AI-generated, the not so fancy one by me :)