Data Detective Loop: the secret weapon of competent data scientists
Updated: Jan 14, 2022
It's rarely about technical skills
The number one gap between an incompetent data scientist versus a competent data scientist is rarely technical skills, but their data-driven problem-solving skills. And the difference is day and night.
Data scientists with low data-driven problem-solving skills:
They have the technical know-how but have difficulty starting and finishing real-world projects.
They feel they work harder than necessary, but their effort is not appreciated.
They feel left out - like everyone else knows the rule of a game that they are unaware of
They receive low salary increases and few career opportunities.
Data scientists who are highly skilled in data-driven problem-solving skills:
They are confident about and capable of delivering any real-world project.
They feel doing meaningful work while receiving the recognition they deserve
They know how to make things happen with others. And their voices matter.
They receive salary raises consistently and are given the best career opportunities.
Data-driven problem-solving skills are rarely taught in tutorials, courses, or even educational programs. Many experienced data scientists develop these skills through years of hands-on work. You have to put in the work and learn from your successes as well as failures.
Meanwhile, there is a huge difference between how fast they develop these skills.
Meet Dave and Eva
We have worked with a senior data scientist - let's call him Dave.
Trained as an engineer, Dave knows all about the hottest ML libraries. He also spends many hours after work to keep up with the latest papers. He is even eager to do amazing work with all the technical knowledge he has. However, after three years of working on various projects, he still made minimal impact in his company, and never received any raise or recognition.
Compare that to Eva (also not her real name).
Eva is a junior data scientist who comes from a non-STEM field. She coded in python for the first time during her onsite job interview. But in less than two years, she ended up developing models that outperform flagship solutions made by two venture-backed startups. She also received very generously raises two years in a row.
After working with and coaching many data scientists like Dave and Eva, we realized that the difference between them is not due to their academic background, technical skills, personality, or talent.
It all comes down to how quickly and effectively you go through the Data Detective Loop when doing data science work.
Dave's downward spiral
When Dave starts a project, he immediately goes through the latest research papers and blog posts. He wants to find the latest technique that was used to solve some similar problem elsewhere. He then plans a complicated solution based on the technique. The project takes twice long as originally planned. During the week before the final deadline, Dave sleeps less than three hours every day, as he fanatically tries to make the solution work.
Dave finally presents his work. But he is surprised by the lack of enthusiasm from his stakeholders. He receives some high-level and ambiguous advice that he is not sure what he can do with. The work is pushed to a repository but never used again. "I guess I have learned my lessons," Dave says to himself, but only to go through the same loop again and again in his future projects.
Eva's Data Detective Loop
Eva starts a project by quickly developing a good understanding of the data, systems, and people involved. Based on this knowledge, she plans a simple solution that can solve the problem, and then systematically iterates towards the final solution. Four weeks into the project, her latest iteration already reaches the offline evaluation criteria.
During the project presentation, stakeholders keep asking questions to learn about her work and give her new ideas on how to solve the problem even better. She also receives practical advice from her team lead. When the next project comes, Eva will be even more confident in her ability to deliver great data science work.
How this plays out over 2-4 years of career
It typically takes a junior data scientist 2-4 years to advance to the next level in their career. So let's see how things play out for Dave and Eva over the course of 2-4 years:
Again, regardless of academic background, technical skills, personality, or talent, you can develop data-driven problem-solving skills. It all comes down to how quickly and effectively you go through the Data Detective Loop.
How to make your Data Detective Loops quick and effective?
For two competent data scientists who are working on the same project, their Data Detective Loops will look completely different from the eyes of an uninformed observer. However, their Loops are based on the same principles and are both effective.
For a very long period of time, we have tried to teach these principles by explaining them to data scientists, helping that once they have understood the principles, they will be able to apply that to all the projects they work on.
It turned out that we were so wrong. It is unrealistic to expect people to learn the principles in isolation without guiding them through specific situations and applications in their daily work. That is also why no book or course can teach this key skill on the market.
Instead, after leading and mentoring data scientists on more than 100 projects, we realised that project-based mentoring is the most reliable way to help new and aspiring data scientists learn how to solve real-world problems and deliver tangible business value, using the technical skills they already have.
If you already have a team lead or mentor that regularly gives you constructive feedback and actionable advice on data-driven problem-solving, relax. You are already going through the loops as fast as you can. It will only be a matter of time before you notice your new superpower.
Meanwhile, if you don't get such feedback or advice regularly, you have three options:
Option 1: Figure it out yourself
Most junior DS did this, especially when there is no other senior data scientist in their companies or teams. If you take this option, make sure you avoid becoming Dave by being extremely open to feedback. Be prepared to spend 2-4 years to figure it out.
Option 2: Find a mentor yourself.
Ask experienced data scientists you know to mentor you. If you don't have one in your company or network, find them at meetups, events, or online communities. However, set realistic expectations for yourself: competent data scientists tend to be busy and paid extremely well. You really need to do your best to elaborate on why the mentorship you propose is a win-win situation not only for you but also your potential mentor.
We also strongly suggest you only work with mentors who have done at least 20 different applied data science projects. Having experience mentoring other data scientists on at least 5 different projects would be a great plus. Working with a data scientist mentor who has less applied DS experience than this is probably not worth it.
Option 3: Apply for the NDS Edge mentoring program
We think both option 1 and 2 are suboptimal solutions. That is why we created the NDS Data Detective course to help professionals and graduates transition into a data science job in 9-12 months. See how you can do that in our webinar.
Conclusion
Regardless of which option you take, keep the Data Detective Loop in mind whenever you are working on a data science problem. Fast and effective Data Detective Loops will save you from much frustration and many sleepless nights while putting you on a fast track career in data science.
Thanks for reading and check out some of our other articles below.