I have encountered big companies that are willing to state that ‘Data Science does not work for the water industry’ and after spending several years in this field as a data scientist I understand where this is coming from and will attempt to break down what this statement may actually mean.

I have not yet encountered deep familiarity with AI and Data Science in the senior management of water companies and a lot of the time AI is perceived as some sort of magic, a silver bullet that could solve many of the problems being faced, without understanding the depth and complexity of building an AI solution. I think this is one of the bigger reasons that create unrealistic expectations and ultimately put data science projects at disadvantage. Similarly, for a data scientist taking on such project, there are usually different expectations of what is available on the table versus what is actually put on the table. And the single biggest differentiator, that can both be a silver bullet or a formula for disaster, is data quality.

A good analogy for the relationship between data science projects and data quality is running and walking, and everyone knows running does not go too well if you cannot walk. As a company, if you pick your battles wisely, and if you even have the choice to pick your battles, there are pockets of insights where rather complete and clean data is readily available, and your data scientist can really woo you by not having to spend over 95% of their time on data enrichment and instead deliver a great product within days. In my experience, naturally, you do not pick your battles as a solution provider, your customers do, and that’s only natural and is where things become a bit tricky – often there are great ambitions as well as great competition to deliver solutions but without any solid data quality foundation, where it is also not easy to enrich the data to make up for that, and this is where expectations start to drift away from reality, sometimes resulting in painful realisation of how much more foundation work needs to be done in order to create those ‘magic’ solutions that would woo your customers.

A data scientist most often exercises huge amounts of creativity, exploring various data, techniques, and methods to solve one of those problems, and probably with little appreciation from the senior management if they don’t truly understand what the path to success takes to be achieved. Working with weak data quality foundations and weak data science company structure is usually a bad thing, but sometimes and personally it can provide great challenges that push one to their limit to work around those challenges and come up with a solution that can provide great satisfaction and sense of achievement, but that’s more from a personal perspective as it does not change the fact that serious inefficiencies and delays have to be incurred to patch up a working solution sometimes. And often it would prevent great data science talent from entering the playing field unfortunately. In my experience the best data scientists are often not in the water industry and what we are talking about may be a one good reason for this, it would be difficult to attract a good data scientist without understanding their challenges and providing a solid structure where they can focus less on data quality and more on the more exciting AI / Machine learning bits.

If a water sector insight provider wants to succeed in having highly competitive data science capabilities, then understanding and addressing data quality should be a high priority rather than an afterthought and taken into account as early as project inception to create those risk mitigation plans and have a better map of how the project would actually go. Sometimes leaders like to say that they are focused on AI and that this is the future and it’s being moved to the core of the business, just because that’s the thing to say nowadays, but the investment in data science does not match what it takes to be successful. Sometimes there’s a single data scientist and no data science leader in the executive team that can slow down incorporation and dissemination of awareness, good planning and deeper understanding of the data science projects, and speed really matters nowadays. Equally, depending on the what the starting point of the journey is for a company, to invest into data science means to invest in solid digitalisation first to enhance quality in all stages of data collection, transformation and storage that would consequently be one of the stronger enablers of data science projects and also be more attractive to good talent.

By Asen Stoimenov

Asen Stoimenov is a Sr. Software Engineer at Skewb. To speak with him about this topic in more detail, you can connect with him on LinkedIn.