Modern data solution development has surprisingly many similarities with the innovation process. In this blog, I describe how to recognize innovation phases when developing data and analytics cases since oftentimes, the development and business goals live in somewhat separate silos. When innovation phases are recognized in the data solution development, innovations can be managed and seamlessly connected to the business processes. Utilizing agile, modern development project management methodologies makes innovation part of the development culture and ensures the integration of data into the business.

Innovation process

Companies in different industries have different development processes and backgrounds, which usually reflect their core business and culture of the company.  Regardless of the industry and scale of innovation and development work, different development phases can be identified. Identification can become very abstract because different phases have different purposes, and it is not a straightforward process. It is a more cyclic process of cyclic phases with constant feedback to the different phases than a linear process. The classical innovation cycle fails to capture complexity, but it provides a general framework for identifying different development phases with different purposes.

Therefore, it is possible to describe the innovation framework only on a high level, identify different general innovation phases, and adapt the innovation process to one’s specific needs. Data and analytics development projects have a lot in common with software development due to the nature of programming work, but also by agile methodologies that come along with the programming. These practices may differ from companies' typical way of working. Therefore, it is important to identify the development phases and make them transparent for different functions.  

Below, I describe the innovation process but one might recognize many similarities with the development projects.

Identifying the phases

It is important to understand and recognize what stage of the development is currently running. If these phases are not defined by the company’s development process, there are characteristics that usually give hints. Determining the development phase is valuable for various reasons. Firstly, it puts the development efforts in the context, which guides discussions focusing on the right topics. For example, different stakeholders often bring up their concerns about different uncertainties at the beginning of the development. Secondly, identifying the development phases addresses the concerns but also puts them in a timely order for the development backlog.  

Ideation and discovery phase

It is extremely important to recognize the difference between assumptions and insights. Ideas should be considered assumptions until they are proven to be insights. Furthermore, assumptions may be true or false, which enables A/B hypothesis testing. Sometimes, even if the insights are seemingly valid, it may be necessary to provide extra evidence to gain relevant stakeholder support.  

Typical discovery and ideation features are:

• Idea or inspiration for a new thing; feature, solution, product
• Idea of improving an existing solution
• Problem to be solved
• Can be tested by the rough concept
• Scoping of the idea
• Prioritization  

Proof of Concept phase

First development activities focus on solving uncertainties, most commonly by creating proof of concepts that are mainly about learning. The learning focuses on whether the initial assumptions were true or false and why so.  

Often referred fail fast -ideology refers to this stage of development. Fast experimentation through concepting is essential. A/B hypothesis testing of assumptions will result in confirmation of whether the assumption is true or false. Both scenarios provide valuable learnings, explaining why such a result of the proof of the concept was realized.  

When shown that the initial assumption was incorrect, it enables a valuable train of thought: why was the initial assumption incorrect, and what was misinterpreted? This leads to pivoting the concept in the correct direction or rejecting it. This creates a natural way for the Plan-Do-Check-Act (PDCA) -cycle, which is at the very core of agile development. Correct use of Proof of Concepts fosters good development culture by, if not creating evidence for the right development initiative, stating that the development effort was not vain because it created valuable learnings.  

Typical features of Proof of Concept phases:

• Testing if ideas can be turned into reality
• Concepting and prototyping
• Testing if the prototype can meet initial specifications
• Evaluation of the potential of the idea  
• Preliminary value realization models  
• Discovering and understanding the uncertainties and risks
• Helps evaluate potential value propositions

Validating concepts for developing MVPs

After the successful proof of concept iteration rounds, the development focuses on validating the concept for real-world challenges. The development focus should start by addressing the most critical uncertainties of business and technical feasibilities. Again, it is important to gain insights through learning. In other words, development work can be seen as risk management. Usually, this stage of development is seen as Minimum Viable Product development, where the minimal amount of needed features are taken to a level where the intended customer can pilot and test the solution in a real environment. This is the phase where many underlying needs are learned. The development process should be agile and adjust and improve concepts based on these learnings. This stage of development also often highlights other stakeholders' involvement in development activities. How data and analytics aligns or changes processes owned by the stakeholders.  Stakeholders may be from different organizations such as sales and marketing, purchasing, quality, compliance, etc.  

Typical features of the Minimum Viable Product phase:

• Technical feasibility studies
• Pilot studies and experiments in real environment
• Business feasibility
• Addressing most critical uncertainties
• Adjust and improve the concepts based on the learnings
• Creating only the necessary features needed for validating the concept in real environment
• Reducing uncertainty and risk mitigation for further development  
• Validated concepts or proof of concepts with acceptable levels of uncertainty for further development

Developing solutions

After the concepts are validated, development can proceed to production. The rest of the required features are developed. In addition to technology development, the development  must consider the necessary aspects for delivering solution to the end users.  The greatest uncertainties are already mitigated in previous phases which creates increased predictability for the development schedule. Prematurely proceeding with the development phase can cause unexpected delays, realization of undetected risks, and scope changes. Taking the solutions in production typically involves necessary quality assurance activities, fulfilling regulatory compliance requirements, writing instructions, preparing training activities, finalizing business cases, marketing activities, and whatever the pre-requirements deployment may require, depending on the industry and scope of the solution. The development phase usually involves more collaboration between different stakeholders and organizations; therefore, predictability helps project management.

Typical features of the Development phase:

• Finalizing developed solution  
• Developed solutions with value realization models, including value propositions
• Finalised business case
• Testing
• Project management activities
• Fulfillment of the deployment needs and requirements

Deployment of the solutions

Deployment is making new solutions available to users. These activities naturally depend on the type of solution and industry. It requires change management activities, new process diagrams, and retiring the replaced solutions. These activities are called, in some industries, New Product Introduction (NPI) or design transfer, where solutions are taken into use by the end users. From the development perspective, a solution needs to be promoted to the users, and extra support should be provided for possible hotfixes. Collecting user feedback and monitoring adoption and impact offers valuable information for future innovation and development.

Typical features of the Deployment phase:

• Make the solution available to users
• New solution launch activities
• Promotion and support of the solutions.  
• Hand-over activities
• Maintenance responsibilities and beginning of the life cycle management
• Capture new knowledge from deployment to improve solutions, develop relationships, and trigger new opportunities
• Realized financial and non-financial value
• Insights and new knowledge to improve solutions

These phases are essential parts of the innovation process but also the development project. Innovation as a word may have different connotations depending on who to ask.  The technical committee, ISO TC 279 of the International Standardization Committee, defines innovation in the standard ISO 56000:2020 [2] as "a new or changed entity realising or redistributing value."

Yes, there is an ISO standardization of Innovation Management. It may be often forgotten that innovation is a manageable and systematic way of working. Furthermore, it is about systematically creating something new and changing the existing. It is often more than just developing the technical solution, and the impact should be managed to realize and redistribute the new value created. By nature, the innovation process is an incremental agile development process driven by hard work and constant learning, which may even turn out to be a radical change for existing businesses.  

Data cases as innovation cases

Here at Brightly, we have the privilege to help the largest organizations on their data journey. We have learned that many companies are currently clarifying their data vision and strategy and are in the middle of a transformation journey to align their data initiatives with their overall business objectives. This is a dynamic process and requires constant refinement.

It can be agreed that getting the data is now easier than ever. Success stories, modern data architectures, constantly evolving IoT solutions, and even faster evolving AI technologies provide vast possibilities for developing businesses and generating information previously unavailable. However, aligning data initiatives to the business objectives may be more tricky than initially assumed, especially if the core business is not data-driven just yet.  

Similarities Between Data Initiatives and Innovation

It is striking to recognize how data and analytic initiatives have common characteristics with innovation. Getting the new data for decision-making is directly realising or redistributing value by creating new insights. New insights drive change – and that’s what innovation is all about, realising new value to change something for better. When change is driven by innovations, it is really difficult to imagine transferring back to the old. For example, move from modern BI reporting back to Excel reporting.  

Expectations for adaptation of new technologies described by Gartner, happens on a smaller scale within the companies. Data and analytics technologies may inflate stakeholder’s expectations. An example of inflated expectations is advances in generative AI, which should be regarding Gartner, hitting the peak of inflated expectations. When those expectations are not met comes disappointment as a trough of disillusionment. It takes some time before the data and analytics development outcomes reach the Plateau of Productivity. This is a typical cycle of any technology innovation.

Finally, one common feature of innovations and data and analytical initiatives is that often they are overestimated in the short term and undervalued in the long term.  

When considering data and analytics initiatives as innovation projects - taking into account all different aspects of a typical innovation process - they become more manageable and connectable to business, respectively. Often beneficial for innovation processes is to interrelate with existing processes, which are most commonly research and development processes.

Getting started

The development  of data and analytics initiatives is comparable to any other development initiative, and common development phases can be identified. Respectively, all development activities should have a connection to the business and take account of people who are using the new solutions. One very good starting point is to place data or analytics ideas in the Business Model Canvas and see what is really known about the assumpted data use case.

Steve Blank & Pete Newellsuggest that each development idea should be described as a Business Model Canvas (S. Blank & P. Newell, 2017 ). It creates questions about what, to/by whom, and how and brings business benefits and costs to the equation. Whenever any of these questions can’t be answered with certainty,  it is a research or development initiative. It is noteworthy that business canvas highlights also other than technical uncertainties building a business case around the value proposition. How can these uncertainties be resolved? For example, design thinking and double diamond methodologies are commonly used tools to figure out what is creating new value for the customers or internal stakeholders and define the problem to be solved.

Conclusion

In this article, I highlighted similarities in data development initiatives and innovation:

• Data provides new insights
• Data-driven initiatives drive change  
• Data initiatives may have inflated expectations
• Overestimating benefits in the short term and underestimating in the long run

Leveraging design methodologies is pivotal in delineating business requirements and effectively addressing business challenges. Supported by state-of-the-art agile development project management, this synergy enables fast experimentation and learning through conceptualization. The ultimate objective is to uncover authentic business needs and gain valuable insights. Agile development methodologies further enhance the process by facilitating rapid development, consistently delivering tangible outcomes that allow for the ongoing evaluation and deployment of solutions to the business.  

These methodologies quickly illustrate that data initiative development is usually a cross-functional effort. Therefore, selecting appropriate methods to adapt to changing user requirements from different stakeholders is crucial. When accepting data cases as innovation initiatives, management of the development process will become a sensible process where continuous learning becomes an integral part of development.  

Changing the status quo is always a challenge, but we at Brightly are committed to helping our clients from the early innovation phases to taking the solutions to production. We at Brightly are helping our customers by using modern design methods accompanied by modern agile development project management approaches to foster fast experimentation, enable learning, and eventually develop the right solutions for production.

References

SFS-EN ISO 56002:2021:en Innovation management. Innovation management system. Guidance (ISO 56002:2019) https://www.iso.org/obp/ui/#iso:std:iso:56000:ed-1:v1:en:term:3.1.1 Cited 15.1.2024

Blank S & Newell P (2017), Harvard Business Review, What Your Innovation Process should Look Like https://hbr.org/2017/09/what-your-innovation-process-should-look-like  

Gartner hype cycle https://en.wikipedia.org/wiki/Gartner_hype_cycle Cited 15.1.2024