By Youko Watari, Technical Marketing Manager
Agile development and its 12 principles have been adopted everywhere, and data warehousing is not an exception. The Agile data warehousing publications out there mostly focus on how to adopt Agile tools and techniques such as the backlog, user stories, and iterations into the data warehouse development process, or how to adopt Agile in the data model design process. One key aspect, however, I see missing in them is how important it is to choose the right data warehousing technologies in order to maximize the success in Agile data warehousing.
Considering the technology aspect of data warehousing is an important point because the successful Agile transformation must occur in the organization as a whole. I believe in order to adopt Agile development for your data warehousing, your foundational infrastructure including the use of hardware and database software must also be agile. A few obvious examples of the agile-ready technologies would be cloud technology and Software-as-a-Service (SaaS).
Another key technology trend of late that is very much in line with the spirit of Agile is software-defined anything (SDx), which provides dynamic and faster ways of configuring and delivering systems. I believe using software-defined technology within your data warehouse environment is well suited for supporting the collaborative, fast-paced, and time-boxed Agile development framework. Looking back at some of my past analytics and data warehousing projects, I can see that software-defined warehouse capabilities can solve some of the common infrastructure challenges we all may be familiar with, such as:
- “We can’t use the Production server and its data as a sandbox or for prototyping.” Usually if you want to explore new possibilities in data analysis by combining existing production data with new untested data sources, the development team needs to request for a new sandbox and go through a pain of loading the data. Instead, why not use a software-controlled data lab capability and carve out a sandbox within the Production system? This will shorten the design and development time and increase the level of collaboration between the development team and stakeholders since the data they use and see is real. This technology could also be very useful if the team is running a risk-based spike in which new and exploratory analytics are tested in order to reduce late failures.
- “We can’t start development for a few months because it will take a long time to provision a new development server.” A common approach for setting up a new development or test environment is to provision a physical server dedicated to this purpose. This may be an easier way to prevent possibilities for security and access violations of Production data or resource contention with the Production users. Drawbacks of this approach however are timeliness needed for Agile development and cost effectiveness. Instead, consider creating secure development and test environments with fine-grained resource and workload management capability within the existing warehousing system using software controls. This will free the project from unnecessary logistics constraints in the project timeline.
- “We need to purchase a powerful system now so that it will accommodate all the users and usage down the road, but we are cost strapped.” One of the key benefits of adopting Agile is being able to realize faster ROI compared with the traditional waterfall style projects because it allows faster deliveries of business value in small increments. The data warehouse system should also adopt this approach and achieve lower TCO and faster ROI by using a database technology that allows the system capacity delivered just in time with a scalability that is graceful and liner. At the same time, a software control should allow managing CPU, I/O, and task priorities at a finer level so that user experience will stay consistent through the growth. An example of this would be setting a minimum response time for workloads so the user perception of the performance will stay consistent regardless of the resource availability and system growth.
Agile data warehousing is not a pipe dream. However you must equip your team with the right understanding of the Agile framework as well as with the right technology that is primed for agility. Agility in data warehousing is achievable today through the software-defined warehouse.
Youko Watari is the Technical Marketing Manager at Teradata responsible for Teradata Database and other core software products. She is a certified Project Management Professional (PMP®), PMI-Agile Certified Practitioner (PMI-ACP) ®, and Certified Scrum Product Owner® (CSPO).
 “The Agile Manifesto” © 2001, Kent Beck, et al (http://www.agilealliance.org/the-alliance/the-agile-manifesto/)