AGILE MATURITY LEVELS AND ITS CORRELATION TO INCREASED THROUGHPUT Anil Varre Lyudmila Mishra Agenda        Agile Maturity Project Overview Technology Stack Project Goals and Challenges Agile Tool Set Process to Achieve Maturity Conclusion Agile Maturity • • Core goal of an Agile team should be to build a deployable product in every iteration through all the phases Maturity Levels as defined by us Level 1 = Continuous Build • Unit Tests and Code Quality Tests with every commit Level 2 = Level 1 + Continuous Deployment • All components are rebuilt and release candidates are deployed to a systems integration environment at least once a day. Level 3 = Level 2 + Continuous Testing • Automated integrations tests, smoke tests, security scans and performance tests run after every deployment to systems integration environment. Level 4 = Level 3 + Continuous Release • Automated promotion of release candidates to a UAT/Pre-Prod environment with regular sign-offs is what defines “Done”. Project Overview   Nature: Complete Re-Architecture of 10 year old platform Size :  60+ team size (developers, content, design, QA)  Duration 2 Years  Multi-Million Dollar Budget  Team Distribution - Onsite, Offshore, and Specialists/Consultants Projects Goals Business Goals Better, Faster, Cheaper System and its delivery Quality Goals Feasibility Accountability Functionality Testability Maintainability Simplicity Scalability. Extensibility Project Challenges    Distributed Teams Organization Culture Poor reliability in terms of  Functionality  Delivery Time  Quality  Varied Technology Stack Technology Stack Complex Problem -> Simple Solution • The Problem Statement: Define a way to deliver HIGH QUALITY, VALUABLE, software in an EFFICIENT, FAST, RELIABLE fashion. • The Solution Adopted: Develop a Build, Deploy, Test and Release Process that  is fully automated  produces release worthy artifacts regularly  is repeatable and reliable  provides quick feedback on functionality, quality, and performance  increases visibility into the health of the project  clearly articulates team velocity and cadence Tool Set Adopted • Project Management and Issue Tracking – • – • JUnit Cobertura – – – – – PMD CPD Jdepend Java NCSS (Non Commenting Source Statements) CCN (Cyclomatic Complexity Number also know as McCabe Metric) Veracode Code Testing – – – – – JUnit Selenium Fitnesse JMeter DTM Data Generator Productivity – • • – – SVN Build And Deployment Tools – – – • DBDeploy Source Control Management – • Ivy Artifactory Tattletale Database Change Management – • JProfiler Dependency Management – • Custom Eclipse Settings Code Profiling – Code Review/Quality – • JIRA Code Coverage – • ANT Shell Script Suite Hudson Version Controlled Developer Image Hudson • • • • • • • • • • • • • Continuous Integration Project Health Dashboard Code Metrics and Quality Continuous Builds & Deploy Create New Projects JBoss Restarts Migrate Data Bases Running & Scheduling Batch Test Data Management in Data Base Deploying Content Single - Push Button Deploy Selenium Regression Tests Leader Board Level 1 – Achieving Continuous Build The Practices  Every developer runs pre-commit tests on local environment.  On successful pre-commit tests, developer commits code to central SCM repository  CI server (Hudson) triggers new build when new revision is detected.  CI server repeats units tests, code review and code analysis, publishes reports for the project and makes it visible to all.  On successful tests, new source packages, binaries and javadocs, etc are published to a common repository so they may used by downstream components.  CI server continues to build downstream components to ensure working software exists Characteristics of the Build Scripts         • Common build scripts for all components belonging to the same technology type Checked in along with same repository as code base. Branched and tagged along with code in order to guarantee repeatable builds. Continually evolve along with codebase. Implemented incrementally to fulfill goals of a specific maturity level Executes in stages as the change follows through the delivery pipeline Complete ideally in 90 seconds, but no longer than 5 mins. Failed builds alert the entire team, including managers and testers, not just developers. Chose Ant as the build scripting tool of choice • • • Quick and easy to develop customized scripts Consistent scripting language irrespective of technology used Easy to maintain by wide variety of team members Level 2 – Achieving Continuous Deployment The Practices  CI server kicks off Stage 1 build scripts – this time carefully tagging everything and exporting tagged source for builds.  CI server publishes artifacts as release candidates on successful build and stage 1 tests.  CI server executes next stage build scripts to create deployment packages or installers.    The exact same installer is used to for deployment into every environment – nothing is rebuilt from this point forward for a release candidate. CI server reconfigures the systems integration environment using reference implementations scripts of all application containers, web servers, etc that are checked in to source control. CI server deploys/installs the new packages on systems integration server  All environment specific configurations are provided as input to the installer using configuration and property files. Characteristics of the Deployment Scripts  Minimal deployment instructions with appropriate levels of automation to reduce chances of errors.  Minimal set of deployment tools to reduce need for extensive training and documentation.         Same deployment tools for each component irrespective of technology to maintain consistent instructions with reduced complexity. Use of standardized reference implementations of software platforms to ensure cohesiveness between all environments. Use of automated pre-requisite environment verification testing to ensure fewer environment based deployment issues. Ability to deploy an individual sub-system when possible without having to alter or update other sub-systems. Ability to deploy entire system by chaining installation of pre-requisite software environments and all subsystems to ensure ability to create the entire system on a brand new hardware platform easily and quickly. Ability to use a unified deployment process into each environment to reduce variability between how changes are applied within the promotion path. Ability to rollback back changes and restore to previous state of environment easily and quickly with simple and few instructions. Ability to query the differences between the post-deployment state and the pre-deployment state. Level 3 – Achieving Continuous Testing The Practices  CI server runs stage 2 tests on systems integration server         integration tests, acceptance tests, performance tests On successful tests, installers are marked as ready to promote and archived to “release staging area”. On a pre defined schedule CI server promotes the latest releasedistribution to a QA Test environment using the exact same deployment packages and automated deployment scripts. CI server repeats acceptance tests. CI server executes automated regression tests. QA team performs manual exploratory tests  Automation remains in pause mode until next signal from QA team. Level 4 - Achieving Continuous Delivery  Develop a Deployment Pipeline that satisfies business needs and reflects team cadence  Begin with manual steps and then script EVERYTHING incrementally.  Push Button Release to any environment   Use schedulers and chain scripts to force movement of releases along pipeline Manage everything in source control  environments and their configurations,  Development, build, test tools, etc,  build and deployment scripts,  test scripts,  documentation, An example Lessons Learned     Despite the following we achieved agility:  No formal CSM/CSP/etc. training for staff  No dedicated product management per team  No dedicated resources for project teams  Not all roles staffed Highly self directed and motivated team is necessary Tools and best practices are simply adopted – no management mandate is necessary Think big but act in small incremental steps Conclusion      12 Releases since project launch on Nov 1st Only one trouble ticket from end users since release! First attempt with Veracode scan resulted in AAA rating Performance test results consistently meet business requirements 100% code coverage all through project lifecycle   <5% PMD and CPD violation throughout project cycle    Easy to refactor code, extend functionality Easy maintenance Self documented architecture and code Easy to integrate new members into team