Canary Deployments Are a Myth: The Evolution to Progressive Delivery and Open Feature

Introduction

Canary deployments have long been a cornerstone of modern software delivery, enabling teams to test new versions in production with minimal risk. However, as applications grow in complexity and cloud-native architectures evolve, the limitations of traditional canary deployments have become increasingly apparent. This article explores the challenges of canary deployments, the rise of progressive delivery, and the role of Open Feature in enabling flexible, safe, and scalable software updates within the CNCF ecosystem.

Core Concepts

Canary Deployments

Canary deployments involve routing a subset of traffic to a new version of an application to validate its stability before full rollout. While this approach reduces risk, it is often insufficient for addressing the complexities of modern systems.

Progressive Delivery

Progressive delivery extends beyond canary deployments by decoupling deployment from feature release. It enables teams to control the rollout of new features through structured phases, ensuring alignment with business goals and user experience.

Open Feature

Open Feature is a framework for managing feature flags, allowing teams to toggle features on or off dynamically. This provides granular control over feature visibility, enabling precise targeting of user groups and seamless rollback in case of issues.

CNCF (Cloud Native Computing Foundation)

CNCF provides the foundational technologies for cloud-native systems, including Kubernetes, Helm, and Argo Rollouts. These tools are critical for implementing progressive delivery and managing complex deployment workflows.

Technical Analysis

Challenges with Canary Deployments

1. Incomplete Code Path Coverage:

   • Subset traffic may not trigger all code paths, leading to undetected bugs. For example, Octopus Deploy’s early SAS product testing uncovered critical errors due to incomplete coverage.
   • Mitigation requires deliberate testing strategies to ensure comprehensive validation.

2. Architectural Limitations:

   • Modern architectures like Kubernetes require tools such as Argo Rollouts for canary support, while traditional VM environments lack native capabilities.
   • Databases like OLTP systems do not support multi-version execution, complicating canary workflows.

3. User Experience Fragmentation:

   • Random routing can lead to inconsistent client states and cross-device/Session discrepancies.
   • Internal teams may encounter conflicting feature versions, causing confusion.

Applicable Scenarios

1. Transactional Applications:

   • Use cases like Google Search or Kubernetes upgrades benefit from canary deployments by gradually shifting traffic to new clusters.

2. Business Logic Stability:

   • Updates with unchanged logic, such as Oracle to .NET migrations, allow rapid rollback if issues arise.

Progressive Delivery Practices

1. Release Rings System:

   • Structured rollout phases (Staff → Insiders → Early Adopters → All Users) ensure controlled feature adoption. For example, Octopus Deploy’s 24-hour full rollout process.

2. Feature Flags as Core Mechanism:

   • Decouple deployment from feature release, enabling dynamic toggling (e.g., 5% testing, 100% release).
   • Retain rollback capabilities by switching back to previous versions if a feature fails.

3. Case Study: SAS Product Architecture:

   • Cell-based architecture with per-customer instances allows granular control over feature rollout.
   • Maintenance windows (e.g., midnight UTC peak) ensure minimal disruption.
   • Feature flags manage phased releases for employees, early adopters, and public users.

Challenges and Solutions

1. Maintenance Window Constraints:

   • Full rollout requires waiting for all instances to complete maintenance, often within 24 hours.

2. Feedback Mechanisms:

   • Use feature flags to collect early user feedback, such as during new UI development.
   • Targeted disabling of specific user groups for negative use cases.

3. Version Conflict Management:

   • Avoid state inconsistencies by using feature flags to control feature activation.
   • Default to disabling features if they fail, ensuring system stability.

Technical Conclusion

• Canary Deployments’ Limitations: They cannot address all deployment challenges, especially when new features are involved.
• Progressive Delivery’s Value: Enables precise, controlled feature rollouts via feature flags, reducing risk and improving user experience.
• Open Feature’s Necessity: Provides flexible feature toggling and risk mitigation, essential for complex cloud-native systems.
• Architectural Considerations: Choose deployment strategies based on application requirements, leveraging CNCF tools for scalability and reliability.

Feature Flag Management and CI/CD Practices

Early Adopter Programs:

• Use feature flags to target specific user groups (e.g., personal instances, customer demos, early adopters) for gradual feature release.
• Example: Personal instances remain enabled, while customer demos require full user experience simulation.

Code and CI/CD Integration:

• Trunk-Based Development: Merge short-lived feature branches (2–3 days) into the main branch continuously.
• Testing Strategy: Include feature flag logic in unit and integration tests, focusing on critical paths rather than all combinations.
• Internal Dog Food Testing: Enable all feature flags in main deployment instances to ensure stability.

Self-Hosted Product Challenges:

• Upgrade Delays: Self-hosted customers may delay upgrades for weeks or months, requiring careful feature flag lifecycle management.
• Flag Cleanup: Remove deprecated flags quarterly during product releases, establishing internal processes to avoid long-term accumulation.

Database and Feature Flag Coordination:

• Schema Changes: Coordinate database updates with feature flags to control when changes take effect.
• Isolated Databases: Allow per-user database copies to manage feature rollout scope.

Open Feature vs. Canary Deployments

• Open Feature Advantages: Achieve 90% canary deployment effectiveness in weeks, simplifying multi-application, database, and production testing complexities.
• Canary Deployment Challenges: Require rigorous processes and resources to handle application, database, and production testing.

Architectural Design Considerations

• Feature Flag Design Principles: Create context-specific flags rather than generic toggles, enabling precise user targeting and pausing.
• Code Structure: Integrate all changes into the main branch, using feature flags to enable "invisible updates" without disrupting CI/CD pipelines.