This conversation discusses the challenges of dealing with flaky tests and the impact they have on the development process. The speaker shares their experience of struggling with identifying and fixing flaky situations, as well as the psychological and operational impact it had on them. They explain how they decentralized the problem by defining test health and flakiness, building a service catalog, and routing alerts to the appropriate teams. The speaker also discusses the importance of approaching engineering problems with calmness and the need to improve code organization. They highlight the efforts made to speed up the CI suite and reduce test coupling. The conversation concludes with a discussion on introducing static analysis tools and the cultural change and adoption process. In this conversation, Puneet shares his experience of implementing changes to improve the CI/CD process. The transfer of ownership from the central team to developers was facilitated by minimizing additional work through documentation and providing examples. Quantifying the impact and value of the changes helped in gaining developer buy-in. Enforcing ownership and responsibility was driven top-down by management, with metrics and performance indicators supporting the shift. The differentiation between critical and non-critical tests allowed for targeted alert routing and escalation. The involvement of managers and automation helped create urgency and behavior change. Puneet advises focusing on solving patterns rather than instances and investing time in learning from others' experiences.

This episode is also available as podcast on all major platforms, including Spotify, Apple Podcasts, and Amazon Music.

This episode's sponsor is ⁠⁠⁠BlueJay by 10xEngg⁠⁠⁠, they are building a very promising product "Bluejay" that makes it easier for developers to create and manage alerts. If you aren't using them, you are missing out!! It automatically handles the creation of alerts and adds a ton of intelligence to them going forward! Request a demo from them TODAY!

Chapters in this video:
00:00 Introduction
00:59 Challenges with Flaky Tests
02:23 Decentralizing the Problem
04:17 Defining Test Health and Flakiness
05:16 Building a Service Catalog
06:16 Approaching Engineering Problems with Calmness
07:44 Routing Alerts and Ownership
08:39 The Impact of Routed Alerts
09:09 The Challenges of Alert Routing
10:34 Improving Code Organization
11:33 Identifying and Fixing Flaky Situations
16:19 Predicting Test Failures
17:18 Debugging Flaky Tests
18:16 Struggling with Building a Complete Process
19:44 Escalating the Problem to Developers
21:09 The Impact of Flaky Tests on Production
23:36 The Inconvenience of Flaky Tests
25:02 The Cycle Times of CI Runs
26:27 Speeding Up the CI Suite
28:21 Improving MongoDB Performance
30:40 Reducing Test Execution Time
32:08 Reducing Test Coupling
35:30 Introducing Static Analysis Tools
39:19 Adopting Linters and Formatters
41:42 Cultural Change and Adoption
46:29 Improving Pre-Commit Checks
47:57 Transfer of Ownership
48:26 Minimizing Additional Work for Developers
49:26 Documenting for Easy Learning
50:50 Quantifying the Impact and Value
51:48 Enforcing Ownership and Responsibility
52:43 Metrics and Management Support
53:13 Measuring Flakiness and Ownership
54:09 Creating Urgency and Behavior Change
55:07 Differentiating Critical and Non-Critical Tests
56:06 Failure Scenarios and Ownership
57:36 Escalation and Manager Involvement
59:01 Democratizing Alerts and Involvement
59:56 Differentiating Infrastructure and Application Failures
01:03:20 Solving Patterns, Not Instances
01:05:46 Learning from Others' Experiences
01:06:15 Investing Time in RCA and Learning
01:07:34 Avoiding Overcommitment and Seeking Feedback


Find me on other platforms:
LinkedIn: ⁠⁠⁠https://www.linkedin.com/in/bkdonline/⁠⁠⁠
Twitter: ⁠⁠⁠https://twitter.com/PakkaNimbu⁠⁠⁠
Telegram: ⁠⁠⁠https://t.me/pakkanimbu⁠⁠⁠
Youtube: https://youtube.com/@pakkanimbu

Ver todos los episodios