Description
Python's Global Interpreter Lock is a friend and rival at the same time. We, as developers, can focus on the design and implementation of applications without the hassle of memory management. On the other side, we complain about the GIL as the limiting factor of performance sensitive applications. Therefore, it is common to refactor parts of systems when the system doesn't perform or scale enough anymore. The refactoring often includes the switch of the used concurrency paradigms like replacing multithreading with multiprocessing or asyncio. Another option is moving logic of CPU-bound workload into C extensions or a full rewrite in a "GIL-free" language. But how do you know that the GIL is the actual performance bottleneck?
While scaling and developing performance sensitive components in Python, my colleagues and I often also assumed the GIL as cause of our performance problems because it is a common and simple answer for this usually complex and varied problems. Instead of starting a rewrite or major refactoring, we took a step back and tried to prove our assumption. With the result that analyzing the impact of the GIL contention on the overall performance is a very interesting problem without common practices or easy usable set of tools that support Python developers. Within this talk, I will share and explain the methods and tools, which we use to analyze the relevance of the GIL on our application performance and how it helped us to stay focused on the actual problematic areas of our applications that required improvements to meet our performance goals.