Teamcity build chain5/31/2023 In a slightly simplified form the process is as follows: So the waiting reason “Waiting for the build queue distribution process” basically shows how much time TeamCity was processing other builds in the queue before it reached the current build. ![]() In TeamCity, all the queued builds are being processed by a single thread in a loop. We started showing the reasons why the queued build does not start on the queued build page:Īfter that we finally started to see that with a large number of builds in the build queue the reason “Waiting for the build queue distribution process” was taking an unusually long time. When we realized that we’re wasting time, we did one relatively simple improvement that helped us a lot. Even though internally TeamCity had all the knowledge about duration of different stages of the queued build, this information was shown in the user interface only after the build had already finished and at this point it’s too late to look for a bottleneck. How much time was spent on the queue processing? Again, only one of the server logs has this information. How much time was spent waiting for a shared resource? Unfortunately, it’s impossible to see it anywhere in the UI, but one can go to the server logs and find it there. How much time was spent on checking for changes? Well, one can go to the build log and see this information there. In any case we have to check a lot of different things to find out why the build is not starting. The problem can be caused by some misconfiguration or there can be a performance problem in TeamCity itself. So when we get a complaint that a build does not start, it’s not immediately clear where the problem is. Finally, when all the conditions are met, the queue optimization kicks in and checks whether the build should be started at all or whether an already finished build could be reused instead (if this finished build has the same revision).Įxperience of analyzing issues reported by our users shows that their configuration can be quite tricky and sometimes it is relying on nuanced behavior of some of the TeamCity features. If there are snapshot dependencies, they should be finished too. If the build depends on some shared resources, then these resources should become available. ![]() For instance, cloud agents should be launched, fresh commits and settings should be fetched from VCS repositories. ![]() The waiting reasonsįor TeamCity to start a build, a lot of conditions should be satisfied. Here is the rundown of how we approached the problem. The build queue bottleneck was so annoying that we’ve decided to dig deeply into the core code of the product. It would not be an exaggeration to say that this issue was driving some of us crazy for quite some time. Hi! I’m waiting for my build to start for 20 minutes, is it intended? During this time I see the agent summary changes, but the configurations don’t start. Why does TeamCity start new builds so slowly? 20 minutes already, agents are idle. īuilds seem to keep piling up in queue and taking longer and longer, are we heading toward a reboot? I see that there are similar problems above with reaching limits but I don’t see such messages, just `No estimate` yet. In this moment, messages from the JetBrains developers were piling up in the teamcity-buildserver Slack channel:įolks, my build is already more than 2 hours in queue. It’s all part of the standard TeamCity functionality, except for that it receives its new features and updates on a daily basis, as we use it for our internal dogfooding.Īt some point, this scale started posing some difficulties for us, as we noticed that during peak hours, newly triggered builds were sitting in the queue for half an hour or more, and basically did not start at all unless you manually moved it to the top. This is all handled by 2000+ build agents (although this number is not static as a lot of them are launched on demand and run builds in hosted environments, such as AWS).Īll of that is handled by our internal CI/CD server called buildserver which is running as a multi-node setup orchestrated by TeamCity. In total, it amounts to 7,500 projects, 50,000+ build configurations (jobs), and around 65,000 builds per day on average. At JetBrains, we use a single TeamCity installation to build all of our products – including the giants like IntelliJ IDEA, P圜harm, Kotlin, as well as hundreds of plugins, and a lot of internal products and services, such as the website or this blog.
0 Comments
Leave a Reply. |