No, I was using plain poll(). That was the default and I didn't change that (because I have no idea what the difference is...). I changed that to devpoll() now and it looks like the problem is solved. I did at least 15 config changes with graceful restarts and not once it got blocked and also no timeouts (I am now using the stock admin_php again)!
What is the difference between poll and devpoll? Is there a difference in performance?