We firstly have a button and label of the list on the top, and to hold it on top we need the spacer which comes after these.
Well, that's the problem. Your code calls addWidget() instead of insertWidget() (see my post from earlier in this thread), which means that the spacer stays at the top of the vbox and pushes both up and down. The button and label get pushed up and the tasks get pushed down. When you add new tasks, this results in the window expanding. When you remove a task, the spacer just expands to add more space between the top and the remaining tasks and the window doesn't shrink.

When you add a task, you need to insert it -above- the spacer so that the spacer is always the last thing in the vbox. When you remove a task, you need to resize the window so the spacer will shrink.