They are frightened because they are seeing their parents at one of the rare times when mother and father are attending to themselves and not counting the children at all.
And the children understand, unconsciously, that if a parent can abandon them for five minutes, the possibility that he will abandon them in future for five years is on the table.
And that is frightening.
Obviously a child has to prepare from the moment of birth for separation from his parents, but that process has to be gradual and slow and held in an embrace.
So do me a favour, and don't listen to all the idiots who tell you that "you have to expose the children to the truth," and that "children need to learn that arguments are part of life."
And if your partner is getting on your nerves, go and thrash it out in another room.