Wikipedia.org says;

Stack effect is the movement of air into and out of buildings, chimney, flue gas stacks, or other containers, and is driven by buoyancy. Buoyancy occurs due to a difference in indoor-to-outdoor air density resulting from temperature and moisture differences. The result is either a positive or negative buoyancy force. The greater the thermal difference and the height of the structure, the greater the buoyancy force, and thus the stack effect. The stack effect is also referred to as the “chimney effect”, and it helps drive natural ventilation, infiltration and fires.

Stack effect in flue gas stacks and chimneys
Large temperature differences between the outside air and the flue gases can create a strong stack effect in chimneys for buildings using a fireplace for heating. Fireplace chimneys can sometimes draw in more cold outside air than can be heated by the fireplace, resulting in a net heat loss.

So in a nutshell:

Chimney sweeping

Stack effect of a chimney

When you start a fire in your fireplace the hot air from the fire rises and pushes the cold air out of the flue. As the fire burns it pulls in air from the home consuming the available oxygen (O2). This combustion air is then drafted up and out of the chimney with the other products of combustion (smoke, heat, etc…). As air in the home is displaced (sent up the flue), “make up” air has to come in from somewhere to replace it.

This “make up” air can come in through leaky doors or windows, can be pulled in through vents and ducts or other penetrations in the envelope of the home or be pulled down the chimney flue in use causing back puffing. Outside air is generally colder (that’s why we’re burning the fireplace) and can actually result in a net loss of heat in the home.