Why is my 1996 Honda Accord overheating only while I am idling?

I checked and both fans come on to cool the engine.
Also when I checked the anti freeze,
The reserve was filled to max but when I checked under the cap there was absolutely nothing in there.
After checking online I just realized I should have been filling the antifreeze from where the cap is and not the reserve.

So should I try and put my anti freeze in under the cap?
Or would a coolant flush be more helpful?
Or is it a water pump problem?

Read more: Why is my 1996 Honda Accord overheating only while I am idling?