Antidepressants have an estrogenic nature as I mentioned before. There are even studies showing that a tiny dose of 25 mg Venlafaxine can stop hot flashes in androgen depletion therapy in prostate cancer patients as well as stopping hot flashes in postmenopausal women. Not to mention that some of them can trigger elevated estrogen, prolactin and LH, causing gynecomastia in some unfortunate men. So, it looks to me that they act like SERMs, except that when you quit SERMs, your estrogen gets back to normal level.
Somehow, antidepressant act like irreversible SERMs, but when you quit SSRIs for most people, hormones are within normal range. Which makes me believe that they change estrogen receptor and/or androgen receptor densities.
The common thing they share with PFS is changing serum estradiol versus testosterone ratio for the duration of therapy. But here's what I don't understand: if the receptors are upregulated, wouldn't homeostasis cause an overall decrease in testosterone synthesis as a compensatory reaction? This is not seen here. How about exogenic testosterone? it often worsens the condition.
Personal anecdote:
Prior to any antidepressant intake, I tried Clomiphene briefly for reasons. It made me completely asexual to the point of finding the female genitalia unpleasant to look at. It felt like how PSSD feels like now. I blamed it on elevated estrogen and when I quit taking it, my libido came back with a vengeance. Something to think about.