If this is an AW-55 transmission like the XC, that driver's side axle takes a lot to get it out.
In the AW-55 transmission, there is a metal cap on each spider gear. That and the spider gear shaft are in the way of trying to "punch" it out from the other side. The standard "pickle fork special tool" doesn't work well. A big cold chisel has the right angle/wedge to "pop" the axle.
If you've tried the punch from the other side, you probably messed up that metal seal. It's an odd part; keeps fluid from getting to the splines, so they run dry and rust, and it doesn't show up on the ATRA manual for the AW-55. There's an internal o-ring for the axle, which is how most cars seal the axle and splines. The transmission seal is actually running on the outside of the spider gear.
I would simply put the passenger side axle back in, drive it, and check for leaks. If you've got a leak, replace that internal o-ring.
And for the driver's side? Yeah, if there's a broken stub of the axle in the splines, it's held in by a circlip and rust. You'll need to weld on a bung/stud or drill and tap a stud into it so that you can get a slide hammer on it...
2016 Tundra Crewmax 4WD 1794
2005 MB S600 (90K, Michelin AS3, M1 0W40)
2002 V70-XC (218K, AltiMax RT43, Castrol Edge 0W40)
2002 V70-T5 (189K, IPD bars, Bilsteins, Contis, Mobil 1 0W30)
1932 Packard Sedan (straight 8, dual sidemounts, original paint and interior, Shell Rotella 15W40)