No further replies in 4 hrs, so I'll chime in...
First off, your profile indicates 18 yrs old. If so and if this TJ is your very first vehicle - nice choice!
As to the described and shown symptoms, I tend to agree (as a non-expert) with
@hear that it's transmission related. My own 2005 TJ purchased 4+ mos ago was purchased with no service history and had sluggish power "delivery". It upshifted smoothly but took too much throttle pedal for downshifts - unless lifting off and back on the gas as you've mentioned. Do you know prior related service history on yours? How does the trans fluid look? Mine had no pink dye remaining whatsoever and was fairly dark. As part of a long list of deferred DIY maintenance projects, I lowered the trans support (Rubicon Express belly skid), dropped and drained the trans pan, replaced with Dorman's equipped with drain plug plus a new Mopar trans filter, followed by refill of the trans pan with ATF+4 (prox 4 qts) plus flushing the torque converter (much more than 4 add'l qts until "pink" using the forum method outlined elsewhere). The thick goop on bottom of that pan was nasty, however only a bit of metallic particulate was evident in the sludge and magnet and with no "chunks" (didn't cut open the filter for inspection). Now a few hundred miles later, downshifts come much sooner and smoothly, without excessive throttle pedal "coaxing". I'm reasonably certain my DIY trans service was the first in 91k total miles. As miles continue to accrue, I'm hoping accumulated varnish on valve body components and solenoids will slowly dissolve. I'm not expecting "like new" functionality on a highly modified trail rig which has clearly seen lots of big rock action, however incremental improvements will be appreciated.
So then, depending on what's known (or not) on yours, perhaps consider the above during diagnostic evaluation...