NO non-locking trem system is gonna stay in tune as well as a locker like a Floyd.
It doesn't matter the nut material, name, manufacturer, slot widths (though too narrow will make it worse of course)...if you bend like a maniac and dive like Eddie, they will go out of tune.
You can make it as good as it can be by compensating a bit. Like for me I usually install strings, stretch the living HELL out of them, tune, stretch, DIVEBOMB, tune. stretch, divebomb, rinse repeat.
That way, after you, say bend the heck out of your G string and it comes back flat, all you have to do is hit the whammy bar and it returns to pitch. You can tune it so that it comes back after a bend instead. These methods work but can be a pita for you and your audience/listener.
If you switch from stock .009s on an SE (or the earlier USA models that came with .009s...they were NO different) then of course your nut will need slots widened. The whole nut, neck, or guitar does NOT need replacing. Well unless you need a locking trem...which is what you need if you are going to be pulling "across the board" bends and to the point of flop divebombs and you want it to return to perfect tuning.
Like I said...with an SE running .009s, well...my SE's tuning stability is as good (or bad depending on how you look at it) as my USA PRS trem models. Going to fatter strings than any nut's slots are cut, on ANY brand's guitar is gonna need some work if you don't want the strings to bind in there.
OP you said you were looking for something in between a stoptail and a Floyd. Sounds like you actually wanted something between a Floyd and a Floyd...or, a Floyd. A non-locking trem is not going to ever be as stable as a locker. Nut sauce, widened slots, tusq, bone, Teflon, locking tuners, trem springs, tremol-no.....NONE of that stuff is going to turn a non-locker into a locker.
Now...all that said, with all the (misdirected) complaining about the SE nuts and slots...but, now that I think about it...probably the only people REALLY complaining are a relative handful of us forum members...a drop in the bucket? ...but if the complaints are many, PRS...why not just cut wider slots in the SE nuts? It costs the same $ to run a wider cutting tool through the thing as it does to run a narrower tool through?. If the nut material is different than that on the USA Core models...just CHANGE it to the SAME nuts!! How much could that possibly cost per unit? a few PENNIES? Literally? Or maybe NONE more? If they doubled their order of USA nuts I'm sure their price would come down? And it would be more than double since they sell more SEs than core USA models.
It would: quiet the complainers. improve the SE line. maybe even SAVE the company a few bucks. If they use "inferior" materials making the SE nuts...but say they perform just as well...why not put them on the USA models too?
Personally I think all this (it's in all the forums..prs, fender, acoustic guitars....) nut complaining is overrated. IMO they have NO affect on tone (except MAYBE slightly only on open strings) and if they are of at least decent quality...if they are properly cut..one works as well as the next. Assuming a baseline of acceptable quality. You know...I don't think a soft plastic or UHMW nut would last very long or work best with a trem but...
So PRS...IF there is a decent amount of negative buzz floating around regarding the SEs' nuts...why not just DUMP them and use the USA nuts on the SEs???????? It would quiet all the negative rumblings about PRS (when "civilians" hear people complain about PRS SE nuts they don't know it's just the SE and just the nut..they just hear .."PRS = BAD".
How much would it cost? Wouldn't it be worth it? WHY give the nay-sayers an open doorway? SLAM it in our faces by just equipping SEs with USA nuts...slotted for .010s. People would HAVE to stop saying PRS (SEs) are BAD...and .009s work fine in a nut slotted for .010s.
I must be stupid. Seems pretty obvious.
OR...just keep it up as things are now and keep hearing all the BAD publicity about PRS guitars. WHY wouldn't you (PRS) just put "proper" nuts on the SEs? If it would dig-in to the profit margins too much...pass-on the $.03 to us the consumer. We won't even NOTICE it. Still having trouble understanding how PRS would NOTICE it on a per guitar basis. To me it seems like a VERY good return on investment. PRS spends a couple of pennies per guitar to quiet the (correct in thinking) complainers! Seems like a win win. If they sell ONE extra SE that they wouldn't otherwise have sold, they might pay for EVERY USA nut for an SE for he whole year.
SO...
-NO non-locker is gonna be as stable as a Floyd. Don't expect it to. If you need to go from huge bends to huge bombs...probably get a Floyd.
-The PRS SE nuts are fine. For me they work as well (and poorly) as the USA nuts. IF you go with larger than stock strings though, obviously you need to give the nut some attention.
-PRS, even though the SE nuts are fine quality-wise..why not just go ahead and use the same nuts as on the USA models? It will cost you NOTHING..or NEXT to nothing and it will once and for all put an END to all of this " PRS SE nuts are terrible....PRS SE nuts need to be replaced immediately...blah blah blah blah...". Why do you insist on staying with the "complaint soliciting" inferior SE nuts? Makes absolutely NO sense. AGAIN...surely pennies per guitar. If that's gonna put you in the po house...we'll pay it for you. Heck I'd even pay an extra 5 BUCKS if it would extinguish all the SE nut fires...and it WOULD. Period! Done. End of the SE nut saga for GOOD!
Or just keep putting "inferior" nuts with narrow slots on the SEs so people can keep complaining and "techs" can eat.
Broken record time....NO a USA nut will not actually HELP or be a big improvement. It will function the SAME. And it too will bind if you run fatter strings in it..but it WILL KILL all the negative talk and complaining.