Thanks for the replies. To respond to everything:
1. After removing the stock strings, which I assume were 9's, I tried D'Addario Extra Light Gauge Chromes, Flat Wound, 10-14-20w-28-38-48. That is when I noticed the problem that it wouldn't tune and the bridge kept leaning forward. I have many guitars with tremolo bridges and never had a problem with 10 gauge strings, or even heavier. So this was new to me.
2. The nut is fine, no need to file it for the slightly bigger strings.
3. I realized that the other 5 strings in this set were larger than the standard 10 string set, so I tried Ernie Ball Slinky Cobalt 10-13-17-26-36-46. Still had the problem but not as bad.
4. It occurred to me that the extra spring supplied with the adjustment tools in the gig bag might be for this purpose. I put it on (moving the middle spring to the adjacent spot so it's balanced, with the middle spring post emply) so now I have 4 springs. That seemed to solve the problem. But of course the trem bar is stiffer now. I liked how it felt with 3 springs.
Regarding these points made in your comments:
"...you will have to adjust the tensioning plate inside the trem cavity to being it back level with the body of the guitar. The llate is level to the guitar, not the bar."
I see two screws in there attached to the plate with the springs; that makes sense.
"...you're going to have to adjust the trem claw to make sure the bridge sits parallel to the top of the body..."
I don't know what you mean by "trem claw." Do you mean the same thing as above-- the two screws in the back cavity?
"You'll have to tighten the screws in the back of the guitar to give them more tension so it sits level"
Do you mean the 3 allen screws? I assume you don't mean the intonation screws. How does this interact with the two screws inside the cavity? Which should I adjust first? If I adjust these two things, can I go back to 3 springs?
Is there a written procedure somewhere? I looked all over the PRS website and couldn't find one. I wish PRS included a setup booklet with the guitar like Fender does, or at least had a procedure easy to find on the website.
After all this is sorted out, I can adjust the neck angle; I've done that many times so I'm not worried about that. But I assume I should do the trem adjustments first. Thanks again for all your help.