So am I really ridiculously screwed if I can’t remember anything I learned in physics?

No! I’m sure you remember more than you think you do. You don’t need to remember every single detail and you’re professors will probably do a little refresher of the essentials. And it’s always easier to relearn something than to learn it for the first time.