[activity] remove ref for now (#32645)

Followup from https://github.com/facebook/react/pull/32499

Manual mode is unused and has some bugs such as revealing hidden
boundaries when manually toggling. We also want to change how manual
mode works, and do some refactors to Activity to make it easier to
support. For now we'll remove it, then add it back after the other
changes we have planned.