SIGN IN SIGN UP

refactor(search): one tokenizer entry point for the module

`run()`'s pre-checks tokenized with plain `tokenize` while `extract_pattern_path`
used `claims_literal_dash_dash`, so the two could disagree about whether a `--`
belonged to a flag -- and the args were tokenized twice per invocation either
way. Both go through `tokenize_search_args` now.

Also drops a `let args = args.clone()` left behind when the restore moved to the
top of `run()`: a full copy of the argument vector on every `rtk grep`/`rtk rg`,
under a comment describing a restore that no longer happened there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
N
Nicolas Le Cam committed
e507aa50d92f03ef68de8a520c2440dc5f18f407
Parent: 0907594