mirror of
https://github.com/chatboxai/chatbox.git
synced 2026-03-27 10:51:18 +00:00
0f906a74af
* fix(token): estimate attachment tokens using preview rules Token estimation in input box now matches actual send payload behavior: - Large files (>500 lines) with tool-capable models use preview tokens - Small files always use full tokens - Includes wrapper token overhead in estimation Changes: - Add shared attachment-payload module with truncation constants - Extend preprocessing to compute lineCount/byteLength/preview tokens - Implement estimateTokensFromMessagesForSendPayload() function - Wire payload-aware estimation into useTokenCount and InputBox - Add comprehensive tests (94 tests passing) * feat(token): implement token estimation system redesign - Add new token-estimation module with background async calculation - Implement computation queue with priority and concurrency control - Add task executor and result persister with debounce - Create useTokenEstimation hook replacing useTokenCount - Update TokenCountMenu with calculation status indicator - Integrate into InputBox component - Simplify preprocessing to calculate only current tokenizer - Add dev tools and logging for debugging - Mark old useTokenCount as deprecated The new system calculates tokens in the background without blocking UI, supports session-based cancellation, and provides detailed progress info. * fix(token): initialize executor and improve calculation status UI - Fix critical bug: executor was never initialized, causing tasks to get stuck - Rename token_estimation_dev_tools.ts to token_estimation_init.ts - Call initializeExecutor() and setResultPersister() on app startup - Initialize in ALL environments, not just development - Add Loader indicator to InputBox token display when calculating - Show '~' prefix on token numbers during calculation - Add detailed progress display in modal (completed/total context messages) * fix(token): calculate current input tokens inline instead of async The constructedMessage only exists in React state, not in the store, so async task execution was failing with 'message not found'. Now calculate tokens synchronously for current input. * fix(routes): clear draft when sending message from homepage Pass onUserMessageReady callback from InputBox to submitNewUserMessage so that clearDraft() is called after message is sent successfully. Bug: When sending a message from homepage and then clicking 'New Chat', the draft text remained in the input box because onUserMessageReady callback was not being passed through. * fix(token-estimation): fix isCalculating state and cancelBySession race condition - Fix isCalculating to include pendingTasks from analysis result, not just queue status - Fix cancelBySession to only cancel when sessionId actually changes (not on unmount) - This fixes the bug where calculating state was never shown due to timing issues * fix(token-estimation): use throttle for intermediate progress display - Change flush strategy from debounce (500ms) to throttle (1000ms) - Guarantees flush at least once per second during calculation - Remove redundant invalidateQueries() call (setQueryData already updates cache) - Restore maxConcurrency to 4 This fixes the issue where users couldn't see intermediate token counts during long calculations because debounce kept resetting the timer. * docs: add token estimation docs and React Query best practices - Add comprehensive token estimation system documentation (docs/token-estimation.md) - Add React Query data management section to AGENTS.md - Prefer setQueryData over invalidateQueries for performance - Document query key patterns and usage examples - Update result-persister tests for throttle behavior (was debounce) * fix(token-estimation): reduce max concurrency to 1 and add task execution interval for progress visualization * fix(logger): change debug logging to use console.debug directly * fix(token-estimation): fix session switching to properly isolate token calculation state - Add getStatusForSession(sessionId) method to return session-specific queue status - Replace broken prevSessionIdRef logic with cleanup function (works with React key remount) - Clear cancelledSessions when re-enqueueing tasks (fixes React Strict Mode double mount) - Update useTokenEstimation to use session-scoped status with sessionId dependency Fixes: Token calculation status incorrectly showing previous session's state after switching * fix(token-estimation): prioritize newest messages and cancel tasks when context changes - Reverse priority order: newest messages now have highest priority (calculated first) - Add retainOnlyMessages() to cancel tasks for messages no longer in context - Call retainOnlyMessages when contextMessages changes (e.g., maxContextMessageCount reduced) - Fixes negative progress display (-190/10) when reducing context limit - Update tests for new priority order * fix(context-management): remove any types in attachment-payload * fix(ui): prevent negative progress in token count menu * docs(token-estimation): fix maxConcurrency docs and add code block languages * chore: fix typo in gitignore (.sysyphus -> .sisyphus) * fix(token-estimation): use TextEncoder for correct UTF-8 byte length content.length returns UTF-16 code units, not actual byte size. For non-ASCII characters (e.g., Chinese), this gives incorrect results. Now matches the ground truth in task-executor.ts.
180 lines
2.1 KiB
Plaintext
180 lines
2.1 KiB
Plaintext
# Logs
|
|
logs
|
|
*.log
|
|
npm-debug.log*
|
|
yarn-debug.log*
|
|
yarn-error.log*
|
|
lerna-debug.log*
|
|
|
|
# Diagnostic reports (https://nodejs.org/api/report.html)
|
|
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
|
|
|
|
# Runtime data
|
|
pids
|
|
*.pid
|
|
*.seed
|
|
*.pid.lock
|
|
.DS_Store
|
|
|
|
# Directory for instrumented libs generated by jscoverage/JSCover
|
|
lib-cov
|
|
|
|
# Coverage directory used by tools like istanbul
|
|
coverage
|
|
*.lcov
|
|
|
|
# nyc test coverage
|
|
.nyc_output
|
|
|
|
# node-waf configuration
|
|
.lock-wscript
|
|
|
|
# Compiled binary addons (https://nodejs.org/api/addons.html)
|
|
build/Release
|
|
|
|
# Dependency directories
|
|
node_modules/
|
|
jspm_packages/
|
|
|
|
# TypeScript v1 declaration files
|
|
typings/
|
|
|
|
# TypeScript cache
|
|
*.tsbuildinfo
|
|
|
|
# Optional npm cache directory
|
|
.npm
|
|
|
|
# Optional eslint cache
|
|
.eslintcache
|
|
|
|
# Optional REPL history
|
|
.node_repl_history
|
|
|
|
# Output of 'npm pack'
|
|
*.tgz
|
|
|
|
# Yarn Integrity file
|
|
.yarn-integrity
|
|
|
|
# dotenv environment variables file
|
|
.env
|
|
.env.test
|
|
|
|
# parcel-bundler cache (https://parceljs.org/)
|
|
.cache
|
|
|
|
# next.js build output
|
|
.next
|
|
|
|
# nuxt.js build output
|
|
.nuxt
|
|
|
|
# vuepress build output
|
|
.vuepress/dist
|
|
|
|
# Serverless directories
|
|
.serverless/
|
|
|
|
# FuseBox cache
|
|
.fusebox/
|
|
|
|
# DynamoDB Local files
|
|
.dynamodb/
|
|
|
|
# Webpack
|
|
.webpack/
|
|
|
|
# Electron-Forge
|
|
out/
|
|
|
|
publish.sh
|
|
build.sh
|
|
build-web.sh
|
|
|
|
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
|
|
|
|
# dependencies
|
|
/node_modules
|
|
/.pnp
|
|
.pnp.js
|
|
|
|
# testing
|
|
/coverage
|
|
/test/output
|
|
|
|
# next.js
|
|
/.next/
|
|
/out/
|
|
|
|
# production
|
|
/build
|
|
|
|
# misc
|
|
.DS_Store
|
|
*.pem
|
|
|
|
# debug
|
|
npm-debug.log*
|
|
yarn-debug.log*
|
|
yarn-error.log*
|
|
.pnpm-debug.log*
|
|
|
|
# local env files
|
|
.env*.local
|
|
|
|
# IDE / Editor
|
|
.idea
|
|
.dir-locals.el
|
|
|
|
# vercel
|
|
.vercel
|
|
|
|
# typescript
|
|
*.tsbuildinfo
|
|
next-env.d.ts
|
|
|
|
tmp
|
|
|
|
# Logs
|
|
logs
|
|
*.log
|
|
|
|
# Runtime data
|
|
pids
|
|
*.pid
|
|
*.seed
|
|
|
|
# Coverage directory used by tools like istanbul
|
|
coverage
|
|
.eslintcache
|
|
|
|
# Dependency directory
|
|
# https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git
|
|
node_modules
|
|
|
|
# OSX
|
|
.DS_Store
|
|
|
|
release/app/dist
|
|
release/build
|
|
.erb/dll
|
|
|
|
.idea
|
|
npm-debug.log.*
|
|
*.css.d.ts
|
|
*.sass.d.ts
|
|
*.scss.d.ts
|
|
|
|
dist
|
|
|
|
electron-builder.env
|
|
|
|
# Share VS Code settings
|
|
# .vscode
|
|
|
|
|
|
**/routeTree.gen.ts
|
|
|
|
.env.sentry-build-plugin
|
|
.sisyphus/ |