merge develop to master #8
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ensure all user-facing strings are covered
…one' option from docs and tests
…up backups via TCP
…ckups to OSS or stream via TCP with i18n support
- Add --existed-backup parameter to upload existing xtrabackup backups - Support both OSS upload and TCP streaming modes - Add version management with VERSION file and version.sh script - Add --version/-v command line options - Update Makefile with version management targets - Update documentation in both Chinese and English READMEs - Set version to 1.0.0-alpha for pre-release testing
Feature/upload existed file
…date docs to match - build outputs fixed 'backup-helper' name again - remove versioned filename notes from README.md and README.en.md - keep --version output minimal (version only)
… resolved-version target
…slate comments to English
- Add real-time progress display with upload speed and percentage - Add automatic IO bandwidth detection with dd benchmark (3x average) - Add manual rate limiting with --io-limit flag - Add --auto-limit-rate for automatic bandwidth limiting (80% of detected) - Add --estimated-size for accurate progress calculation - Calculate backup size from MySQL datadir or existing backup files - Update progress every 500ms to reduce console flooding - Show final statistics (total size, duration, average speed) - Add bandwidth.go module for IO detection - Add progress.go for progress tracking - Add datadir.go for size calculation - Update README with new features and examples - Add mysql-backup-helper to .gitignore
- Add GetAvailablePort() to automatically find available port - Add GetLocalIP() to get local IP address - Support --stream-port=0 to auto-find available port - Display listening IP and port in output - Update StartStreamServer to return actual port and IP - Update README with auto port detection examples - Remove requirement to specify port when using stream mode
- Change startTime to be initialized only on first Update() call - Use sync.Once to ensure timer starts exactly when data flows - Avoid counting initialization time in progress duration - Handle case when no data is transferred (startTime is zero)
- Remove default 9999 assignment for StreamPort in SetDefaults() - Update port selection logic to only use config value when explicitly set (>0) - Preserve streamPort=0 to trigger auto-find available port - Fix issue where auto port detection was blocked by default value
- Remove mysql-backup-helper binary from version control - File is already in .gitignore and should not be tracked
- Add comprehensive logging to diagnose dd command execution failures - Log Linux vs macOS style dd command attempts and their results - Log parsing process for both Linux and macOS output formats - Log test results with success/failure details - Display full output lines when parsing fails for debugging
- Remove '2>&1' from dd command arguments as CombinedOutput() already merges stdout/stderr - This was causing 'unrecognized operand' error on macOS - Add output logging for both Linux and macOS dd attempts
- Extend parseDDOutput to handle GB/s, MB/s, and KB/s units - Fixes parsing failure on macOS where dd outputs speed in GB/s format - Example: '1.1 GB/s' is now correctly parsed and converted to bytes/s
…ate limiting - Remove dd disk stress testing to avoid impacting production systems - Add real-time IO monitoring using iostat (Linux/macOS compatible) - Implement dynamic rate adjustment based on IO utilization: * Automatically reduce rate by 10% when IO > 80% * Gradually restore rate by 10% when IO < 56% * Minimum protection: never below 10% of original limit - Add RateLimitedWriter for TCP stream rate limiting - Update OSS upload to use dynamic rate limits per part - Update documentation with new monitoring and adjustment behavior
- Fix TCP stream rate limiting to work without autoLimitRate flag - Fix OSS upload rate limiting with client-side RateLimitedReader - Add unit parsing for --io-limit (supports B, KB, MB, GB, TB, KiB, MiB, GiB, TiB) - Add detailed debug logging for iostat output and parsing - Change --io-limit from Int64 to String to support unit parsing - Add RateLimitedReader for client-side OSS upload throttling - Improve rate limiting logic: always apply if ioLimit is set, not just when autoLimitRate is enabled
- Add ParseRateLimit function to support different units (KB/s, MB/s, GB/s, TB/s) - Change --io-limit from Int64Var to StringVar to accept unit-based values - Ensure rate limiting works for both OSS upload and TCP stream modes - Apply rate limiting even when auto-limit-rate is disabled - Fix missing check() function in utils/mysql.go Examples: --io-limit 100MB/s --io-limit 1GB/s --io-limit 500KB/s
- Add IO utilization (util%) from iostat to progress display - Add current rate limit display in progress bar - Update ProgressTracker with SetIOInfo method to update IO stats - Start background goroutine to periodically update IO info every 2 seconds - Support both OSS upload and TCP stream modes - Display format: Progress: X / Y (Z%) - Speed - Duration - IO: X% - Limit: X/s
- Remove auto-limit-rate feature and IO monitoring - Support io-limit=-1 to disable rate limiting completely - Update default rate limit to 200MB/s - Simplify progress display (remove IO utilization and limit info) - Update documentation and examples
- Merge tcp_client.go and tcp_receiver.go into tcp_stream.go - Rename StartStreamServer to StartStreamSender for clarity - Extract rate limiting code from progress.go to rate_limit.go - Remove bandwidth.go (no longer used after removing auto-limit-rate) - Remove AutoLimitRate field from config.go - Add --download mode for receiving backup data via TCP stream - Add --output parameter for download mode (supports stdout via '-') - Add RateLimitedReader for download mode rate limiting
- Add --download and --output parameters to command-line arguments - Add download mode usage examples - Update progress tracking description to include download mode - Update rate limiting description to include download mode
- Add outputHeader() to download mode - Fix progress tracker to show 'Download completed' and 'Total downloaded' for download mode - Clear progress line before displaying completion message for better formatting
- Add ParseSize function to parse size strings with units (KB, MB, GB, TB) - Change --estimated-size from Int64Var to StringVar - Update README examples to show unit usage - Support same units as io-limit: B, KB, MB, GB, TB
- Add DownloadOutput field to Config struct - Support downloadOutput in config.json (command line takes precedence) - Update example-config.json to include downloadOutput field - Ensure proper priority: command line > config file > default path
- Add --extract-to parameter for direct extraction to directory - Add --extract-compress-type parameter for compressed backups - Implement ExtractXbstream and ExtractXbstreamWithDecompress functions - Support zstd and qpress compression types - Update README with stream extraction examples and notes
- Add SetOutputToStderr() method to ProgressTracker - When --output - is used, output progress to stderr instead of stdout - This allows clean piping to xbstream or other tools - Update README with examples of using --output - with xbstream
…ping - Change StartStreamReceiver info messages to stderr - Change header output to stderr when outputting to stdout - Change IO limit messages to stderr when outputting to stdout - Ensure stdout only contains pure backup data for xbstream
Feature/progress
|
fanke seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.