20250728_00 - Release

unixwzrd · unixwzrd · commit 1982bbae612f · 2025-07-28T13:55:40.000-05:00
- Switched from Unidecode to ftfy: Replaced aggressive Unicode-to-ASCII conversion with intelligent text fixing
- Preserves Extended ASCII: Now correctly preserves 8-bit extended ASCII characters (128-255) like é, ñ, ü, etc.
- Smarter Unicode Handling: Only converts problematic Unicode characters while preserving intentional extended ASCII usage
- Updated Dependencies: Replaced Unidecode dependency with ftfy in requirements.txt
- Maintains AI Artifact Removal: Still removes smart quotes, EM/EN dashes, and other "AI tells" as designed
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,8 +1,18 @@
 # Changelog for UnicodeFix
 
+## 2025-07-28
+
+### **Extended ASCII Preservation Fix**
+
+- **Switched from Unidecode to ftfy:** Replaced aggressive Unicode-to-ASCII conversion with intelligent text fixing
+- **Preserves Extended ASCII:** Now correctly preserves 8-bit extended ASCII characters (128-255) like é, ñ, ü, etc.
+- **Smarter Unicode Handling:** Only converts problematic Unicode characters while preserving intentional extended ASCII usage
+- **Updated Dependencies:** Replaced `Unidecode` dependency with `ftfy` in requirements.txt
+- **Maintains AI Artifact Removal:** Still removes smart quotes, EM/EN dashes, and other "AI tells" as designed
+
 ## 2025-07-23
 
-**Test Suite Fixes & Validation**
+### **Test Suite Fixes & Validation**
 
 - **Fixed Default Scenario:** Corrected test script to properly handle default behavior (creates `.clean.ext` files) without using `-o` flag that caused errors with multiple files
 - **Cascading File Prevention:** Added filtering to prevent processing already-cleaned `.clean.ext` files in subsequent test runs
@@ -19,7 +29,7 @@
 
 ## 2025-07-22
 
-**Major Release - "Enough of Your AI Nonsense" Edition**
+### **Major Release - "Enough of Your AI Nonsense" Edition**
 
 - **CLI Supercharged:** Added new power flags:
   `-i` / `--invisible` (preserve zero-width/invisible Unicode)
diff --git a/bin/cleanup-text.py b/bin/cleanup-text.py
@@ -22,13 +22,13 @@
 import re
 import sys
 
-# Check for unidecode dependency early, with a clear message if missing
+# Check for ftfy dependency early, with a clear message if missing
 try:
-    from unidecode import unidecode  # noqa: F401
+    import ftfy
 except ImportError:
     print(
-        "[✗] Missing dependency: 'Unidecode'. Please install it with:\n"
-        "    pip install Unidecode\n"
+        "[✗] Missing dependency: 'ftfy'. Please install it with:\n"
+        "    pip install ftfy\n"
         "Or install all requirements with:\n"
         "    pip install -r requirements.txt",
         file=sys.stderr
@@ -63,6 +63,10 @@ def clean_text(text: str, preserve_invisible: bool = False) -> str:
     Returns:
         str: The cleaned text with normalized ASCII characters
     """
+    # Use ftfy for intelligent text fixing and normalization
+    text = ftfy.fix_text(text)
+    
+    # Handle specific cases that unidecode might not handle perfectly
     replacements = {
         '\u2018': "'", '\u2019': "'",  # Smart single quotes
         '\u201C': '"', '\u201D': '"',  # Smart double quotes
@@ -153,11 +157,14 @@ def main():
         # No files provided: filter mode (STDIN to STDOUT)
         raw = sys.stdin.read()
         cleaned = clean_text(raw, preserve_invisible=args.invisible)
-        # Add or suppress newline at EOF based on -n/--no-newline
+
+        # Handle newline at EOF based on -n/--no-newline
         if not args.no_newline:
-            cleaned = ensure_single_newline(cleaned)
-        else:
-            cleaned = cleaned.rstrip('\r\n')
+            # Only add newline if there isn't one already
+            if not cleaned.endswith('\n'):
+                cleaned += '\n'
+        # If --no-newline is specified, leave the file exactly as is (no changes to newlines)
+        
         sys.stdout.write(cleaned)
         return
 
diff --git a/requirements.txt b/requirements.txt
@@ -1 +1 @@
-Unidecode
+ftfy
