LICENSE.md
6 days ago
README.md
6 days ago
compat-utf8.php
6 days ago
composer.json
6 days ago
utf8-encoder.php
6 days ago
utf8.php
6 days ago
README.md
198 lines
| 1 | --- |
| 2 | slug: encoding |
| 3 | title: Encoding |
| 4 | install: wp-php-toolkit/encoding |
| 5 | |
| 6 | see_also: |
| 7 | - html | HTML | Normalize incoming text before HTML tokenization. |
| 8 | - xml | XML | Keep invalid bytes out of XML streams. |
| 9 | - dataliberation | DataLiberation | Clean content before importing it into WordPress. |
| 10 | --- |
| 11 | |
| 12 | UTF-8 validation and scrubbing with a pure-PHP fallback when <code>mbstring</code> is unavailable. Detects malformed bytes and replaces them per the Unicode maximal-subpart algorithm. |
| 13 | |
| 14 | ## Why this exists |
| 15 | |
| 16 | <p>Every parser in this toolkit eventually has to decide what to do with text bytes. XML rejects malformed UTF-8. JSON and databases can fail late. CSS, HTML, WXR, and Blueprint validation all need consistent answers about whether a string is well-formed Unicode.</p> |
| 17 | |
| 18 | <p>The Encoding component provides the small UTF-8 primitives the rest of the toolkit can share: validate bytes, scrub invalid sequences, scan code points, and detect Unicode noncharacters. When <code>mbstring</code> is available it can delegate to it; when it is not, the component uses its own byte scanner so behavior stays available in restricted PHP environments.</p> |
| 19 | |
| 20 | <p>Historically, this became the common foundation for Blueprint validation and CSS/XML processing, replacing ad hoc Unicode helpers with the WordPress core UTF-8 routines used here.</p> |
| 21 | |
| 22 | ## Validating UTF-8 before storing it |
| 23 | |
| 24 | <p><code>wp_is_valid_utf8()</code> rejects overlong sequences, surrogate halves, and stray ISO-8859-1 bytes. Use it as a guard in front of any code path that assumes UTF-8 (database, JSON, XML).</p> |
| 25 | |
| 26 | <!-- snippet: |
| 27 | filename: validate.php |
| 28 | runnable: true |
| 29 | --> |
| 30 | ```php |
| 31 | <?php |
| 32 | require '/php-toolkit/vendor/autoload.php'; |
| 33 | |
| 34 | use function WordPress\Encoding\wp_is_valid_utf8; |
| 35 | |
| 36 | $samples = array( |
| 37 | 'ASCII' => 'just a test', |
| 38 | 'UTF-8 pencil' => "\xE2\x9C\x8F", |
| 39 | 'latin-1 byte' => "B\xFCch", |
| 40 | 'overlong slash' => "\xC1\xBF", |
| 41 | 'surrogate half' => "\xED\xB0\x80", |
| 42 | ); |
| 43 | |
| 44 | foreach ( $samples as $label => $bytes ) { |
| 45 | echo sprintf( "%-14s %s\n", $label . ':', wp_is_valid_utf8( $bytes ) ? 'valid' : 'invalid' ); |
| 46 | } |
| 47 | ``` |
| 48 | |
| 49 | <!-- expected-output --> |
| 50 | ``` |
| 51 | ASCII: valid |
| 52 | UTF-8 pencil: valid |
| 53 | latin-1 byte: invalid |
| 54 | overlong slash: invalid |
| 55 | surrogate half: invalid |
| 56 | ``` |
| 57 | |
| 58 | ## Scrubbing invalid bytes with U+FFFD |
| 59 | |
| 60 | <p>Replace each ill-formed sequence with the Unicode replacement character. Useful right before serializing to XML, JSON, or sending to an LLM that will choke on broken bytes.</p> |
| 61 | |
| 62 | <!-- snippet: |
| 63 | filename: scrub.php |
| 64 | runnable: true |
| 65 | --> |
| 66 | ```php |
| 67 | <?php |
| 68 | require '/php-toolkit/vendor/autoload.php'; |
| 69 | |
| 70 | use function WordPress\Encoding\wp_scrub_utf8; |
| 71 | |
| 72 | $broken = "the byte \xC0 should not be here."; |
| 73 | echo wp_scrub_utf8( $broken ) . "\n"; |
| 74 | |
| 75 | echo wp_scrub_utf8( ".\xE2\x8C\xE2\x8C." ) . "\n"; |
| 76 | ``` |
| 77 | |
| 78 | <!-- expected-output --> |
| 79 | ``` |
| 80 | the byte � should not be here. |
| 81 | .��. |
| 82 | ``` |
| 83 | |
| 84 | ## Detecting Unicode noncharacters |
| 85 | |
| 86 | <p>Code points like U+FFFE, U+FFFF, and the U+FDD0–U+FDEF block are valid Unicode scalar values but forbidden in XML and unwelcome in many interchange formats. Check for them before serializing user-submitted content into strict XML, WXR, or other systems that reject noncharacters.</p> |
| 87 | |
| 88 | <!-- snippet: |
| 89 | filename: noncharacters.php |
| 90 | runnable: true |
| 91 | --> |
| 92 | ```php |
| 93 | <?php |
| 94 | require '/php-toolkit/vendor/autoload.php'; |
| 95 | |
| 96 | use function WordPress\Encoding\wp_has_noncharacters; |
| 97 | |
| 98 | $samples = array( |
| 99 | 'normal text' => 'normal text', |
| 100 | 'U+FFFE' => "oops \u{FFFE}", |
| 101 | 'U+FDD0' => "hi \u{FDD0} bye", |
| 102 | ); |
| 103 | |
| 104 | foreach ( $samples as $label => $text ) { |
| 105 | echo sprintf( "%-12s %s\n", $label . ':', wp_has_noncharacters( $text ) ? 'reject' : 'ok' ); |
| 106 | } |
| 107 | ``` |
| 108 | |
| 109 | <!-- expected-output --> |
| 110 | ``` |
| 111 | normal text: ok |
| 112 | U+FFFE: reject |
| 113 | U+FDD0: reject |
| 114 | ``` |
| 115 | |
| 116 | ## Three-way pipeline: validate, scrub, then check noncharacters |
| 117 | |
| 118 | <p>Real-world inputs are messy: an old WXR export, a CSV with mixed encodings, a paste from Word. Combination of validate + scrub + noncharacter-check covers the three classes of breakage that bite later.</p> |
| 119 | |
| 120 | <!-- snippet: |
| 121 | filename: pipeline.php |
| 122 | runnable: true |
| 123 | --> |
| 124 | ```php |
| 125 | <?php |
| 126 | require '/php-toolkit/vendor/autoload.php'; |
| 127 | |
| 128 | use function WordPress\Encoding\wp_is_valid_utf8; |
| 129 | use function WordPress\Encoding\wp_scrub_utf8; |
| 130 | use function WordPress\Encoding\wp_has_noncharacters; |
| 131 | |
| 132 | $inputs = array( |
| 133 | 'good' => 'Café', |
| 134 | 'latin1' => "caf\xE9", |
| 135 | 'overlong' => "x\xC1\xBFy", |
| 136 | 'noncharac' => "hi \u{FFFE} there", |
| 137 | ); |
| 138 | |
| 139 | foreach ( $inputs as $label => $bytes ) { |
| 140 | $valid = wp_is_valid_utf8( $bytes ); |
| 141 | $cleaned = wp_scrub_utf8( $bytes ); |
| 142 | $weird = wp_has_noncharacters( $cleaned ); |
| 143 | echo sprintf( "%-10s valid=%s noncharacter=%s -> %s\n", $label, $valid ? 'Y' : 'N', $weird ? 'Y' : 'N', $cleaned ); |
| 144 | } |
| 145 | ``` |
| 146 | |
| 147 | <!-- expected-output --> |
| 148 | ``` |
| 149 | good valid=Y noncharacter=N -> Café |
| 150 | latin1 valid=N noncharacter=N -> caf� |
| 151 | overlong valid=N noncharacter=N -> x��y |
| 152 | noncharac valid=Y noncharacter=Y -> hi there |
| 153 | ``` |
| 154 | |
| 155 | ## Salvaging a legacy ISO-8859-1 column inside a UTF-8 corpus |
| 156 | |
| 157 | <p>Old WordPress databases sometimes mix encodings: most rows are UTF-8 but a few were stored as latin-1. Detect the bad rows with <code>wp_is_valid_utf8()</code> and only re-encode those.</p> |
| 158 | |
| 159 | <!-- snippet: |
| 160 | filename: mixed-encoding.php |
| 161 | runnable: true |
| 162 | --> |
| 163 | ```php |
| 164 | <?php |
| 165 | require '/php-toolkit/vendor/autoload.php'; |
| 166 | |
| 167 | use function WordPress\Encoding\wp_is_valid_utf8; |
| 168 | use function WordPress\Encoding\wp_scrub_utf8; |
| 169 | |
| 170 | $rows = array( |
| 171 | 1 => 'Plain ASCII', |
| 172 | 2 => 'Café', |
| 173 | 3 => "caf\xE9", |
| 174 | 4 => "weird \xC0 byte", |
| 175 | ); |
| 176 | |
| 177 | foreach ( $rows as $id => $value ) { |
| 178 | if ( wp_is_valid_utf8( $value ) ) { |
| 179 | echo "#$id ok: $value\n"; |
| 180 | continue; |
| 181 | } |
| 182 | $converted = @iconv( 'ISO-8859-1', 'UTF-8', $value ); |
| 183 | if ( false !== $converted && wp_is_valid_utf8( $converted ) ) { |
| 184 | echo "#$id recovered as latin1: $converted\n"; |
| 185 | } else { |
| 186 | echo "#$id unrecoverable, scrubbing: " . wp_scrub_utf8( $value ) . "\n"; |
| 187 | } |
| 188 | } |
| 189 | ``` |
| 190 | |
| 191 | <!-- expected-output --> |
| 192 | ``` |
| 193 | #1 ok: Plain ASCII |
| 194 | #2 ok: Café |
| 195 | #3 recovered as latin1: café |
| 196 | #4 recovered as latin1: weird À byte |
| 197 | ``` |
| 198 |