Articles
Convert Unicode to ASCII7 using AnyASCII Java Library
AnyAscii provides ASCII-only replacement strings for practically all Unicode characters.
"Sanitizing inputs" is critical and it goes beyond checking for SQLi and XSS. For example, some filename naming conventions that work on one OS may not work on another. (Don't name your file "NUL.txt" on Windows or you won't be able to delete it.) As a precaution, we save uploaded files to an offline directory using a random ASCII7 filename and store the original uploaded name separately. If the file is to be available for downloading and is required to retain the original file name, we use CFContent headers or the AHREF "download" parameter to rename it on-the-fly external of the file system.
Some UTF-8/16 characters are intentionally used to impersonate other characters in order to slip past spam & keyword filters. (We also see this behavior a lot with comment form spammers.)
We identified some abuse on our mail servers due to URLs that contained Punycode to spoof official looking domains like Google, PayPal and Adobe. It's difficult to write rules based on what you think you are seeing when the characters aren't what they seem.
When editing scripts using VSCode, you can edit the settings.json file to enable highlighting certain UTF-8 and Unicode characters-especially invisible, ambiguous, or non-ASCII ones. (This isn't enabled by default.) Additional extensions like Render Special Characters, Gremlins and Highlight Bad Chars are available and offer additional configuration.
"editor.unicodeHighlight.invisibleCharacters": true,
"editor.unicodeHighlight.ambiguousCharacters": true,
"editor.unicodeHighlight.nonBasicASCII": true
If you store names in a database using VARCHAR instead of NVARCHAR, your data could become mangled and a Polish last name like "Łukasinski" will become unsearchable and be stored as "�ukasinski". If using Microsoft SQL Server, you can use "COLLATE Latin1_General_CI_AI" to search for "Milliere" and also return accented versions like "Millière", but your mileage may vary and some ColdFusion processes may encode the entities as "Millière".
We processed some UTF-8 CSV data that contain SMS text responses and the third-party executables we used failed when attempting to process the data due to the inclusion of emoji symbols.
So how can you protect your web application, your files, your data and your clients? My preference has been to convert to ASCII7. I've used Junidecode (Java) for years, but it crashes when encountering an emoji or punycode. I recently discovered anyASCII (Java) and found it to be faster and also be fully compatible with emoji & punycode characters. For emojis, it converts the symbols to the "alias" text label.
Source Code and Demo
<!--- AnyASCII Demo
Last Edit: 2025-07-03T19:21:44.37-07:00; Saved 2025-07-03T19:22:08.373-07:00 --->
<!--- AnyASCII Demo
Last Edit: 2025-07-03T19:19:03.613-07:00; Saved 2025-07-03T19:19:10.285-07:00 --->
<!--- AnyASCII Demo
Last Edit: 2025-07-03T19:11:14.43-07:00; Saved 2025-07-03T19:11:28.245-07:00 --->
<h2>AnyASCII: Converts Unicode characters to their best ASCII representation</h2>
<p><b>Live Demo:</b> <a href="https://anyascii.com/">https://anyascii.com/</a><br>
<b>Repository:</b> <a href="https://github.com/anyascii/anyascii" target="_blank" rel="nofollow noopener noreferrer">https://github.com/anyascii/anyascii</a></p>
<fieldset><legend>Example Usage with ColdFusion</legend>
<pre>/*
javapaths = ["D:\JavaLibaries\anyascii\anyascii-0.3.3.jar"];
javaloader = new javaloader.JavaLoader(javapaths);
anyAscii = javaloader.create("com.anyascii.AnyAscii");
*/
anyAscii = createobject("java", "com.anyascii.AnyAscii");
asciiText = anyAscii.transliterate( inputString );</pre></fieldset>
<h2>Unit Test Examples</h2>
<p><button id="toggleButton" type="button">High ASCII Highlight: Disabled</button> <i><b>(Click this button to identify non-ASCII7 characters.)</b></i></p>
<cfscript>
// javapaths = ["D:\JavaLibraries\anyascii\anyascii-0.3.3.jar"];
// javaloader = new javaloader.JavaLoader(javapaths);
// anyAscii = javaloader.create("com.anyascii.AnyAscii");
anyAscii = createobject("java", "com.anyascii.AnyAscii");
tests = [
["name": "Default example", "value": "άνθρωποι"]
,["name": "emojis", "value": "👑 🌴"]
,["name": "emoji with text", "value": "I like 🍕"]
,["name": "misc", "value": "☆ ♯ ♰ ⚄ ⛌"]
,["name": "Letterlike", "value": "№ ℳ ⅋ ⅍"]
,["name": "division slash", "value": "readme ∕ .txt"]
,["name": "division slash", "value": "cámera"]
,["name": "Telephone symbol", "value": "℡"]
,["name": "Trademark and copyright symbol", "value": "™ ® ©"]
,["name": "Pseudolocalization", "value": "śƥàm ƒĩĺţēŕ"]
,["name": "Enclosed Alphanumerics", "value": "ⓔⓧⓐⓜⓟⓛⓔ"]
,["name": "Punycode (L with stroke)", "value": "gmaił.com"]
,["name": "Punycode (Cyrillic o)", "value": "gооgle.com"]
,["name": "Punycode (Latin small capital G)", "value": "ɢoogle.com"]
,["name": "Punycode (Latin small letter B with Dot below)", "value": "adoḅe.com"]
,["name": "Punycode (Evilurl demo)", "value": "ɡіᴛһսᖯ.com"]
,["name": "Accented characters", "value": "ÀÁÂÃÄÅÆÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝàáâãäåæèeêëìíîïñòóôõöøùúûüý'""-..."]
,["name": "Braille", "value": "⠏⠗⠑⠍⠊⠑⠗"]
,["name": "China's Capital", "value": "北亰"]
,["name": "Japanese Name", "value": "Mr. まさゆき たけだ"]
,["name": "Diacritic marks", "value": "côte d'Azur"]
,["name": "Polish", "value": "Łukasinski"]
,["name": "Mathematical Bold Fraktur", "value": "𝕮𝖆𝖑𝖑𝖊 𝕭𝖑𝖆𝖓𝖈𝖔𝖘, 𝕮𝖔𝖘𝖙𝖆 𝕽𝖎𝖈𝖆"]
];
writeoutput("<table id=""anyAsciiTable"" border=""1"" padding=""3"" cellspacing=""0""><thead><tr><th>Test Name</th><th>Input</th><th>Output</th><th>Input<br>Length</th><th>Output<br>Length</th><th>HashCode<br>Equality</th><th>Duration</th></tr></thead><tbody>");
for (test in tests){
timeStart = gettickcount();
asciiText = anyAscii.transliterate(test.value);
timeDuration = gettickcount() - timeStart;
isSame = asciiText.hashCode() eq test.value.hashCode();
writeoutput("<tr><td>#test.Name#</td><td>#encodeforhtml(test.value)#</td><td>#asciiText#</td><td>#len(test.value)#</td><td>#len(asciiText)#</td><td>#isSame#</td><td>#timeDuration#</td></tr>");
}
writeoutput("</tbody></table>");
</cfscript>
<style>
#anyAsciiTable {width:100%;}
#anyAsciiTable tbody tr td {word-wrap: break-word; overflow-wrap:break-word; white-space:normal; max-width: 250px; }
#anyAsciiTable tbody tr:nth-child(even) {background-color: #f2f2f2;}
.dark-theme #anyAsciiTable tbody tr:nth-child(even) {background-color:#333;}
#anyAsciiTable tbody .highlight {background-color: yellow; font-weight: bold; color:red;}
#anyAsciiTable tbody td:nth-of-type(4), #anyAsciiTable tbody td:nth-of-type(5), #anyAsciiTable tbody td:nth-of-type(6), #anyAsciiTable tbody td:nth-of-type(7) {text-align:center;}
#toggleButton {padding:10px; margin:10px; font-weight:bold;}
</style>
<script defer>
function attachToggleHighlight(button, table) {
var isHighlighted = false;
var originalContents = [];
/* Store original cell contents */
var rows = table.rows;
for (var i = 0; i < rows.length; i++) {
originalContents[i] = [];
var cells = rows[i].cells;
for (var j = 0; j < cells.length; j++) {
originalContents[i][j] = cells[j].innerHTML;
}
}
button.onclick = function() {
isHighlighted = !isHighlighted;
for (var i = 0; i < rows.length; i++) {
var cells = rows[i].cells;
for (var j = 0; j < cells.length; j++) {
if (isHighlighted) {
cells[j].innerHTML = originalContents[i][j].replace(/[^\x00-\x7F]/g, function(match) {
return '<span class="highlight">' + match + '</span>';
});
} else {
cells[j].innerHTML = originalContents[i][j];
}
}
}
button.textContent = isHighlighted ? 'High ASCII Highlight: Enabled' : 'High ASCII Highlight: Disabled';
};
}
document.addEventListener('DOMContentLoaded', function() {
var button = document.getElementById('toggleButton');
var table = document.getElementById('anyAsciiTable');
if (!button || !table) {
console.error('Button or table not found. Ensure elements with IDs "toggleButton" and "myTable" exist.');
return;
}
button.disabled = true;
window.onload = function() {
button.disabled = false;
attachToggleHighlight(button, table);
};
});
</script>