Articles
generateEmailHashCode UDF
Useful for identifying and blocking comment form abuse; SQL queries using INTs are faster
The generateEmailHashCode user-defined function (UDF) processes an email address by extracting the domain, sanitizing the username by removing periods (dots) and any part after the first "+", converting both parts to lowercase, and then returning a consistent Java-generated integer hash code of the resulting string. This process aims to create a consistent hash value for email addresses that might have variations in case or the presence of periods or "+"-suffixes in the username.
We log all web-based contact form submissions and store submitted emails along with the sanitized, normalized integer. This has enabled us to 1) easily block submissions whenever abusers change the email by using variations of the period or 2) identify when users create multiple accounts using the "+" suffix. We also publish a file named Comment-Spam-Email-Addresses.txt that contains email addresses and domains of repeat abusers.
While reviewing the logs of failed contact form submissions, I identified a couple email address variations that were exploiting some Gmail features in an attempt to bypass our filters. (Gmail has a "plus" feature and ignores periods in addresses.) A SQL query using REPLACE to remove all periods revealed that this comment form spammer had performed 279 attempts using 162 variations of their 15 character gmail username in an effort to circumvent our filters. We log the full email address that was posted and, when matching via SQL solely using the email addresses, it appeared as each email address was only used 2-4 times... versus the 279 obfuscated attempts.
To better identify & highlight abusers via SQL queries, an EmailHash (INT) column has been added to the database table. When searching or logging the email address, the value is sanitized (remove + string and . from the username) and then a java hashCode is generated. Using integers to join database records is much faster than using varchar and has lower storage requirements.
While there's a very slight risk of collision, I'm not aware of any other numeric hashing solution. I've tested this on multiple ColdFusion/Lucee installations and Java's hashCode has returned consistent results regardless of platform and/or JDK. I've used hashCode in some other ColdFusion applications and even have some other blog posts regarding it:
- Hash strings to INT using Java
- Caching Repetitive UDF Access to Request Scope Using Argument Stringified+Hashed Key
Source Code
<cfscript>
public numeric function generateEmailHashCode(required string email) hint="I sanitize email and generate a unique integer hash using java hashCode" {
local.d = tostring(listlast(trim(arguments.email), "@"));
local.u = (listlen(arguments.email,"@") gt 1) ? trim(listfirst(arguments.email, "@")) : "";
local.u = listfirst(javacast("string", local.u), "+").replaceAll("\.", "");
local.e = lcase(trim(local.u)) & "@" & lcase(trim(local.d));
return javacast("int", local.e.hashCode());
}
</cfscript>
Demo
<cfscript>
tests = [
"alberthanshi.n49@gmail.com", "albertha.nshi.n49@gmail.com", "albertha.n.s.h.in.4.9@gmail.com", "alberth.an.s.h.in4.9@gmail.com", "alberth.an.s.h.i.n.49@gmail.com", "alberth.a.ns.hi.n.4.9@gmail.com", "albert.hans.hi.n.4.9@gmail.com", "alber.th.a.nsh.in.49@gmail.com", "alber.th.a.n.s.hin.4.9@gmail.com", "alber.th.a.n.s.h.in49@gmail.com", "alber.t.ha.n.shin.49@gmail.com", "albe.rthan.sh.in49@gmail.com", "albe.rthan.s.hi.n4.9@gmail.com", "albe.rth.a.n.shi.n4.9@gmail.com", "albe.rt.hans.hin.4.9@gmail.com", "albe.rt.hans.h.i.n.4.9@gmail.com", "albe.rt.ha.nshin.49@gmail.com", "albe.rt.ha.nsh.in49@gmail.com", "albe.rt.h.an.sh.in.4.9@gmail.com", "albe.rt.h.a.n.s.hi.n.4.9@gmail.com", "albe.r.tha.ns.hi.n49@gmail.com", "albe.r.tha.n.s.h.i.n4.9@gmail.com", "albe.r.t.hans.h.in4.9@gmail.com", "alb.erth.ans.h.i.n49@gmail.com", "alb.erth.a.ns.hi.n49@gmail.com", "alb.erth.a.ns.hi.n4.9@gmail.com", "alb.erth.a.n.shi.n.4.9@gmail.com", "alb.ert.hans.hi.n49@gmail.com", "alb.ert.han.shin49@gmail.com", "alb.ert.h.ans.h.in4.9@gmail.com", "alb.ert.h.a.ns.h.in.49@gmail.com", "alb.er.thanshin49@gmail.com", "alb.er.tha.nshin49@gmail.com", "alb.er.th.a.nsh.in49@gmail.com", "alb.er.th.a.ns.hin.4.9@gmail.com", "alb.er.t.han.s.hin.49@gmail.com", "alb.er.t.ha.nshin4.9@gmail.com", "alb.er.t.ha.nshi.n.4.9@gmail.com", "alb.er.t.ha.ns.hin4.9@gmail.com", "alb.er.t.h.ansh.in49@gmail.com", "alb.er.t.h.an.s.h.in.49@gmail.com", "alb.e.rth.an.sh.in49@gmail.com", "alb.e.rth.a.n.s.hin49@gmail.com", "alb.e.rt.h.an.s.hin.49@gmail.com", "alb.e.r.th.a.nshin.4.9@gmail.com", "alb.e.r.t.h.a.ns.h.i.n49@gmail.com", "al.bertha.n.shin49@gmail.com", "al.bert.ha.n.s.hin4.9@gmail.com", "al.bert.h.an.sh.i.n.49@gmail.com", "al.bert.h.a.n.s.h.in.4.9@gmail.com", "al.ber.tha.nshin49@gmail.com", "al.ber.tha.n.shin4.9@gmail.com", "al.ber.th.a.n.s.hi.n49@gmail.com", "al.ber.th.a.n.s.hi.n.49@gmail.com", "al.ber.t.h.an.sh.i.n.4.9@gmail.com", "al.ber.t.h.an.s.hin4.9@gmail.com", "al.ber.t.h.a.ns.hi.n4.9@gmail.com", "al.ber.t.h.a.ns.h.in49@gmail.com", "al.be.rthan.sh.i.n49@gmail.com", "al.be.rtha.n.s.hin.49@gmail.com", "al.be.rth.ans.h.in.4.9@gmail.com", "al.be.r.th.a.n.shi.n.4.9@gmail.com", "al.be.r.t.h.an.sh.i.n.49@gmail.com", "al.be.r.t.h.a.n.shin.49@gmail.com", "al.b.ertha.n.s.hin.49@gmail.com", "al.b.erth.a.n.s.hi.n.4.9@gmail.com", "al.b.ert.hansh.in49@gmail.com", "al.b.er.thanshi.n.4.9@gmail.com", "al.b.er.than.s.hin.4.9@gmail.com", "al.b.er.th.a.nsh.in49@gmail.com", "al.b.er.t.h.a.nshin4.9@gmail.com", "al.b.er.t.h.a.n.shin4.9@gmail.com", "al.b.e.rth.anshi.n.49@gmail.com", "al.b.e.rth.ansh.i.n4.9@gmail.com", "al.b.e.rth.a.nshin.49@gmail.com", "al.b.e.r.than.shin49@gmail.com", "al.b.e.r.t.h.an.s.hin.4.9@gmail.com", "al.b.e.r.t.h.a.ns.hi.n.49@gmail.com", "a.lberthans.h.in.49@gmail.com", "a.lberthan.s.h.i.n.4.9@gmail.com", "a.lberth.a.nsh.i.n4.9@gmail.com", "a.lber.tha.ns.h.in4.9@gmail.com", "a.lber.t.ha.n.s.h.in.4.9@gmail.com", "a.lber.t.h.ansh.i.n49@gmail.com", "a.lber.t.h.an.s.hin4.9@gmail.com", "a.lber.t.h.a.n.shi.n.49@gmail.com", "a.lbe.rth.ans.hi.n49@gmail.com", "a.lbe.rth.a.nsh.in4.9@gmail.com", "a.lbe.rt.hansh.i.n4.9@gmail.com", "a.lbe.rt.hans.hin4.9@gmail.com", "a.lbe.rt.hans.hi.n4.9@gmail.com", "a.lbe.rt.han.sh.in4.9@gmail.com", "a.lbe.r.thansh.in.49@gmail.com", "a.lbe.r.than.s.hin4.9@gmail.com", "a.lbe.r.tha.nsh.in.49@gmail.com", "a.lbe.r.tha.n.sh.in4.9@gmail.com", "a.lbe.r.t.ha.ns.hin4.9@gmail.com", "a.lbe.r.t.ha.n.s.h.i.n.49@gmail.com", "a.lbe.r.t.h.anshi.n4.9@gmail.com", "a.lbe.r.t.h.ansh.in49@gmail.com", "a.lbe.r.t.h.a.ns.h.i.n49@gmail.com", "a.lb.erth.ansh.i.n.49@gmail.com", "a.lb.erth.a.nsh.i.n49@gmail.com", "a.lb.ert.han.shi.n.49@gmail.com", "a.lb.ert.han.sh.in4.9@gmail.com", "a.lb.ert.han.sh.i.n4.9@gmail.com", "a.lb.ert.h.an.sh.i.n49@gmail.com", "a.lb.ert.h.a.ns.h.in.4.9@gmail.com", "a.lb.er.th.a.ns.hin49@gmail.com", "a.lb.er.th.a.n.s.hi.n.49@gmail.com", "a.lb.er.t.han.s.h.in.4.9@gmail.com", "a.lb.er.t.ha.nshin49@gmail.com", "a.lb.er.t.ha.nsh.in49@gmail.com", "a.lb.er.t.ha.nsh.in.4.9@gmail.com", "a.lb.er.t.h.a.n.shin.49@gmail.com", "a.lb.e.rthans.hi.n4.9@gmail.com", "a.lb.e.rthan.shin.49@gmail.com", "a.lb.e.rth.anshin4.9@gmail.com", "a.lb.e.rth.a.n.shi.n49@gmail.com", "a.lb.e.rt.ha.n.s.hi.n.49@gmail.com", "a.lb.e.rt.h.ansh.in4.9@gmail.com", "a.lb.e.rt.h.ansh.in.49@gmail.com", "a.lb.e.rt.h.a.ns.hi.n49@gmail.com", "a.lb.e.r.thansh.i.n49@gmail.com", "a.lb.e.r.th.a.n.sh.i.n.4.9@gmail.com", "a.lb.e.r.t.hanshin.4.9@gmail.com", "a.lb.e.r.t.h.a.ns.h.i.n.49@gmail.com", "a.lb.e.r.t.h.a.n.sh.i.n.4.9@gmail.com", "a.l.bertha.n.shin.49@gmail.com", "a.l.berth.an.sh.in4.9@gmail.com", "a.l.berth.an.s.hin4.9@gmail.com", "a.l.berth.an.s.h.in.4.9@gmail.com", "a.l.bert.h.ans.hi.n.49@gmail.com", "a.l.ber.tha.nsh.i.n49@gmail.com", "a.l.ber.tha.n.shin.4.9@gmail.com", "a.l.ber.th.an.s.h.i.n.49@gmail.com", "a.l.ber.t.hanshi.n.49@gmail.com", "a.l.ber.t.han.s.hi.n.4.9@gmail.com", "a.l.ber.t.han.s.h.i.n.4.9@gmail.com", "a.l.ber.t.h.anshi.n.49@gmail.com", "a.l.be.rt.han.sh.i.n49@gmail.com", "a.l.be.rt.h.a.n.shi.n49@gmail.com", "a.l.be.r.t.han.s.hin4.9@gmail.com", "a.l.b.ertha.nsh.in49@gmail.com", "a.l.b.ertha.nsh.i.n.49@gmail.com", "a.l.b.erth.ansh.i.n4.9@gmail.com", "a.l.b.ert.han.s.hin49@gmail.com", "a.l.b.ert.ha.ns.h.in.4.9@gmail.com", "a.l.b.ert.ha.ns.h.i.n4.9@gmail.com", "a.l.b.ert.ha.n.shin4.9@gmail.com", "a.l.b.ert.h.anshi.n49@gmail.com", "a.l.b.er.tha.n.shi.n4.9@gmail.com", "a.l.b.er.tha.n.sh.i.n.49@gmail.com", "a.l.b.er.tha.n.s.hin49@gmail.com", "a.l.b.er.t.h.a.nshin4.9@gmail.com", "a.l.b.er.t.h.a.ns.hin.4.9@gmail.com", "a.l.b.er.t.h.a.n.s.hi.n.49@gmail.com", "a.l.b.e.rthanshi.n.4.9@gmail.com", "a.l.b.e.rth.an.shin.49@gmail.com", "a.l.b.e.rt.h.an.s.h.i.n49@gmail.com", "a.l.b.e.r.t.hansh.i.n49@gmail.com", "a.l.b.e.r.t.ha.n.sh.i.n.4.9@gmail.com"
];
cr = 0;
testCount = arraylen(tests);
writeoutput("<h2>Testing #testCount# 'diffferent' email addresses</h2>");
for (test in tests){
cr += 1;
writeoutput("<div>[#cr#/#testCount#] #test# = #generateEmailHashCode(test)#</div>");
}
</cfscript>