Spam Prevention

The crate.spam package bundles a set of independent, composable helpers for filtering bots and abusive traffic. Each module does one thing and can be used on its own — mix and match to match your threat model.

Module	Purpose
crate.spam.antispam	Client IP extraction, simple bot heuristics, tarpit, offense tracking
crate.spam.rateLimit	Fixed-window per-key rate limiter, plus a two-window traffic gate
crate.spam.banChecker	Admin-managed ban list for emails, domains, and IPs
crate.spam.domainCheck	Cached DNS existence check for a domain
crate.spam.dnsbl	DNSBL (Spamhaus, SpamCop, …) lookup
crate.spam.stopForumSpam	StopForumSpam blocklist lookup
crate.auth.challenges.altcha	ALTCHA proof-of-work challenge — see ALTCHA Module
crate.spam.spamfilter	Heuristic classification of form fields — see Spam Filter

All caching helpers use TimedCache, so lookups only hit the network (or DNS resolver) on cache misses.

crate.spam.antispam

Helpers that run at the HTTP edge: figure out who the client actually is, decide whether it smells like a bot, optionally stall the caller, and remember repeat offenders.

Import

import crate.spam.antispam;

extractClientIp

Returns the real client IP, honoring X-Forwarded-For only when the immediate peer is in the trusted-proxy list (or is on a private network when trustPrivateNetworks is true). Untrusted peers never get to inject headers.

string extractClientIp(HTTPServerRequest req, ProxyConfig proxy);
string extractClientIp(HTTPServerRequest req, const string[] trustedProxies = [],
                      bool trustPrivateNetworks = false);

auto ip = extractClientIp(req, ProxyConfig(["10.0.0.1"], true));

ProxyConfig

Field	Type	Default	Description
trustedProxies	const(string[])	[]	Peer addresses whose X-Forwarded-For we honor
trustPrivateNetworks	bool	false	Also trust any peer on RFC1918 / ULA / loopback

isPrivateAddress

Returns true for loopback (127.0.0.0/8, ::1), RFC1918 IPv4 (10/8, 172.16/12, 192.168/16), IPv6 ULA (fc00::/7), and link-local IPv6 (fe80::/10). Useful outside the proxy flow too.

bool isPrivateAddress(string ip);

isBotRequest

Cheap heuristic: flags the request as a bot if it’s missing Accept-Language, missing User-Agent, or the UA matches any of defaultBotUaPatterns (python-requests, curl/, go-http-client, okhttp). Pass your own patterns to override.

bool isBotRequest(HTTPServerRequest req, const string[] userAgentPatterns = defaultBotUaPatterns);

tarpit

Sleeps for config.base + random(0, jitterSeconds) and then writes a 204 No Content. Used to waste a detected bot’s time without signaling the block.

void tarpit(HTTPServerResponse res, TarpitConfig config);

TarpitConfig fields: base (default 25.seconds), jitterSeconds (default 5).

OffenseTracker

Thin wrapper over a TimedCache!bool. Mark IPs that triggered any of your spam rules, then check future requests against the same cache to decide whether to tarpit or outright block.

auto tracker = new OffenseTracker(new TimedCache!bool(null, 1.hours, 5000));

if(tracker.isKnown(ip)) {
    tarpit(res, TarpitConfig());
    return;
}

if(spamFilter.classify(fields, config).isSpam) {
    tracker.mark(ip);
}

matchesBotUserAgent / parseForwardedForFirstHop

Lower-level helpers used by isBotRequest and extractClientIp. Exposed so you can reuse them in custom middleware.

crate.spam.rateLimit

Import

import crate.spam.rateLimit;

RateLimiter

Fixed-window limiter keyed by arbitrary string (IP, user id, endpoint, …). Each key gets limit allowances per window; the count resets on the first call after the window elapses.

auto rl = new RateLimiter(10, 1.minutes);

if(!rl.allow(clientIp)) {
    res.statusCode = 429;
    return;
}

Constructor arg	Default	Description
limit	required	Max calls per window
window	required	Duration of the window
maxEntries	100_000	Upper bound before a reap pass cleans stale keys

Empty keys always pass (fail-open — useful when extractClientIp can’t determine a value).

allow counts the request and returns false once the key is over the limit. reached(key) counts the request too but answers the opposite question — “has this key hit the threshold?” — and treats an empty key as reached, so an unkeyable client cannot slip past a gated check.

TrafficGate

A burst gate over two windows. A key is busy once it crosses either threshold, which catches both a short spike and a slow grind:

auto gate = new TrafficGate(
    new RateLimiter(3, 1.minutes),
    new RateLimiter(60, 1.hours));

if(gate.busy(clientIp)) {
    // serve a challenge, tarpit, or degrade the response
}

busy always records the request on both limiters, so neither window loses a count when the other has already tripped.

crate.spam.banChecker

An admin-managed deny list stored in a crate, checked by email, email domain, or IP.

Import

import crate.spam.banChecker;

BanChecker

auto checker = new BanChecker(bannedIdentityCrate);

if(checker.isBanned(email, clientIp)) {
    res.statusCode = 403;
    return;
}

The backing crate holds documents with a value field; isBanned matches the lowercased, trimmed email, the domain extracted from it, and the trimmed IP against that field in one query. Blank candidates are dropped, and a request with nothing to match on is not banned.

enforceNotBanned(email, ip) is the throwing form — it raises ForbiddenException with the message This account has been banned.

crate.spam.domainCheck

Import

import crate.spam.domainCheck;

DomainChecker

Checks whether a domain has an A/AAAA record. Positive and negative results are cached; transient DNS failures fail open (return true and are not cached) so a flaky resolver doesn’t lock out legitimate domains for the whole TTL.

auto checker = new DomainChecker(10.minutes, 1000, 2.seconds);

if(!checker.exists(extractDomain(email))) {
    return badRequest("Unknown email domain");
}

Constructor arg	Default	Description
ttl	10.minutes	Cache lifetime for positive/negative results
maxSize	1000	Max cached entries
dnsTimeout	2.seconds	Per-lookup DNS timeout

A second constructor accepts a bool delegate(string) lookup for testing.

Result semantics:

• Domain resolves → true (cached)
• Definitive NXDOMAIN / “host not found” → false (cached)
• Timeout / network / other error → true (not cached)

crate.spam.dnsbl

Import

import crate.spam.dnsbl;

DnsblChecker

Classic DNSBL: reverse the IPv4 octets, append a blocklist zone (zen.spamhaus.org, bl.spamcop.net, …), try to resolve. A successful resolution means “listed.” Returns on the first listed blocklist. IPv6 is not supported by most free DNSBLs and is treated as not-listed.

auto dnsbl = new DnsblChecker(["zen.spamhaus.org", "bl.spamcop.net"]);

if(dnsbl.isListed(clientIp)) {
    tarpit(res, TarpitConfig());
    return;
}

Constructor arg	Default	Description
blocklists	["zen.spamhaus.org"]	DNSBL zones to query
ttl	1.hours	Cache lifetime
maxSize	10_000	Max cached entries
dnsTimeout	2.seconds	Per-lookup DNS timeout

reverseIpv4

Returns "4.3.2.1" for "1.2.3.4", or "" if the input isn’t a well-formed IPv4 string.

crate.spam.stopForumSpam

Import

import crate.spam.stopForumSpam;

StopForumSpamChecker

Queries the free StopForumSpam HTTP API with an IP and optional email. Results are cached (default 6 hours) to stay within the free-tier rate limit. Fails open — network errors, non-200 responses, and parse failures all return “not listed.”

auto sfs = new StopForumSpamChecker();

if(sfs.isSpam(clientIp, submittedEmail)) {
    return forbidden();
}

Constructor arg	Default	Description
apiUrl	"https://api.stopforumspam.org/api"	API endpoint
ttl	6.hours	Cache lifetime
maxSize	10_000	Max cached entries

A second constructor accepts a string delegate(string url) fetcher for testing.

Putting It Together

A middleware that layers several checks — cheap ones first, expensive ones last:

import crate.spam.antispam;
import crate.spam.rateLimit;
import crate.spam.domainCheck;
import crate.spam.dnsbl;
import crate.spam.stopForumSpam;

auto proxy   = ProxyConfig(["10.0.0.1"], true);
auto rl      = new RateLimiter(20, 1.minutes);
auto domains = new DomainChecker();
auto dnsbl   = new DnsblChecker();
auto sfs     = new StopForumSpamChecker();
auto tracker = new OffenseTracker(new TimedCache!bool(null, 1.hours, 5000));

void guard(HTTPServerRequest req, HTTPServerResponse res) {
    auto ip = extractClientIp(req, proxy);

    if(tracker.isKnown(ip) || isBotRequest(req)) {
        tarpit(res, TarpitConfig());
        return;
    }

    if(!rl.allow(ip)) {
        res.statusCode = 429;
        return;
    }

    if(dnsbl.isListed(ip) || sfs.isSpam(ip)) {
        tracker.mark(ip);
        tarpit(res, TarpitConfig());
        return;
    }

    // proceed to the real handler
}