How can I find non-ASCII characters in text files?

Question

Is there a tool that can scan a small text file and look for any character not in the simple ASCII character set?

A simple Java or Groovy script would also do.

Answer 1

Well, it's still here after an hour, so I may as well answer it. Here's a simple filter that prints only non-ASCII characters from its input, and gives exit code 0 if there weren't any and 1 if there were. Reads from standard input only.

#include <stdio.h>
#include <ctype.h>

int main(void)
{
    int c, flag = 0;

    while ((c = getchar()) != EOF)
        if (!isascii(c)) {
            putchar(c);
            flag = 1;
        }

    return flag;
}

Answer 2

Just run $JDK_HOME/bin/native2ascii on the text file and search for "\u" in the output file. I'm assuming you want to find it so you can escape it anyway and this will save you a step. ;)

Answer 3

I have no idea if this is legit, casting each char to an int and using a catch to identify things that fail. I'm also too lazy to write this in java so have some Groovy

def chars = ['Ã', 'a', 'Â', 'ç', 'x', 'o', 'Ð'];

chars.each{
    try{ def asciiInt = (int) it }
    catch(Exception e){ print it + " "}
}

==> Ã Â ç Ð

Answer 4

In Java (assuming the string is specified as the first command-line argument:

public class Main
{
    public static void main(String[] args)
    {
        String stringToSearch = args[0];
        int len = stringToSearch.length();
        for (int i = 0; i < len; i++)
        {
            char ch = stringToSearch.charAt(i);
            if (ch >= 128) // non-ascii
            {
                System.out.print(ch + " ");
            }
        }
        System.out.println();
    }
}

To make this your own, replace stringToSearch with whatever you need.

Answer 5

A simple groovy example:

def str = [ "this doesn't have any unicode", "this one does ±ÁΘ·€ÔÅ" ]

str.each {
    if( it ==~ /[\x00-\x7F]*/ ) {
        println "all ascii: $it"
    } else {
        println "NOT ASCII: $it"
    }
}

It's as simple as this bit here: it ==~ /[\x00-\x7F]*/

Edit: I forgot to include a version for files. Oops:

def text = new File(args[0]).text
if( text ==~ /[\x00-\x7F]*/ ) {
    println "${args[0]} is only ASCII"
    System.exit(0)
} else {
    println "${args[0]} contains non-ASCII characters"
    System.exit(-1)
}

That version can be used as a command line script, and includes an exit status so it can be chained.