23

I am only using the 128 character set defined in the original ANSI standard.

But as a whole how are the files implemented differently.

I am not concerned with the display, i.e. if a tab is displayed with 6 or 8 characters but the actual internal representation in memory

One difference I've heard is the use of \r\n (Windows) vs. \n for line termination (Linux).

user1699905
  • 1
  • 5
  • I think the byte order mark is killing my #!(first line) in my php files I transferred over from windows to linux. The whole file works but it can not find the interpreter as it should. If I specefically make sure to encode in ANSI by selecting the encoding method in notepad is it true ASCII or does Windows do something else –  Jun 09 '11 at 23:52
  • See if you have bomstrip on your Gnu/Linux box. It is part of Debian (and at least some others), but may need installing. It is needed because Microsoft erroneously adds a BOM to the start of utf-8 files. – ctrl-alt-delor Mar 18 '18 at 22:24

5 Answers5

24

"Unicode" on Windows is UTF-16LE, and each character is 2 or 4 bytes. Linux uses UTF-8, and each character is between 1 and 4 bytes.

"The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)"

Ignacio Vazquez-Abrams
  • 111,361
  • 10
  • 201
  • 247
20

Line breaks

Windows uses CRLF (\r\n, 0D 0A) line endings while Unix just uses LF (\n, 0A).

Character Encoding

Most modern (i.e., since 2004 or so) Unix-like systems make UTF-8 the default character encoding.

Windows, however, lacks native support for UTF-8. It internally works in UTF-16, and assumes that char-based strings are in a legacy code page. Fortunately, Notepad is capable of reading UTF-8 files; unfortunately, "ANSI" encoding is still the default.

Problematic Special Characters

U+001A SUBSTITUTE

Windows (rarely) uses Ctrl+Z as an end-of-file character. For example, if you type a file at the command prompt, it will be truncated at the first 1A byte.

On Unix, Ctrl+Z is nothing special.

U+FEFF ZERO WITH NO-BREAK SPACE (Byte-Order Mark)

On Windows, UTF-8 files often start with a "byte order mark" EF BB BF to distinguish them from ANSI files.

On Linux, the BOM is discouraged because it breaks things like shebang lines in shell scripts. Plus, it'd be pointless to have a UTF-8 signature when UTF-8 is the default encoding anyway.

Giacomo1968
  • 53,069
  • 19
  • 162
  • 212
user46971
  • 1,289
  • 1
  • 10
  • 11
  • 1
    Ctrl-Z works on windows just like Ctrl-D ( or whatever character you have bound to EOF with `stty` ) does on Linux: the console driver translates it to end of file. The literal character does not appear in the input stream; it just causes read() to return 0. – psusi Jun 08 '11 at 01:22
  • I think the byte order mark is killing my #!(first line) in my php files I transferred over from windows to linux. The whole file works but it can not find the interpreter as it should. If I specefically make sure to encode in ANSI by selecting the encoding method in notepad is it true ASCII or does Windows do something else? –  Jun 09 '11 at 23:52
  • 3
    It worth mentioning that the pseudo-term “ANSI code page”, although still appears in such programs as Notepad, is utterly a misnomer, and Microsoft admitted this long ago. See http://en.wikipedia.org/wiki/Windows_code_page for details. – Incnis Mrsi Aug 18 '15 at 12:17
  • utf-8 does not have a BOM, but MS-Windows inserts one. Making it not true utf-8. One of the rules of utf-8 is that any file that could be represented in ascii, is bit for bit identical in utf-8. Also you can start reading utf-8 at any point in the stream. – ctrl-alt-delor Mar 18 '18 at 22:22
  • As of 2020 or so, it is no longer true that Windows lacks native support for UTF-8; there is an option to select UTF-8 as the system's "ANSI" codepage. (It often breaks legacy "not quite Unicode" programs in surprising ways, though.) – u1686_grawity Feb 26 '23 at 11:08
  • (Though even besides that, it hasn't been completely true for a while – the built-in Windows functions for codepage conversion like "MultiByteToWideChar()" have had CP_UTF8 as an option for much longer, which is how Notepad loads and saves UTF-8 files; it's the ability to directly pass UTF-8 filenames to the system without converting them to UTF-16 first that's new.) – u1686_grawity Feb 26 '23 at 11:10
  • @user46971 there is an accidental critical typo, “ZERO **WITH** NO-BREAK SPACE” should read “ZERO **WIDTH** NO-BREAK SPACE”. I’m not allowed to make single-character edits to other people’s posts and I will not concoct spurious changes just to bypass the rules. Would you mind correcting the typo? Thank you. – Euro Micelli Feb 27 '23 at 06:53
4

One difference I've hear is the use of \r\n (Windows) vs. \n for line breaks (Linux).

Yes. Most UNIX text editors will handle this automatically, Windows programmers editors may handle this, general text editors (base Notepad) will not.

Windows seems to also need the EOF (Ctrl-Z) as END OF FILE in some contexts, whereas you'll probably never see it on UNIX.

Remember that MacOS X is now UNIX underneath, so it uses UNIX line endings. Though before OS X (MacOS 9 and below) it had its own ending (\r)

EDIT: in other format CR and LF:

  • \n is ASCII 0x0A, Line Feed (LF)
  • \r is ASCII 0x0D, Carriage return (CR)
Rich Homolka
  • 31,057
  • 6
  • 55
  • 80
  • Where are \r\n and \n in the ASCII character set? http://en.wikipedia.org/wiki/File:ASCII_Code_Chart.svg –  Jun 07 '11 at 21:31
  • 2
    @Chris \n is ASCII 0x0A, Line Feed. \r is ASCII 0x0D, Carriage return – Rich Homolka Jun 07 '11 at 21:36
  • @Rich What about EOF? Is this an ANSI character? –  Jun 07 '11 at 21:39
  • it appears to be the ASCII character SUB –  Jun 07 '11 at 21:45
  • No, EOF is not an ASCII character. EOF is usually signaled out of band ( by read returning 0 and not filling the buffer with anything ), but some old DOS routines used 0xFF/-1 to signal EOF in band. – psusi Jun 08 '11 at 01:20
  • to user. and @psusi https://en.wikipedia.org/wiki/Control-Z "In some operating systems [e.g. DOS and cmd/win32 console] , Control+Z is used to signal an end-of-file, and thus known as the EOF character" <-- And also, if you try from cmd / win32 console.. copy con a.a and you can use ctrl-z to signal you're finished.. it takes it like an end of file.. though indeed ^Z (26th letter of the alphabet is z) so ascii 26. Ctrl-Z looks like SUB . I see this listing http://www.asciitable.com/ doesn't call it EOF and maybe most don't maybe none do. – barlop Dec 07 '15 at 17:30
  • 2
    @barlop, the terminal translates the keystroke ( it is normally ctrl-d on unix systems ) into EOF, unless this control key has been disabled. The application reads an EOF rather than the actual key you hit. That is to say, `read()` returns zero bytes instead of any specific character. – psusi Dec 07 '15 at 23:55
  • @psusi What ascii char is it returning to the program? EOF isn't on asciitable.com – barlop Dec 08 '15 at 03:01
  • 1
    @barlop, that is what I have been saying: it does not return *any* character. read() returns the number of bytes that it stored in your buffer. On EOF, it simply gives you zero bytes. That is the signal that you have reached the end of the file, and that there is nothing more to read. – psusi Dec 09 '15 at 02:38
  • Windows and DOS never need 0x1A to terminate the file, as per your linked wikipedia article – phuclv Nov 22 '16 at 08:31
  • Dos was based on CP/M, CP/M file sizes were in blocks. So a text file would end with `control z`, to indicate it ended before the end of the block. It sometimes lives on on Microsofts Windows, but was never needed even in DOS. – ctrl-alt-delor Mar 18 '18 at 22:28
  • @ctrl-alt-delor: Text files were often exchanged between computers using XMODEM, which would pad file sizes to the next multiple of 128 bytes, without regard for the underlying file system. When XMODEM programs sent files that were not a multiple of 128 bytes, they would pad with one or more 0x1A bytes, but receiving programs would retain all of the padding bytes since they would have no way of knowing whether the original file was a binary file that happened to contain those byte values. – supercat Feb 27 '23 at 17:09
1

What Unicode encoding is used is not OS based.

Even Windows notepad.exe has options listed- (i'll put in brackets what notepad means by that) ANSI(not unicode), Unicode(notepad means Unicode LE), Unicode Big Endian(BE), UTF-8

ANSI isn't unicode it involves a very limited number of characters so lets put that aside.

But see even notepad can do LE, or BE, or UTF-8

And notepad aside, UTF-8 can be with or without a BOM.

And I use Windows with Cygwin though Windows ports may well do \r\n even when you specify \n Have seen sed do that.

There is no one rule of what Unicode encoding a particular OS uses. It wouldn't be a very flexible OS if there was.

To really see the differences know the Software, what Encoding a piece of software uses or offers.

Get Cygwin and xxd, and/or a hex editor and look at what is really inside the file. Use the 'file' command to help identify a file. Then you actually see what UTF 16bit LE is. What UTF 16bit BE is. What UTF-8 is (and UTF-8 can be with or without a BOM).

Sometimes you can tell notepad to save as unicode(by which notepad means unicode 16 bit little endian), and it won't. But choose a unicode font like arial unicode, and copy in some unicode characters from charmap and it will.. And a good way to see what notepad or whatever software is doing, is by looking at the hex of a file

C:\asdf>notepad.exe a.a

C:\asdf>file a.a
a.a; Little-endian UTF-16 Unicode text, with no line terminators

C:\asdf>type a.a
aaa慡ൡ <-- though displayed aaa followed by some boxes in my cmd window
C:\asdf>

C:\asdf>xxd a.a
0000000: fffe 6100 6100 6100 6161 610d            ..a.a.a.aaa.

C:\asdf>

^^ The portion of the byte that stores the 61 is the lower value portion which with LE is stored first.

The dd command (a *nix command I run from cygwin within windows) can switch it 

C:\asdf>xxd -p a.a
fffe6100610061006161610d

C:\asdf>file a.a
a.a; Little-endian UTF-16 Unicode text, with no line terminators

C:\asdf>dd if=a.a conv=swab of=a.a2
0+1 records in
0+1 records out
12 bytes (12 B) copied, 0 seconds, Infinity B/s

C:\asdf>type a.a2
a  a a aaa
C:\asdf>xxd -p a.a2
feff00610061006161610d61

C:\asdf>file a.a2
a.a2; Big-endian UTF-16 Unicode text, with no line terminators

C:\asdf>

And notepad itself can save as UTF-16 Big Endian or UTF-16 Little Endian or UTF-8

enter image description here

If you're a technical person or even just a notepad user, you're not bound to one encoding because of your OS!

I suppose UTF-8 makes more sense than UTF-16, UTF-16 would use 16 bits even for characters that should only need 8 bits. Also though, bear in mind that charmap shows the UTF-16 code.

Sublime(A windows text editor) saves unicode as UTF-8 by default.

I use Windows and sometimes unicode, and i'm using UTF-8 mostly.

And as Windows is that technically flexible, linux is at least as technically flexible!

barlop
  • 23,380
  • 43
  • 145
  • 225
  • Did you write the commands `file` and `type` inside the Cygwin prompt? – Vesnog Aug 15 '16 at 22:45
  • `xxd` and `type` commands are missing in standard Cygwin installation I presume. Apart from that I want to reproduce your results. – Vesnog Aug 15 '16 at 23:48
  • 1
    @Vesnog `type` is a standard command built into cmd.exe `xxd` is most likely not installed with cygwin by default, but when you install cygwin or after it, if you start cygwin setup you get a long list of commands you can install for use in cygwin, and just type xxd into the cygwin setup search box and it comes up. xxd is also available from after the installation of vim7 so you could obtain it from there too. – barlop Aug 16 '16 at 02:05
  • 1
    @Vesnog you can run cygwin commands inside cygwin or outside cygwin. If you run them outside cygwin then add `c:\cygwin\bin` (if that's where cygwin's bin subdirectory is), into your path. Also any internal cmd command like 'type' or 'dir', or any external exe like calc.exe(windows calculator) can be run/launched from within cygwin. Pretty much anything that can be run from cygwin can be run from cmd and vice versa. If you wanted to use bash then use cygwin and if you ran into issues with single vs double quotes then run cygwin commands within cygwin and cmd ones within cmd. – barlop Aug 16 '16 at 02:07
  • I was able to reproduce the examples, yet I have a question. How does `dd` command change the encoding by swapping bits and is the encoding always hardcoded into the file and can be read in a hexdump such as the one obtained by `xxd`? – Vesnog Aug 17 '16 at 23:01
  • @Vesnog it's swapping adjacent bytes but not the bits within each byte. You kind of answer part of your own question. You ask "How does dd command change the encoding by swapping bits" Indeed, dd changes the encoding by swapping each pair of adjacent bytes. So if one byte is 00 and the next byte is 61 then it swaps that pair. Each set of 2 bytes gets rearranged. The encoding isn't hardcoded, but order of every single bit and every byte, is "hard coded".The dd command is reading it and writing a new file. xxd is showing what is truly in the file, and you can figure out the encoding from that – barlop Aug 17 '16 at 23:35
  • 1
    @Vesnog xxd can write a file too, e.g. `echo 61|xxd -r -p>a.a` then try `type a.a` So you can actually get a byte dump with xxd -p, rearrange or modify the bytes then feed it into xxd -r -p and get a new different file with a different encoding or different data based on the old data. The "file" command is figuring out the encoding, based on the bytes. – barlop Aug 17 '16 at 23:44
0

Linux uses UTF-8, and each character is between 1 and 6 bytes,not between 1 and 4 bytes.

U00000000 - U0000007F: 0xxxxxxx
U00000080 - U000007FF: 110xxxxx 10xxxxxx
U00000800 - U0000FFFF: 1110xxxx 10xxxxxx 10xxxxxx
U00010000 - U001FFFFF: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
U00200000 - U03FFFFFF: 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
U04000000 - U7FFFFFFF: 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
showkey
  • 89
  • 4
  • 16
  • 40