27

I have generated a wordlist.txt of 11 GB by crunch-3.6. When I try to open the file with Vi or gedit, I run into problems because of the file size. How can I view this file?

Nathan Smith
  • 257
  • 1
  • 4
  • 11
vais sai
  • 273
  • 1
  • 3
  • 6
  • 8
    I don't think it's a duplicate. Although its closely related and some answers might be helpful, this one is about viewing and not editing. The accepted answer is not helpful for viewing. – pLumo Nov 09 '17 at 09:07
  • 1
    I agree. Viewing a file and editing a file are not the same thing, and the distinction only becomes more important when the file is very large. – Eliah Kagan Nov 14 '17 at 15:58

2 Answers2

48

Don't use a text editor for viewing text.

There are better tools:

View files with less (Scroll with Space, End, Home, PageUp, PageDown; Search with "/something" ; Leave with q).

From less manual:

Less does not have to read the entire input file before starting, so with large input files it starts up faster than text editors like vi (1).

Usage:

less wordlist.txt

Consider the use of less -n:

-n or --line-numbers:

Suppresses line numbers. The default (to use line numbers) may cause less to run more slowly in some cases, especially with a very large input file. Suppressing line numbers with the -n option will avoid this problem.

(thanks for suggesting -n option @pipe)

Use grep to get only the lines you're interested in:

# Show all Lines beginning with A:
grep "^A:"  wordlist.txt

# Show all Lines ending with x and use less for better viewing
grep "x$"  wordlist.txt | less

Use head or tail to get the first or last n lines

head wordlist.txt
tail -n 200 wordlist.txt

For editing text, refer to this question.

pLumo
  • 26,204
  • 2
  • 57
  • 87
  • 4
    Note that `more` will load the whole file in memory, unlike `less`. If it's that big, I'd advise against using it. The best option would definitely be to use `grep`. – Nepho Nov 09 '17 at 13:06
  • I removed `more` as it does not really add any benefit compared to less – pLumo Nov 09 '17 at 13:17
  • 1
    My comment might actually be false. I started checking both `more` and `less` source code, and both seem to use either `open` or `fopen`, which doesn't change anything regarding the way the file is loaded, as far as I know – Nepho Nov 09 '17 at 13:17
  • 2
    Agreed. `more` is a very dated utility, if `less` is available I don't think of any reason why you'd use `more`. – Nepho Nov 09 '17 at 13:18
  • 1
    @Nepho, `more` has one huge advantage over `less`: it doesn't support the `LESSOPEN` input preprocessor. If you're trying to view the raw text of a man page or something, it's much faster to type `more my_docs.man` than to dig through the `less` documentation to figure out how to suppress the preprocessor. – Mark Nov 09 '17 at 23:34
  • 1
    I suggest invoking _less_ using something like `less -nS` for huge files, to speed things up a little. – pipe Nov 10 '17 at 12:09
  • thanks@ pipe. I added -n, but didn't find information on why -S should provide better performance. – pLumo Nov 10 '17 at 12:22
11

Often, just "grep" is enough to find what you need.

If you need more "context" around a particular line, then use "grep -n" to find the line numbers of the lines of interest, then use sed to print out a "chunk" of the file around that line: 

$ grep -n 'word' file 
123:A line with with word in it 

$ sed -n '120,125p' file 
A line 
Another line
The line before
A line with with word in it 
The line after
Something else
Phill W.
  • 211
  • 1
  • 2