Chapter 3 Study Guide

CNT-2311-Chapter 3

Exploring Linux Filesystems

Objectives

• Comparison of Windows and Linux filesystems
• Navigate the directory structure using relative and absolute pathnames
• Types of files
• View filenames and file types
• Use shell wildcards
• Display contents of files
• Search files for regular expressions using grep
• Use vi to modify text files
• Alternatives to vi
• File Hierarchy

Comparison of Windows v. Linux filesystems

Windows File System Attributes

• Directory structure - volume based, drive letters assigned at boot
• Can not natievly read linux file system
• c:\ is root directory
• "\" is the file seperator = c:\logman.exe
• file names are not case sensitive
• have hidden files

Linux File System Attributes

• unified directory structure - single rood directory
• every directory is a sub-directory of /
• Drives and directories must be mounted to be seen
• Can read FAT, NTFS, EXT2,3,4 SQUASHFS, lsofs, sysfs, procfs file systems
• "/" is the file separator = /home/randy/documents/test.txt
• file names are case sensitive
• hidden files start with "."

Navigate the directory structure using relative and absolute pathnames 

start in home dir

 use ~ to reference home directories

present dir = pwd

/home/randy/

change dir = cd

cd /etc - change directory to /etc

relative path name = path to the file in relation to the current working directory

".." references the next higher up dir

tab-completion!

Supported File Systems

 Linux supports numerous file system types Ext2: This is like UNIX file system. It has the concepts of blocks, inodes and directories. Ext3: It is ext2 filesystem enhanced with journaling capabilities. Journaling allows fast file system recovery. Supports POSIX ACL (Access Control Lists). Isofs (iso9660): Used by CDROM file system. Sysfs: It is a ram-based filesystem initially based on ramfs. It is use to exporting kernel objects so that end user can use it easily. Procfs: The proc file system acts as an interface to internal data structures in the kernel. It can be used to obtain information about the system and to change certain kernel parameters at runtime using sysctl command. For example you can find out cpuinfo with following command:

Hierarchical File system

Often referred to as the "FHS" At the very top or the beginning is your / (forward slash) which represents your root directory. 1. / – Root Every single file and directory starts from the root directory. Only root user has write privilege under this directory. Please note that /root is root user’s home directory, which is not same as /.

2. /bin – User Binaries Contains binary executables. Common linux commands you need to use in single-user modes are located under this directory. Commands used by all the users of the system are located here. For example: ps, ls, ping, grep, cp.

3. /sbin – System Binaries Just like /bin, /sbin also contains binary executables. But, the linux commands located under this directory are used typically by system aministrator, for system maintenance purpose. For example: iptables, reboot, fdisk, ifconfig, swapon

4. /etc – Configuration Files Contains configuration files required by all programs. This also contains startup and shutdown shell scripts used to start/stop individual programs. For example: /etc/resolv.conf, /etc/logrotate.conf

5. /dev – Device Files Contains device files. These include terminal devices, usb, or any device attached to the system. For example: /dev/tty1, /dev/usbmon0

6. /proc – Process Information Contains information about system process. This is a pseudo filesystem contains information about running process. For example: /proc/{pid} directory contains information about the process with that particular pid. This is a virtual filesystem with text information about system resources. For example: /proc/uptime

7. /var – Variable Files var stands for variable files. Content of the files that are expected to grow can be found under this directory. This includes — system log files (/var/log); packages and database files (/var/lib); emails (/var/mail); print queues (/var/spool); lock files (/var/lock); temp files needed across reboots (/var/tmp);

8. /tmp – Temporary Files Directory that contains temporary files created by system and users. Files under this directory are deleted when system is rebooted.

9. /usr – User Programs Contains binaries, libraries, documentation, and source-code for second level programs. /usr/bin contains binary files for user programs. If you can’t find a user binary under /bin, look under /usr/bin. For example: at, awk, cc, less, scp /usr/sbin contains binary files for system administrators. If you can’t find a system binary under /sbin, look under /usr/sbin. For example: atd, cron, sshd, useradd, userdel /usr/lib contains libraries for /usr/bin and /usr/sbin /usr/local contains users programs that you install from source. For example, when you install apache from source, it goes under /usr/local/apache2 10. /home – Home Directories

Home directories for all users to store their personal files. For example: /home/john, /home/nikita

11. /boot – Boot Loader Files Contains boot loader related files. Kernel initrd, vmlinux, grub files are located under /boot For example: initrd.img-2.6.32-24-generic, vmlinuz-2.6.32-24-generic

12. /lib – System Libraries Contains library files that supports the binaries located under /bin and /sbin Library filenames are either ld* or lib*.so.* For example: ld-2.11.1.so, libncurses.so.5.7

13. /opt – Optional add-on Applications opt stands for optional. Contains add-on applications from individual vendors. add-on applications should be installed under either /opt/ or /opt/ sub-directory.

14. /mnt – Mount Directory Temporary mount directory where sysadmins can mount filesystems.

15. /media – Removable Media Devices

Temporary mount directory for removable devices. For examples, /media/cdrom for CD-ROM; /media/floppy for floppy drives; /media/cdrecorder for CD writer

16. /srv – Service Data srv stands for service. Contains server specific services related data.

Types of files used by Linux

Text, Binary, Executable

Directory = still just a file

Linked = reference other files, shortcut

Special device files = reference hard disks or ports, contained in /dev

Named pipes = communication between process in memory

Created using mkfifo or mknod One process is a reader other is a writer Sockets = named pipes between remote computers Filenames Filename facts 255 max characters alphanumeric, _ -  .   may or may not have extension to denote type Lots of different extensions 

Links for Additional Info on File Extension

• http://learn.clemsonlinux.org/wiki/File_extensions 
• http://www.debianhelp.co.uk/fileext.htm 
• http://filext.com/alphalist.php?extstart=%5EL  (not limited to linux) 

Commands to view filenames and file types

p*wd - Show current directory

• pwd - /home/randy

• ls -ltr   - Show listing and give color codes to information

listing including file permissions, ownership and date/time stamp.

• Executable files: Green

• Normal file : Normal

• Directory: Blue

• Symbolic link : Cyan

• Pipe: Yellow

• Socket: Magenta

• Block device driver: Bold yellow foreground, with black background

• Character device driver: Bold yellow foreground, with black background

• Orphaned syminks : Blinking Bold white with red background

• Missing links ( - and the files they point to) : Blinking Bold white with red background

• Archives or compressed : Red (.tar, .gz, .zip, .rpm

• Image files : Magenta (.jpg, gif, bmp, png, tif)

View filenames and file types

• use ls to list file and directories

* ls /etc/bob = command argument * Different file types are color coded * ls = pwd

•  ls -F = gives file type, special character is listed on the left of the name = Metacharacters=

• @ = linked file

• = exe

• / = subdirectory

• = is a socket

• | = named pipe

Other files do not have special character appended to them - they can be anything not listed above

• ls -l = ls long listing

All ls Options

* -a =Lists all filenames
* -A = Lists most filenames - excludes hidden and special files
* -C = Lists filenames in column format
* --color=n =Lists filenames without color
* -d =Lists directory names instead of their contents
* -f = Lists all filenames without sorting
* -F = Lists filenames classified by file type
* --full-time = Lists filenames in long format and displays the full modification time
* -l = Lists filenames in long format
* -lh = Lists filenames in long format in easy-to-read file sizes
* -lG, -l, -o  = Lists filenames in long format but omits group info
* -r = Lists filenames in reverse sorted order
* -R = Lists filenames in the specified directory and all subdirectories
* -s = Lists filenames with file sizes in KB
* -S = Lists filenames by size
* -t = Lists filenames sorted by modification time
* -U = Lists filenames without sorting
* -x = Lists filenames in rows 

When listing files/directories a special character on the left indicates what type of file it is

• d = dir
• L = linked
• b & c = special device files
• n = named pipe
• s = socket
• - = other - txt, bin

• file command = Gives file analysis info, type

Other Attributes listing Options

• Perms
• Hard link count
• Owner, group,
• Size
• Mod. Time

Display contents of files

• vi - editing a file
• emas - alternative to vi
• cat - concatenation - similar to "type" in DOS/Windows.
• tac - reverse concatenation - types file in reverse
• head - displays the top ten lines of a file
• tail - displays the bottom ten lines of a file
• more - displays file page by page
• less - similar to more but allow line by line manipulation

Display contents of Binary files

• strings = searches binary file for text
• od = displays file in octal format
• od -x = displays contents in hex

Key Terms

  The tab completion feature is something your really going to like, it makes life much easier for you, rather than typing every single command you type the first character or two and hit the tab key and it finishes it for you.

• ~ metacharacter = represent user's home dir
• Absolute pathname - full path to a file or starting from /
• Binary data file - contains machine language (1 or 0)
• cat - display file contents
• cd - change directory
• Command mode - vi mode for text editing
• Concatenation - joining of text
• egrep = grep -E
• emacas = ediotr macros editor, think of as word pad
• fgrep = grep -F
• Gedit = GUI text editor
• Head - displays first 10 lines, can specify # of lines
• Insert mode - vi mode only allows text into file, no other options
• less command - display file pg. by pg. cursor navigation
• Linked file - file that represents another file
• ll command = ls -l
• more command = displays file pg. by pg. & line by line
• Named pipe file - temp connection that sends info from 1 cmd or process in memory to another, *can represent a file
• Nano editor - terminal text editor, uses ctrl keys for functions and navigation
• od command - displays file contents in octal format
• Parent dir - one dir level up
• Regular expressions - special metacharacters
• Relative pathname - path to a folder relative to the pwd
• Socket file = named pipe between 2 computers
• Special device file - files to identify hardware
• Strings command - search & display text in a bin file
• Tab-completion - press tab to complete path in the shell
• tac command = cat backwards
• Tail cmd = display last 10 lines
• Text tools = program to create, modify, & search txt files
• Wildcard metacharacters = used to match certain characters 

Key Commands

We have some of the key commands listed alphabetically here but the cat command is much like the tac command where cat is used to display the entire contents of a text file the tac command begins with the last line of a file and ends with the the first line of the file. (Just like the word cat backwards is tac)   The " global regular expression print" or grep command searches files for patterns of characters using regular expressions. The egrep & fgrep are other variants of grep that we will talk about later. The print working directory is a useful command to see where you are in a current directory in the tree. And ls command is another useful used for listing files in a given directory.

*cat = used to create, display, copy, redirect output, and combine text file

Syntax: cat filename cat options filename cat file1 file2 cat file1 file2 > newcombinedfile

Example: read or read the contents of files, enter: $ cat /etc/passwd

Example: The output from cat command is written to /tmp/text.txt file instead of being displayed on the monitor screen. You can view /tmp/text.txt using cat command itself: $ cat /etc/passwd > /tmp/test.txt

Example: combining files. The original file or files are not modified or deleted. In this example, cat will concatenate copies of the contents of the three files /etc/hosts, /etc/resolv.conf, and /etc/fstab: $ cat /etc/hosts /etc/resolv.conf /etc/fstab

Example: Redirect output. redirect the output using shell standard output redirection: $ cat /etc/hosts /etc/resolv.conf /etc/fstab > /tmp/outputs.txt $ cat /tmp/outputs.txt

Example: Use a pipe to filter data. In this example send output of cat to the less command using a shell pipe as the file is too large for all of the text to fit on the screen at a time: $ cat /etc/passwd | less

Example: Use cat command for file creation. To create a file called foo.txt, enter: $ cat > foo.txt

*cd (change directory)

Syntax: cd [directory] Example: Used to go back one directory on the majority of all Unix shells. It is important that the space be between the cd and the .. $ cd ..

Example: When in a Korn shell to get back one directory used to go back one directory. $ cd -

Example: go back one directory and then go into the home/users/computerhope directory. $ cd ../home/users/computerhope

Example: go back two directories. $ cd ../../

Example: Move to home directory. $cd

*grep = See section "Search files for regular expressions using grep (Global Regular Expressions)"

normal use of the grep command. grep searchtext filenames

This is an example of a common usage of grep: grep apple fruitlist.txt

An example of a search argument that has exact text with multiple words: grep "apple orange" fruitlist.txt

An example of using grep to search multiple files- in this case, text files: grep apple *.txt

Example of grep ignoring the case of the search: grep -i apple fruitlist.txt

Line matches exactly - in this case the word apple is the only word on that line: grep - x apple fruitlist.txt

Example of grep search that finds all lines that do not match: grep -v apple fruitlist.txt

Examples of searching for text with spaces on either side: grep " apple " fruitlist.txt grep -w apple fruitlist.txt

Example of searching with wildcards: grep " ap.le " fruitlist.txt

Example of grep with extended regular expressions: egrep "(apple|orange)" fruitlist.txt

*file = Tells you if the object you are looking at is a file or if it is a directory.

Example: What may appear when running file with a wildcard for all files. $ file * shutdown.htm: HTML document text si.htm: HTML document text side0.gif: GIF image data, version 89a, 107 x 18 robots.txt: ASCII text, with CRLF line terminators routehlp.htm: HTML document text rss: setgid directory

Example: What may appear when running the above example; running the file command listing any file ending with .txt. $file *.txt output: form.txt: news or mail text friend.txt: news or mail text ihave.txt: news or mail text index.txt: ASCII Java program text, with very long lines, with CRLF line terminators jargon.txt: news or mail text news.txt: Non-ISO extended-ASCII C program text, with very long lines, with CRLF line terminators newsdata.txt: Non-ISO extended-ASCII English text, with very long lines, with CRLF line terminators qad.txt: news or mail text refrence.txt: news or mail text robots.txt: ASCII text, with CRLF line terminators stopwords.txt: ASCII English text, with CRLF line terminators yhelp.txt: news or mail text

*head = Displays the first ten lines of a file, unless otherwise stated.

Syntax = head [-number | -n number] filename Example: Display the first fifteen lines of myfile.txt. $ head -15 myfile.txt

*more = Displays text one screen at a time.

Example: Displaying the file myfile.txt at line three. $ more +3 myfile.txt

*less = Opposite of the more command.

Example: Open file.txt in less. $ less file.txt

*ls = Lists a directories contents. See Section, "Commands to view filenames and file types"

*pwd = Short for print working directory, displays current directory name on the screen.

$ pwd

*strings = Prints each string of printable characters in a file to the screen. Used to extract binary information from files.

Syntax: strings [options] file_name(s)

• Sources:

http://www.cyberciti.biz/faq/howto-use-cat-command-in-unix-linux-shell-script/ http://www.computerhope.com/unix/ http://www.linfo.org/strings.html

Use vi to modify text files

What is vi?

The default editor that comes with the UNIX operating system is called vi (visual editor). [Alternate editors for UNIX environments include pico and emacs, a product of GNU.] The UNIX vi editor is a full screen editor and has two modes of operation: Command mode commands which cause action to be taken on the file, and Insert mode in which entered text is inserted into the file. In the command mode, every character typed is a command that does something to the text file being edited; a character typed in the command mode may even cause the vi editor to enter the insert mode. In the insert mode, every character typed is added to the text in the file; pressing the <Esc> (Escape) key turns off the Insert mode. While there are a number of vi commands, just a handful of these is usually sufficient for beginning vi users. To assist such users, this Web page contains a sampling of basic vi commands. The most basic and useful commands are marked with an asterisk (* or star) in the tables below. With practice, these commands should become automatic. NOTE: Both UNIX and vi are case-sensitive. Be sure not to use a capital letter in place of a lowercase letter; the results will not be what you expect.

To Get Into and Out Of vi

To Start vi

To use vi on a file, type in vi filename. If the file named filename exists, then the first page (or screen) of the file will be displayed; if the file does not exist, then an empty file and screen are created into which you may enter text.

• vi filename edit filename starting at line 1

	vi -r filename	recover filename that was being edited when system crashed

To Exit vi

Usually the new or modified file is saved when you leave vi. However, it is also possible to quit vi without saving the file. Note: The cursor moves to bottom of screen whenever a colon (:) is typed. This type of command is completed by hitting the <Return> (or <Enter>) key.

• :x<Return> quit vi, writing out modified file to file named in original invocation

	:wq<Return>	quit vi, writing out modified file to file named in original invocation
	:q<Return>	quit (or exit) vi

• :q!<Return> quit vi even though latest changes have not been saved for this vi call

Moving the Cursor

Unlike many of the PC and MacIntosh editors, the mouse does not move the cursor within the vi editor screen (or window). You must use the the key commands listed below. On some UNIX platforms, the arrow keys may be used as well; however, since vi was designed with the Qwerty keyboard (containing no arrow keys) in mind, the arrow keys sometimes produce strange effects in vi and should be avoided. If you go back and forth between a PC environment and a UNIX environment, you may find that this dissimilarity in methods for cursor movement is the most frustrating difference between the two. In the table below, the symbol ^ before a letter means that the <Ctrl> key should be held down while the letter key is pressed.

• j or <Return>

 [or down-arrow]	move cursor down one line

• k [or up-arrow] move cursor up one line
• h or <Backspace>

 [or left-arrow]	move cursor left one character

• l or <Space>

 [or right-arrow]	move cursor right one character

• 0 (zero) move cursor to start of current line (the one with the cursor)
• $ move cursor to end of current line

	w	move cursor to beginning of next word
	b	move cursor back to beginning of preceding word
	:0<Return> or 1G	move cursor to first line in file
	:n<Return> or nG	move cursor to line n
	:$<Return> or G	 move cursor to last line in file

Screen Manipulation

The following commands allow the vi editor screen (or window) to move up or down several lines and to be refreshed.

	^f	move forward one screen
	^b	move backward one screen
	^d	move down (forward) one half screen
	^u	move up (back) one half screen
	^l	redraws the screen
	^r	redraws the screen, removing deleted lines

Adding, Changing, and Deleting Text

Unlike PC editors, you cannot replace or delete text by highlighting it with the mouse. Instead use the commands in the following tables. Perhaps the most important command is the one that allows you to back up and undo your last action. Unfortunately, this command acts like a toggle, undoing and redoing your most recent action. You cannot go back more than one step.

• u UNDO WHATEVER YOU JUST DID; a simple toggle

The main purpose of an editor is to create, add, or modify text for a file. Inserting or Adding Text

The following commands allow you to insert and add text. Each of these commands puts the vi editor into insert mode; thus, the <Esc> key must be pressed to terminate the entry of text and to put the vi editor back into command mode.

• i insert text before cursor, until <Esc> hit

	I	insert text at beginning of current line, until <Esc> hit

• a append text after cursor, until <Esc> hit

	A	append text to end of current line, until <Esc> hit

• o open and put text in a new line below current line, until <Esc> hit
• O open and put text in a new line above current line, until <Esc> hit

Changing Text

The following commands allow you to modify text.

• r replace single character under cursor (no <Esc> needed)

	R	replace characters, starting with current cursor position, until <Esc> hit
	cw	change the current word with new text, 

starting with the character under cursor, until <Esc> hit

	cNw	change N words beginning with character under cursor, until <Esc> hit; 
 e.g., c5w changes 5 words
	C	change (replace) the characters in the current line, until <Esc> hit
	cc	change (replace) the entire current line, stopping when <Esc> is hit
	Ncc or cNc	change (replace) the next N lines, starting with the current line,

stopping when <Esc> is hit

Deleting Text

The following commands allow you to delete text.

• x delete single character under cursor

	Nx	delete N characters, starting with character under cursor
	dw	delete the single word beginning with character under cursor
	dNw	delete N words beginning with character under cursor; 
 e.g., d5w deletes 5 words
	D	delete the remainder of the line, starting with current cursor position

• dd delete entire current line

	Ndd or dNd	delete N lines, beginning with the current line; 
 e.g., 5dd deletes 5 lines

Cutting and Pasting Text

The following commands allow you to copy and paste text.

	yy	copy (yank, cut) the current line into the buffer
	Nyy or yNy	copy (yank, cut) the next N lines, including the current line, into the buffer
	p	put (paste) the line(s) in the buffer into the text after the current line

Other Commands

Searching Text

A common occurrence in text editing is to replace one word or phase by another. To locate instances of particular sets of characters (or strings), use the following commands.

	/string	 search forward for occurrence of string in text
	?string	 search backward for occurrence of string in text
	n	 move to next occurrence of search string
	N	 move to next occurrence of search string in opposite direction

Determining Line Numbers

Being able to determine the line number of the current line or the total number of lines in the file being edited is sometimes useful.

	:.=	 returns line number of current line at bottom of screen
	:=	 returns the total number of lines at bottom of screen
	^g	provides the current line number, along with the total number of lines,

in the file at the bottom of the screen Saving and Reading Files

These commands permit you to input and output files other than the named file with which you are currently working.

	:r filename<Return>	 read file named filename and insert after current line 

(the line with cursor)

	:w<Return>	write current contents to file named in original vi call
	:w newfile<Return>	write current contents to a new file named newfile
	:12,35w smallfile<Return>	write the contents of the lines numbered 12 through 35 to a new file named smallfile
	:w! prevfile<Return>	write current contents over a pre-existing file named prevfile

Regular Expressions

Search files for regular expressions using grep (Global Regular Expressions)

• grep allows you to search through a file for text argument.
• grep "localhost" /etc/hosts

return lines that include localhost in the file /etc/hosts

• grep -v "localhost" /etc/hosts

return lines that DONT include localhost in the file /etc/hosts

• grep -l "LOCALHOST" /etc/hosts

return lines that include localhost in the file /etc/hosts that is not case-sensitive

• grep -i "text" "File"

to search for text that is not case sensitive in a file since the default is case sensitive 

• grep "text" (quotation marks)

will search for text patterns like: texting, subtext all consist of the word text

• grep " text " (quotations with a space before and after)

will search for the exact text in a line

• grep " s.text " (quaotations with a space and a period)

will search for any pattern of text that starts with the letter s and ends with text: subtext, supertext

• grep "^T " (quotation, carat, space quotation)

To view lines that start with the letter T

Searching with extended regular expressions with egrep

• egrep "(help | hope)" (quotation, parenthesis, help, space, pipe, space, hope, parenthesis, qotation)

To view lines that contain the text "help" or "hope"

Commands to Display Contents of Binary Files

• strings = searching binary file for text
• od = displays file in octal format
• od -x = displays contents in hex

Searching for Text within Files

• Tools: grep, awk, sed, vi, emacs, ex, ed, C++, PERL, tcl

• Regular expressions

• Wildcard metacharacters are interpreted by the shell

• Regular expressions are interpreted by text tool program

• Wildcard metacharacters match characters in file & dir names

• Regular expressions match charters within file

• Wildcard metacharacters typically have different definitions than regular expressions

• metacharacters

• There are more regular expression metacharacters than wildcard metacharacters

Alternatives to vi

Emacs

• Not installed by default in Fedora 13.
• Can run in a GUI environment.
• More GUI than vi.
• Written in C and Emacs Lisp.
• Licensed GNU GPL
• Users can combine commands into macros to automate task.

gedit editor

• Default text editor for GNOME desktop environment.
• Does not have the advanced functionality that vi or Emacs has.
• Has ability to install additional Gedit plugins.
• One of the easiest editors to use.

nano editor

• Based on pine UNIX editor.
• Provides shortcut hints at the bottom of the screen.
• Has graphical text editor.

Summary

• Linux file system is a hierarchy,  series of directories

paths are absolute or relative

• Many types of files - text, scripts, executable, dirs, linked, special device
• ls - view file names and many options to modify view
• Wildcard metacharacters help to select multiple files
• Regular expression metacharacters are used in many ways.
• Text files can be viewed differently with commands such as, head, tail, cat, tac, more and less.
• vi is the most common text editor however GUI options exits  

References

• Anatomy of the Linux File System

http://www.ibm.com/developerworks/linux/library/l-linux-filesystem/

• File extensions

http://learn.clemsonlinux.org/wiki/File_extensions

http://www.debianhelp.co.uk/fileext.htm

• Linux/Unix/BSD Post-Exploitation Command List

https://docs.google.com/document/d/1ObQB6hmVvRPCgPTRZM5NMH034VDM-1N-EWPRz2770K4/edit?hl=en_US

• vi tutorial

http://www.unix-manuals.com/tutorials/vi/vi-in-10-1.html

• VI Cheatsheet

 http://www.atmos.albany.edu/deas/atmclasses/atm350/vi_cheat_sheet.pdf

• Learning the vi and Vim Editors

http://www.amazon.com/Learning-Vim-Editors-Arnold-Robbins/dp/059652983X/ref=sr_1_1?s=books&ie=UTF8&qid=1326846342&sr=1-1 

• Command Guides

http://wiki.ihcnt.net/w/Linux_command_guide

http://wiki.ihcnt.net/w/Linux-command-list 

• Emac Guide

http://sean.wenzel.net/docs/emacs/quick_reference/ 

• 15 Practical examples of ls commands

http://www.thegeekstuff.com/2009/07/linux-ls-command-examples/

• 50 Most Frequently Used UNIX / Linux Commands http://www.thegeekstuff.com/2010/11/50-linux-commands/