techne/bash.org

Bash

• Split large text file into smaller files with equal number of lines
• Loop through lines of file
• Use grep to find URLs from HTML file

Split large text file into smaller files with equal number of lines

split -l 60 bigfile.txt prefix-

Loop through lines of file

while read line; do
    echo "$line";
done </path/to/file.txt

Use grep to find URLs from HTML file

cat urls.html | grep -Eo "(http|https)://[a-zA-Z0-9./?=_%:-]*"

• grep -E: egrep
• grep -o: only output what has been grepped
• (http|https): either http OR https
• a-zA-Z0-9: match all lowercase, uppercase, and digits
• .: match period
• /: match slash
• ?: match ?
• =: match =
• _: match underscore
• %: match percent
• :: match colon
• -: match dash
• *: repeat the […] group any number of times