Automation Tool with bash shell

Linux command for remove duplicate lines in a file

awk '!seen[$0]++' file.txt

Repeat a loop to solve the list

for URL in `cat /tmp/filelist.txt`; do something good; echo $URL" is done " >> /checklist-ok.txt; done

Crawl the url list with http response code

ND=576220
for i in $(seq 1 $END); do out=$(curl -s -o /dev/null --head -w "%{http_code}" -X POST "https://example.com/?p=$i";); echo $i"|"$out >> listposts.txt ; done

Remove blank lines in file

sed '/^$/d' input.txt > output.txt

Delete lines match pattern in text file

sed -i.bak '/pattern to match/d' ./infile

Count how many times a string appear in a cell of excel:

=(LEN(A2)-LEN(SUBSTITUTE(A2, "happy","")))/LEN("happy")

…Saved!