Linux command for remove duplicate lines in a file Repeat a loop to solve the list Crawl the url list with http response code Remove blank lines in file Delete lines match pattern in text…