Bash Cheatsheet

This article is an ongoing article to collect usefull bash scripts and helps to reduce overhead in my brain

general

fdupes -r -f . | grep -v '^$' | xargs rm -v #find duplicates and delete them
find $PATH -type f -empty
find $PATH -type d -empty -delete
find $FOLDER -type f | wc -l #counts number of files in a folder
mkdir `date '+%Y-%m-%d'`_test # create a folder with current date and a test string
cp `ls ./ | head -n 100` 100/

Copy file

If you copy a large amount of files and want to find out which is currently copied:

ls -l /proc/27554/fd
lrwx------    1 admin    administ        64 Nov  3 07:39 0 -> /dev/pts/0
lrwx------    1 admin    administ        64 Nov  3 07:39 1 -> /dev/pts/0
lrwx------    1 admin    administ        64 Nov  3 07:39 2 -> /dev/pts/0
lr-x------    1 admin    administ        64 Nov  3 07:39 3 -> ....
l-wx------    1 admin    administ        64 Nov  3 07:39 4 -> .....

Where 27554 is the PID of your cp process

MySql

Size of databases in MB

SELECT table_schema "database", sum(data_length + index_length)/1024/1024 "size in MB" FROM information_schema.TABLES GROUP BY table_schema;

Grant access:

CREATE USER 'user'@'%' IDENTIFIED BY PASSWORD '*WHATSOEVER';

Get all users:

mysql> select * from mysql.user;

Convert

PGP

Encrypt a file for multiple recipients:

gpg --output filename_output.gpg --encrypt --recipient foo@bar.a --recipient a@b.d --recipient c@d.ef input.filename.zip

Vms

Access user to shared folders

sudo adduser $USER vboxsf

change keyboard layout permanent

sudo dpkg-reconfigure keyboard-configuration

Rsync

rsync -vzr viper@zoo.basf.net:/mnt/disk1/viper ./

SSH tunnel

Host UltimateDestinationHost
  ProxyCommand ssh GATEWAYSYSTEM -W %h:%p

key forwarding:

ssh-add ./ssh/id_rsa

pear with proxy

pear config-set http_proxy http://username:password@SERVER-IP_ODER_NAME:PORT

proxy

/etc/envirenment
export http_proxy=SERVER-IP_ODER_NAME:PORT
export https_proxy=SERVER-IP_ODER_NAME:PORT
export ftp_proxy=SERVER-IP_ODER_NAME:PORT
export HTTP_PROXY=$http_proxy
export HTTPS_PROXY=$https_proxy
export FTP_PROXY=$ftp_proxy

processes

ls -l /proc/$PID # to find executed bin
netstat -tupan # to get reports

screen

 list screens: screen -ls
screen -x #open a screen
screen -S <$NAME> # start a new screen with name
Ctl-a d #detach from screen

external

compressed

tar -zxf file.tar.gz
zip -P $PASSWORD-r $OUTPUT.zip $INPUTFILE

Filesize

du -h --max-depth=1 /
To get the size of all folders within /
or:
du -s -h ./

curl

behind a proxy: curl --proxy http://$YOURPROXY:$PORT/ $CURLPARAM
no proxy: curl --noproxy $YOURDESTINATION $CURLPARAM

for-Loop

for i in `dir -1`; do $COMMAND $i; done
for i in *; do $COMMAND $i; done
find . -type f -exec $COMMAND;
for i in $( seq -w 0 999 ); do echo $i ; done | more #print 0-999 to the bash or anywhere else

grep / fgrep

grep for all keywords within the keywords.txt

fgrep -rf keywords.tst foldertobesuarched/
cat $FILE | grep '$TERM1\|$TERM2' | sort | uniq
finds two terms and display only unique results
grep $TERM -A N -B M # show lines before and after
strings file | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" # IP adresses only
grep -v "`cat ignore`" to_be_searched.txt # search but ignore all terms in the given file

grep -r -Eo '(http|https)://[^ "]+' . > ../all_links
grep -Eo '(http|https)://[^ "]+' all_links > all_links_real
cat all_links_real | sort | uniq  > all_links_sorted
grep -Eo '(http|https)://[^/"]+' all_links_sorted.txt |  uniq -c | sort -nr |

awk

$ head -1 $FILE | awk '{print $4, $5}'
[16/Nov/2009:09:06:28 +0300]

all unique IPs: 
awk '{print $1}' $FILE | sort | uniq

How many requests:
awk '{print $7}' $FILE | sort | uniq -c | sort -nr | head

Response codes:
awk '{print $9}' $FILE| sort | uniq -c | sort -nr | head#
awk '($9="404") {print $7}' $FILE | sort | uniq -c | sort -nr | head

Images / DD / raw

qemu

Convert a vmdk to an raw / dd image

qemu-img convert -p -O raw source.vmdk destination.raw
    (0.00/100%)

zip

zip -P $password $RESULT.zip $FILE1 $FILE2...
unzip \*.zip # unzip all zips in one folder
7z x $FILENAME.7z #extracts a 7zip file more info

python

http_proxy=http://$YOURPROXY:$PORT/ python $YOURPYTHONSCRIPT.py #python behind a proxy

Disable proxy for a request:

import requests
import os
os.environ['no_proxy'] = '127.0.0.1,localhost,local.domain.net'
...

Regex

https://regex-golang.appspot.com/assets/html/index.html

Logging / Debug

https://docs.python.org/2.4/lib/minimal-example.html

PIP

pip install -–proxy=user:pass@$YOURPROXY:$PORT $PACKAGE #pip behind a proxy
pip freeze # will show you all installed python packages

dig

dig alexanderjaeger.de MX @8.8.8.8 #asks a specific nameserver for results (also usable within your own network and a local DNS resolver)

SCP

scp $USERNMAME@$REMOTEHOST:foo.txt /$DESTINATION/ -R 3-R to do it recursive

git

Find original URL of the repo: git config --get remote.origin.url
git config --global http.proxy http://$YOURPROXY:$PORT/

apt pgp key adden

sudo apt-key adv --keyserver-options http-proxy=http://$YOURPROXY:$PORT/ --keyserver keyserver.ubuntu.com --recv-keys $GPG_KEY

External useful cheatsheets:

http://cli.learncodethehardway.org/bash_cheat_sheet.pdf

Reading logs

lnav

Simply awesome: http://lnav.org/

Related posts to Bash Cheatsheet