cool_things [Nebula Cyber Wiki]

Site Tools


cool_things

Put some links to cool stuff here and let others know what you are checking out


Networking

  1. VPN
  2. 802.1x
    1. Freeradius: Decent FreeRadius setup for 802.1x: https://blog.svedr.in/posts/freeradius-peapv0+mschapv2-howto.html
  3. Firewalls
    1. iptables
      1. Country based IP blocking with xtables-addons-common http://xmodulo.com/block-network-traffic-by-country-linux.html

Development

  1. Programming Languages
  2. Package Managers
    1. Yarn package mangager: https://github.com/yarnpkg/yarn
    2. Brew(package manager macOS):https://brew.sh/
    3. Chocolatey(package manager for windows):https://chocolatey.org/
    4. GNU package manager(sort of): https://www.gnu.org/software/gsrc/
    5. Conda: https://docs.conda.io/en/latest/ Package, dependency and environment management for any language—Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN
  3. Code Libraries
    1. Beautiful soup: https://www.crummy.com/software/BeautifulSoup/bs4/doc/ pulling data from html
    2. Scrapy: https://scrapy.org/ python web scraper
  4. Web Dev
    1. Grapesjs Website framework builder: https://grapesjs.com/

Development Operations

  1. Serverless
  2. Web Servers
    1. Expose local sites to internet through Ngrok servers:https://ngrok.com/
  3. Identity Management
    1. Linux
  4. Automation
    1. Jenkins(automation server): https://jenkins.io/
    2. Puppet(Automatic machine configuration): https://puppet.com/
  5. Distributed Computing
    1. Apache hadoop(distrubuted computing):http://hadoop.apache.org/
  6. Containers
    1. Docker(link to docs):https://docs.docker.com/
    2. Container Orchestration
  7. Cloud Orchestration
    1. Cloud Infrastructure Automation - https://www.hashicorp.com/
  8. Database
    1. Redis(in memory database):https://redis.io/
  9. Proxy
    1. Caching proxy for the Web supporting HTTP, HTTPS, FTP, and more: http://www.squid-cache.org/

——————————————————————————————————————-

Security

  1. Blogs/Github
    1. OWASP(great for security research): https://www.owasp.org/index.php/Main_Page
    2. Krebs on Security - https://krebsonsecurity.com/
    3. Jason Donenfeld - https://github.com/zx2c4
  2. Frameworks/Best Practice
    1. OWASP(great for security research): https://www.owasp.org/index.php/Main_Page
  3. Tools
    1. Packet Sniffers
      1. Wireshark packet sniffer: https://www.wireshark.org/
      2. Another good TCPDump tutorial: https://danielmiessler.com/study/tcpdump/
      3. Wireshark alternative NetworkMiner: https://www.netresec.com/?page=NetworkMiner
    2. Powershell
      1. Powershell Empire: https://github.com/EmpireProject/Empire Empire is a PowerShell and Python post-exploitation agent
      2. Bloodhound: https://github.com/BloodHoundAD/BloodHound Six Degrees of Domain Admin
      3. Deathstar: https://github.com/byt3bl33d3r/DeathStar Automate getting Domain Admin
    3. Web
      1. IoT
        1. Shodan Internet of things search: https://www.shodan.io/
      2. Scan site for threats - Threatcrowd - https://www.threatcrowd.org/
      3. Vuln scans - HackerTarget - https://hackertarget.com/
      4. Dark Owl: https://www.darkowl.com/ automatically, anonymously and continuously collects, indexes and ranks actionable data from the darknet
    4. C2
      1. Caldera: https://github.com/mitre/caldera (An automated adversary emulation system)
      2. Silent Trinity: https://github.com/byt3bl33d3r/SILENTTRINITY An asynchronous post-exploitation agent powered by Python, IronPython, C# and .NET's DLR
    5. Testing
      1. Atomic: https://github.com/redcanaryco/atomic-red-team Small and highly portable detection tests based on MITRE's ATT&CK.

Encryption

  1. Web

Virtualization


Text Editors

  1. Spacemacs(vim and emacs combined editior) - http://spacemacs.org/
  2. Visual Studio Code - https://code.visualstudio.com/

Data Analytics

  1. Apache spark(data analytics) - https://spark.apache.org/

LabSetup

Install LAMP stack Ubuntu 16.04: https://www.linode.com/docs/web-servers/lamp/install-lamp-stack-on-ubuntu-16-04/


Customize Window manager

i3 Window Manager:https://i3wm.org/


Customize Shell

Oh-my-zsh(zsh shell):https://github.com/robbyrussell/oh-my-zsh


Cheat Sheets for common Utilities

SCP Cheat Sheet:https://gist.github.com/dehamzah/ac216f38319d34444487f6375359ad29

TMUX Cheat Sheet: https://gist.github.com/henrik/1967800

TMUX copy and paste: http://www.rushiagr.com/blog/2016/06/16/everything-you-need-to-know-about-tmux-copy-pasting/

VIM Cheat Sheet: https://vim.rtorr.com/

Spacemacs Cheat Sheet: https://simpletutorials.com/c/3034/Spacemacs+%28Vim+mode%29+Cheatsheet

Regex Cheat Sheet: https://regex101.com


IRC TOOLS

Freenode web irc: https://freenode.net/


Other Tech Magic

IFTTT (If This, Then That): https://ifttt.com


Communities to ask Questions/Learn/Answer Questions

Reddit sysadmins: https://www.reddit.com/r/sysadmin/

Stack Overflow: https://stackoverflow.com/

Information Security Stack: https://security.stackexchange.com/

Server Stack: https://serverfault.com/

Super User Stack: https://superuser.com/

Ask Ubuntu: https://askubuntu.com/


Good Reading

TTY: https://www.linusakesson.net/programming/tty/index.php

Toa of TMUX: https://leanpub.com/the-tao-of-tmux/read


Useful Scripts

Microsoft script center(link takes you to security scripts):https://gallery.technet.microsoft.com/scriptcenter/site/search?f%5B0%5D.Type=RootCategory&f%5B0%5D.Value=security&f%5B0%5D.Text=Security


COMMANDS TO BE ADDED BOLD IS COMMAND ITALICS IS DESCRIPTION

Powershell Get-Filehash one.txt {gives sha256 hash)

apt-cache search sysct


LOOK UP RESEARCH ADD

AWS Security Group stuff

AuthorizeSecurityGroupIngress

- AuthorizeSecurityGroupEgress

- RevokeSecurityGroupIngress

- RevokeSecurityGroupEgress

- CreateSecurityGroup

- DeleteSecurityGroup

Contexts and cross site scripting


Site bookmarks to check out later ( delete after 30 days)

https://cisofy.com/lynis/

https://lucene.apache.org/solr/

https://hackernoon.com/how-ive-captured-all-passwords-trying-to-ssh-into-my-server-d26a2a6263ec

https://alexpearce.me/2016/02/managing-dotfiles-with-stow/

https://www.linode.com/docs/getting-started/

https://securityespresso.org/tutorials/2019/03/22/vpn-server-using-wireguard-on-ubuntu/

https://www.ckn.io/blog/2017/11/14/wireguard-vpn-typical-setup/

https://stackoverflow.com/questions/11530090/adding-a-new-entry-to-the-path-variable-in-zsh

https://skeletonscribe.net/

http://www.pythontutor.com/visualize.html#mode=edit

https://seanmcgary.com/posts/haproxy---route-by-domain-name/

https://stackoverflow.com/questions/4929251/how-to-step-through-python-code-to-help-debug-issues


Ideas

If stack-overflow had a donate-a-dollar button, you would definitely be getting a dollar =). That would be an awesome feature actually - people would use the platform more and earn money

https://blog.jessfraz.com/post/installing-and-using-wireguard/

Twilio setup group message with a phonenumber and then have webhooks from instagram,twitter,whatsapp for sso chatapp with sms support via twilio


Linux Notes

debian

sudo apt-get -y install net-tools

my checklist for the 5 minutes after getting dropped in a random *nix box

- w -f to see who's logged in and how long the server's been up

- ss -tlnp (linux) netstat -an (bsd) to see what processes have listening sockets

- getent passwd to list users, passwd <x> to change, passwd -l <x> to lock an account (on bsd: pw lock <x>)

- visudo to open the sudoers config - remove any users or groups that shouldn't have access to sudo

- crontab -l and atq to see what tasks are scheduled to run later (use sudo -u <user> crontab -l for each user to list that user's crontab, or just poke around /var/spool/cron or /var/cron/tabs or /etc/crontab.d, etc

- iptables -L -n -v to list firewall rules

- check users' (esp. root's) ~/.bash_history and/or ~/.histfile

- edit /etc/groups to manage what users should be in what groups (esp. check on wheel and sudo groups)

- browse journalctl -xef (shift pgup/down) (linux) or /var/log/all.log (bsd) + the rest of /var/log for any funny business

- top or ps to see processes (edited)

- systemctl list-unit-files (recent linux) or service –status-all/ls /etc/rc.d (bsd, olderish linux) to see what services are enabled

- check /etc/rc.conf and /usr/local/etc/rc.conf (both bsd only) for any funny system configuration

- check uname -a, cat /etc/issue, lsb_release -a to find out what kind of *nix this is

- debian/ubuntu: apt-get update -y && apt-get upgrade && apt-get dist-upgrade, freebsd: pkg upgrade, redhat/centos/fedora: yum upgrade/dnf upgrade, suse: zypper upgrade

- ip6tables to check ipv6 stuff - maybe just disable all ipv6 with a sysctl or kernel boot param

- /etc/inetd.conf /etc/inetd.d /etc/xinetd.conf /etc/xinetd.d

fc -l 0 | grep 'thing_i_want'

sum all numbers in a file awk '{n += $1}; END{print n}' file.txt (breaks if non number)

/proc for running processes

/sys for system information (devices,kernel, etc)

uname - : displays information for currently running kernel

lsmod : displays a listing of all currently loaded kernel modules

modinfo : displays information of specific kernel module

modprobe : command used to dynamically load and unload kernel modules

lspci : displays information on pci devices attached

lsusb : displays information on usb devices attached

lscpu : displays information on processors on a system

lsblk : displays information on all block devices on a system

dmesg : tradition utility used for viewing the kernel ring buffer

journalctl -k : systemd utility to view the kernel ring buffer within the systemd journal

runlevel : view your current runlevel

telinit : changes to another runlevel

systemctl list-units-files -t target :show all unit fies for available target

systemctl list-units -t target : show all loaded and active unit files

systemctl get-default : list out the default target

systemctl set-default : Change the default target to a different target

wall : broadcasts a message to all logged in users(after message is typed terminate the message with a Ctrl+D)

acpid : Advanced configuration and Power interface, registers sytem events( such as pressing the power button or clposing laptop lid)

Shutdown Commands

poweroff

telinit 0

shutdown -h +1

systemctl isolate poweroff.target

Reboot Commands

reboot

telinit 6

shutdown -r now

systemctl isolate reboot.target

Linux Directory Structure

/ : Bottom of the directory tree, the 'root'

/var : The variable location, log files and dynamic content (such as web sites) are often found here

/home : The users' home directory, where personal files are stored

/boot : The boot directory, where the Linux kernel and supporting files are stored

/opt : Location used for 'optional' software, often used by third-party software vendors. Enterprise environments make extensive use of this location

/lib : libraries

/usr/lib(32 bit systems) /usr/lib64(for 64 bit systems)

/usr/local/lib

/usr/share

Two library files

Dynamic(ends in .so)

Statically linked ( ends in .a)

ldd : prints out the the shared object (library) dependancies ex. ldd /bin/ls

ldconfig : Configures dynamic linker run-time bindings, creates a cache based on library directories and can show you what is currently cached

/etc/ld.so.conf : Configuration file that points to directories and other configuration files that hold references to library directory locations

LD_LIBRARY_PATH : Legacy environment variable that points to a path where library files can be read from

mount : can be used to mount partitions to directories, or show all existing mounts without any options

lsblk : used to show all block devices on a system and their names

fdisk -l /dev/diskname : Can be used to list out partition information on the specified disk

swapon –summary : Shows a summary of the swap usage on a system, same information can be found in /proc/swaps

LVM : Logic Volume Manager ( Can be used for nearly any mount point except /boot, Allows for rezising of volumes, allows for snapshots)

pvs : Lists out the phsical volumes in a LVM group

vgs : Lists out the volume groups within an LVM group

lvs : Lists out the logical volumes within an LVM group

findmnt /boot : to find boot partition

Legacy GRUB

grub-install [device] : to install legacy grub

grub : invokes the GRUB shell environment

help : Print the help listing for GRUB or get more info on a command i.e. help [command]

find : Search for a file in all partitions and list the device(s) the file is on.

quit : Exit the grub shell

Grub2

/boot/efi : vfat or FAT32 and where image files are stored

/boot/grub2 : grubenv and themes

Red Hat based distros : grub2-<command>

Debian based distros : grub-<command>

grub2-editenv list : view the default boot entry for the grub configuration file

/etc/default/grub

grub2-mkconfig : creates(or updates) a /boot/grub2/grub.cfg file based on entries from /etc/default/grub file

update-grub : command that can be used to update a GRUB2 config after changes to /etc/default/grub have been made

APT

Advanced Package Tool

Reads /etc/apt/sources.list : configuration file that lists out repository locations for packages

apt-get remove : removes package from the system but any config files that came with the package will be left behind

apt-get autoremove : removes rest of dependencies

apt-get purge : removes packages from the system and any associated configuration files

apt-get dist-upgrade : upgrades all packages on the system up to the next release of the distribution

apt-get download : downloads the package does not install it.

apt-cache search : searches through your local apt cache for package that can be installed

apt-cache show : lists out basic information about a package

apt-cache showpkg : displays more technical information about a package

dpkg

dpkg –info : Displays information on a package

dpkg –status : Same as –info, but less detail

dpkg -l : Lists out packages that match the string provided

dpkg -i : Installs the specified package(s)

dpkg -L : Lists out all the files that were installed with a specified package

dpkg -r : Removes a specified package but leaves the configuration files behind

dpkg -P : Removes a specified package and also any configuration files that were installed with it

dpkg -S : Searches through the package database for a file specified and lists out any mentions of the specified file or string

dpkg-reconfigure : Allows for the modification of a package by re-running the applicatons configuration tool

YUM

Yellowdog Updater Modified

Handles RPM package dependencies

Global yum configuration options are set in /etc/yum.conf

Reads repository information from /etc/yum.repos.d/

Caches latest repository information in /var/cache/yum

yum search : searches the yum repos for a specified package

yum info : Lists information about a specified package

yum list installed : Displays all installed packages

yum clean all : Cleans up all of yum's cache information and its local database file

yum remove : Uninstalls a package, leaves dependecies behind

yum autoremove : Uninstalls a package and its dependecies

yum whatprovides : Find out what package provides a specified file name

yum reinstall : Reinstalls a specified package

yum-utils : yum utilties

yumdownloader : downloads a package

Other RPM Package Managers

Zypper

Used on SUSE Linux distrubtions

Exampes

Zypper repos

Zypper install vim

DNF - Dandified yum

Used on Fedora Linux distrubutions ( future replacement for yum in Red Hat Enterprise Linux )

Same command syntax as yum

Red Hat Package Manager (rpm)

.rpm package contains application or utility

default config files

How and where to install files that come with a package

List of dependencies that the package requires

rpm database located in /var/lib/rpm

use rpm –rebuilddb command to repair a corrupted rpm database

rpm -qpi : Displays information on a package

rpm -qpl : List files in a package

rpm -qa : List out all installed packages

rpm -i : Installs a specified package, often combined with other options to provided more verbose output, i.e. rpm -ivh

rpm -U : Upgrades an installed packaged with a newer version

rpm -e : Uninstalls (erases) an installed package

rpm -Va : Verify all installed packages

rpm2cpio : Converts an .rpm file into a cpio archive file, often combined with the cpio command, i.e. rpm2cpio some.rpm | cpio -idmv

Bash Environment

env : Command that displays environment variables

echo : versatile command that can be used to print the value of a variable to the screen

set : Displays shell settingds or shell variables for the session

unset : Removes a variable or custom bash function

shopt : Displays shell options and their current settings

export : Command used to export a variable to the current shell and any new shells started from the current shell

pwd : Displays the full path to the current working directory

which : Used to locate an application file that is located within the user's PATH

type : used to determine if something is a function, file, alias, built-oin, or keywork i,e. type type, type ls, type cd

“weak” quotes : Weak quotes or double quotes, will expand variables, but characters used for path substitution or for pattern matching will not be expanded. e.g: echo “$PATH” will print the PATH

ls “*” will not work because the command will look for a file named *

'strong quotes' : Inside strong or single quotes, nothing is intrepeted. e.g. echo '$PATH' will print $PATH to the screen

history : Command that shows the most recently ran commands

.bash_history : File located in the user's home directory that contains the previously run commands

HISTFILESIZE : Environment variable that determines how many lines .bash_history file will contain

Man Pages

Section 1 : Executable programs or shell commands

Section 2 : System calls - functions provided by the kernel

Section 3 : Library calls - functions within program libraries

Section 4 : Special files - typically those found in /dev

Section 5 : File formats and conventions - for example /etc/passwd and other configuration files

Section 6 : Games

Section 7 : Miscellaneous items and conventions - e.g. man(7), regex(7)

Section 8 : System Administration commands - usually only for root

Section 9 : Kernel routines ( non standard )

man : Command used to open the manual page for a specified command

man -k : Used to search the man pages for a specific keyword

apropos : Links to the man-k command

man [section number] : Open a particular section number for a specified command e.g man 4 synaptics

Basic Commands for Viewing Text Files

zcat : Used to view gzip compressed text files

bzcat : Used to view bunzip2 compressed text files

xzcat : Used to view XZ compressed text files

nl : Prints the number of lines in a file. Output of the command can be modified to fit different cases.

wc : The word count command. Can be used to print the number of words, lines or bytes in a file.

od : The octal dump command. Use to print out a file in octal or many other formats

Message Digests

md5sum : Calculates and checks a file's hash based on the MD5 algorithm

sha265sum : Calculates and checks a file's hash value based on the SHA-2 hash algorithm using 265 bits

sha512sum : Calculates and checks a file's hash value based on the SHA-2 hash algorithm using 512 bits'

Text Manipulation

sort : Command used to sort and/or merge lines of a file

uniq : Displays unique lines of a file

tr : Used to translate(or swap) characters in a file or another character e.g tr 'A-Z' 'a-z' to translate capital to lowercase

cut : Extracts columns or fields of data from a file

cut -f 6 -d ':' /etc/passwd : print only the 6th column of information with : delimiter.

paste : Merge lines of files

sed : The 'stream editor' command. Used for a variety of tasks, commonly utilized to alter text in a file or search and replace words e.g sed 's/desktop/workstation/g' filename replace desktop with workstation globally through file

split : Command that is used to split a file up into individual pieces. By default, each piece of the file contains up to 1,000 lines, but this can be changed to a different value or size.

Basic File Management

ls : Command used to display a list of files and directories

touch : Command that is used to modify a files time stamp but s more often used to create an empty file

cp : The copy command, used to make a copy of the file

rm : The remove command. Used to delete a file from the system

mv : The move command. Used to move or rename a file

file : Command used to try and determine a file's type

cd : The 'change directory' command

mkdir : The 'make directory' command

rmdir : The 'remove directory' command/ WIll not delete a directory with any content

$PATH : Environment variable that describes the directories that the currently logged in user can run applications from without specifying their full path.

Archive and Compression Commands

dd : Copies and converts files. Often used to create files of arbitary size and to backup disk drives

tar : Wraps up files and folders into an archive file. This command does not provide compression on its own

gzip : Command that creates .gz compressed files

gunzip : Command that extracts .gz compressed files

tar -czf Directoryname.tar.gz Directoryname : tar and gzip directory

tar -cjf Directoryname.tar.bz2 Directoryname : tar and bzip2 directory

bzip2 : Command that creates .bz2 compressed files

bunzip2 : Command that extracts .bz2 compressed files

xz : Command that creates .xz compressed files

unxz : Command that extracts .xz compressed files

Finding Files

find -name : find files based on their name

fine -ctime : find files based on the time they were last changed

find -atime : find files based on their access time

find -empty : find files or folders that are empty

find -exec[command]{}\; : The exec statement for the find command that will act on the results of the find command itself.

find . -empty -type f -exec rm -f {} \;

File Globbing

* : Matches zero or more characters

? : Matches any single character

[abc] : Matches any on of the characters in the list

[^abc] : Matches any one character except those | n the list, case sensitive

[0-9] : Matches a range of numbers

ls [Ww]eather[Rr]eport199[0-9]?2017*'

Standard input/output/error

stdin has handle number associated with it : 0

stdout has handle number associated with it : 1

stderr has handle number associated with it : 2

internet-toaster.sh : stderr goes to screen

internet-toaster.sh 2> error.log : stderr gets redirected to error.log file

internet-toaster.sh 2>&1 | less : stderr and stdout gets sent as stdin to the less command'

Redirecting output/input

tee : Reads data from stdin and writes that data to stdout and files. The command is useful for chaining together long commands and viewing output at various stages

xargs : Accepts input from stdin and other commands. Commonly used with the 'find' command(but can be used with other commands as well)

Process Commands

ps : Lists the process status of each running process on the system

top : Interactive, near real-time monitoring of processess running on a system

man proc : Manual page for the /proc psuedo file system directory

man signal : Manual pages for the various signal states of processes

Monitoring process

uptime : View how long the system has been up, how many users are logged in, and CPU load average.

free : View used and available memory and swap space.

pgrep : Find process information based on process name

kill : Send signal(usually SIGTERM) to a process based on PID

pkill : Send a signal(usually SIGTERM) to a process based on process name

killall : Kills all processes based on a name provided as an argument

watch : Runs a command at specificed intervals/ Used to monitor a commands output

screen : A terminal window manager that allows you to run commands in an isolated session

tmux : A modern terminal window manager with extra features.

nohup : command that is preceded by nohup receives singal 1(NOHUP) so that should a terminal window close, the process will still run as long as the login session is not terminated.

bg : Sends a job to the background while it keeps it running

& : When applied to the end of a command, the command is sent to the background returning the use of the shell to the user

fg : Brings a job that is in the background into the foreground

jobs : Displays a listing of jobs that are in the background

nice : Command used to define a new nice level before an application is started

renice : Command used to change the nice level of an application that is already running

Regular Expressions

. : Represents a single character

$ : Search the end of a line

[abc] : Search for specified characters

[^abc] : Search for other characters, but not these

* : Match zero or more preceding characters or expression

man 7 regex : manual pages for regular expression

sed : Stream editor can operate on files using regular expressions

egrep : Command that searches a specified file line by line returning lines that contain a pattern mtching a given regular expression. Equivalent to grep -E

fgrep : Searches based on strings rather than patterns. Also uses file globbing instead of regular expressions. Equivalent to grep -F

vi/vim text editor

vimtutor : Excellent built in tutorial system that walks you through the basics of using Vim from novice to advanced

o in command mode moves cursor to next line

v in visual mode

while in visual mode use y to yank highlighted text and while in command mode use p to paste {tags copy paste visual mode }

MBR partitioning tools

lsblk : Command used to list block devices ( such as hard drives)

fdisk : Legacy command used to create partitions of the MBR (DOS) type

parted : Modern command used to create partitions of MBR or GPT types

Partition Id's

83 - standard Linux filesystems

82 - Linux swap partitions

8e - Linux LVM Volumes

GPT Partition tools

gdisk : This a command in the style of fdisk that can be used to create GPT partitions on disks

parted : The parted command can be used to create both MBR and GPT disk partitions

Swap paritions

mkswap : This command can be used to format a patition to be used as swap space

swapon : Enables a swap partition or file

swapoff : Disables a swap parition or file

Linux filesystems

Non-Journaling : ext2 - legacy Linux file system, released in 1993

Journaling : Uses a journal to keep track of changes that have not yet been written to the file system

Examples :

ext3 : released in 2001, introduced journaling to ext2

ext4 : released in 2006, added extra features, meant to be a 'stop-gap' until a better solution comes along

XFS : created in 1993 by Silicon Graphics, Inc for the IRIX operating system. Ported to Linux in 2001

Brtfs : Uses CoW (Copy on write) , Uses subvolumes( similar to a partition, can be accessed like a directory )

FAT : File allocation table, Linux can use VFAT( virtual file allocation table) which allows for long file names

exFAT : Allows for files larger than 2GB in size Primarily used for external disk drives, thumb drives, etc

mkfs : Creates a new file system on a partition. Can be used like so:

mkfs -t [fs-type]

mkfs.[fstype]

blkid : Displays the file systems universally unique identifier( UUID)

Disk Space Usuage

df : Command that show the available disk space on a file system

du : Command that displays the amount of disk space in use

inode : An inode (index node) stores information about files and folders such as : permissions, ownership, and file type. Most file systems contain a maximum number of inodes that it can contain

Commands that display inode information

ls -i

df -i

du –inodes

Maintaining file system

fsck : File system check utility. Can be onvoked via command line and configured in /etc/fstab. Devices must be unmounted before a check can run

e2fsck : File system check utility for ext2. ext3, and ext4 file systems. Can be used to replay the file systems journal

mke2fs : Utility for creating new ext2,ext3,ext4 file systems

tune2fs : Utility used to adjust parameters on a ext2,ext3,ext4 file system

xfs_repair : utility used to repair file systems

xfs_fsr : Reorganizes data stored in blocks on an XFS file system. Similiar to running a defrag utility on a n MS Windows file system.

xfs_db : Utility used to debug an XFS file system.

Mounting file systems

Each directory docking station and each partition a ship docking into the docking station.

mount : Command used to mount a file system to a mount point

man mount : Manual pages for the mount command. Pay particular attention to the file system independent mount options

umount : Command used to unmount a file system. Can specify eithier the device, label or mount point

/etc/fstab : File system table file. Permanent mount configurtions are set here.

File/Folder permissions

r = read permission

w = write permission

x = execute permission

- = no permission

Octal Permissoins

4 = read permission

2 = write permission

1 = execute permission

0 = no permission

chown : Change the ownership of a file or directory

chmod : Change the mode of a file or directory, which will effect the item's permissions

chgrp : Change the group ownership of a file or folder

SUID : The 'set user id' bit. files iwht an 's' omn the place of the 'x' permission within the user's column have this set. Note thast this will not work on Bash scripts anymore. Also many file systems can be mounted with the 'nosuid' option

SGID : The 'set group id' bit. files and folders with the 's' in the place of 'x' permission within the group's column have this set. This assigns group ownership to the files. Useful for shared group directories.

Sticky Bit L: This permission has a 't' in the place of an 'x' in the others column. This permission only allows the creator of a file to remove the file.

chmod 4760 file.txt : 4 is for the SUID bit

chmod -R 2760 Directory : 2 is for the SGID bit

chmod 1777 important.txt : 1 for the sticky bit

umask : Shows the current umask settings. Can also be used to set a new umask value for the current shell session

Default permissions :

777 = default for directories

666 = default for files

umask value : Subtract the umask value from the default permission to get the currently configured permission value

Permanent Configuration : /etc/bashrc = umask set for whole system

/home/[user]/.bashrc : umask set for individual user

links

ln : Creates a 'hard' link to a file or directory. This type of link will only work on the file system of the originating file

ln -s : Creates a symbolic 'soft' link to a file or directory. Links of this type can traverse file systems

unlink : Removes a link from a file or folder.

file system hierarch

/ : root of filesystem

bin : contains executable programs

boot : contains files necessary for system to boot up. Linux kernel exsits in this directory

dev : directory where all devices are refrenced from

etc : system configuration files and config for services

home : home directory for users personal folder

lib : library files that share code

lib64 : 64 bit library files that share code

media : Where devices such as cd drives/usb drives can be mounted

mnt : other harddrives that you would connect to your system

opt : optional location for applications to be stored if not in bin

proc : provide information about running linux system

root : home directory for root user

sbin : System administrator tools and programs

srv : typically used for server applications such as web servers

sys : Used for information about hardware that is on the system

tmp : Used for applications to store temporary data

usr : contains own set of directory tree similiar to that of root. where more applications are stored along with system documentation and additional configs

var : contains files with variable size such as log files

finding commands

locate : Command that searches a local database of files and folders looking for items that match the search criteria

updatedb : Updates the database that the locate command uses

whereis : This command locates binary,source, and/or manual pages for a command

git

git –version : displays the version of git installed

.git : directory that keeps track of the project, database tracks the files.

branch : line of development

structure of .git

COMMIT_EDITMSG : file of each commit to the repo

HEAD : refrence to current branch working on

config : config for current repo

description : name of repo

hooks : contain scripts to run automated taks

index : keeps track of files in staging area or ready to be commited

info : contains items like exclude files

logs : log files for activities

objects : database of compressed files that are hashed versions of content of files that have been commited. Link to database and working directory

refs : refrence files to branches and tags of repo

git init /path/to/directory : initializes a git repo, eithier by creating a new directory or adding git repo files to an existing directory

git init –bare /path/to/directory : initiliazes a bare git repo, for larger projects, contains no working area

git config : command used to configure various elements of your git environment

git config –list : view your configuration information for your git environment

–global to apply to system

git config core.editor : choose default editor

man git-config

git add : command to used to add files to a git project, adds them to the index file so that theycan be tracked in the staging area.

git status : can be used to see what files are in the staging area.(not commited yet)

git rm : removes a file from a project

git status -s : view the output in shortened format

git status -v : get more verbose output, including what was changed in a file

man git-status

git commit : opens a text editor to prepare for a commit of files in the staging area

git commit -m “Commit Message” : bypasses the editor and performs a message with the specified message

git commit -a -m “message” : commit a modified file in the staging area

man git-commit

.git/info/exclude : original file that contrains file patterns that git will not track

.gitignore : ignore file local to a git repository commonly used to exclude files based on patterns

git check-ignore <pattern> used to debug git ignore to see what is and is not being excluded from git

man gitignore

git tag -a <tag name> -m <message> : creates an annotated tag

git tag : view all tags in the repo

git tag <tag name> -m <message> : create a lightweight tag

git tag -d <tag name> : delete a specific tag

man git-tag : local documentation for the git tag command

git branch <branch name> : creates a new branch of the project tree

git checkout <branch name> : switches to another branch

HEAD : pointer to the current branch being worked on, can use git log and git status to view which branch HEAD is pointing to

man git-branch

man git-checkout

git merge : combines the latest commits from two branches into one branch

git branch -d <branch> : deletes specified branch

man git-merge

git rebase <branch> : replay changes made to one branch over the top of another branch

man git-rebase

git revert <commit> : revert a commit in the project

man git-revert

git diff : view the differences between two commits, files, blobs, or between the working tree and staging area

man git-diff

git gc : git garbage collection command, cleans out old objects that can not be referenced by the database anymore, and compresses contents withing the .git directory to save disk space

git gc –prune : by default, cleans out objects that are older than two weeks

man git-gc

git log : view the git repo history

git log –graph : show a textual graph of a projects commit history

git log –stat : show a statistics of the files with each commit

git log –pretty=format : format the output of a git log command to display specific fields

git clone <local repo> <new repo> : clone local repo

man git-clone

git clone <remote URL> : clone remote repo

git remote -v : shows remote servers that are being tracked for the current repo and their latest statistics

git fetch : fetches new commit information down from the remote server for the current repository, does not commit anything to local database

git push -u <remote> <local branch> : pushes local changes to the upstream git repo

man git-push

man git-fetch

man git-remote

Linux Scripting

echo sh{ot,ort,oot} = shot shoot short printed to console

echo st{il,al}l = still stall printed to console

echo “${!HO*}” = will print all env variables that start with HO, not explicit values but names of environment variables

Mac Commands

control+a = home

control+e = end

AWK Magic

awk 'BEGIN { TOTAL=0} { if ($0 == $0+0) { TOTAL+=$0 } else { print $0, “is not a number” }} END {print TOTAL}' data.txt : sum all numbers in a file

Perl snippets

cat /tmp/data.txt | perl -ne '$sum+=$_; END {print “$sum”}' : sum all numbers in a file


AWS Notes

cant run port scans no DNS zone walking

sts - simple token service

IAM

implicity deny overrides implicit allow

always use ssh forwarding

MFA should be on

programmatic access = api,cli, dev tools

management console = web browser

policies cannot be attached directly to resources like ec2

Statement element is main element for policy

permissions for IAM must be explicilty added, before that they have no permissions

managed and custom polcies

managed : AWS managed

custom : user managed

cross account access

identity federation

s3 bucket policies

This sample bucket policy allows for both SSE-S3 and SSE-KMS based encrypted objects while denying everything else:

{
    "Version": "2012-10-17",
    "Id": "PutObjPolicy",
    "Statement": [
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::your-bucket/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": [
                        "AES256",
                        "aws:kms"
                    ]
                }
            }
        },
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::your-bucket/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"
                }
            }
        }
    ]
}

security groups work like stateless firewalls

create different VPC's for work (production,development,testing,etc)

inpsector - investigage aws resources

WAF rules are based on conditions such as IP addresses,HTTP headers,HTTP body.Uniform Resource Identifier,SQL Injection,Cross site scripting

Integrated with AWS services, cloudfront,API gateway, Application load balancer.

General Purpose - A1,T2,T3,M4,M5

EC2 instances - Compute optimized C4,C5

Memory Optimized - R4, R5, x1,X1e, High memory, z1d

Accelerated Computing - P2,P3 General purpose GPU

G3 : Graphics intensive workloads

F1: Customizable hardware acceleration

Storage optimized :

H1 : Up to 16tb of local HDD storage

D2: Up to 64TB of local HDD storage

i3 : NVMe SSD-Backed (High IOPS at low cost)

Instance sizes : increased number of vCPUs,memory,Networking performance, Clock speed

NAT gateways and Bastion hosts :

Bastion Hosts : “Gate” that protects our infrastructure but allows access for updates or other management

Used to control remote access

Can have elastic IP , Can have standby bastion hosts for high availability


Goals

Get better at C,Python,Java and GO

Linux sys admin

10k in investment profit

cool_things.txt · Last modified: 2020/01/01 18:05 by 72.21.196.67