Forward (left to right) pipeline

head FileName | sort | awk '{print}'

You can use a backwards redirection (not sure that's the official name) to provide STDIN to a utility.

sort < FileName

FileName is the STDIN for sort.

STDIN will be executed before pipes.

sort < FileName | tail

So this will sort the file, then print the tail.

You can group commands with parentheses

tail < (sort FileName)

So this will sort the file, then print the tail.
At least, I've read that you can do this. It doesn't work in my environment (Bash 4.1.2).
- "You cannot pass file descriptors over SSH. <(...) creates virtual files on your system and does not have a meaning when executing on a remote system." That would explain it, I'm using a remote system.

Save the output of a utility to a file

sort FileName > NewFileName

The crond daemon is the background service that enables cron.
Crond checks for jobs in these locations:
- /var/spool/cron (individual user jobs)
- /etc/cron.d (service and application jobs)
- /etc/anacrontab (special case: service and application jobs)

Open cron job editor:

crontab -e FileName

When you save and exit, the crond daemon is restarted and will see your updates.

Crontab uses Vi as its editor.

By default, cron jobs are run as the user who created them.

Example of structure:

 # Setup default environment that these scheduled commands will run in
 SHELL=/bin/bash
 MAILTO=root@example.com
 PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
 
 # For details see "crontab man"
 
 # Example of job definition:
 # .---------------- minute (0 - 59)
 # |  .------------- hour (0 - 23)
 # |  |  .---------- day of month (1 - 31)
 # |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
 # |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
 # |  |  |  |  |
 # *  *  *  *  * user-name  command-to-be-executed
 
 # backup using the rsbu program to the internal 4TB HDD and then 4TB external
 01 01 * * * /usr/local/bin/rsbu -vbd1 ; /usr/local/bin/rsbu -vbd2
 
 # Set the hardware clock to keep it in sync with the more accurate system clock
 03 05 * * * /sbin/hwclock --systohc
 
 # Perform monthly updates on the first of the month
 # 25 04 1 * * /usr/bin/dnf -y update

MAILTO sets where the results of the cron job will be sent.
Results will include the output you'd see if you ran the command manually.

Even when you set the PATH, it is recommended to still give the fully qualified path to each command.

Note that semicolon (;) is used to separate multiple commands.

You can leave any/all time specifications as *.

You can list time specifications:

 # Run quarterly reports
 02 03 1 1,4,7,10 * /usr/local/bin/reports.sh

You can use a range:

 # Remind me each hour of the work-day
 01 09-17 * * * /usr/local/bin/hourlyreminder.sh

You can run every x-th time-unit:

 # Run every 5 minutes, but only during even-numbered hours between 8am and 6pm
 */5 08-18/2 * * * /usr/local/bin/mycronjob.sh

The division must have a remainder of 0.

anacron will run tasks that were skipped because the computer was off or too busy.

Missed jobs will be ran only once - not as many times as were missed.

Sort the lines in a file

sort FileName

grep [options] pattern input_file_names

grep -e patternA -e patternB input_file_names
grep --regex=patternA --regex=patternB input_file_names

or

grep -f pattern_file_name input_file_names
grep --file=pattern_file_name input_file_names

Where pattern_file_name contains a list of patterns, separated by endlines.

Ignore case
-i or --ignore-case

Select non-matching lines instead
-v or --invert-match

Select whole word matches only
-w or --word-regexp

Select whole line matches only
-x or --line-regexp

Output just the count of matching lines per file
-c or --count

Output list of files containing no matches only
-L or --files-without-match

Output list of files containing a match only
-l or --files-with-matches

Stop searching a file after x matching lines are found
-m x or --max-count=x

Print just the matching section of lines
-o or --only-matching

Print line number by each output (starting at 1)
-n or --line-number

Include x lines after each matching line
-A x or --after-context=x

Include x lines before each matching line
-B x or --before-context=x

Include x lines before and after each matching lines
-C x or --context=x or -x

Prints results to STDOUT

awk '{print}' FileName

With nothing else specified, this will just print the file.

Awk will split lines into fields (default delimiter is whitespace).

The fields are accessible as $1, $2, etc.
These are the Field Variables.
$0 holds the entire line.

Print just the 1st and 4th columns of data.

awk '{print $1,$4}' FileName

Specifying a different field delimiter.

awk 'BEGIN { FS="|" } ; { print $1 }' FileName

This will only print lines that contain the regex

awk '/regex here/ {print}' FileName

-1 to suppress column 1 (lines unique to File 1)
-2 to suppress column 2 (lines unique to File 2)
-3 to suppress column 3 (lines shared by both files)

Getting the lines that only appear in File 1

comm -23 File1 File2

--check-order will check that the files are sorted
--nocheck-order will not check that the files are sorted

--output-delimiter=X sets the column delimiter

Replace text based on regular expressions. By default, sed will only replace the FIRST occurrence of a substring in each line.

Replace substring "current" with string "new" throughout old_filename. Save the output to new_filename.

sed s/current/new/ old_filename > new_filename

Use quotes if you have metacharacters in the regular expression.

sed 's/current/new/' old_filename > new_filename

The regular expression delimiter is the first character after s. You can use any delimiter.

sed 's_/usr/bin_/common/bin_' old_filename > new_filename

sed 's:/usr/bin:/common/bin:' old_filename > new_filename

The error "unterminated 's' command" means you forgot a delimiter.

Use & to refer to the matched string, so you can use it in the replacement.

sed 'a/[aeiou]*/(&)/' old_filename > new_filename

Use escaped () to mark multiple patterns for later use. They will be refered to as \1 \2...\9. 9 is the limit.

sed 's:\([aeiou]*\).*:\1:' old_filename > new_filename

Addressing specified which lines to apply the replacement to.

--only make replacements in lines that contain 'IF'
sed '/IF/s/current/new/' old_filename > new_filename

The global flag replaces ALL instances of matching strings in each line.

sed 's/current/new/g' old_filename > new_filename

Replace just the Xth instance in each line.

sed 's/current/new/X' old_filename > new_filename

--just the 4th 'current' is replaced
sed 's/current/new/4' old_filename > new_filename

Option -n means don't print output. Flag p means print only lines that contain a match.

sed -n 's/current/new/p' old_filename > new_filename

--print just lines with matches, like grep
sed -n 's/current/ p' old_filename > new_filename

--print just lines without a match
sed -n 's/current/ !p' old_filename > new_filename

Flag w means print only the matches. If you use multiple flags, w must be the last one.

sed -n 's/current/new/w' old_filename > new_filename

The ignore case flag.

sed 's/current/new/I' old_filename > new_filename

Delete selected text

sed '/text/ d' FileName

Make multiple substitutions at once.

sed -e 's/A/a/' -e 's/B/b/' old_filename > new_filename

--or specify a file of endline-delimited substitution expressions
sed -f pattern_filename old_filename > new_filename

Only apply changes to the Xth line.

sed 'X s/current/new/' old_filename > new_filename

--only the 3rd line
sed '3 s/current/new/' old_filename > new_filename

--only lines 3 through 90 inclusive
sed '3,90 s/current/new/' old_filename > new_filename

--only lines 3 through end of file
sed '3,$ s/current/new/' old_filename > new_filename

Only apply changes where the line starts with '#'.

sed '/^#/ s/current/new/' old_filename > new_filename

Only apply changes that occur between START and STOP. So, the from line that includes START to the next line that includes STOP. This can occur multiple times per file.

sed '/START/,/STOP/ s/current/new/' old_filename > new_filename

--from the 3rd line to the next instance of /STOP/
sed '3,/STOP/ s/current/new/' old_filename > new_filename

Delete all lines from the 3rd to the end of file.

sed '3,$ d' old_filename > new_filename

Remove blank and empty lines

sed '/^\s*$/d' FileName

Users are identified by a unique username and a unique id (UID). A UID is a positive integer.

Create a new user

useradd USERNAME
useradd [OPTIONS] USERNAME

User options start with the defaults in "/etc/default/useradd". [OPTIONS] specified in the useradd command will override the defaults.

See also "/etc/login.defs" for password-related settings.

Adding a user will update "/etc/passwd", "/etc/shadow", "/etc/group", and "/etc/gshadow".

Create user's home directory at "/home/USERNAME"

useradd -m USERNAME
useradd --create-home USERNAME

The home dir will be initialized with files copied from "/etc/skel".

Specify a different home directory

useradd -m -d /SOME/PATH USERNAME
useradd --create-home --home /SOME/PATH USERNAME

Specify the user's UID

useradd -u 123 USERNAME
useradd --uid 123 USERNAME

By default, the next available UID will be used.

Specify the user's group

useradd -g 123 USERNAME
useradd --gid 123 USERNAME

By default, a new group will be created with the same name as the user.

Add a comment or description

useradd -c "a b c" USERNAME
useradd --comment "a b c" USERNAME

Comments are saved in "/etc/passwd".

Set the new user's password

passwd USERNAME

Enter the password when prompted.

Change user's home directory

usermod -d /SOME/PATH USERNAME

id USERNAME      #display UID, GID, and groups
id -u USERNAME   #display UID
id -gn USERNAME  #display GID

chown changes the ownership/permissions on a file/directory.

View current permissions

ls -l

chown [OPTIONS] USER[:GROUP] FILE_OR_DIRECTORY

Recursively change ownership, starting from this point and going down

chown -R USER[:GROUP] FILE_OR_DIRECTORY

vim FILENAME

i = enter Insert mode
Escape = exit Insert mode

(in Normal mode)
w = write/save file to current filename
w FILENAME = write/save file to FILENAME
q = quit vim
q! = quit even with unsaved changes
wq! = write then quit

(in Insert or Normal mode)

h = move left
j = move down
k = move up
l = move right

0 = move to beginning of line
$ = move to end of line
gg = move to beginning of file
G = move to end of file

w = move forward one word
b = move back one word

`. = move to last edit

(in Insert mode)

u = undo last edit
Ctrl-r = redo last edit

dw = delete a word
dd = delete a line
d0 = delete to beginning of line
d$ = delete to end of line
dgg = delete to beginning of file
dG = delete to end of file

(linux is not recognizing LAN connection)

list connections

ifconfig

should at least see a "lo" for local host
may also see "eth0" for ethernet connection

list PCI devices

lspci

lists all devices, such as VGA Controller, Audio, Ethernet, USB, SATA, PCI Bridge, etc
(I'm seeing the two expected ethernet controllers listed)

list hardware, filtering for network

sudo lshw -c network

(I'm seeing "*-network DISABLED" at the top)
look at the configuration line to see what driver is needed
(I need e1000e 3.2.6-k or r8169

list drivers

lsmod | sort

see if the driver listed in lshw is also here
(I see both drivers I could use listed here)

seems that "ifconfig" has been deprecated
and "ip" is recommended instead

list connections

ip link

(now I see "lo" local connection plus my two ethernet options, both say "state DOWN")

enable a DOWN connection

sudo ip link set <LINKID> up 

(YESSSSSS, I can now see this connection listed under "ifconfig" and router connection has lit up)
(but I don't see an "inet" line for it, with the ip address)

(still seems to not be connected to the internet, for instance cannot install new package)

(rebooting)
(ethernet connection is DOWN again after reboot, so set it to UP again)
("ip addr" shows "LOWER_UP" which blog says means the other end of the connection exists)

check that ethernet card is working

ethtool <LINKID>
 #example: ethtool eth0

(dislays "Link detected: yes" so that is working)

TODO try rebooting the router

Turning services on and off

//get status
sudo systemctl status {service name}

//start, stop, restart
sudo systemctl start {service name}
sudo systemctl stop {service name}
sudo systemctl restart {service name}

What version of the server is running?

lsb_release -a

Get useful log info about a service

journalctl -xe | grep {service name}

ex: "journalctl -xe | grep mssql"

what version of a library is installed?

ldconfig -p | grep {package name}

search everywhere for where a particular library version is installed

find / -name {library}-{version} 2>/dev/null

Ex: find / -name liblber-2.4.so.2 2>/dev/null

Virtual hosting is hosting multiple domain names from one IP address. Apache's virtual host directs requests to different folders, based on the domain name.

Apache's top-level directory for websites is "/var/www"
Create subdirectories here for each domain name. Include a "public_html" directory nested in each one.

Set the ownership on the "public_html" directory to the user who owns that website.

sudo chown -R {user}:{group} /var/www/example.com/public_html

Make sure the new directories are reable by all users. This may already be set correctly.

sudo chmod -R 755 /var/www

Upload an html file to the public_html directory for testing.

Make a copy of "/etc/apache2/sites-available/000-default.conf" for each domain name.

sudo cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/example.com.conf

Edit the file with this template

<VirtualHost *:80>
    ServerAdmin admin@example.com
    ServerName example.com
    ServerAlias www.example.com
    DocumentRoot /var/www/example.com/public_html
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Enable the virtual hosts (creates symlinks with sites-enabled)

sudo a2ensite example.com.conf

Restart Apache for the changes to take effect

sudo service apache2 restart
 #or
sudo systemctl reload apache2

Disable the virtual host

sudo a2dissite example.com.conf

To test this setup, you can hard-code your local computer's host file. (This is not referring to the linux server, but to the computer you will test with.)
- Linux
- edit "/etc/hosts" to include a line like "111.111.111.111 example.com"
- Windows
- run notepad as Administrator to open file
- edit "c:\windows\system32\drivers\etc\hosts" to include a line like "111.111.111.111 example.com"

(1)
One-time setup: register Microsoft key and feed

wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
sudo mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/ubuntu/18.04/prod.list 
sudo mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
sudo chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
sudo chown root:root /etc/apt/sources.list.d/microsoft-prod.list

Install .Net Core runtime

sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install aspnetcore-runtime-2.1

That's it. You do not need the SDK on a production server.
Verify installation

dotnet --info
dotnet --list-sdks
dotnet --list-runtimes

(2)
Deploy the app:
- run a Release build
- copy the build files (found in /bin) to the appropriate folder on the server
- run the app with

dotnet My_App.dll

- navigate to the app in browser on the local server machine at "http://{server_address}:{port}"

(3)
Setup the app to run and be monitored in the background
Install Supervisor

sudo apt-get install supervisor

Initialize folder "/var/aspnetcore" for all .Net Core apps

sudo mkdir /var/aspnetcore/{my_app}

Grant access to the user for this site

sudo chown -R {user}:{group} /var/aspnetcore/{my_app}

Copy your build (/bin) files into this directory.

Init supervisor conf file

sudo vim /etc/supervisor/conf.d/{my_app}.conf

[program:{process_name}]
command=/usr/bin/dotnet /var/aspnetcore/{my_app}/{my_app}.dll --urls "http://*:5000"
directory=/var/aspnetcore/{my_app}
autostart=true
autorestart=true
stderr_logfile=/var/log/{my_app}.err.log
stdout_logfile=/var/log/{my_app}.out.log
environment=ASPNETCORE_ENVIRONMENT=Production,CONNECTIONSTRINGS__DEFAULT="{connection_string}"
user={app_user}
stopsignal=INT

Note that "environment" is used to set environment variables for this application.

environment=KEYA=valA,KEYB=valB

The values can optionally be enclosed in double quotes ("")

Start Supervisor

sudo service supervisor start

Restart Supervisor

sudo supervisorctl reload

Watch Supervisor logs to see app startup

sudo tail -f /var/log/supervisor/supervisord.log
sudo tail -f /var/log/{my_app}.out.log 

?? supervisor log says it is running as root because no user was specified, but i did specify user ??

!! it works! I can reach the web api from another computer on port 5000 !

ok, the service part stopped working after rebooting the server
stay calm
the port 80 static html parts are still working
- the Supervisor app is in a start/die loop on "address 5000 already in use"

Install Net Tools to see what is using that port

sudo apt install net-tools

netstat -ltnp | grep :5000

It says "tcp6" which made me think of the nginx configuration that included ip4 and ip6
- manually stopped nginx
- confirmed service is now reachable from another computer on port 5000
- removing the port 5000 config from nginx
- restarted nginx, confirmed 5000 service is still available externally
- trying a reboot again
- hmm, now port 80 sites AND 5000 service are having trouble
- shit I can't connect to my server
- oh shit
- ok, nobody panic, I manually turned it off and on again and everything is working
- trying system reboot again
- everything is still working

Supervisor hosts locally only
Example /etc/supervisor/conf.d/{sitename}.conf

[program:{sitename}]
command=/usr/bin/dotnet /var/aspnetcore/{sitename}/{program}.dll --urls "http://127.0.0.1:5100"
directory=/var/aspnetcore/{sitename}
autostart=true
autorestart=true
stderr_logfile=/var/log/{sitename}.err.log
stdout_logfile=/var/log/{sitename}.out.log
environment=ASPNETCORE_ENVIRONMENT=Production
user=abound
stopsignal=INT

Apache2 listens on port 80
Example /etc/apache2/sites-available/{sitename}.conf

<VirtualHost *:80>
    ServerName {domainname}
    ProxyPreserveHost On
    ProxyPass / http://127.0.0.1:5100/
    ProxyPassReverse / http://127.0.0.1:5100/
    ErrorLog /var/log/apache2/{sitename}-error.log
    CustomLog /var/log/apache2/{sitename}-access.log common
</VirtualHost>

Directory browsing is when the web user can view the list of files in a directory on your server.

To disable directory browsing for all sites hosted on the server:
- open "/etc/apache2/apache2.conf"
- change line "Options Indexes FollowSymLinks" to "Options FollowSymLinks"
- restart apache2 service for changes to take effect

Instructions for Ubuntu + Apache2

1) Generate a Private Key file and CSR (Certificate Signing Request)
- command prompt: sudo openssl req -new -newkey rsa:2048 -nodes -keyout {name}.key -out {name}.csr

Place these files in a location that only Root can access.

The {name}.key file is the private key file.
The {name}.csr file is the certificate signing request. This is used to order an SSL Certificate from a signing authority, and to encrypt messages that only the private key can decrypt.

When asked for the Common Name, enter the fully qualified Domain Name.
Ex: google.com
Ex: *.google.com to include all subdomains

Country Code list: https://www.digicert.com/kb/ssl-certificate-country-codes.htm

**instructions paused, looking at certbot option

https://certbot.eff.org/

easy generation and installation of certificates, with auto-renewal

mostly followed this
[Installing A New Hard Drive]

to get a list of the devices

sudo lshw -C disk

"-C disk" will filter by the description, if you don't see your disk then look without the filter as well
this will give you the "/dev/{device}" label

command line to partition disk

sudo parted /dev/{device}

follow the rest of the instructions from the webpage for this part

then command line formatting

sudo mkfs -t ext4 /dev/{device}



find the uuid of the device, which is more consistent after reboots than the "/dev/{device}" name


sudo blkid



create a mount point

add a directory under "/media" with a name of your choice


sudo mkdir /media/{name}



configure the drive to be mounted on each server startup


sudo vim /etc/fstab

and add a line like


/dev/disk/by-uuid/{uuid} /media/{name} ext4 defaults 0 2

the "2" here tells it to run a disk check on each server start, you can set to "0" to skip that



reboot server