Read documentation (manual) for a program
man PROGRAM
What kernel is running?
uname -srm
Outputs kernel name, kernel version, and hardware type.
What distribution is running?
cat /etc/os-release
What version of apache is installed?
apache2 -v
How much disk space is used/free?
df
df -h #human readable
How to install new updates for everything?
apt-get update
apt-get -y update #assume "yes" to all questions
If you do kernel updates, "reboot" after to pick up the changes.
What is the status of MS Sql Server?
systemctl status mssql-server.service
Reboot server
sudo reboot
What is installed?
apt list --installed
Print current location (Print Working Directory)
pwd
Jump to root dir
cd /
Jump to your home dir
cd
OR
cd ~
Up one level
cd ..
Back up once in location history
cd -
Utilities in Linux can be formed into a pipeline where the output of one command becomes the input of another.
Forward (left to right) pipeline
head FileName | sort | awk '{print}'
You can use a backwards redirection (not sure that's the official name) to provide STDIN to a utility.
sort < FileName
FileName is the STDIN for sort.
STDIN will be executed before pipes.
sort < FileName | tail
So this will sort the file, then print the tail.
You can group commands with parentheses
tail < (sort FileName)
So this will sort the file, then print the tail.
At least, I've read that you can do this. It doesn't work in my environment (Bash 4.1.2).
- "You cannot pass file descriptors over SSH. <(...) creates virtual files on your system and does not have a meaning when executing on a remote system." That would explain it, I'm using a remote system.
Save the output of a utility to a file
sort FileName > NewFileName
Cron is for scheduling recurring tasks.
The crond daemon is the background service that enables cron.
Crond checks for jobs in these locations:
- /var/spool/cron (individual user jobs)
- /etc/cron.d (service and application jobs)
- /etc/anacrontab (special case: service and application jobs)
Open cron job editor:
crontab -e FileName
When you save and exit, the crond daemon is restarted and will see your updates.
Crontab uses Vi as its editor.
By default, cron jobs are run as the user who created them.
Example of structure:
# Setup default environment that these scheduled commands will run in
SHELL=/bin/bash
MAILTO=root@example.com
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
# For details see "crontab man"
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command-to-be-executed
# backup using the rsbu program to the internal 4TB HDD and then 4TB external
01 01 * * * /usr/local/bin/rsbu -vbd1 ; /usr/local/bin/rsbu -vbd2
# Set the hardware clock to keep it in sync with the more accurate system clock
03 05 * * * /sbin/hwclock --systohc
# Perform monthly updates on the first of the month
# 25 04 1 * * /usr/bin/dnf -y update
MAILTO sets where the results of the cron job will be sent.
Results will include the output you'd see if you ran the command manually.
Even when you set the PATH, it is recommended to still give the fully qualified path to each command.
Note that semicolon (;) is used to separate multiple commands.
You can leave any/all time specifications as *.
You can list time specifications:
# Run quarterly reports
02 03 1 1,4,7,10 * /usr/local/bin/reports.sh
You can use a range:
# Remind me each hour of the work-day
01 09-17 * * * /usr/local/bin/hourlyreminder.sh
You can run every x-th time-unit:
# Run every 5 minutes, but only during even-numbered hours between 8am and 6pm
*/5 08-18/2 * * * /usr/local/bin/mycronjob.sh
The division must have a remainder of 0.
anacron will run tasks that were skipped because the computer was off or too busy.
Missed jobs will be ran only once - not as many times as were missed.
Search input files for lines containing a match to a given pattern list. Matching lines are (by default) copied to standard output.
grep [options] pattern input_file_names
grep -e patternA -e patternB input_file_names
grep --regex=patternA --regex=patternB input_file_names
or
grep -f pattern_file_name input_file_names
grep --file=pattern_file_name input_file_names
Where pattern_file_name contains a list of patterns, separated by endlines.
Ignore case
-i or --ignore-case
Select non-matching lines instead
-v or --invert-match
Select whole word matches only
-w or --word-regexp
Select whole line matches only
-x or --line-regexp
Output just the count of matching lines per file
-c or --count
Output list of files containing no matches only
-L or --files-without-match
Output list of files containing a match only
-l or --files-with-matches
Stop searching a file after x matching lines are found
-m x or --max-count=x
Print just the matching section of lines
-o or --only-matching
Print line number by each output (starting at 1)
-n or --line-number
Include x lines after each matching line
-A x or --after-context=x
Include x lines before each matching line
-B x or --before-context=x
Include x lines before and after each matching lines
-C x or --context=x or -x
Awk can manipulate text files.
Prints results to STDOUT
awk '{print}' FileName
With nothing else specified, this will just print the file.
Awk will split lines into fields (default delimiter is whitespace).
The fields are accessible as $1, $2, etc.
These are the Field Variables.
$0 holds the entire line.
Print just the 1st and 4th columns of data.
awk '{print $1,$4}' FileName
Specifying a different field delimiter.
awk 'BEGIN { FS="|" } ; { print $1 }' FileName
This will only print lines that contain the regex
awk '/regex here/ {print}' FileName
Diff returns the differences between two files, compared line-by-line.
Diff tells you what changes need to be made to make the files identical.
The files must be pre-sorted.
Cmp will compare two files byte-by-byte.
Comm will compare two files line-by-line.
It outputs (A) the lines just in File 1 (B) the lines just in File 2 (C) the lines in both files.
The files must be pre-sorted.
If the comparison seems to not be working, check how lines are ended. Line-Feed vs Carriage-Return-Line-Feed, for instance.
-1 to suppress column 1 (lines unique to File 1)
-2 to suppress column 2 (lines unique to File 2)
-3 to suppress column 3 (lines shared by both files)
Getting the lines that only appear in File 1
comm -23 File1 File2
--check-order will check that the files are sorted
--nocheck-order will not check that the files are sorted
--output-delimiter=X sets the column delimiter
Sed = Stream EDitor. It is for modifying files automatically.
Replace text based on regular expressions. By default, sed will only replace the FIRST occurrence of a substring in each line.
Replace substring "current" with string "new" throughout old_filename. Save the output to new_filename.
sed s/current/new/ old_filename > new_filename
Use quotes if you have metacharacters in the regular expression.
sed 's/current/new/' old_filename > new_filename
The regular expression delimiter is the first character after s. You can use any delimiter.
sed 's_/usr/bin_/common/bin_' old_filename > new_filename
sed 's:/usr/bin:/common/bin:' old_filename > new_filename
The error "unterminated 's' command" means you forgot a delimiter.
Use & to refer to the matched string, so you can use it in the replacement.
sed 'a/[aeiou]*/(&)/' old_filename > new_filename
Use escaped () to mark multiple patterns for later use. They will be refered to as \1 \2...\9. 9 is the limit.
sed 's:\([aeiou]*\).*:\1:' old_filename > new_filename
Addressing specified which lines to apply the replacement to.
--only make replacements in lines that contain 'IF'
sed '/IF/s/current/new/' old_filename > new_filename
The global flag replaces ALL instances of matching strings in each line.
sed 's/current/new/g' old_filename > new_filename
Replace just the Xth instance in each line.
sed 's/current/new/X' old_filename > new_filename
--just the 4th 'current' is replaced
sed 's/current/new/4' old_filename > new_filename
Option -n means don't print output. Flag p means print only lines that contain a match.
sed -n 's/current/new/p' old_filename > new_filename
--print just lines with matches, like grep
sed -n 's/current/ p' old_filename > new_filename
--print just lines without a match
sed -n 's/current/ !p' old_filename > new_filename
Flag w means print only the matches. If you use multiple flags, w must be the last one.
sed -n 's/current/new/w' old_filename > new_filename
The ignore case flag.
sed 's/current/new/I' old_filename > new_filename
Delete selected text
sed '/text/ d' FileName
Make multiple substitutions at once.
sed -e 's/A/a/' -e 's/B/b/' old_filename > new_filename
--or specify a file of endline-delimited substitution expressions
sed -f pattern_filename old_filename > new_filename
Only apply changes to the Xth line.
sed 'X s/current/new/' old_filename > new_filename
--only the 3rd line
sed '3 s/current/new/' old_filename > new_filename
--only lines 3 through 90 inclusive
sed '3,90 s/current/new/' old_filename > new_filename
--only lines 3 through end of file
sed '3,$ s/current/new/' old_filename > new_filename
Only apply changes where the line starts with '#'.
sed '/^#/ s/current/new/' old_filename > new_filename
Only apply changes that occur between START and STOP. So, the from line that includes START to the next line that includes STOP. This can occur multiple times per file.
sed '/START/,/STOP/ s/current/new/' old_filename > new_filename
--from the 3rd line to the next instance of /STOP/
sed '3,/STOP/ s/current/new/' old_filename > new_filename
Delete all lines from the 3rd to the end of file.
sed '3,$ d' old_filename > new_filename
Remove blank and empty lines
sed '/^\s*$/d' FileName
jq is a command line tool for parsing, selecting, and re-arranging json data.
jq is actually a c library available for any platform.
I'm putting it here because it is very much a command line tool friendly to pipes.
Users are identified by a unique username and a unique id (UID). A UID is a positive integer.
Create a new user
useradd USERNAME
useradd [OPTIONS] USERNAME
User options start with the defaults in "/etc/default/useradd". [OPTIONS] specified in the useradd command will override the defaults.
See also "/etc/login.defs" for password-related settings.
Adding a user will update "/etc/passwd", "/etc/shadow", "/etc/group", and "/etc/gshadow".
Create user's home directory at "/home/USERNAME"
useradd -m USERNAME
useradd --create-home USERNAME
The home dir will be initialized with files copied from "/etc/skel".
Specify a different home directory
useradd -m -d /SOME/PATH USERNAME
useradd --create-home --home /SOME/PATH USERNAME
Specify the user's UID
useradd -u 123 USERNAME
useradd --uid 123 USERNAME
By default, the next available UID will be used.
Specify the user's group
useradd -g 123 USERNAME
useradd --gid 123 USERNAME
By default, a new group will be created with the same name as the user.
Add a comment or description
useradd -c "a b c" USERNAME
useradd --comment "a b c" USERNAME
Comments are saved in "/etc/passwd".
Set the new user's password
passwd USERNAME
Enter the password when prompted.
Change user's home directory
usermod -d /SOME/PATH USERNAME
id USERNAME #display UID, GID, and groups
id -u USERNAME #display UID
id -gn USERNAME #display GID
chown changes the ownership/permissions on a file/directory.
View current permissions
ls -l
chown [OPTIONS] USER[:GROUP] FILE_OR_DIRECTORY
Recursively change ownership, starting from this point and going down
chown -R USER[:GROUP] FILE_OR_DIRECTORY
A shell text editor.
Open a file with vim
vim FILENAME
i = enter Insert mode
Escape = exit Insert mode
(in Normal mode)
w = write/save file to current filename
w FILENAME = write/save file to FILENAME
q = quit vim
q! = quit even with unsaved changes
wq! = write then quit
(in Insert or Normal mode)
h = move left
j = move down
k = move up
l = move right
0 = move to beginning of line
$ = move to end of line
gg = move to beginning of file
G = move to end of file
w = move forward one word
b = move back one word
`. = move to last edit
(in Insert mode)
u = undo last edit
Ctrl-r = redo last edit
dw = delete a word
dd = delete a line
d0 = delete to beginning of line
d$ = delete to end of line
dgg = delete to beginning of file
dG = delete to end of file
Virtual hosting is hosting multiple domain names from one IP address. Apache's virtual host directs requests to different folders, based on the domain name.
Apache's top-level directory for websites is "/var/www"
Create subdirectories here for each domain name. Include a "public_html" directory nested in each one.
Set the ownership on the "public_html" directory to the user who owns that website.
sudo chown -R {user}:{group} /var/www/example.com/public_html
Make sure the new directories are reable by all users. This may already be set correctly.
sudo chmod -R 755 /var/www
Upload an html file to the public_html directory for testing.
Make a copy of "/etc/apache2/sites-available/000-default.conf" for each domain name.
sudo cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/example.com.conf
Edit the file with this template
<VirtualHost *:80>
ServerAdmin admin@example.com
ServerName example.com
ServerAlias www.example.com
DocumentRoot /var/www/example.com/public_html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
Enable the virtual hosts (creates symlinks with sites-enabled)
sudo a2ensite example.com.conf
Restart Apache for the changes to take effect
sudo service apache2 restart
#or
sudo systemctl reload apache2
Disable the virtual host
sudo a2dissite example.com.conf
To test this setup, you can hard-code your local computer's host file. (This is not referring to the linux server, but to the computer you will test with.)
- Linux
- edit "/etc/hosts" to include a line like "111.111.111.111 example.com"
- Windows
- run notepad as Administrator to open file
- edit "c:\windows\system32\drivers\etc\hosts" to include a line like "111.111.111.111 example.com"
(1)
One-time setup: register Microsoft key and feed
wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
sudo mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/ubuntu/18.04/prod.list
sudo mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
sudo chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
sudo chown root:root /etc/apt/sources.list.d/microsoft-prod.list
Install .Net Core runtime
sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install aspnetcore-runtime-2.1
That's it. You do not need the SDK on a production server.
Verify installation
dotnet --info
dotnet --list-sdks
dotnet --list-runtimes
(2)
Deploy the app:
- run a Release build
- copy the build files (found in /bin) to the appropriate folder on the server
- run the app with
dotnet My_App.dll
- navigate to the app in browser on the local server machine at "http://{server_address}:{port}"
(3)
One-time setup: reverse proxy
- recommended for security of dynamic web apps
- required for HTTPS
3.1) Install Apache2 (already done with general server setup)
3.2) Enable Apache2 modules
sudo a2enmod rewrite
sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod headers
sudo a2enmod ssl
sudo service apache2 restart
!! TODO LATER continue in https://odan.github.io/2018/07/17/aspnet-core-2-ubuntu-setup.html from "Configure SSL"
Configure nginx as reverse proxy, to forward certain urls to local services
- leave "/etc/nginx/sites-available/default" alone
- add a conf file for this service
sudo vim /etc/nginx/sites-available/api.{my_app}.com.conf
With contents
server {
listen 5000;
server_name api.{my_app}.com;
root /var/www/{my_app}.com/public_html;
location / {
proxy_pass http://localhost:5000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
Verify setup is ok
sudo nginx -t
Start or restart nginx to pick up changes
sudo service nginx start
sudo service nginx restart
If you make a conf file per port/website, then add a symlink from "/etc/nginx/sites-enabled" to each file in "/etc/nginx/sites-available". One conf file per port.
If you are also running Apache2 for port 80, make sure nginx is not listening for port 80.
!! but now when i start "dotnet my_app.dll" i get Kestral error port already in use !!
- stopped nginx - verified dotnet dll can start again
- maybe I just need to configure proxy in apache2
<VirtualHost *:80>
# The ServerName directive sets the request scheme, hostname and port that
# the server uses to identify itself. This is used when creating
# redirection URLs. In the context of virtual hosts, the ServerName
# specifies what hostname must appear in the request's Host: header to
# match this virtual host. For the default virtual host (this file) this
# value is not decisive as it is used as a last resort host regardless.
# However, you must set it for any further virtual host explicitly.
#ServerName www.example.com
ServerAdmin abound@namesabound.com
ServerName namesabound.com
ServerAlias www.namesabound.com
DocumentRoot /var/www/namesabound.com/public_html
# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the loglevel for particular
# modules, e.g.
#LogLevel info ssl:warn
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
# For most configuration files from conf-available/, which are
# enabled or disabled at a global level, it is possible to
# include a line for only one particular virtual host. For example the
# following line enables the CGI configuration for this host only
# after it has been globally disabled with "a2disconf".
#Include conf-available/serve-cgi-bin.conf
</VirtualHost>
<VirtualHost *:5000>
ProxyPreserveHost On
ProxyPass / http://127.0.0.1:5000/
ProxyPassReverse / http://127.0.0.1:5000/
ErrorLog /var/log/apache2/namesabound-error.log
CustomLog /var/log/apache2/namesabound-access.log common
</VirtualHost>
Verify apache2 config is ok
sudo apachectl configtest
Restart apache2 service
sudo service apache2 restart
https://hbhhathorn.medium.com/install-an-asp-net-core-web-api-on-linux-ubuntu-18-04-and-host-with-nginx-and-ssl-2ed9df7371fb
https://webdock.io/en/docs/how-guides/shared-hosting-multiple-websites/how-configure-nginx-to-serve-multiple-websites-single-vps
https://odan.github.io/2018/07/17/aspnet-core-2-ubuntu-setup.html
(4)
Setup the app to run and be monitored in the background
Install Supervisor
sudo apt-get install supervisor
Initialize folder "/var/aspnetcore" for all .Net Core apps
sudo mkdir /var/aspnetcore/{my_app}
Grant access to the user for this site
sudo chown -R {user}:{group} /var/aspnetcore/{my_app}
Copy your build (/bin) files into this directory.
Init supervisor conf file
sudo vim /etc/supervisor/conf.d/{my_app}.conf
[program:{process_name}]
command=/usr/bin/dotnet /var/aspnetcore/{my_app}/{my_app}.dll --urls "http://*:5000"
directory=/var/aspnetcore/{my_app}
autostart=true
autorestart=true
stderr_logfile=/var/log/{my_app}.err.log
stdout_logfile=/var/log/{my_app}.out.log
environment=ASPNETCORE_ENVIRONMENT=Production,CONNECTIONSTRINGS__DEFAULT="{connection_string}"
user={app_user}
stopsignal=INT
Note that "environment" is used to set environment variables for this application.
environment=KEYA=valA,KEYB=valB
The values can optionally be enclosed in double quotes ("")
Start Supervisor
sudo service supervisor start
Watch Supervisor logs to see app startup
sudo tail -f /var/log/supervisor/supervisord.log
sudo tail -f /var/log/{my_app}.out.log
?? supervisor log says it is running as root because no user was specified, but i did specify user ??
!! it works! I can reach the web api from another computer on port 5000 !
ok, the service part stopped working after rebooting the server
stay calm
the port 80 static html parts are still working
- the Supervisor app is in a start/die loop on "address 5000 already in use"
Install Net Tools to see what is using that port
sudo apt install net-tools
netstat -ltnp | grep :5000
It says "tcp6" which made me think of the nginx configuration that included ip4 and ip6
- manually stopped nginx
- confirmed service is now reachable from another computer on port 5000
- removing the port 5000 config from nginx
- restarted nginx, confirmed 5000 service is still available externally
- trying a reboot again
- hmm, now port 80 sites AND 5000 service are having trouble
- shit I can't connect to my server
- oh shit
- ok, nobody panic, I manually turned it off and on again and everything is working
- trying system reboot again
- everything is still working
Supervisor hosts locally only
Example /etc/supervisor/conf.d/{sitename}.conf
[program:{sitename}]
command=/usr/bin/dotnet /var/aspnetcore/{sitename}/{program}.dll --urls "http://127.0.0.1:5100"
directory=/var/aspnetcore/{sitename}
autostart=true
autorestart=true
stderr_logfile=/var/log/{sitename}.err.log
stdout_logfile=/var/log/{sitename}.out.log
environment=ASPNETCORE_ENVIRONMENT=Production
user=abound
stopsignal=INT
Apache2 listens on port 80
Example /etc/apache2/sites-available/{sitename}.conf
<VirtualHost *:80>
ServerName {domainname}
ProxyPreserveHost On
ProxyPass / http://127.0.0.1:5100/
ProxyPassReverse / http://127.0.0.1:5100/
ErrorLog /var/log/apache2/{sitename}-error.log
CustomLog /var/log/apache2/{sitename}-access.log common
</VirtualHost>
Directory browsing is when the web user can view the list of files in a directory on your server.
To disable directory browsing for all sites hosted on the server:
- open "/etc/apache2/apache2.conf"
- change line "Options Indexes FollowSymLinks" to "Options FollowSymLinks"
- restart apache2 service for changes to take effect
Instructions for Ubuntu + Apache2
1) Generate a Private Key file and CSR (Certificate Signing Request)
- command prompt: sudo openssl req -new -newkey rsa:2048 -nodes -keyout {name}.key -out {name}.csr
Place these files in a location that only Root can access.
The {name}.key file is the private key file.
The {name}.csr file is the certificate signing request. This is used to order an SSL Certificate from a signing authority, and to encrypt messages that only the private key can decrypt.
When asked for the Common Name, enter the fully qualified Domain Name.
Ex: google.com
Ex: *.google.com to include all subdomains
Country Code list: https://www.digicert.com/kb/ssl-certificate-country-codes.htm
**instructions paused, looking at certbot option
https://certbot.eff.org/
easy generation and installation of certificates, with auto-renewal