home | prev | next

Apache deployment

Initial Setup

This document describes how to deploy the CDCS using uWSGi and Apache. Currently you can deploy the Material Data Configuration System (MDCS) and the Materials Registry and Repository (MRR). Both are deployed using uWSGI, which is a fast, self-healing and developer/sysadmin-friendly application container server coded in pure C. It is served thru Apache proxies. This document will show how to serve using Apache.

  1. Upload project to desired directory (with http access), in this example we called it mdcs. If you are deploying this, this directory must be in the Server Root.

    $ git clone https://github.com/usnistgov/mdcs mdcs --branch master
    cd mdcs
  2. Install required packages for the project. This includes Python modules or required software such as databases.

  3. Run installation scripts for project provided with the project download.

  4. Install Apache 2.4 (information in the guide is for Apache 2.4) to serve MDCS and MMR over the web.

Virtual Environment

It is recommended you use a virtual environment that can be built separate from the system binaries, to make sure the apache user can access all needed binaries. Again, the Environment MUST BE in the Server Root path, in this example we have created the environment in /var/www/html/env,with a Server Root of /var/www/html and are using Anaconda, but any environment creator will work.

1. Install Anaconda
$ wget https://repo.anaconda.com/archive/Anaconda2-5.1.0-Linux-x86_64.sh
$ bash Anaconda2-5.2.0-Linux-x86_64.sh
2. Update Conda
$ conda update -n base conda
3. Create Environment with python 2.7
$ conda create -y -p /var/www/html/env python=2.7
4. Enable environment
source activate /var/www/html/env

Databases and Django

1. Install mongodb
$ sudo apt-get install -y mongodb-org
2. Start Mongo
3. Create Admin User
$ mongo
use admin
db.createUser(
{
    user: "admin",
    pwd: "admin",
    roles: [ { role: "userAdminAnyDatabase", db: "admin"},"backup","restore"]
}
)
exit
4. Create MDCS user

6. Celery, Redit, and Git

Celery, Redis and Git may already be installed on your system, if they are, you can use what is installed, otherwise install:

1. Install Git
$ apt-get install git
2. Install Celery using pip:
$ pip install Celery
3. Run Celery
$ celery worker -E --app=mdcs -l info -B --purge --logfile=/home/www/html/mdcs/logs/celery.log --workdir=/home/www/html/mdcs --detach
Note: mdcs is the project name
4. Install Redis:
$ wget http://download.redis.io/redis-stable.tar.gz
$ tar xvzf redis-stable.tar.gz
cd redis-stable
$ make install
4. Run Redis
$ redis-server --daemonize yes

7. uWSGI

The following is the config file for uWSGi distributed with CDCS (mdcs/mdcs.ini). Again, it is installed in /var/www/html/mdcs/ which MUST be in the Server Root path.

In this example: ServerRoot /var/www/html and it runs as user/group www-data/www-data

mdcs.ini:

[uwsgi]
protocol=http
buffer-size=32768
master = true
processes = 5
socket=XXX.XXX.XXX.XXX(this is your IP address):8000 (this is the port)
chmod-socket = 664
vacuum = true
die-on-term = true
pidfile=/var/run/mdcs.pid
chdir=/var/www/html/mdcs
pythonpath=/var/www/html/env/lib/python2.7/site-packages
wsgi-file=/var/www/html/mdcs/mdcs/mdcs.wsgi
uid=www-data
gid=www-data
check-static=/var/www/html/mdcs
thunder-lock=true
enable-threads=true
virtualenv=/var/www/html/env
static-map=/static=/var/www/html/mdcs/static.prod
static-check=/var/www/html/mdcs/static.prod
static-check=/var/www/html/mdcs
daemonize=/var/www/html/mdcs/logs/uwsgi.log
Install uwsgi
$ pip install uwsgi
Start MDCS
$ uwsgi --ini /var/www/html/mdcs/mdcs/mdcs.ini

Settings.py

Make the following edits to settings.py.

1. Set secret key
SECRET_KEY = <secret_key>
2. Set host name
ALLOWED_HOSTS = ['HOSTNAME.SITE.COM']
3. Activate HTTPS
os.environ['HTTPS'] = 'on'
4. Secure cookies
CSRF_COOKIE_SECURE = True
CSRF_COOKIE_AGE = None
SESSION_COOKIE_SECURE = True
SESSION_EXPIRE_AT_BROWSER_CLOSE = True
SESSION_COOKIE_AGE = 604800
5. Set x-frame options
X_FRAME_OPTIONS = 'SAMEORIGIN'
6. Set User Database or use default
** PostGRES   (Can be MYSQL also)**
DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'USER':"mgiUser",
            'PASSWORD': "pass4mgiUser123",
            'NAME': 'mgi',
        }
    }
 
Or Use the Default Sqlite
 
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}
 
 

Host

  1. Install Apache 2.4 on host.

  2. Install mod_wsgi for Apache.

  3. Create or upload certificate, key and certificate chain for HTTPS configuration.

NOTE: Chmod 600 [account] the certificate, key and certificate chain.

Apache

1. Install Apache packages
  $ sudo apt-get update && sudo apt-get upgrade
  $ sudo apt-get install apache2 apache2-doc apache2-utils apache2-dev
2. Apache2.conf
3. uwsgi support:
  $ apt-get install libapache2-mod-uwsgi
4. Perl support:
  $ sudo apt-get install $ libapache2-mod-perl2
5. Python support:
  $ sudo apt-get install libapache2-mod-python
6. open firewall ports 80 & 443
  $ sudo ufw allow 'Apache Full'
  $ sudo ufw status
7. DO NOT USE with self-signed Certs
  SSLUseStapling On
  SSLStaplingCache shmcb:logs/ssl_stapling(32768)
8. Install mod_wsgi
9. Configure apache2.conf
  #
  # The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
  #
  LockFile ${APACHE_LOCK_DIR}/accept.lock
Pid file
  #
  # PidFile: The file in which the server should record its process
  # identification number when it starts.
  # This needs to be set in /etc/apache2/envvars
  #
  PidFile ${APACHE_PID_FILE}
Time out and keep-alive configurations
  #
  # Timeout: The number of seconds before receives and sends time out.
  #
  Timeout 300
 
  #
  # KeepAlive: Whether or not to allow persistent connections (more than
  # one request per connection). Set to "Off" to deactivate.
  #
  KeepAlive On
 
  #
  # MaxKeepAliveRequests: The maximum number of requests to allow
  # during a persistent connection. Set to 0 to allow an unlimited amount.
  # We recommend you leave this number high, for maximum performance.
  #
  MaxKeepAliveRequests 100
 
  #
  # KeepAliveTimeout: Number of seconds to wait for the next request from the
  # same client on the same connection.
  #
  KeepAliveTimeout 5
Server-Pool Size Regulation (MPM specific)
Prefork MPM
  # StartServers: number of server processes to start
  # MinSpareServers: minimum number of server processes which are kept spare
  # MaxSpareServers: maximum number of server processes which are kept spare
  # MaxClients: maximum number of server processes allowed to start
  # MaxRequestsPerChild: maximum number of requests a server process serves
  <IfModule mpm_prefork_module>
      StartServers          5
      MinSpareServers       5
      MaxSpareServers      10
      MaxClients          150
      MaxRequestsPerChild   0
  </IfModule>
Worker MPM
  # StartServers: initial number of server processes to start
  # MinSpareThreads: minimum number of worker threads which are kept spare
  # MaxSpareThreads: maximum number of worker threads which are kept spare
  # ThreadLimit: ThreadsPerChild can be changed to this maximum value during a
  #              graceful restart. ThreadLimit can only be changed by stopping
  #              and starting Apache.
  # ThreadsPerChild: constant number of worker threads in each server process
  # MaxClients: maximum number of simultaneous client connections
  # MaxRequestsPerChild: maximum number of requests a server process serves
  <IfModule mpm_worker_module>
      StartServers          2
      MinSpareThreads      25
      MaxSpareThreads      75
      ThreadLimit          64
      ThreadsPerChild      25
      MaxClients          150
      MaxRequestsPerChild   0
  </IfModule>
Event MPM
  # StartServers: initial number of server processes to start
  # MinSpareThreads: minimum number of worker threads which are kept spare
  # MaxSpareThreads: maximum number of worker threads which are kept spare
  # ThreadsPerChild: constant number of worker threads in each server process
  # MaxClients: maximum number of simultaneous client connections
  # MaxRequestsPerChild: maximum number of requests a server process serves
  <IfModule mpm_event_module>
      StartServers          2
      MinSpareThreads      25
      MaxSpareThreads      75
      ThreadLimit          64
      ThreadsPerChild      25
      MaxClients          150
      MaxRequestsPerChild   0
  </IfModule>
Apache environment variables
  # These need to be set in /etc/apache2/envvars
  User ${APACHE_RUN_USER}
  Group ${APACHE_RUN_GROUP}
Access to files
  #
  # AccessFileName: The name of the file to look for in each directory
  # for additional configuration directives.  See also the AllowOverride
  # directive.
  #
 
  AccessFileName .htaccess
  #
  # The following lines prevent .htaccess and .htpasswd files from being
  # viewed by Web clients.
  #
  <Files ~ "^\.ht">
      Order allow,deny
      Deny from all
      Satisfy all
  </Files>
MIME types
  #
  # DefaultType is the default MIME type the server will use for a document
  # if it cannot otherwise determine one, such as from filename extensions.
  # If your server contains mostly text or HTML documents, "text/plain" is
  # a good value.  If most of your content is binary, such as applications
  # or images, you may want to use "application/octet-stream" instead to
  # keep browsers from trying to display binary files as though they are
  # text.
  #
  # It is also possible to omit any default MIME type and let the
  # client's browser guess an appropriate action instead. Typically the
  # browser will decide based on the file's extension then. In cases
  # where no good assumption can be made, letting the default MIME type
  # unset is suggested  instead of forcing the browser to accept
  # incorrect  metadata.
  #
  DefaultType None
Host name lookups
  #
  # HostnameLookups: Log the names of clients or just their IP addresses
  # e.g., www.apache.org (on) or 204.62.129.132 (off).
  # The default is off because it'd be overall better for the net if people
  # had to knowingly turn this feature on, since enabling it means that
  # each client request will result in AT LEAST one lookup request to the
  # nameserver.
  #
  HostnameLookups Off
Error Log
  # ErrorLog: The location of the error log file.
  # If you do not specify an ErrorLog directive within a <VirtualHost>
  # container, error messages relating to that virtual host will be
  # logged here.  If you *do* define an error logfile for a <VirtualHost>
  # container, that host's errors will be logged there and not here.
  #
  ErrorLog ${APACHE_LOG_DIR}/error.log
  #
  # LogLevel: Control the number of messages logged to the error_log.
  # Possible values include: debug, info, notice, warn, error, crit,
  # alert, emerg.
  #
  LogLevel warn
Module configuration
  # Include module configuration:
  Include mods-enabled/*.load
  Include mods-enabled/*.conf
Port configuration
  # Include list of ports to listen on and which to use for name based vhosts
  Include ports.conf
Log variable name references
  #
  # The following directives define some format nicknames for use with
  # a CustomLog directive (see below).
  # If you are behind a reverse proxy, you might want to change %h into %{X-Forwarded-For}i
  #
  LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
  LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
  LogFormat "%h %l %u %t \"%r\" %>s %O" common
  LogFormat "%{Referer}i -> %U" referer
  LogFormat "%{User-agent}i" agent
Inclusions
  # Include of directories ignores editors' and dpkg's backup files,
  # see the comments above for details.
 
  # Include generic snippets of statements
  Include conf.d/
Virtual host configurations
  # Include the virtual host configurations:
  Include sites-enabled/
 
  <Directory />
      Options FollowSymLinks
      AllowOverride None
      Order deny,allow
      Deny from all
  </Directory>
 

Configure httpd.conf

1. Server
ServerRoot "/opt/httpd/current"
ServerAdmin HOSTADMIN@HOST.COM
2. Load modules
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule filter_module modules/mod_filter.so
LoadModule mime_module modules/mod_mime.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule headers_module modules/mod_headers.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule status_module modules/mod_status.so
LoadModule info_module modules/mod_info.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule dir_module modules/mod_dir.so
LoadModule alias_module modules/mod_alias.so
LoadModule wsgi_module modules/mod_wsgi.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
3. User/group settings
<IfModule unixd_module>
    User mdcsuser
    Group mdcsuser
</IfModule>
4. Directory configurations
<Directory />
AllowOverride none
Require all denied
</Directory>
5. Directory index file
<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>
6. File access
<Files ".ht*">
    Require all denied
</Files>
7. Error logs
ErrorLog "logs/error_log"
LogLevel warn
8. Log configurations
<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common
    <IfModule logio_module>
        LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>
    CustomLog "logs/access_log" common
</IfModule>
9. Mime configurations
<IfModule mime_module>
    TypesConfig conf/mime.types
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
</IfModule>
10. SSL configurations
<IfModule ssl_module>
    SSLRandomSeed startup builtin
    SSLRandomSeed connect builtin
</IfModule>
11. Server tokens
ServerTokens prod
12. Tracing configuration
TraceEnable off
13. Rewrite settings
# RewriteEngine On
# RewriteCond %{HTTPS} off
# RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
14. Location configurations
<Location "/server-status">
    SetHandler server-status
    Require local
</Location>
<Location "/server-info">
    SetHandler server-info
    Require local
</Location>
15. SSL configurations
    SSLCipherSuite HIGH:MEDIUM:!MD5:!RC4:!SHA1
    SSLProxyCipherSuite HIGH:MEDIUM:!MD5:!RC4:!SHA1
    SSLHonorCipherOrder on
    SSLProtocol TLSv1.1 +TLSv1.2
    SSLProxyProtocol TLSv1.1 +TLSv1.2
    SSLPassPhraseDialog  builtin
    SSLSessionCache        "shmcb:/opt/httpd/current/logs/ssl_scache(512000)"
    SSLSessionCacheTimeout  300
    SSLUseStapling On
    SSLStaplingCache "shmcb:/opt/httpd/current/logs/ssl_stapling(32768)"
16. Global configurations
    # Other global configuration directives
    ServerTokens ProductOnly
    TraceEnable off
    #Header always append X-Frame-Options "SAMEORIGIN"
17. Virtual host configuration files
    # Virtual host include files
    Include "/basedirectory/*-vhost.conf"

Configure *-vhost.conf

1. Port configurations
# Ports to listen on
Listen 443
Listen 80
Port 443
2. Virtual host configurations
<VirtualHost _default_:443>
    RequestHeader set X_FORWARDED_PROTO 'https'
    ServerAdmin ADMIN@SITE.COM
    ErrorLog "/basedirectory/projectdirectory/logs/error_log"
    TransferLog "/basedirectory/projectdirectory/access_log"
    SSLEngine on
    SSLCertificateFile "/basedirectory/CERTIFICATE.crt"
    SSLCertificateKeyFile "/basedirectory/KEY.key"
    SSLCertificateChainFile "basedirectory/intermediate-ca.crt"
    <FilesMatch "\.(cgi|shtml|phtml|php)$">
        SSLOptions +StdEnvVars
    </FilesMatch>
    BrowserMatch "MSIE [2-5]" \
    nokeepalive ssl-unclean-shutdown \
    downgrade-1.0 force-response-1.0
    CustomLog "/basedirectory/projectdirectory/logs/ssl_request_log" \
    "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
    WSGIScriptAlias / /basedirectory/projectdirectory/mdcs/mgi/wsgi.py
    <Directory /basedirectory/projectdirectory/mdcs/mgi>
        <Files wsgi.py>
            Require all granted
        </Files>
        <LimitExcept GET POST HEAD>
            Require all denied
            # Order allow,deny
            # deny from all
        </LimitExcept>
    </Directory>
    ServerName HOSTNAME.SITE.COM
    <Directory /basedirectory/projectdirectory/mdcs/static/>
    Options -Indexes -Includes -ExecCGI -FollowSymLinks
    Require all granted
    <LimitExcept GET POST HEAD>
        Require all denied
        # Order allow,deny
        # deny from all
    </LimitExcept>
    </Directory>
    path=/path/to/current/lib/python2.7
</VirtualHost>
Port 80 (rerouted to secure port)
<VirtualHost *:80>
    ServerName HOSTNAME.SITE.COM
    Redirect permanent / https://HOSTNAME.SITE.COM
</VirtualHost>
 

Postgres

  1. If Postgres is not installed, install the package.
  2. Create database: mgi
  3. Create username and password for PostGRES
  4. Enter connection details in settings.py.
 DATABASES = {
     'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'USER':"<postgres_user>",
   'PASSWORD': "<postgres_password>",
   'NAME': 'mgi',
    }
  }