This document describes how to deploy the CDCS using uWSGi and Apache. Currently you can deploy the Material Data Configuration System (MDCS) and the Materials Registry and Repository (MRR). Both are deployed using uWSGI, which is a fast, self-healing and developer/sysadmin-friendly application container server coded in pure C. It is served thru Apache proxies. This document will show how to serve using Apache.
Upload project to desired directory (with http access), in this example we called it mdcs. If you are deploying this, this directory must be in the Server Root.
$ git clone https://github.com/usnistgov/mdcs mdcs --branch master$ cd mdcs
Install required packages for the project. This includes Python modules or required software such as databases.
Run installation scripts for project provided with the project download.
Install Apache 2.4 (information in the guide is for Apache 2.4) to serve MDCS and MMR over the web.
It is recommended you use a virtual environment that can be built separate from the system binaries, to make sure the apache user can access all needed binaries. Again, the Environment MUST BE in the Server Root path, in this example we have created the environment in /var/www/html/env,with a Server Root of /var/www/html and are using Anaconda, but any environment creator will work.
$ wget https://repo.anaconda.com/archive/Anaconda2-5.1.0-Linux-x86_64.sh$ bash Anaconda2-5.2.0-Linux-x86_64.sh
$ conda update -n base conda
$ conda create -y -p /var/www/html/env python=2.7
$ source activate /var/www/html/env
$ sudo apt-get install -y mongodb-org
$ mongod --config /var/www/html/mdcs/mongo/conf/mongodb.conf --fork
$ mongouse admindb.createUser
$ mongouse mgidb.createUserexit
$ git clone https://github.com/MongoEngine/django-mongoengine.git$ cd django-mongoengine$ python setup.py install$ apt-get install libpcre3$ apt-get install libpcre3-de$ apt-get install uwsgi-plugin-python$ apt-get install libapache2-mod-wsgi
Celery, Redis and Git may already be installed on your system, if they are, you can use what is installed, otherwise install:
$ apt-get install git
$ pip install Celery
$ celery worker -E --app=mdcs -l info -B --purge --logfile=/home/www/html/mdcs/logs/celery.log --workdir=/home/www/html/mdcs --detachNote: mdcs is the project name
$ wget http://download.redis.io/redis-stable.tar.gz$ tar xvzf redis-stable.tar.gz$ cd redis-stable$ make install
$ redis-server --daemonize yes
The following is the config file for uWSGi distributed with CDCS (mdcs/mdcs.ini). Again, it is installed in /var/www/html/mdcs/ which MUST be in the Server Root path.
In this example: ServerRoot /var/www/html and it runs as user/group www-data/www-data
mdcs.ini:
[uwsgi]protocol=httpbuffer-size=32768master = trueprocesses = 5socket=XXX.XXX.XXX.XXX(this is your IP address):8000 (this is the port)chmod-socket = 664vacuum = truedie-on-term = truepidfile=/var/run/mdcs.pidchdir=/var/www/html/mdcspythonpath=/var/www/html/env/lib/python2.7/site-packageswsgi-file=/var/www/html/mdcs/mdcs/mdcs.wsgiuid=www-datagid=www-datacheck-static=/var/www/html/mdcsthunder-lock=trueenable-threads=truevirtualenv=/var/www/html/envstatic-map=/static=/var/www/html/mdcs/static.prodstatic-check=/var/www/html/mdcs/static.prodstatic-check=/var/www/html/mdcsdaemonize=/var/www/html/mdcs/logs/uwsgi.log
$ pip install uwsgi
$ uwsgi --ini /var/www/html/mdcs/mdcs/mdcs.ini
Make the following edits to settings.py.
SECRET_KEY = <secret_key>
ALLOWED_HOSTS = ['HOSTNAME.SITE.COM']
os.environ['HTTPS'] = 'on'
CSRF_COOKIE_SECURE = TrueCSRF_COOKIE_AGE = NoneSESSION_COOKIE_SECURE = TrueSESSION_EXPIRE_AT_BROWSER_CLOSE = TrueSESSION_COOKIE_AGE = 604800
X_FRAME_OPTIONS = 'SAMEORIGIN'
** PostGRES (Can be MYSQL also)**DATABASES = {'default': {'ENGINE': 'django.db.backends.postgresql_psycopg2','USER':"mgiUser",'PASSWORD': "pass4mgiUser123",'NAME': 'mgi',}}Or Use the Default SqliteDATABASES = {'default': {'ENGINE': 'django.db.backends.sqlite3','NAME': os.path.join(BASE_DIR, 'db.sqlite3'),}}
Install Apache 2.4 on host.
Install mod_wsgi for Apache.
Create or upload certificate, key and certificate chain for HTTPS configuration.
NOTE: Chmod 600 [account] the certificate, key and certificate chain.
$ sudo apt-get update && sudo apt-get upgrade$ sudo apt-get install apache2 apache2-doc apache2-utils apache2-dev
KeepAlive Off
Edit /etc/apache2/mods-available/mpm_prefork.conf
# prefork MPM# StartServers: number of server processes to start# MinSpareServers: minimum number of server processes which are kept spare# MaxSpareServers: maximum number of server processes which are kept spare# MaxRequestWorkers: maximum number of server processes allowed to start# MaxConnectionsPerChild: maximum number of requests a server process serves<IfModule mpm_prefork_module>StartServers 4MinSpareServers 20MaxSpareServers 40MaxRequestWorkers 200MaxConnectionsPerChild 4500</IfModule>
Edit /etc/apache2/mods-available/mpm_event.conf
# event MPM# StartServers: initial number of server processes to start# MinSpareThreads: minimum number of worker threads which are kept spare# MaxSpareThreads: maximum number of worker threads which are kept spare# ThreadsPerChild: constant number of worker threads in each server process# MaxRequestWorkers: maximum number of worker threads# MaxConnectionsPerChild: maximum number of requests a server process serves<IfModule mpm_event_module>StartServers 2MinSpareThreads 25MaxSpareThreads 75ThreadLimit 64ThreadsPerChild 25MaxRequestWorkers 150MaxConnectionsPerChild 3000</IfModule>
Enable modules:
$ sudo a2dismod mpm_event$ sudo a2enmod mpm_prefork
$ apt-get install libapache2-mod-uwsgi
$ sudo apt-get install $ libapache2-mod-perl2
$ sudo apt-get install libapache2-mod-python
$ sudo ufw allow 'Apache Full'$ sudo ufw status
SSLUseStapling OnSSLStaplingCache shmcb:logs/ssl_stapling(32768)
When Apache starts up it has to read the various Certificate (see SSLCertificateFile) and Private Key (see SSLCertificateKeyFile) files of the SSL-enabled virtual servers.
Because for security reasons the Private Key files are usually encrypted, mod_ssl needs to query the administrator for a Pass Phrase in order to decrypt those files.
SSLPassPhraseDialog builtin
This configures the storage type of the global/inter-process SSL Session Cache.
This cache is an optional facility which speeds up parallel request processing. For requests to the same server process (via HTTP keep-alive), OpenSSL already caches the SSL session information locally. But because modern clients request inlined images and other data via parallel requests (usually up to four parallel requests are common) those requests are served by different pre-forked server processes.
This makes use of a high-performance cyclic buffer (approx. size bytes in size) inside a shared memory segment in RAM (established via /path/to/datafile) to synchronize the local OpenSSL memory caches of the server processes. This is the recommended session cache. To use this, ensure that mod_socache_shmcb is loaded.
SSLSessionCache "shmcb:logs/ssl_scache(512000)"SSLSessionCacheTimeout 300
This directive overrides the behavior of TRACE for both the core server and mod_proxy. The default TraceEnable on permits TRACE requests per RFC 2616, which disallows any request body to accompany the request. TraceEnable off causes the core server and mod_proxy to return a 405 (Method not allowed) error #to the client.
TraceEnable off
This directive controls whether Server response header field which is sent back to clients includes a description of the generic OS-type of the server as well as information about compiled-in modules.
# ServerTokens Prod[uctOnly]# Server sends (e.g.): Server: ApacheServerTokens ProductOnly
The mod_wsgi package provides an Apache module that implements a WSGI compliant interface for hosting Python based web applications on top of the Apache web server.`server
Note it requires apache2-dev installed above
$ pip install mod_wsgi
Or,
$ sudo apt-get install libapache2-mod-wsgi
If you get Python.h not found:
$ sudo apt-get install python-dev
Create or upload certificate, key and certificate chain for HTTPS configuration.
a2enmod ssla2enmod ssla2enmod headers
See http://httpd.apache.org/docs/2.2/ for detailed information about the directives and /usr/share/doc/apache2-common/README.Debian.gz about
Summary of how the Apache 2 configuration works in Debian:
# /etc/apache2/# |-- apache2.conf# | `-- ports.conf# |-- mods-enabled# | |-- *.load# | `-- *.conf# |-- conf.d# | `-- *# `-- sites-enabled# `-- *
apache2.conf:
The binary:
# ServerRoot: The top of the directory tree under which the server's# configuration, error, and log files are kept.## NOTE! If you intend to place this on an NFS (or otherwise network)# mounted filesystem then please read the LockFile documentation (available# at <URL:http://httpd.apache.org/docs/2.2/mod/mpm_common.html#lockfile>);# you will save yourself a lot of trouble.## Do NOT add a slash at the end of the directory path.##ServerRoot "/etc/apache2"
## The accept serialization lock file MUST BE STORED ON A LOCAL DISK.#LockFile ${APACHE_LOCK_DIR}/accept.lock
## PidFile: The file in which the server should record its process# identification number when it starts.# This needs to be set in /etc/apache2/envvars#PidFile ${APACHE_PID_FILE}
## Timeout: The number of seconds before receives and sends time out.#Timeout 300## KeepAlive: Whether or not to allow persistent connections (more than# one request per connection). Set to "Off" to deactivate.#KeepAlive On## MaxKeepAliveRequests: The maximum number of requests to allow# during a persistent connection. Set to 0 to allow an unlimited amount.# We recommend you leave this number high, for maximum performance.#MaxKeepAliveRequests 100## KeepAliveTimeout: Number of seconds to wait for the next request from the# same client on the same connection.#KeepAliveTimeout 5
# StartServers: number of server processes to start# MinSpareServers: minimum number of server processes which are kept spare# MaxSpareServers: maximum number of server processes which are kept spare# MaxClients: maximum number of server processes allowed to start# MaxRequestsPerChild: maximum number of requests a server process serves<IfModule mpm_prefork_module>StartServers 5MinSpareServers 5MaxSpareServers 10MaxClients 150MaxRequestsPerChild 0</IfModule>
# StartServers: initial number of server processes to start# MinSpareThreads: minimum number of worker threads which are kept spare# MaxSpareThreads: maximum number of worker threads which are kept spare# ThreadLimit: ThreadsPerChild can be changed to this maximum value during a# graceful restart. ThreadLimit can only be changed by stopping# and starting Apache.# ThreadsPerChild: constant number of worker threads in each server process# MaxClients: maximum number of simultaneous client connections# MaxRequestsPerChild: maximum number of requests a server process serves<IfModule mpm_worker_module>StartServers 2MinSpareThreads 25MaxSpareThreads 75ThreadLimit 64ThreadsPerChild 25MaxClients 150MaxRequestsPerChild 0</IfModule>
# StartServers: initial number of server processes to start# MinSpareThreads: minimum number of worker threads which are kept spare# MaxSpareThreads: maximum number of worker threads which are kept spare# ThreadsPerChild: constant number of worker threads in each server process# MaxClients: maximum number of simultaneous client connections# MaxRequestsPerChild: maximum number of requests a server process serves<IfModule mpm_event_module>StartServers 2MinSpareThreads 25MaxSpareThreads 75ThreadLimit 64ThreadsPerChild 25MaxClients 150MaxRequestsPerChild 0</IfModule>
# These need to be set in /etc/apache2/envvarsUser ${APACHE_RUN_USER}Group ${APACHE_RUN_GROUP}
## AccessFileName: The name of the file to look for in each directory# for additional configuration directives. See also the AllowOverride# directive.#AccessFileName .htaccess
## The following lines prevent .htaccess and .htpasswd files from being# viewed by Web clients.#<Files ~ "^\.ht">Order allow,denyDeny from allSatisfy all</Files>
## DefaultType is the default MIME type the server will use for a document# if it cannot otherwise determine one, such as from filename extensions.# If your server contains mostly text or HTML documents, "text/plain" is# a good value. If most of your content is binary, such as applications# or images, you may want to use "application/octet-stream" instead to# keep browsers from trying to display binary files as though they are# text.## It is also possible to omit any default MIME type and let the# client's browser guess an appropriate action instead. Typically the# browser will decide based on the file's extension then. In cases# where no good assumption can be made, letting the default MIME type# unset is suggested instead of forcing the browser to accept# incorrect metadata.#DefaultType None
## HostnameLookups: Log the names of clients or just their IP addresses# e.g., www.apache.org (on) or 204.62.129.132 (off).# The default is off because it'd be overall better for the net if people# had to knowingly turn this feature on, since enabling it means that# each client request will result in AT LEAST one lookup request to the# nameserver.#HostnameLookups Off
# ErrorLog: The location of the error log file.# If you do not specify an ErrorLog directive within a <VirtualHost># container, error messages relating to that virtual host will be# logged here. If you *do* define an error logfile for a <VirtualHost># container, that host's errors will be logged there and not here.#ErrorLog ${APACHE_LOG_DIR}/error.log
## LogLevel: Control the number of messages logged to the error_log.# Possible values include: debug, info, notice, warn, error, crit,# alert, emerg.#LogLevel warn
# Include module configuration:Include mods-enabled/*.loadInclude mods-enabled/*.conf
# Include list of ports to listen on and which to use for name based vhostsInclude ports.conf
## The following directives define some format nicknames for use with# a CustomLog directive (see below).# If you are behind a reverse proxy, you might want to change %h into %{X-Forwarded-For}i#LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combinedLogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combinedLogFormat "%h %l %u %t \"%r\" %>s %O" commonLogFormat "%{Referer}i -> %U" refererLogFormat "%{User-agent}i" agent
# Include of directories ignores editors' and dpkg's backup files,# see the comments above for details.# Include generic snippets of statementsInclude conf.d/
# Include the virtual host configurations:Include sites-enabled/<Directory />Options FollowSymLinksAllowOverride NoneOrder deny,allowDeny from all</Directory>
ServerRoot "/opt/httpd/current"ServerAdmin HOSTADMIN@HOST.COM
LoadModule authn_file_module modules/mod_authn_file.soLoadModule authn_core_module modules/mod_authn_core.soLoadModule authz_host_module modules/mod_authz_host.soLoadModule authz_groupfile_module modules/mod_authz_groupfile.soLoadModule authz_user_module modules/mod_authz_user.soLoadModule authz_core_module modules/mod_authz_core.soLoadModule access_compat_module modules/mod_access_compat.soLoadModule auth_basic_module modules/mod_auth_basic.soLoadModule reqtimeout_module modules/mod_reqtimeout.soLoadModule filter_module modules/mod_filter.soLoadModule mime_module modules/mod_mime.soLoadModule log_config_module modules/mod_log_config.soLoadModule env_module modules/mod_env.soLoadModule headers_module modules/mod_headers.soLoadModule setenvif_module modules/mod_setenvif.soLoadModule version_module modules/mod_version.soLoadModule unixd_module modules/mod_unixd.soLoadModule status_module modules/mod_status.soLoadModule info_module modules/mod_info.soLoadModule autoindex_module modules/mod_autoindex.soLoadModule dir_module modules/mod_dir.soLoadModule alias_module modules/mod_alias.soLoadModule wsgi_module modules/mod_wsgi.soLoadModule rewrite_module modules/mod_rewrite.soLoadModule socache_shmcb_module modules/mod_socache_shmcb.so
<IfModule unixd_module>User mdcsuserGroup mdcsuser</IfModule>
<Directory />AllowOverride noneRequire all denied</Directory>
<IfModule dir_module>DirectoryIndex index.html</IfModule>
<Files ".ht*">Require all denied</Files>
ErrorLog "logs/error_log"LogLevel warn
<IfModule log_config_module>LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedLogFormat "%h %l %u %t \"%r\" %>s %b" common<IfModule logio_module>LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio</IfModule>CustomLog "logs/access_log" common</IfModule>
<IfModule mime_module>TypesConfig conf/mime.typesAddType application/x-compress .ZAddType application/x-gzip .gz .tgz</IfModule>
<IfModule ssl_module>SSLRandomSeed startup builtinSSLRandomSeed connect builtin</IfModule>
ServerTokens prod
TraceEnable off
# RewriteEngine On# RewriteCond %{HTTPS} off# RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
<Location "/server-status">SetHandler server-statusRequire local</Location><Location "/server-info">SetHandler server-infoRequire local</Location>
SSLCipherSuite HIGH:MEDIUM:!MD5:!RC4:!SHA1SSLProxyCipherSuite HIGH:MEDIUM:!MD5:!RC4:!SHA1SSLHonorCipherOrder onSSLProtocol TLSv1.1 +TLSv1.2SSLProxyProtocol TLSv1.1 +TLSv1.2SSLPassPhraseDialog builtinSSLSessionCache "shmcb:/opt/httpd/current/logs/ssl_scache(512000)"SSLSessionCacheTimeout 300SSLUseStapling OnSSLStaplingCache "shmcb:/opt/httpd/current/logs/ssl_stapling(32768)"
# Other global configuration directivesServerTokens ProductOnlyTraceEnable off#Header always append X-Frame-Options "SAMEORIGIN"
# Virtual host include filesInclude "/basedirectory/*-vhost.conf"
The configuration in this file may not be needed if the user has access to all Apache configuration files. This is to create a buffer between users and the main Apache configuration. All information can be included in httpd.conf.
This file is included in the httpd.conf by linking it with ‘Include /path/to/vhost.conf’.
# Ports to listen onListen 443Listen 80
Port 443
<VirtualHost _default_:443>
RequestHeader set X_FORWARDED_PROTO 'https'
ServerAdmin ADMIN@SITE.COM
ErrorLog "/basedirectory/projectdirectory/logs/error_log"TransferLog "/basedirectory/projectdirectory/access_log"
SSLEngine onSSLCertificateFile "/basedirectory/CERTIFICATE.crt"SSLCertificateKeyFile "/basedirectory/KEY.key"SSLCertificateChainFile "basedirectory/intermediate-ca.crt"
<FilesMatch "\.(cgi|shtml|phtml|php)$">SSLOptions +StdEnvVars</FilesMatch>
BrowserMatch "MSIE [2-5]" \nokeepalive ssl-unclean-shutdown \downgrade-1.0 force-response-1.0CustomLog "/basedirectory/projectdirectory/logs/ssl_request_log" \"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
WSGIScriptAlias / /basedirectory/projectdirectory/mdcs/mgi/wsgi.py
<Directory /basedirectory/projectdirectory/mdcs/mgi><Files wsgi.py>Require all granted</Files><LimitExcept GET POST HEAD>Require all denied# Order allow,deny# deny from all</LimitExcept></Directory>
ServerName HOSTNAME.SITE.COM
<Directory /basedirectory/projectdirectory/mdcs/static/>Options -Indexes -Includes -ExecCGI -FollowSymLinksRequire all granted<LimitExcept GET POST HEAD>Require all denied# Order allow,deny# deny from all</LimitExcept></Directory>
path=/path/to/current/lib/python2.7
</VirtualHost>
Port 80 (rerouted to secure port)
<VirtualHost *:80>ServerName HOSTNAME.SITE.COMRedirect permanent / https://HOSTNAME.SITE.COM</VirtualHost>
DATABASES = {'default': {'ENGINE': 'django.db.backends.postgresql_psycopg2','USER':"<postgres_user>",'PASSWORD': "<postgres_password>",'NAME': 'mgi',}}