PM Software Stack

PMSS, short of PM Software Stack is a compilation of scripts to manage single server side seedbox configuration, turning instantly and conveniently an regular Debian 5.0/6.0 server into a seedbox. The PMSS does not contain multi-server management utilities nor clustering support out-of-box, but does include PM Master GUI automatic remote update features. Debian 6.0 squeeze support is considered stable, as well as Debian 5.0. PMSS currently only supports rTorrent+ruTorrent stack. Multi-Server management is done via API with central server, with strict authentication and limited scope for security.

General features

• Semi-automatic installation
• Fast and easy creation of new user accounts with quota and rTorrent resource limits, etc.
• Easy user resource reconfiguration
• Autonomous, idempotent Lighttpd configuration
• Autonomous configuration of basic server features
• Compiling rTorrent + libTorrent + Quota + XMLRPC from source.
• Multiple redundancy levels for rTorrent
• Lighttpd redundancy / autorestart
• Basic server optimizations to maximize performance
• Terminate, Suspend/Unsuspend users
• Webdav for users
• Other basic management scripts

Using PMSS

See Category:PM Software Stack Guides

Changelog

Changes 2013

18/5/2013

• install.sh: moving around php5 installation due to continuing OVH template issues

13/5/2013

• install.sh: looks like the OVH Debian 6 wants to install apache2 with php5-geoip, so moved around the php5 modules and after their installation remove apache2
• install.sh: Added code for removing OVH installed Apache2 which is quite hard to get fully removed, and causes PMSS installation to fail

11/5/2013

• update-step2: Added packages required for autodl irssi
• update-step2: Added creation of /etc/autodl.cfg

8/5/2013

• trafficStats month data traffic is now for the past month, not since beginning of calendar month.
• addUser: fixed typo (use of undefined constant)
• addUser: fixed argv optionals checking
• userConfig: added several checks for user existing
• created userTrafficLimit under utils to set the traffic limit
• addUser: fixed typo in pathname for traffic limit data
• userConfig: added setting traffic limit
• userConfig: now restarts rTorrent (near instant)
• created trafficLimits cron which checks traffic limits. Communications module is still required before traffic limits can be enabled
• update-step2: added smartmontools to pkgs to be installed

20/4/2013

• trafficStats now parses 23k lines before 1st of August, 2013 because the traffic logging speed has been lowered at 17th April update -> showing just 1½ weeks initially. Time limit for that so it will not slow down always.

17/4/2013

• Added lib/traffic.php for trafficStatistics class
• Refactoring showTraffic
• stats.php shows this traffic data on info page now
• Added cron/trafficStatistics.php which saves traffic data to .trafficData periodically (once per hour)
• Decreased traffic data saving to once per 5 minutes
• Also now monitoring since beginning of the month
• showTraffic now uses the "cached" data saved by trafficStatistics
• util/userTransfer.php now puts in request for rTorrent restart after transfer is complete
• addUser optional argument for traffic limit in TBs
• update-step2 now checks, creates and chmods a bunch of required directories

13/1/2013

• Added new remote calls to serverApi, had forgotten this
• Dropped traffic logging to 4 min intervals, unfortunately this means showTraffic for 15mins is totally skewed, and maybe noticeable skew on the hourly
• updateQuotas cron now every second minute
• checkInstances now every 10mins from 15mins, making faster 2nd level redundancy for rTorrent
• changed updateQuota cron's API calls to base the call time to modify time of /scripts ie. roughly time the system was installed. This way the requests don't get pooled up at same minute from all servers

12/1/2013

Release of 10th of Jan features as well. Several bug fixes in this release.

• showTraffic now prunes erroneous data from the log
• trafficLog cron now checks for erroneous data and logs it
• updateQuotas sends the user quota and df data only once per hour
• Absolute path to iptables on setupNetwork + makeMonitoringRules so it actually works on boot time :)

10/1/2013

Waiting for release, pending server side support.

• updateQuotas now reports user quota to API server as well
• updateQuotas reports df to API server
• lib/users.php error message when issues with /home
• setupNetwork won't try to enact empty ruleset anymore
• fileCopy updated help message. This is pretty much useless nowadays in any cas and might be deprecated/removed in future release

9/1/2013

One of the DCs changed debian installation template so fixes for that. Issues with rTorrent compilation, lighttpd etc.

• Changes on install.sh package removals (apache2 not getting removed??)
• Removed munin installation from install.sh
• Added installation of pkg-config make openssl
• Fixed on update-step2 rtorrent compilation
• Install.sh now executes /scripts/update.php but does not do API key setup, it's done on the update script

5/1/2013

Quick performance related hot-fix, and one performance update.

• trafficLog now deletes the traffic log file from tmp, this was causing slowdowns etc.
• rTorrent peers better configuration, min from 14 to 6 and max from 21 to 32 per "resource block" (Resource block=250Mb of rTorrent ram). This affects only new accounts, reconfigured and recreated.

2/1/2013

Work on the new API - note this is still largely untested so far

• API create jobs dir + chmod added
• API function to make calls to server added
• API function to create authorization for remote calls added
• API make call userTraffic added
• trafficLog cron fix users check
• trafficLog reports traffic to server
• serverApi typos fixed (had not been tested before)
• serverApi create dirs bugs fixed
• serverApi logger was missing newline and has been added
• serverApi some commenting added
• fixed timestamp on scriptsInc writeLog
• trimming some elements in serverApi which due to unrelated reasons have newline at end

1/1/2013

This update also releases the changes of 7/12/2012 which were postponed due to QA reasons.

• Removal of netcat packages in update-step2 and install script
• Added count checks for users in makeMonitoringRules and recreate lighttpd rules scripts
• Copied some standard packages from update-step2 to install.sh
• Removed some apt packages from update-step2 which has been on install.sh for a *Really* long time
• Removed some irrelevant apt-get lines from install -> duplication
• Added p7zip-full package to update-step2 installation
• Fix showtraffic weekly data, was only considering ~2/5ths of the data
• ShowTraffic compares now user count, not the string returned in order to determine if there is users setup
• More uniform ShowTraffic display output, which is also programmatically easy to parse

Changes 2012

7/12/2012

This update is pending for release.

• Removal of root.cron.additional config from usage, as it does not conform to good practices
• Removal of deprecated code from update-step2 relating to vnstat and root.cron.additional
• Removing root.cron.additional on update-step2
• Removal of deprecated code from setupRootCron
• Removal of /var/run/screen/S-[username] during user termination
• Executing setupNetwork on user addition
• Added tor repos for debian squeeze
• Update-Step2 now checks squeeze repos against the actual configured, not hard coded sha1 string
• Removal of deprecated apt update from beginning of update-step2
• Some commenting and refactoring on update-step2
• Remove munin if this is a .pulsedmedia.com server
• Adduser now logs to /var/log/pmss
• Create remote api key
• Salt for local api key
• Salt for remote api key
• Enhanced algo for remote and local api key, more factors etc. making it more distinguished
• chmod api keys
• Update-Step2: if no remote api key present, remove local key and execute API key creation
• rename lib/api.php to lib/serverApi.php
• Full rewrite of serverApi, pretty much complete server api infrastructure including command map for: addUser, terminateUser, setupNetwork, recreateUser

1/11/2012

• Removed updating from OVH kernel to standard debian kernel
• No fair share / QoS rules done if OVH kernel is present
• Setup network (monitoring, QOS) on update step 2
• Changed root.cron.additional vnstat lines to conform cron.d format
• If debian 6 removal of vnstat -u cronjobs
• setupRootCron both cron force-reload and restart to ensure new cron will be enabled
• Temporarily disable all QoS & Fair share, unintentional effects

31/10/2012

• Display traffic used per user on week, daily, hourly and 15min basis. Note: 15min is slightly inaccurate due to datalogging delays (2min interval), hourly is within acceptable error margin
• Fine-tuning of fair share
• QoS is always setup and cleared.
• recreateUser: Added a check if backup dir already exists.

30/10/2012

• Fixed var/www/index.php redirection
• Added script to create traffic monitoring iptables rules
• Created /scripts/cron
• moved updateQuotas, checkInstances, checkHttpd to cron dir
• created /var/log/pmss
• Changed checkInstances + checkHttpd logging to /var/log/pmss dir
• Refactored cron tasks to utilize /etc/cron.d NOTE: If you have customized root crontab this may overwrite the changes!
• Removed "tester" from user filtration list and added srvapi, pmcseed, pmcdn to the filtration list
• Beginnings of network management cron
• If an OVH kernel is detected on Deb Squeeze, install standard and set it as default. Note: No Deb5 support yet - only changed IF vanilla kernel is not installed
• Added packages ethtool, elinks, links, lynx
• Added Basic QoS to ensure quality of service and fairness in BW sharing - enabled at boot time. Note: Does not work on OVH kernel - OVH has removed support for these in their kernel
• Cron jobs now also save their error output to log files
• Traffic logging script and /var/log/pmss/traffic directory for traffic log files.

14/10/2012

• pulsedBox authentication pruning to filter/fix the most common user errors

22/09/2012

• Fixed quota parsin in GUI when using LVM device /dev/mapper/ type addressing
• Updated installer to install rTorrent 0.9.2 and libTorrent 0.13.2
• Fixed updater setup of squeeze repos
• ruTorrent update, does include saving user preferences etc. to the new one
• Fixed a bug relating to .pulsedmedia mgmt features, incorrect order for permissions setting
• Update-step2 mild refactoring
• Removed useless apt-get install from update.php
• Added ffmpeg along with screenshots plugin
• Removed unpack plugin (way too high resource consumption)
• getfile plugin included now
• Applied quota parsing fixed to hddquota plugin and made slightly more dynamic
• Lowered time before displaying welcome page quota information

17/09/2012

• Removed proxy from soft.sh as this hasn't been included for sometime now.
• Updated rTorrent to 0.9.2 and libTorrent to 0.13.2 official releases

10/06/2012

• PEX + DHT enabled by default. DHT port same as SCGI port for now. Template changed - not the actual code, so in all versions the default has changed. Server specific configuration can disable.

22/05/2012

• Quota display bugs fixed: Servers with UUID assignment instead of device and in some cases collecting the total quota limit had last number cut off

24/04/2012

• Munin configuration updates for .pulsedmedia.com
• Updated method of checking ruTorrent version to avoid "updates" when none is needed
• More conditions for user space update, if removed user but something remains as backup / as a failure of user termination script
• Refactored a bit update-step2 fixing when .rtorrent.rc is updated

06/04/2012

• Compile rTorrent always when version mismatch during update, even if empty/not installed
• Added download speed limit when downloading via ajaxManager
• Changed chat tab to use username instead of random

27/03/2012

• Update debian 5 repos to the archive ones
• On install also enable fastcgi-php (Squeeze)
• Get quota from apt repo, and do not compile on install (Lenny support is now limited)
• Updated squeeze repos
• Slightly refactored update-step2 and install

21/03/2012

• Removed autotools + cpuload plugin from skel rutorrent

20/03/2012

• rTorrent + libtorrent version to be changed to GIT 2011/11/08 which is slightly updated from release 0.8.9, few bug fixes etc. GIT information retained if you want to check validity.
• Removal of cpuload + autotools rutorrent plugins. CPULoad being misleading at best, autotools haven't worked for ages and developer is MIA.
• Changed fetching libmediainfo packages from pulsedmedia.com instead of sourceforge.
• Choosing libmedia info + libs based on debian version. Now Debian 6.0 should be fully supported in updates.
• Creating symlink /usr/local/bin/mktorrent -> /usr/bin/mktorrent thus fixing ruTorrent torrent creation in some Debian 6.0 systems (with updated ruTorrent+rTorrent)
• Cleaning installed libtorrent

04/03/2012

• Allow key based authentication on SSH
• Pulsed Media specific management code - not added unless server is for .pulsedmedia.com
• Added srvmgmt to filtered users --- NOTICE: srvadmin user account will be deprecated in a future release

12/02/2012

• added scripts inc for files to be updated/copied on update

10/02/2012

• Fixed torrent creation from ruTorrent
• Fixed quota plugin (JSON parser has been changed to strict parsing)
• Updated quota plugin version information
• Fixed ajax manager recycle bin
• Made ruTorrent to update browser cache
• Created framework for easy user space files updating during server update (files dependant upon server upgrade)
• Removed recreateLighttpdConfig + lighttpd restart from update step 2 (should by now be in main update in 99.9%+ servers)
• Changed update.php to do lighttpd force-reload instead of restart
• Refactored update-step2 slightly: Moved specific files, and user specific stuff into single loop
• Added several packages to be installed during server installation, which were only on update. They still remain on update as well before better solution can be done (dselect)
• Filtering backup user dirs out from user list
• During scripts update, also update system packages

09/02/2012

• Reversing yesterday's update due to security concerns
• Removed installer speedups due to stability concerns

08/02/2012

• Fixed typo in install script preventing quota from being compiled

05/02/2012

• Lighttpd session "stability" upgrade, fixing issue causing session timeouts/re-requiring authentication

04/02/2012

• Added mediainfo
• Added server root document to redirect to pulsedmedia.com
• Added php5-geoip extension to be installed
• Simplified update-step2 scripts being installed :)
• Removed apt dist-upgrade from install
• Simplified (less) apt-get lines during install
• Reordered apt-get commands during install for swifter installation
• Removed some "junk" code from install
• Updated copyright text in install.sh
• Added some information on header of install.sh
• Install.sh some additional comments
• Made installation slightly quicker by usage of background processes (BETA, if your disk is extremely slow this may cause installation to fail)
• Reordering other items in install.sh to make installation faster
• Some preliminary work for supporting whitelabeling
• Removed umask, hash interval, hash max tries rtorrent configures (deprecated)
• Updated rTorrent + ruTorrent to versions 0.9.0/0.13.0

Note: Please report any issues due to the upgrade of rTorrent immediately to Pulsed Media. The patch has been tested on a small group of servers, but all situations are hard to foresee.

31/01/2012

• Fixed some typos in update-step2

28/01/2012

• Added iotop and cfv to be installed from apt repo.
• Added -m toggle for htpasswd because for some reason this is not default in Debian Lenny
• Added SSL support (self signed cert), just change http to https
• Added Webdav support, accessible from "/webdav-USERNAME/" *** BETA: Permission issues, your mileage will vary
• Removed SSL key creation from install
• Update-step2 creates SSL key automatically utilizing the server hostname, removes old deprecated key

Changes 2011

15/12/2011

• Added .irssi/config to user dirs, autojoining to #PulsedMedia on Freenode when irssi is launched
• Updated GUI IRC links, Chat tab and Adobe download links, with other small additional information and updates.
• Removed Network Storage tab from GUI because Dropbox is blocking pageviews coming from our system for unknown reason.
• Added Proxy information on GUI
• Update-step2 sees if .irssi/config exists and if it does not copies it there

30/11/2011

• Chosen rTorrent SCGI port range increased from 4k to 8k.
• "fractional blocks" for config. Resource calcs are based on number of blocks, and fractions were not allowed prior.

29/11/2011

• During update also update user crontabs. Added to both update.php and step2, to be removed from step2 after this update has seen significant adoption.
• Restarting lighttpd during update, also done doubly for now
• Moved setup api key to regular update script. In several months it will potentially be removed from update all together as it's a installation step, and included to update older nodes.
• Scripts permissions update
• updateQuotas.php -> cron to update all user quotas, as discreet user from central cron to ensure it gets updated.
• Updated root & user crontabs to reflect updateQuotas

28/11/2011

• quotaFix now removes aquota*new files to avoid collision and thus failed update via quotacheck
• userTorrents.php : Lists the count of torrents each user has, for quicker analysis usage stats

23/11/2011

• Permissions setup script added, which is utilized by recreateUser.
• recreateUser.php fixes

22/11/2011

• Created recreateUser.php which recreates home dir, preserving data+session if existing. ALPHA

19/11/2011

• Fixed saving API Key.

17/11/2011

• Suspend and Unsuspend scripts fixed
• recreateLighttpd backup fixed
• Added access format of: servername/user-username/, for example: localhost/user-johndoe/. Thus supporting non FQDN setups
• addUser.php calls recreateLighttpd instead of just adding the hostname, this ensures lighttpd config is always up-to-date.
• Added to both update steps recreateLighttpd. It will remain on step2 for the next ~6months to ensure every system out there will have the update update.php
• Removed the default "testing" torrents
• Upload slots global, by default 6 times the non-global upload slots. It is "forcefully" inserted even if template does not it have for the moment. this forcefull insert will be removed in the future.
• checkInstances now also checks rtorrent configs, and if changed lists them on /root/changedConfigs for admin review

25/8/2011

Streamlined installer for faster setups :) First release of API, just one command now. Some things moved to BETA status and considered feature complete. Many bug fixes, additional commenting etc. also added

• fixed userTransfer chowning lacking -R, thus UID was transferred for many files from the other server causing incorrect permissions
• wrapped vnstat install into if !file_exists to see if vnstat is installed already.
• added username 'srvadmin' to filter list on systemUsers
• removed from installer editing munin.conf -> if used can be done manually. Interim solution.
• put install vim as first thing on installer
• setting node hostname straight after installing vim
• moved setting up lighttpd as last thing
• scripts, skel, etc are installed earlier in installation
• SSL cert + quota as last items on setup
• Executing update-step2 on installer (so you don't need to execute updater after setup to get everything in)
• /scripts/util/setupRootCron.php -> creating root conf moved out of update.php
• setupSkelPermissions.php, same reason as root cron setup
• added setupRootCron and setupSkelPermissions to installer
• Idempotent lighttpd configurator feature completed. Backups the config first.
• Terminate user moved to beta status, as using lighttpd configurator now and restarting lighttpd.
• Added for terminate second argument, if 2nd argument is --confirm, straight accepts termination without asking for approval, for making automation easier.
• checkHttpd timer increased to 3 seconds before restarting lighttpd
• terminateUser timer increased to 5 seconds after trying to kill all user processes
• setupApiKey -> creates node key for API. Invoked upon setup.
• added authorization key
• removed /var/www/index.lighttpd.html
• added /var/www/index.php redir to pulsedmedia.com
• added beginnings of remote API for easier remote control management, by default in debug mode for now

8/6/2011

Bug fixes.

terminateUser.php:

• Was not actually completing shell userdeletion
• Added killing user processes (forgotten)
• added sleeping 2 seconds to allow time for rTorrent/other processes by user to die
• Added 2nd attempt to userdel after data removal
• Fixed some typos in texts
• Added check for user actually exists

NOTE: Still Alpha, non feature complete and not recommended for production use.

• checkHttpd.php added sleep between lighttpd kill and restart, to avoid PID file mismatch scenario

update-step2.php:

• First time being used
• update.php moved cron & ftp config stuff AFTER step2
• Install vnstat + vim (vim not by default on Debian 6)
• Setup interface for vnstat
• Configure vnstat for Debian 6
• Make vnstat cron jobs for Debian 5: twice a minute update (thus no support for 1GigE interface)
• Debian 5 add cron jobs to root.cron.additional config, create if needed

31/5/2011

• GUI welcome page checking if proper quota information was passed has been added.
• GUI welcome page malformed serialized data fix
• GUI welcome page quota percentage missing % char
• GUI welcome page quota text styles: White and location

30/5/2011

• GUI update script quota checking added (doesn't try to update when user is over quota burst limit, no more blanking GUI when update happens!)
• Users default quota information update rate has been increased to 10mins
• HDD Quota plugin commenting added
• HDD Quota plugin parser bug with bursted amounts fixed
• HDD Quota now shows USED/QUOTA instead of FREE/QUOTA. Makes more sense.
• HDD Quota "title" fixed (showing quota info) to be whole area of the meter
• Welcome page quota meter added :) Basic graph, larger and constantly visible text for quota info :)
• Basic idempotent lighttpd configurator (alpha, not suitable for production, TESTING only)

23/5/2011

• terminateUser.php http password deletion fix, text fixes, executability.
• checkHttpd.php output lighttpd output, killall -9 lighttpd to ensure shutdown, http fetch mute removed so error gets logged
• update.php: removed apt updates because soft.sh already does this
• crons: increased check http to 4 minute intervals (avoid http timeout period issues)

18/5/2011

Temporary fix to bypass userDb until fully utilized. You may use this update or use /scripts/util/populateUserDb.php to fix the issue.

• Changed listUsers.php to always use systemUsers.php, as fixing userDb saving introduced a bug where ie. checkInstances was not working fully (using userDb, which lacks most users in most servers)

16/5/2011

Few bug fixes and minor features introduced.

• Update script order of items.
• Update script apt-get options for additional package installation lacked auto confirm.
• Update script support for "step 2", /scripts/util/update-step2.php is included if present, thus supporting dynamic updates
• Update script execute binary line added, so it can actually be executed simply by ./update.php
• /scripts permission fix, only root should have anykind of access (no execute or read for others)
• Support for root additional items via /etc/seedbox/config/root.cron.additional
• Add user script minor cleanup, additional commenting.
• Suspend script comments
• simple terminate user script, ALPHA -> not feature complete yet.

Updating from 27/03/2011 release or earlier

cd ~; bash soft.sh; php /scripts/update.php

Updating from 15/5/2011 release

php /scripts/update.php

15/5/2011

Couple important new features introduced, several bug fixes.

• Added lighttpd status checking and auto-restart (very rudimentary)
• Fixed typo in /scripts/util/userConfig.php
• Added /etc/seedbox/config/root.cron
• /scripts/util/quotaFix.php now echoes results (also turns quota off, no need to turn it off first)
• proftpd config update (remote) with MaxClientsPerUser, MaxClientsPerHost, run /scripts/util/ftpConfig.php
• users class now supresses error if runtime db is not found
• users class now saves the user db upon save
• users class deconstructor added to save users db upon exit
• modules support code begun (version for highly dynamic with multiple hooks etc.)
• Changed update.php to a very basic one, as the remote capabilities are not being employed currently
• Rudimentary module /etc/seedbox/module/basic/addUser.php for ultra simplistic user addition module (ie. custom configs, limitations).
• Basic suspend & unsuspend capabilities (untested ALPHA)
• skel permissions fixed

updating

bash soft.sh; php /scripts/update.php

27/03/2011

This is primarily a bug fix and code quality release.

• Set shell to /bin/bash upon user creation
• Set shell to /bin/bash when using userConfig
• quotaFix.php now turns quota off before attempting quota check
• quotaFix.php now displays quota info before and after check
• Lots of comments added to all code files touched
• userConfig.php $filesLimitPerGb was undefined - hardcoded 500
• update.php minor changes
• default version now set to 1, in preparation to take it into production
• startRtorrent: codestyle fixes, Made error messages more clear, named rTorrent screen session to "rtorrent"