PM Software Stack

PMSS, short of PM Software Stack is a compilation of scripts to manage single server side seedbox configuration, turning instantly and conveniently an regular Debian 6.0/7.0 server into a seedbox. The PMSS does not contain multi-server management utilities nor clustering support out-of-box, but does include PM Master GUI automatic remote update features. Debian 6.0 squeeze support is considered stable. PMSS currently only supports rTorrent+ruTorrent stack. Multi-Server management is done via API with central server, with strict authentication and limited scope for security.

General features

• Semi-automatic installation
• Fast and easy creation of new user accounts with quota and rTorrent resource limits, etc.
• Easy user resource reconfiguration
• Autonomous, idempotent Lighttpd configuration
• Autonomous configuration of basic server features
• Compiling rTorrent + libTorrent + Quota + XMLRPC from source.
• Multiple redundancy levels for rTorrent
• Lighttpd redundancy / autorestart
• Basic server optimizations to maximize performance
• Terminate, Suspend/Unsuspend users
• Webdav for users
• Other basic management scripts

Using PMSS

See Category:PM Software Stack Guides

Changelog

Changes 2014

31/08/2014

• Minor revisions to install.sh and code cleanup

12/06/2014

• terminateUser.php: added groupdel as well for the user group

09/06/2014

• Slight refactoring of terminateUser.php
• terminateUser.php: --confirm now first, before username

02/06/2014

• Update on versioning issue with installing libmediainfo, mediainfo and libzen. fixed and these will be now installed on wheezy as well.
• libdigest-sha1-perl pkg doesn't exist on wheezy so it was preventing several other packages from installing
• libdigest-sha-perl pkg now installed on wheezy

11/02/2014

• Styling of info tab, info.php/stats.php

29/01/2014

• pulsedBox upload: Fixed for new style urls, should work with both subdomain and subfolder setups now.
• Installer package updates so that PMSS should install into Debian 7 without a hitch now, previously 2 packages were missing.

Changes 2013

21/12/2013

• userTransfer.php: Tries to retry 20 times with 15-120sec delay. Necessary since OVH has introduced new bw limitations: Connections are cut at random, exact rules unknown but result is that transferring user data connections are cut at random. This has been tested via multiple DCs to be true, from multiple OVH DCs.
• userTransfer.php: At the end display user's old server password IN CASE the 20 retries is not sufficient.

03/10/2013

• install.sh: updated libxmlrpc URI to the new one.

29/9/2013

• update-step2: Added wheezy repos
• update-step2: Made wheezy repo update
• update-step2: Some refactoring in regards of repo updates
• cron/trafficLog: Updated traffic parsing to support wheezy
• install.sh: Moved apt-get install rar unrar to it's own command due to OVH wheezy template issues
• install.sh: Removed some commented out things, blast from past type of things
• install.sh: Updated queue read ahead to double size
• install.sh: Updated header comments

18/5/2013

• install.sh: moving around php5 installation due to continuing OVH template issues

13/5/2013

• install.sh: looks like the OVH Debian 6 wants to install apache2 with php5-geoip, so moved around the php5 modules and after their installation remove apache2
• install.sh: Added code for removing OVH installed Apache2 which is quite hard to get fully removed, and causes PMSS installation to fail

11/5/2013

• update-step2: Added packages required for autodl irssi
• update-step2: Added creation of /etc/autodl.cfg

8/5/2013

• trafficStats month data traffic is now for the past month, not since beginning of calendar month.
• addUser: fixed typo (use of undefined constant)
• addUser: fixed argv optionals checking
• userConfig: added several checks for user existing
• created userTrafficLimit under utils to set the traffic limit
• addUser: fixed typo in pathname for traffic limit data
• userConfig: added setting traffic limit
• userConfig: now restarts rTorrent (near instant)
• created trafficLimits cron which checks traffic limits. Communications module is still required before traffic limits can be enabled
• update-step2: added smartmontools to pkgs to be installed

20/4/2013

• trafficStats now parses 23k lines before 1st of August, 2013 because the traffic logging speed has been lowered at 17th April update -> showing just 1½ weeks initially. Time limit for that so it will not slow down always.

17/4/2013

• Added lib/traffic.php for trafficStatistics class
• Refactoring showTraffic
• stats.php shows this traffic data on info page now
• Added cron/trafficStatistics.php which saves traffic data to .trafficData periodically (once per hour)
• Decreased traffic data saving to once per 5 minutes
• Also now monitoring since beginning of the month
• showTraffic now uses the "cached" data saved by trafficStatistics
• util/userTransfer.php now puts in request for rTorrent restart after transfer is complete
• addUser optional argument for traffic limit in TBs
• update-step2 now checks, creates and chmods a bunch of required directories

13/1/2013

• Added new remote calls to serverApi, had forgotten this
• Dropped traffic logging to 4 min intervals, unfortunately this means showTraffic for 15mins is totally skewed, and maybe noticeable skew on the hourly
• updateQuotas cron now every second minute
• checkInstances now every 10mins from 15mins, making faster 2nd level redundancy for rTorrent
• changed updateQuota cron's API calls to base the call time to modify time of /scripts ie. roughly time the system was installed. This way the requests don't get pooled up at same minute from all servers

12/1/2013

Release of 10th of Jan features as well. Several bug fixes in this release.

• showTraffic now prunes erroneous data from the log
• trafficLog cron now checks for erroneous data and logs it
• updateQuotas sends the user quota and df data only once per hour
• Absolute path to iptables on setupNetwork + makeMonitoringRules so it actually works on boot time :)

10/1/2013

Waiting for release, pending server side support.

• updateQuotas now reports user quota to API server as well
• updateQuotas reports df to API server
• lib/users.php error message when issues with /home
• setupNetwork won't try to enact empty ruleset anymore
• fileCopy updated help message. This is pretty much useless nowadays in any cas and might be deprecated/removed in future release

9/1/2013

One of the DCs changed debian installation template so fixes for that. Issues with rTorrent compilation, lighttpd etc.

• Changes on install.sh package removals (apache2 not getting removed??)
• Removed munin installation from install.sh
• Added installation of pkg-config make openssl
• Fixed on update-step2 rtorrent compilation
• Install.sh now executes /scripts/update.php but does not do API key setup, it's done on the update script

5/1/2013

Quick performance related hot-fix, and one performance update.

• trafficLog now deletes the traffic log file from tmp, this was causing slowdowns etc.
• rTorrent peers better configuration, min from 14 to 6 and max from 21 to 32 per "resource block" (Resource block=250Mb of rTorrent ram). This affects only new accounts, reconfigured and recreated.

2/1/2013

Work on the new API - note this is still largely untested so far

• API create jobs dir + chmod added
• API function to make calls to server added
• API function to create authorization for remote calls added
• API make call userTraffic added
• trafficLog cron fix users check
• trafficLog reports traffic to server
• serverApi typos fixed (had not been tested before)
• serverApi create dirs bugs fixed
• serverApi logger was missing newline and has been added
• serverApi some commenting added
• fixed timestamp on scriptsInc writeLog
• trimming some elements in serverApi which due to unrelated reasons have newline at end

1/1/2013

This update also releases the changes of 7/12/2012 which were postponed due to QA reasons.

• Removal of netcat packages in update-step2 and install script
• Added count checks for users in makeMonitoringRules and recreate lighttpd rules scripts
• Copied some standard packages from update-step2 to install.sh
• Removed some apt packages from update-step2 which has been on install.sh for a *Really* long time
• Removed some irrelevant apt-get lines from install -> duplication
• Added p7zip-full package to update-step2 installation
• Fix showtraffic weekly data, was only considering ~2/5ths of the data
• ShowTraffic compares now user count, not the string returned in order to determine if there is users setup
• More uniform ShowTraffic display output, which is also programmatically easy to parse

Changes 2012

7/12/2012

This update is pending for release.

• Removal of root.cron.additional config from usage, as it does not conform to good practices
• Removal of deprecated code from update-step2 relating to vnstat and root.cron.additional
• Removing root.cron.additional on update-step2
• Removal of deprecated code from setupRootCron
• Removal of /var/run/screen/S-[username] during user termination
• Executing setupNetwork on user addition
• Added tor repos for debian squeeze
• Update-Step2 now checks squeeze repos against the actual configured, not hard coded sha1 string
• Removal of deprecated apt update from beginning of update-step2
• Some commenting and refactoring on update-step2
• Remove munin if this is a .pulsedmedia.com server
• Adduser now logs to /var/log/pmss
• Create remote api key
• Salt for local api key
• Salt for remote api key
• Enhanced algo for remote and local api key, more factors etc. making it more distinguished
• chmod api keys
• Update-Step2: if no remote api key present, remove local key and execute API key creation
• rename lib/api.php to lib/serverApi.php
• Full rewrite of serverApi, pretty much complete server api infrastructure including command map for: addUser, terminateUser, setupNetwork, recreateUser

1/11/2012

• Removed updating from OVH kernel to standard debian kernel
• No fair share / QoS rules done if OVH kernel is present
• Setup network (monitoring, QOS) on update step 2
• Changed root.cron.additional vnstat lines to conform cron.d format
• If debian 6 removal of vnstat -u cronjobs
• setupRootCron both cron force-reload and restart to ensure new cron will be enabled
• Temporarily disable all QoS & Fair share, unintentional effects

31/10/2012

• Display traffic used per user on week, daily, hourly and 15min basis. Note: 15min is slightly inaccurate due to datalogging delays (2min interval), hourly is within acceptable error margin
• Fine-tuning of fair share
• QoS is always setup and cleared.
• recreateUser: Added a check if backup dir already exists.

30/10/2012

• Fixed var/www/index.php redirection
• Added script to create traffic monitoring iptables rules
• Created /scripts/cron
• moved updateQuotas, checkInstances, checkHttpd to cron dir
• created /var/log/pmss
• Changed checkInstances + checkHttpd logging to /var/log/pmss dir
• Refactored cron tasks to utilize /etc/cron.d NOTE: If you have customized root crontab this may overwrite the changes!
• Removed "tester" from user filtration list and added srvapi, pmcseed, pmcdn to the filtration list
• Beginnings of network management cron
• If an OVH kernel is detected on Deb Squeeze, install standard and set it as default. Note: No Deb5 support yet - only changed IF vanilla kernel is not installed
• Added packages ethtool, elinks, links, lynx
• Added Basic QoS to ensure quality of service and fairness in BW sharing - enabled at boot time. Note: Does not work on OVH kernel - OVH has removed support for these in their kernel
• Cron jobs now also save their error output to log files
• Traffic logging script and /var/log/pmss/traffic directory for traffic log files.

14/10/2012

• pulsedBox authentication pruning to filter/fix the most common user errors

22/09/2012

• Fixed quota parsin in GUI when using LVM device /dev/mapper/ type addressing
• Updated installer to install rTorrent 0.9.2 and libTorrent 0.13.2
• Fixed updater setup of squeeze repos
• ruTorrent update, does include saving user preferences etc. to the new one
• Fixed a bug relating to .pulsedmedia mgmt features, incorrect order for permissions setting
• Update-step2 mild refactoring
• Removed useless apt-get install from update.php
• Added ffmpeg along with screenshots plugin
• Removed unpack plugin (way too high resource consumption)
• getfile plugin included now
• Applied quota parsing fixed to hddquota plugin and made slightly more dynamic
• Lowered time before displaying welcome page quota information

17/09/2012

• Removed proxy from soft.sh as this hasn't been included for sometime now.
• Updated rTorrent to 0.9.2 and libTorrent to 0.13.2 official releases

10/06/2012

• PEX + DHT enabled by default. DHT port same as SCGI port for now. Template changed - not the actual code, so in all versions the default has changed. Server specific configuration can disable.

22/05/2012

• Quota display bugs fixed: Servers with UUID assignment instead of device and in some cases collecting the total quota limit had last number cut off

24/04/2012

• Munin configuration updates for .pulsedmedia.com
• Updated method of checking ruTorrent version to avoid "updates" when none is needed
• More conditions for user space update, if removed user but something remains as backup / as a failure of user termination script
• Refactored a bit update-step2 fixing when .rtorrent.rc is updated

06/04/2012

• Compile rTorrent always when version mismatch during update, even if empty/not installed
• Added download speed limit when downloading via ajaxManager
• Changed chat tab to use username instead of random

27/03/2012

• Update debian 5 repos to the archive ones
• On install also enable fastcgi-php (Squeeze)
• Get quota from apt repo, and do not compile on install (Lenny support is now limited)
• Updated squeeze repos
• Slightly refactored update-step2 and install

21/03/2012

• Removed autotools + cpuload plugin from skel rutorrent

20/03/2012

• rTorrent + libtorrent version to be changed to GIT 2011/11/08 which is slightly updated from release 0.8.9, few bug fixes etc. GIT information retained if you want to check validity.
• Removal of cpuload + autotools rutorrent plugins. CPULoad being misleading at best, autotools haven't worked for ages and developer is MIA.
• Changed fetching libmediainfo packages from pulsedmedia.com instead of sourceforge.
• Choosing libmedia info + libs based on debian version. Now Debian 6.0 should be fully supported in updates.
• Creating symlink /usr/local/bin/mktorrent -> /usr/bin/mktorrent thus fixing ruTorrent torrent creation in some Debian 6.0 systems (with updated ruTorrent+rTorrent)
• Cleaning installed libtorrent

04/03/2012

• Allow key based authentication on SSH
• Pulsed Media specific management code - not added unless server is for .pulsedmedia.com
• Added srvmgmt to filtered users --- NOTICE: srvadmin user account will be deprecated in a future release

12/02/2012

• added scripts inc for files to be updated/copied on update

10/02/2012

• Fixed torrent creation from ruTorrent
• Fixed quota plugin (JSON parser has been changed to strict parsing)
• Updated quota plugin version information
• Fixed ajax manager recycle bin
• Made ruTorrent to update browser cache
• Created framework for easy user space files updating during server update (files dependant upon server upgrade)
• Removed recreateLighttpdConfig + lighttpd restart from update step 2 (should by now be in main update in 99.9%+ servers)
• Changed update.php to do lighttpd force-reload instead of restart
• Refactored update-step2 slightly: Moved specific files, and user specific stuff into single loop
• Added several packages to be installed during server installation, which were only on update. They still remain on update as well before better solution can be done (dselect)
• Filtering backup user dirs out from user list
• During scripts update, also update system packages

09/02/2012

• Reversing yesterday's update due to security concerns
• Removed installer speedups due to stability concerns

08/02/2012

• Fixed typo in install script preventing quota from being compiled

05/02/2012

• Lighttpd session "stability" upgrade, fixing issue causing session timeouts/re-requiring authentication

04/02/2012

• Added mediainfo
• Added server root document to redirect to pulsedmedia.com
• Added php5-geoip extension to be installed
• Simplified update-step2 scripts being installed :)
• Removed apt dist-upgrade from install
• Simplified (less) apt-get lines during install
• Reordered apt-get commands during install for swifter installation
• Removed some "junk" code from install
• Updated copyright text in install.sh
• Added some information on header of install.sh
• Install.sh some additional comments
• Made installation slightly quicker by usage of background processes (BETA, if your disk is extremely slow this may cause installation to fail)
• Reordering other items in install.sh to make installation faster
• Some preliminary work for supporting whitelabeling
• Removed umask, hash interval, hash max tries rtorrent configures (deprecated)
• Updated rTorrent + ruTorrent to versions 0.9.0/0.13.0

Note: Please report any issues due to the upgrade of rTorrent immediately to Pulsed Media. The patch has been tested on a small group of servers, but all situations are hard to foresee.

31/01/2012

• Fixed some typos in update-step2

28/01/2012

• Added iotop and cfv to be installed from apt repo.
• Added -m toggle for htpasswd because for some reason this is not default in Debian Lenny
• Added SSL support (self signed cert), just change http to https
• Added Webdav support, accessible from "/webdav-USERNAME/" *** BETA: Permission issues, your mileage will vary
• Removed SSL key creation from install
• Update-step2 creates SSL key automatically utilizing the server hostname, removes old deprecated key

Changes 2011

15/12/2011

• Added .irssi/config to user dirs, autojoining to #PulsedMedia on Freenode when irssi is launched
• Updated GUI IRC links, Chat tab and Adobe download links, with other small additional information and updates.
• Removed Network Storage tab from GUI because Dropbox is blocking pageviews coming from our system for unknown reason.
• Added Proxy information on GUI
• Update-step2 sees if .irssi/config exists and if it does not copies it there

30/11/2011

• Chosen rTorrent SCGI port range increased from 4k to 8k.
• "fractional blocks" for config. Resource calcs are based on number of blocks, and fractions were not allowed prior.

29/11/2011

• During update also update user crontabs. Added to both update.php and step2, to be removed from step2 after this update has seen significant adoption.
• Restarting lighttpd during update, also done doubly for now
• Moved setup api key to regular update script. In several months it will potentially be removed from update all together as it's a installation step, and included to update older nodes.
• Scripts permissions update
• updateQuotas.php -> cron to update all user quotas, as discreet user from central cron to ensure it gets updated.
• Updated root & user crontabs to reflect updateQuotas

28/11/2011

• quotaFix now removes aquota*new files to avoid collision and thus failed update via quotacheck
• userTorrents.php : Lists the count of torrents each user has, for quicker analysis usage stats

23/11/2011

• Permissions setup script added, which is utilized by recreateUser.
• recreateUser.php fixes

22/11/2011

• Created recreateUser.php which recreates home dir, preserving data+session if existing. ALPHA

19/11/2011

• Fixed saving API Key.

17/11/2011

• Suspend and Unsuspend scripts fixed
• recreateLighttpd backup fixed
• Added access format of: servername/user-username/, for example: localhost/user-johndoe/. Thus supporting non FQDN setups
• addUser.php calls recreateLighttpd instead of just adding the hostname, this ensures lighttpd config is always up-to-date.
• Added to both update steps recreateLighttpd. It will remain on step2 for the next ~6months to ensure every system out there will have the update update.php
• Removed the default "testing" torrents
• Upload slots global, by default 6 times the non-global upload slots. It is "forcefully" inserted even if template does not it have for the moment. this forcefull insert will be removed in the future.
• checkInstances now also checks rtorrent configs, and if changed lists them on /root/changedConfigs for admin review

25/8/2011

Streamlined installer for faster setups :) First release of API, just one command now. Some things moved to BETA status and considered feature complete. Many bug fixes, additional commenting etc. also added

• fixed userTransfer chowning lacking -R, thus UID was transferred for many files from the other server causing incorrect permissions
• wrapped vnstat install into if !file_exists to see if vnstat is installed already.
• added username 'srvadmin' to filter list on systemUsers
• removed from installer editing munin.conf -> if used can be done manually. Interim solution.
• put install vim as first thing on installer
• setting node hostname straight after installing vim
• moved setting up lighttpd as last thing
• scripts, skel, etc are installed earlier in installation
• SSL cert + quota as last items on setup
• Executing update-step2 on installer (so you don't need to execute updater after setup to get everything in)
• /scripts/util/setupRootCron.php -> creating root conf moved out of update.php
• setupSkelPermissions.php, same reason as root cron setup
• added setupRootCron and setupSkelPermissions to installer
• Idempotent lighttpd configurator feature completed. Backups the config first.
• Terminate user moved to beta status, as using lighttpd configurator now and restarting lighttpd.
• Added for terminate second argument, if 2nd argument is --confirm, straight accepts termination without asking for approval, for making automation easier.
• checkHttpd timer increased to 3 seconds before restarting lighttpd
• terminateUser timer increased to 5 seconds after trying to kill all user processes
• setupApiKey -> creates node key for API. Invoked upon setup.
• added authorization key
• removed /var/www/index.lighttpd.html
• added /var/www/index.php redir to pulsedmedia.com
• added beginnings of remote API for easier remote control management, by default in debug mode for now

8/6/2011

Bug fixes.

terminateUser.php:

• Was not actually completing shell userdeletion
• Added killing user processes (forgotten)
• added sleeping 2 seconds to allow time for rTorrent/other processes by user to die
• Added 2nd attempt to userdel after data removal
• Fixed some typos in texts
• Added check for user actually exists

NOTE: Still Alpha, non feature complete and not recommended for production use.

• checkHttpd.php added sleep between lighttpd kill and restart, to avoid PID file mismatch scenario

update-step2.php:

• First time being used
• update.php moved cron & ftp config stuff AFTER step2
• Install vnstat + vim (vim not by default on Debian 6)
• Setup interface for vnstat
• Configure vnstat for Debian 6
• Make vnstat cron jobs for Debian 5: twice a minute update (thus no support for 1GigE interface)
• Debian 5 add cron jobs to root.cron.additional config, create if needed

31/5/2011

• GUI welcome page checking if proper quota information was passed has been added.
• GUI welcome page malformed serialized data fix
• GUI welcome page quota percentage missing % char
• GUI welcome page quota text styles: White and location

30/5/2011

• GUI update script quota checking added (doesn't try to update when user is over quota burst limit, no more blanking GUI when update happens!)
• Users default quota information update rate has been increased to 10mins
• HDD Quota plugin commenting added
• HDD Quota plugin parser bug with bursted amounts fixed
• HDD Quota now shows USED/QUOTA instead of FREE/QUOTA. Makes more sense.
• HDD Quota "title" fixed (showing quota info) to be whole area of the meter
• Welcome page quota meter added :) Basic graph, larger and constantly visible text for quota info :)
• Basic idempotent lighttpd configurator (alpha, not suitable for production, TESTING only)

23/5/2011

• terminateUser.php http password deletion fix, text fixes, executability.
• checkHttpd.php output lighttpd output, killall -9 lighttpd to ensure shutdown, http fetch mute removed so error gets logged
• update.php: removed apt updates because soft.sh already does this
• crons: increased check http to 4 minute intervals (avoid http timeout period issues)

18/5/2011

Temporary fix to bypass userDb until fully utilized. You may use this update or use /scripts/util/populateUserDb.php to fix the issue.

• Changed listUsers.php to always use systemUsers.php, as fixing userDb saving introduced a bug where ie. checkInstances was not working fully (using userDb, which lacks most users in most servers)

16/5/2011

Few bug fixes and minor features introduced.

• Update script order of items.
• Update script apt-get options for additional package installation lacked auto confirm.
• Update script support for "step 2", /scripts/util/update-step2.php is included if present, thus supporting dynamic updates
• Update script execute binary line added, so it can actually be executed simply by ./update.php
• /scripts permission fix, only root should have anykind of access (no execute or read for others)
• Support for root additional items via /etc/seedbox/config/root.cron.additional
• Add user script minor cleanup, additional commenting.
• Suspend script comments
• simple terminate user script, ALPHA -> not feature complete yet.

Updating from 27/03/2011 release or earlier

cd ~; bash soft.sh; php /scripts/update.php

Updating from 15/5/2011 release

php /scripts/update.php

15/5/2011

Couple important new features introduced, several bug fixes.

• Added lighttpd status checking and auto-restart (very rudimentary)
• Fixed typo in /scripts/util/userConfig.php
• Added /etc/seedbox/config/root.cron
• /scripts/util/quotaFix.php now echoes results (also turns quota off, no need to turn it off first)
• proftpd config update (remote) with MaxClientsPerUser, MaxClientsPerHost, run /scripts/util/ftpConfig.php
• users class now supresses error if runtime db is not found
• users class now saves the user db upon save
• users class deconstructor added to save users db upon exit
• modules support code begun (version for highly dynamic with multiple hooks etc.)
• Changed update.php to a very basic one, as the remote capabilities are not being employed currently
• Rudimentary module /etc/seedbox/module/basic/addUser.php for ultra simplistic user addition module (ie. custom configs, limitations).
• Basic suspend & unsuspend capabilities (untested ALPHA)
• skel permissions fixed

updating

bash soft.sh; php /scripts/update.php

27/03/2011

This is primarily a bug fix and code quality release.

• Set shell to /bin/bash upon user creation
• Set shell to /bin/bash when using userConfig
• quotaFix.php now turns quota off before attempting quota check
• quotaFix.php now displays quota info before and after check
• Lots of comments added to all code files touched
• userConfig.php $filesLimitPerGb was undefined - hardcoded 500
• update.php minor changes
• default version now set to 1, in preparation to take it into production
• startRtorrent: codestyle fixes, Made error messages more clear, named rTorrent screen session to "rtorrent"