View source for PM Software Stack

From Pulsed Media Wiki

← PM Software Stack

You do not have permission to edit this page, for the following reason:

---

You can view and copy the source of this page.

PMSS, short of PM Software Stack is a compilation of scripts to manage single server side [[seedbox]] configuration, turning instantly and conveniently an regular [[Debian]] 5.0 [[server]] into a [[seedbox]]. The PMSS does not contain multiserver management utilities nor [[Cluster|clustering]] support out-of-box, but does include [[PM Master GUI]] remote update features. [[Debian]] 6.0 squeeze support is still in unstable status but several production servers already do exist. PMSS currently only supports [[RTorrent|rTorrent]]+[[ruTorrent]] stack. == General features == * Semi-automatic [[Installing PM Software Stack|installation]] * [[Adding a new user|Fast and easy creation of new user accounts]] with [[quota]] and [[rTorrent]] resource limits, etc. * [[Changing user settings|Easy user resource reconfiguration]] * Autonomous, [[idempotent]] [[Lighttpd]] configuration * Autonomous configuration of basic [[server]] features * [[Compiling]] [[RTorrent|rTorrent]] + [[libTorrent (Rakshasa)|libTorrent]] + [[Quota]] + [[XMLRPC]] from [[source]]. * Multiple [[redundancy]] levels for [[rTorrent]] * [[Lighttpd]] [[redundancy]] / autorestart * Basic server optimizations to maximize performance * [[Removing user|Terminate]], [[PMSS Suspending/Unsuspending user|Suspend/Unsuspend]] users * Webdav for users * Other basic management scripts == Using PMSS == See [[:Category:PM Software Stack Guides]] == Changelog == === Changes 2012 === ==== 7/12/2012 ==== This update is pending for release. * Removal of root.cron.additional config from usage, as it does not conform to good practices * Removal of deprecated code from update-step2 relating to vnstat and root.cron.additional * Removing root.cron.additional on update-step2 * Removal of deprecated code from setupRootCron * Removal of /var/run/screen/S-[username] during user termination * Executing setupNetwork on user addition * Added tor repos for debian squeeze * Update-Step2 now checks squeeze repos against the actual configured, not hard coded sha1 string * Removal of deprecated apt update from beginning of update-step2 * Some commenting and refactoring on update-step2 * Remove munin if this is a .pulsedmedia.com server * Adduser now logs to /var/log/pmss * Create remote api key * Salt for local api key * Salt for remote api key * Enhanced algo for remote and local api key, more factors etc. making it more distinguished * chmod api keys * Update-Step2: if no remote api key present, remove local key and execute API key creation ==== 1/11/2012 ==== * Removed updating from OVH kernel to standard debian kernel * No fair share / QoS rules done if OVH kernel is present * Setup network (monitoring, QOS) on update step 2 * Changed root.cron.additional vnstat lines to conform cron.d format * If debian 6 removal of vnstat -u cronjobs * setupRootCron both cron force-reload and restart to ensure new cron will be enabled * Temporarily disable all QoS & Fair share, unintentional effects ==== 31/10/2012 ==== * Display traffic used per user on week, daily, hourly and 15min basis. Note: 15min is slightly inaccurate due to datalogging delays (2min interval), hourly is within acceptable error margin * Fine-tuning of fair share * QoS is always setup and cleared. * recreateUser: Added a check if backup dir already exists. ==== 30/10/2012 ==== * Fixed var/www/index.php redirection * Added script to create traffic monitoring iptables rules * Created /scripts/cron * moved updateQuotas, checkInstances, checkHttpd to cron dir * created /var/log/pmss * Changed checkInstances + checkHttpd logging to /var/log/pmss dir * Refactored cron tasks to utilize /etc/cron.d '''NOTE: If you have customized root crontab this may overwrite the changes!''' * Removed "tester" from user filtration list and added srvapi, pmcseed, pmcdn to the filtration list * Beginnings of network management cron * If an OVH kernel is detected on Deb Squeeze, install standard and set it as default. '''Note: No Deb5 support yet - only changed IF vanilla kernel is not installed''' * Added packages ethtool, elinks, links, lynx * Added Basic QoS to ensure quality of service and fairness in BW sharing - enabled at boot time. '''Note: Does not work on OVH kernel - OVH has removed support for these in their kernel''' * Cron jobs now also save their error output to log files * Traffic logging script and /var/log/pmss/traffic directory for traffic log files. ==== 14/10/2012 ==== * pulsedBox authentication pruning to filter/fix the most common user errors ==== 22/09/2012 ==== * Fixed quota parsin in GUI when using LVM device /dev/mapper/ type addressing * Updated installer to install rTorrent 0.9.2 and libTorrent 0.13.2 * Fixed updater setup of squeeze repos * ruTorrent update, does include saving user preferences etc. to the new one * Fixed a bug relating to .pulsedmedia mgmt features, incorrect order for permissions setting * Update-step2 mild refactoring * Removed useless apt-get install from update.php * Added ffmpeg along with screenshots plugin * Removed unpack plugin (way too high resource consumption) * getfile plugin included now * Applied quota parsing fixed to hddquota plugin and made slightly more dynamic * Lowered time before displaying welcome page quota information ==== 17/09/2012 ==== * Removed proxy from soft.sh as this hasn't been included for sometime now. * Updated rTorrent to 0.9.2 and libTorrent to 0.13.2 official releases ==== 10/06/2012 ==== * PEX + DHT enabled by default. DHT port same as SCGI port for now. Template changed - not the actual code, so in all versions the default has changed. Server specific configuration can disable. ==== 22/05/2012 ==== * Quota display bugs fixed: Servers with UUID assignment instead of device and in some cases collecting the total quota limit had last number cut off ==== 24/04/2012 ==== * Munin configuration updates for .pulsedmedia.com * Updated method of checking ruTorrent version to avoid "updates" when none is needed * More conditions for user space update, if removed user but something remains as backup / as a failure of user termination script * Refactored a bit update-step2 fixing when .rtorrent.rc is updated ==== 06/04/2012 ==== * Compile rTorrent always when version mismatch during update, even if empty/not installed * Added download speed limit when downloading via ajaxManager * Changed chat tab to use username instead of random ==== 27/03/2012 ==== * Update debian 5 repos to the archive ones * On install also enable fastcgi-php (Squeeze) * Get quota from apt repo, and do not compile on install (Lenny support is now limited) * Updated squeeze repos * Slightly refactored update-step2 and install ==== 21/03/2012 ==== * Removed autotools + cpuload plugin from skel rutorrent ==== 20/03/2012 ==== * rTorrent + libtorrent version to be changed to GIT 2011/11/08 which is slightly updated from release 0.8.9, few bug fixes etc. GIT information retained if you want to check validity. * Removal of cpuload + autotools rutorrent plugins. CPULoad being misleading at best, autotools haven't worked for ages and developer is MIA. * Changed fetching libmediainfo packages from pulsedmedia.com instead of sourceforge. * Choosing libmedia info + libs based on debian version. Now Debian 6.0 should be fully supported in updates. * Creating symlink /usr/local/bin/mktorrent -> /usr/bin/mktorrent thus fixing ruTorrent torrent creation in some Debian 6.0 systems (with updated ruTorrent+rTorrent) * Cleaning installed libtorrent ==== 04/03/2012 ==== * Allow key based authentication on SSH * Pulsed Media specific management code - not added unless server is for .pulsedmedia.com * Added srvmgmt to filtered users --- NOTICE: srvadmin user account will be deprecated in a future release ==== 12/02/2012 ==== * added scripts inc for files to be updated/copied on update ==== 10/02/2012 ==== * Fixed torrent creation from ruTorrent * Fixed quota plugin (JSON parser has been changed to strict parsing) * Updated quota plugin version information * Fixed ajax manager recycle bin * Made ruTorrent to update browser cache * Created framework for easy user space files updating during server update (files dependant upon server upgrade) * Removed recreateLighttpdConfig + lighttpd restart from update step 2 (should by now be in main update in 99.9%+ servers) * Changed update.php to do lighttpd force-reload instead of restart * Refactored update-step2 slightly: Moved specific files, and user specific stuff into single loop * Added several packages to be installed during server installation, which were only on update. They still remain on update as well before better solution can be done (dselect) * Filtering backup user dirs out from user list * During scripts update, also update system packages ==== 09/02/2012 ==== * Reversing yesterday's update due to security concerns * Removed installer speedups due to stability concerns ==== 08/02/2012 ==== * Fixed typo in install script preventing quota from being compiled ==== 05/02/2012 ==== * Lighttpd session ''"stability"'' upgrade, fixing issue causing session timeouts/re-requiring authentication ==== 04/02/2012 ==== * Added mediainfo * Added server root document to redirect to pulsedmedia.com * Added php5-geoip extension to be installed * Simplified update-step2 scripts being installed :) * Removed apt dist-upgrade from install * Simplified (less) apt-get lines during install * Reordered apt-get commands during install for swifter installation * Removed some "junk" code from install * Updated copyright text in install.sh * Added some information on header of install.sh * Install.sh some additional comments * Made installation slightly quicker by usage of background processes (BETA, if your disk is extremely slow this may cause installation to fail) * Reordering other items in install.sh to make installation faster * Some preliminary work for supporting whitelabeling * Removed umask, hash interval, hash max tries rtorrent configures (deprecated) * '''Updated rTorrent + ruTorrent to versions 0.9.0/0.13.0''' ''Note:'' Please report any issues due to the upgrade of rTorrent immediately to Pulsed Media. The patch has been tested on a small group of servers, but all situations are hard to foresee. ==== 31/01/2012 ==== * Fixed some typos in update-step2 ==== 28/01/2012 ==== * Added iotop and cfv to be installed from apt repo. * Added -m toggle for htpasswd because for some reason this is not default in Debian Lenny * Added SSL support (self signed cert), just change http to https * Added Webdav support, accessible from "/webdav-'''USERNAME'''/" *** BETA: Permission issues, your mileage will vary * Removed SSL key creation from install * Update-step2 creates SSL key automatically utilizing the server hostname, removes old deprecated key === Changes 2011 === ==== 15/12/2011 ==== * Added '''.irssi/config''' to user dirs, autojoining to #PulsedMedia on Freenode when irssi is launched * Updated GUI IRC links, Chat tab and Adobe download links, with other small additional information and updates. * Removed Network Storage tab from GUI because Dropbox is blocking pageviews coming from our system for unknown reason. * Added Proxy information on GUI * Update-step2 sees if .irssi/config exists and if it does not copies it there ==== 30/11/2011 ==== * Chosen rTorrent SCGI port range increased from 4k to 8k. * "fractional blocks" for config. Resource calcs are based on number of blocks, and fractions were not allowed prior. ==== 29/11/2011 ==== * During update also update user crontabs. Added to both update.php and step2, to be removed from step2 after this update has seen significant adoption. * Restarting lighttpd during update, also done doubly for now * Moved setup api key to regular update script. In several months it will potentially be removed from update all together as it's a installation step, and included to update older nodes. * Scripts permissions update * updateQuotas.php -> cron to update all user quotas, as discreet user from central cron to ensure it gets updated. * Updated root & user crontabs to reflect updateQuotas ==== 28/11/2011 ==== * quotaFix now removes aquota*new files to avoid collision and thus failed update via quotacheck * userTorrents.php : Lists the count of torrents each user has, for quicker analysis usage stats ==== 23/11/2011 ==== * Permissions setup script added, which is utilized by recreateUser. * recreateUser.php fixes ==== 22/11/2011 ==== * Created recreateUser.php which recreates home dir, preserving data+session if existing. ''ALPHA'' ==== 19/11/2011 ==== * Fixed saving API Key. ==== 17/11/2011 ==== * Suspend and Unsuspend scripts fixed * recreateLighttpd backup fixed * Added access format of: '''servername/user-''username''/''', for example: ''localhost/user-johndoe/''. Thus supporting non FQDN setups * addUser.php calls recreateLighttpd instead of just adding the hostname, this ensures lighttpd config is ''always'' up-to-date. * Added to both update steps recreateLighttpd. It will remain on step2 for the next ~6months to ensure every system out there will have the update update.php * Removed the default "testing" torrents * Upload slots global, by default 6 times the non-global upload slots. It is ''"forcefully"'' inserted even if template does not it have for the moment. this forcefull insert will be removed in the future. * checkInstances now also checks rtorrent configs, and if changed lists them on /root/changedConfigs for admin review ==== 25/8/2011 ==== Streamlined installer for faster setups :) First release of API, just one command now. Some things moved to BETA status and considered feature complete. Many bug fixes, additional commenting etc. also added * fixed userTransfer chowning lacking -R, thus UID was transferred for many files from the other server causing incorrect permissions * wrapped [[vnstat]] install into if !file_exists to see if [[vnstat]] is installed already. * added [[username]] 'srvadmin' to filter list on systemUsers * removed from installer editing munin.conf -> if used can be done manually. Interim solution. * put install [[vim]] as first thing on installer * setting [[node]] [[hostname]] straight after installing vim * moved setting up [[lighttpd]] as last thing * scripts, skel, etc are installed earlier in installation * [[SSL]] cert + [[quota]] as last items on setup * Executing update-step2 on installer (so you don't need to execute updater after setup to get everything in) * /scripts/util/setupRootCron.php -> creating root conf moved out of update.php * setupSkelPermissions.php, same reason as root cron setup * added setupRootCron and setupSkelPermissions to installer * [[Idempotent]] lighttpd configurator feature completed. Backups the config first. * Terminate user moved to beta status, as using lighttpd configurator now and restarting lighttpd. * Added for terminate second argument, if 2nd argument is --confirm, straight accepts termination without asking for approval, for making automation easier. * checkHttpd timer increased to 3 seconds before restarting lighttpd * terminateUser timer increased to 5 seconds after trying to kill all user processes * setupApiKey -> creates node key for API. Invoked upon setup. * added authorization key * removed /var/www/index.lighttpd.html * added /var/www/index.php redir to pulsedmedia.com * added beginnings of remote [[API]] for easier remote control management, by default in debug mode for now ==== 8/6/2011 ==== Bug fixes. terminateUser.php: * Was not actually completing shell userdeletion * Added killing user processes (forgotten) * added sleeping 2 seconds to allow time for rTorrent/other processes by user to die * Added 2nd attempt to userdel after data removal * Fixed some typos in texts * Added check for user actually exists '''NOTE: Still Alpha, non feature complete and not recommended for production use.''' * checkHttpd.php added sleep between lighttpd kill and restart, to avoid PID file mismatch scenario update-step2.php: * First time being used * update.php moved cron & ftp config stuff AFTER step2 * Install vnstat + vim (vim not by default on Debian 6) * Setup interface for vnstat * Configure vnstat for Debian 6 * Make vnstat cron jobs for Debian 5: twice a minute update (thus no support for 1GigE interface) * Debian 5 add cron jobs to root.cron.additional config, create if needed ==== 31/5/2011 ==== * GUI welcome page checking if proper quota information was passed has been added. * GUI welcome page malformed serialized data fix * GUI welcome page quota percentage missing % char * GUI welcome page quota text styles: White and location ==== 30/5/2011 ==== * GUI update script quota checking added (doesn't try to update when user is over quota burst limit, no more blanking GUI when update happens!) * Users default quota information update rate has been increased to 10mins * HDD Quota plugin commenting added * HDD Quota plugin parser bug with bursted amounts fixed * HDD Quota now shows USED/QUOTA instead of FREE/QUOTA. Makes more sense. * HDD Quota "title" fixed (showing quota info) to be whole area of the meter * Welcome page quota meter added :) Basic graph, larger and constantly visible text for quota info :) * Basic [[Idempotency|idempotent]] lighttpd configurator (alpha, not suitable for production, TESTING only) ==== 23/5/2011 ==== * terminateUser.php http password deletion fix, text fixes, executability. * checkHttpd.php output lighttpd output, killall -9 lighttpd to ensure shutdown, http fetch mute removed so error gets logged * update.php: removed apt updates because soft.sh already does this * crons: increased check http to 4 minute intervals (avoid http timeout period issues) ==== 18/5/2011 ==== Temporary fix to bypass userDb until fully utilized. You may use this update or use /scripts/util/populateUserDb.php to fix the issue. * Changed listUsers.php to always use systemUsers.php, as fixing userDb saving introduced a bug where ie. checkInstances was not working fully (using userDb, which lacks most users in most servers) ==== 16/5/2011 ==== Few bug fixes and minor features introduced. * Update script order of items. * Update script apt-get options for additional package installation lacked auto confirm. * Update script support for ''"step 2"'', /scripts/util/update-step2.php is included if present, thus supporting dynamic updates * Update script execute binary line added, so it can actually be executed simply by ./update.php * /scripts permission fix, only root should have anykind of access (no execute or read for others) * Support for root additional items via /etc/seedbox/config/root.cron.additional * Add user script minor cleanup, additional commenting. * Suspend script comments * simple terminate user script, ALPHA -> not feature complete yet. '''Updating from 27/03/2011 release or earlier''' cd ~; bash soft.sh; php /scripts/update.php '''Updating from 15/5/2011 release''' php /scripts/update.php ==== 15/5/2011 ==== Couple important new features introduced, several bug fixes. * Added [[lighttpd]] status checking and auto-restart (very rudimentary) * Fixed typo in /scripts/util/userConfig.php * Added /etc/seedbox/config/root.cron * /scripts/util/quotaFix.php now echoes results (also turns [[quota]] off, no need to turn it off first) * [[proftpd]] config update (remote) with MaxClientsPerUser, MaxClientsPerHost, run /scripts/util/ftpConfig.php * users class now supresses error if runtime db is not found * users class now saves the user db upon save * users class deconstructor added to save users db upon exit * modules support code begun (version for highly dynamic with multiple hooks etc.) * Changed update.php to a very basic one, as the remote capabilities are not being employed currently * Rudimentary module ''/etc/seedbox/module/basic/addUser.php'' for ultra simplistic user addition module (ie. custom configs, limitations). * Basic suspend & unsuspend capabilities (untested ALPHA) * skel permissions fixed '''updating''' bash soft.sh; php /scripts/update.php ==== 27/03/2011 ==== This is primarily a bug fix and code quality release. * Set shell to /bin/bash upon user creation * Set shell to /bin/bash when using userConfig * quotaFix.php now turns quota off before attempting quota check * quotaFix.php now displays quota info before and after check * Lots of comments added to all code files touched * userConfig.php $filesLimitPerGb was undefined - hardcoded 500 * update.php minor changes * default version now set to 1, in preparation to take it into production * startRtorrent: codestyle fixes, Made error messages more clear, named rTorrent screen session to "rtorrent"

Return to PM Software Stack.