[Rails] RubyGems runtime memory use in Rails apps...

[Stef T]

Hello Steve,
    No no, your definitely not a pariah, and your emails are definitely 
making it onto the list ;) A crazy thought appears, perhaps woefully and 
stupidly in my head, but, its 1AM EST here, so, forgive me for this, 
but, why not simply use the design pattern of Singleton in 
custom_require.rb if you believe this is the problem ?

    There now follows a rather clunky 'human diff' .. please bear with me :)

    at line 24 (in the require method)
        @gempath_searcher ||= Gem::GemPathSearcher.new
    should become
        @gempath_searcher = Gem::GemPathSearcher.instance

    And then you want to change the module Gem part below to look 
something like;

    module Gem
      require 'singleton'
      #
      # GemPathSearcher has the capability to find loadable files inside
      # gems.  It generates data up front to speed up searches later.
      #
      class GemPathSearcher
            include Singleton

    If you then put a 'puts @gempath_searcher.object_id' after the line 
24 change, and run (say) your ruby ./script/server, you will see a huge 
spew of lines like ;

    stef at stef-desktop:~/work/myTest$ ruby ./script/server
    -606820690
    -606820690
    -606820690

    If you 'undo' those changes, and go back to the good old-fashioned 
code, you will see something like;

    stef at stef-desktop:~/work/myTest$ ruby ./script/server
    -606681356
    -606869086
    -606681356
    -606818976

    Different object_id's means different object's .. so .. by using the 
Singleton design pattern, your gemserver will now only be created 
'once'. Now, of course, wether or not this breaks things horrible OR if 
this does indeed fix your memory leakage, you will have to let us all 
know :)

    Either way, have a nice night, I am shuffling off to bed, for some 
well earned R&R. Let me know how it all works out though please, 
irregardless ;)
    Regards
    Stef
(ps. I could never worship Satan, but only Cthulhu, mean to say, why 
worship a lesser evil ;D
(pps. that -is- a joke for the humour impaired ;)
Steve Longdo wrote:
> Stef!!!!
>
>   Thank you for answering at all!  I am a leper on the list it feels 
> like sometimes.  I am profiling Typo, an app in need of some very 
> "Mature Optimization".  So no worries there.  Locally I use OS X/PPC, 
> Ruby 1.8.4. However I am doing the real profiling against my 
> www.stevelongdo.com <http://www.stevelongdo.com> site on TextDrive.  
> It is also Ruby 1.8.4 and I think a Linux flavor not Solaris 10 yet 
> anyway.
>
> Typo and its external dependant version of Rails were trunk a couple 
> days ago.  TextDrive limits are based on RSZ going over 48MB.  VSZ is 
> almost always lower than that when Ruby decides to allocate some more 
> space and send RSZ over the limit. I am not sure whether they are 
> using Stefan Kaes suggested patched GC or if it is plain vanilla Ruby 
> GC.  I don't for sure how to find that out on TextDrive.
>
> At any rate there is no good reason for my personal Typo weblog to 
> cross into the 48MB zone.  I'll bet all of my content together is 512kB. 
>
> Outside of all that though I am trying to learn more about how 
> RubyGems actually does its business and whether it could do it in less 
> than ~11MB of RAM.  I am barreling into the code now since response to 
> my "plea" for help went unanswered.  I would expect that Shared Host 
> providers would want to have somebody look at this because it should 
> be an issue for all Rails apps not just Typo.  Imagine if we figure 
> out how to shave 10MB off of each Rails app out there?  Shared Hosts 
> become more viable, people send me presents and bow in my presence...
>
> I've gotten a bit further in my research tonight take a look at my 
> latest post: 
> http://www.stevelongdo.com/articles/2006/08/09/rubygems-single-multi-tons-and-memorysize 
> <http://www.stevelongdo.com/articles/2006/08/09/rubygems-single-multi-tons-and-memorysize> 
>
>
> All hail Stef our dark underlord...Oh not affiliated with Satan, I 
> see.  :-P
>
> On 8/8/06, *Stef T* <stef at ummon.com <mailto:stef at ummon.com>> wrote:
>
>
>     Hello Steve,
>         Well, since you asked for help, can I suggest that you provide
>     pertinent details ? OS used, ruby version, and also the example code
>     that you used to profile would be nice ;) I must say, that rubygems
>     hooks into the main process from what I can (hence the require) so I
>     would -expect- it to stay resident in memory. Which reminds me, what
>     memory are you profiling here ? VSZ ? RSZ ? Then again, in the day and
>     age of 1GB in my desktop machine, its probably highly easy to
>     dismiss a
>     RSZ of 10MB ;) Heck, almost all of the servers at work have >4GB.
>     Thats
>     not the point, I know, but, there is no point in chasing rabbits down
>     holes if you don't need to (also called the 'premature optimisation'
>     problem).
>
>         Details details, the devil is in the details*
>         Regards
>         Stef
>     (* this should not be taken as an endorsement of satan nor that I have
>     -the- answer to your problems ;)
>
>     Steve Longdo wrote:
>     > This an open plea for someone with knowledge of RubyGems to explain
>     > why RubyGems stays in memory (over 10MB) after a Rails
>     application is
>     > loaded.  I am unable to figure out why this would be necessary. I
>     > haven't had any response from Jim or Chad from RubyGems about it
>     so I
>     > thought I would throw it open to the list to see if anyone could
>     > educate me or share thier experiences with this.
>     >
>     > There is a more detailed write up on my blog (
>     >
>     http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use
>     >
>     <http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use
>     <http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use>>
>     > ).
>     >
>
> -
> Thanks,
> -Steve
> http://www.stevelongdo.com