[Rails] ActiveRecord retrieval style? Lazy initialtization?

[Jan Prill]

Hi Berin,

aren't you able to use an approach similare to what the paginate method
does?

In a first step cound the records you are going to process then iterate on
them in a paginated way by using 'where id in (id, id, id, id)'

Cheers,
Jan

On 7/27/06, Berin Loritsch <bloritsch at d-haven.org> wrote:
>
>  I'm using ActiveRecord for an Extract and Transform operation, so it is
> outside of its native environment of Rails.  I am running into a problem
> with memory consumption when I retrieve large amounts of data.  By large
> amounts I am talking about roughly 12,700 records that each have a one to
> many relationship with about 15 additional types of information.  One of
> those types happens to be a table with binaries in it.
>
> Based on the way the system seems to be responding, its as if ActiveRecord
> is creating an array and then using lazy initialization for the records.
> Creating the ET app was very easy, but I'm running into a brick wall here.
> I know that ActiveRecord was designed for web systems, so being able to
> process 12,700 complex records is outside of what you might consider normal
> use.
>
> The question I have is whether I can access it in a more fly-weight
> model.  In short, once I'm done with a record I don't need it any more.
> Most database related stuff I've used operates this way.  In short you have
> one object (in Java it would be a ResultSet), and its contents are mutated
> as you iterate over the whole recordset.  Instead of trying to create 12,700
> objects and its ancillary information, merely creating one object and
> changing its value as you iterate.
>
> Am I going to have to go to a lower level API to do what I need?  If so, I
> would still like to use the ActiveRecord API for each individual record....
> Using the low level API to get the list of IDs and then ActiveRecord to
> retrieve just the record for that ID.
>
> There's got to be something I'm missing.
>
> --
>     *Berin Loritsch*
> Owner
>       *Work:* 571-215-7708
>  *Email:* bloritsch at d-haven.com
> *IM:* bloritsch at d-haven.org (MSN)
>  *http://www.linkedin.com/in/bloritsch
> *
> * *  *D-Haven* <http://d-haven.org>
> 20044 Great Falls Forest Dr.
> <http://maps.google.com/maps?q=20044+Great+Falls+Forest+Dr.%2CGreat+Falls%2CVA+22066%2CUSA&hl=en>Great Falls, VA 22066
> USA
>     See who we know in common <http://www.linkedin.com/e/wwk/5678360/> Want
> a signature like this? <http://www.linkedin.com/e/sig/5678360/>
>
> _______________________________________________
> Rails mailing list
> Rails at lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://wrath.rubyonrails.org/pipermail/rails/attachments/20060728/7fe6320c/attachment.html