[Rails] ActiveRecord retrieval style? Lazy initialtization?

[Berin Loritsch]

I'm using ActiveRecord for an Extract and Transform operation, so it is 
outside of its native environment of Rails.  I am running into a problem 
with memory consumption when I retrieve large amounts of data.  By large 
amounts I am talking about roughly 12,700 records that each have a one 
to many relationship with about 15 additional types of information.  One 
of those types happens to be a table with binaries in it.

Based on the way the system seems to be responding, its as if 
ActiveRecord is creating an array and then using lazy initialization for 
the records.  Creating the ET app was very easy, but I'm running into a 
brick wall here.  I know that ActiveRecord was designed for web systems, 
so being able to process 12,700 complex records is outside of what you 
might consider normal use.

The question I have is whether I can access it in a more fly-weight 
model.  In short, once I'm done with a record I don't need it any more.  
Most database related stuff I've used operates this way.  In short you 
have one object (in Java it would be a ResultSet), and its contents are 
mutated as you iterate over the whole recordset.  Instead of trying to 
create 12,700 objects and its ancillary information, merely creating one 
object and changing its value as you iterate.

Am I going to have to go to a lower level API to do what I need?  If so, 
I would still like to use the ActiveRecord API for each individual 
record.... Using the low level API to get the list of IDs and then 
ActiveRecord to retrieve just the record for that ID.

There's got to be something I'm missing.

-- 
*Berin Loritsch*
Owner 	

*Work:* 571-215-7708
	*Email:* bloritsch at d-haven.com <mailto:bloritsch at d-haven.com>
*IM:* bloritsch at d-haven.org (MSN)
*http://www.linkedin.com/in/bloritsch
*
* * 		*D-Haven* <http://d-haven.org>
20044 Great Falls Forest Dr.
<http://maps.google.com/maps?q=20044+Great+Falls+Forest+Dr.%2CGreat+Falls%2CVA+22066%2CUSA&hl=en> 
Great Falls, VA 22066
USA

See who we know in common <http://www.linkedin.com/e/wwk/5678360/> 	Want 
a signature like this? <http://www.linkedin.com/e/sig/5678360/>

-------------- next part --------------
Skipped content of type multipart/related