Lazily column retrieval

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Lazily column retrieval

Gili
Hi,

        I'm looking for something similar to
http://objectstyle.org/jira/secure/ViewIssue.jspa?key=CAY-316 which will
allow me to lazily retrieve certain blob columns. I was told someone on
this list might already have an ExtendedType which does this?

Gili
--
http://www.desktopbeautifier.com/
Reply | Threaded
Open this post in threaded view
|

Re: Lazily column retrieval

Andrus Adamchik
I haven't heard of anyone posting a streaming [C|B]LOB ExtendedType.  
A generic implementation would require some thought, but a simplified  
read-only custom version should be pretty easy to make.

Reading blobs is much easier that writing them. Just implement  
"getClassName()" to return "java.sql.Blob" and "materializeObject
(..)" to call "getBlob()". Don't recall if some DB's (Oracle?)  
required an open connection that created a BLOB for the BLOB object  
to be accessible. If this is the case, I can think of a few  
workarounds for that.

If you are going to try it and willing to share your code, please  
attach to CAY-316.

Thanks
Andrus

On Aug 27, 2005, at 1:05 AM, Gili wrote:

> Hi,
>
>     I'm looking for something similar to http://objectstyle.org/ 
> jira/secure/ViewIssue.jspa?key=CAY-316 which will allow me to  
> lazily retrieve certain blob columns. I was told someone on this  
> list might already have an ExtendedType which does this?
>
> Gili
> --
> http://www.desktopbeautifier.com/

Reply | Threaded
Open this post in threaded view
|

Re: Lazily column retrieval

Gili
Hi Andrus,

        You're one of the main developers of Cayenne if I remember correctly :)
Could you possibly help me add pissimistic locking support to Cayenne?
I've got a use-case that requires it.

        I'm hoping to add both pissimistic locking and streaming BLOBs to the
next release but I am new to Cayenne so I don't know where exactly to start.

Thanks,
Gili

Andrus Adamchik wrote:

> I haven't heard of anyone posting a streaming [C|B]LOB ExtendedType.  A
> generic implementation would require some thought, but a simplified  
> read-only custom version should be pretty easy to make.
>
> Reading blobs is much easier that writing them. Just implement  
> "getClassName()" to return "java.sql.Blob" and "materializeObject (..)"
> to call "getBlob()". Don't recall if some DB's (Oracle?)  required an
> open connection that created a BLOB for the BLOB object  to be
> accessible. If this is the case, I can think of a few  workarounds for
> that.
>
> If you are going to try it and willing to share your code, please  
> attach to CAY-316.
>
> Thanks
> Andrus
>
> On Aug 27, 2005, at 1:05 AM, Gili wrote:
>
>> Hi,
>>
>>     I'm looking for something similar to http://objectstyle.org/ 
>> jira/secure/ViewIssue.jspa?key=CAY-316 which will allow me to  lazily
>> retrieve certain blob columns. I was told someone on this  list might
>> already have an ExtendedType which does this?
>>
>> Gili
>> --
>> http://www.desktopbeautifier.com/
>
>
>

--
http://www.desktopbeautifier.com/
Reply | Threaded
Open this post in threaded view
|

Re: Lazily column retrieval

Andrus Adamchik

On Aug 27, 2005, at 3:41 PM, Gili wrote:

> Could you possibly help me add pessimistic locking support to  
> Cayenne? I've got a use-case that requires it.
>
>     I'm hoping to add both pessimistic locking and streaming BLOBs  
> to the next release but I am new to Cayenne so I don't know where  
> exactly to start.

Hi Gili,

You've already found ExtendedType API. This is a place to start doing  
streaming BLOBs.

I am not a big fun of pessimistic locking (possibility of deadlocks,  
lowered overall throughput and inefficient resource use...) and as a  
result don't have much experience with it. So I haven't given it much  
thought and won't be able to provide a good advice without doing some  
research, including cross-db issues.

However I am all for including this feature for completeness. It can  
definitely be useful. So if you start on the implementation, I'd  
suggest you to subscribe to cayenne-devel and post your questions  
there. Time permitting, myself and, I am sure, others would be able  
to help with integration.

One piece of it is already in place - attributes and relationships  
can be tagged as a part of a lock. This is currently used for  
optimistic locking and can be extended to pessimistic locking (though  
AFAIK not all db allow column-level locking ... this will have to be  
abstracted through DbAdapter SQLActions).

Not sure about the flow, i.e. how the lock is created... Probably  
some method at the DataContext level (DataContext.lock()) that would  
result in all select queries to be executed with "FOR UPDATE" clause  
until the next commit or rollback.

Also if I understand correctly, JDBC Connection that created a lock  
must be the one to unlock it (is that so??) This would be a big  
change to the current model of a "disconnected" DataContext, when  
generally speaking each select or commit operation gets its own  
Connection from the pool. We'll have to open a whole can of worms  
with an open connection tied to an execution thread. Or maybe this is  
not as bad as I think it is...

Andrus

Reply | Threaded
Open this post in threaded view
|

RE: Lazily column retrieval

Gentry, Michael (Contractor)
In reply to this post by Gili
Gili, why is it that you need pessimistic locking over optimistic
locking?  In other words, what is your use-case?  I'd be hard-pressed to
come up with a situation where I'd prefer pessimistic locking.

Thanks,

/dev/mrg


-----Original Message-----
From: Gili [mailto:[hidden email]]
Sent: Saturday, August 27, 2005 3:41 PM
To: [hidden email]
Subject: Re: Lazily column retrieval


Hi Andrus,

        You're one of the main developers of Cayenne if I remember
correctly :)
Could you possibly help me add pissimistic locking support to Cayenne?
I've got a use-case that requires it.

        I'm hoping to add both pissimistic locking and streaming BLOBs
to the
next release but I am new to Cayenne so I don't know where exactly to
start.

Thanks,
Gili

Andrus Adamchik wrote:
> I haven't heard of anyone posting a streaming [C|B]LOB ExtendedType.
A
> generic implementation would require some thought, but a simplified  
> read-only custom version should be pretty easy to make.
>
> Reading blobs is much easier that writing them. Just implement  
> "getClassName()" to return "java.sql.Blob" and "materializeObject
(..)"
> to call "getBlob()". Don't recall if some DB's (Oracle?)  required an
> open connection that created a BLOB for the BLOB object  to be
> accessible. If this is the case, I can think of a few  workarounds for

> that.
>
> If you are going to try it and willing to share your code, please  
> attach to CAY-316.
>
> Thanks
> Andrus
>
> On Aug 27, 2005, at 1:05 AM, Gili wrote:
>
>> Hi,
>>
>>     I'm looking for something similar to http://objectstyle.org/ 
>> jira/secure/ViewIssue.jspa?key=CAY-316 which will allow me to  lazily

>> retrieve certain blob columns. I was told someone on this  list might

>> already have an ExtendedType which does this?
>>
>> Gili
>> --
>> http://www.desktopbeautifier.com/
>
>
>

--
http://www.desktopbeautifier.com/
Reply | Threaded
Open this post in threaded view
|

Re: Lazily column retrieval

Gili

        I've implemented the algorithm found here
http://www.usenix.org/events/usits99/full_papers/bahn/bahn_html/

        I've got two tables:

image_cache_stats, which contains [num_requests, num_hits]
image_cache, which contains cache entries [score, time, cachedImage]

        In a typical request, num_requests is incremented, and a few entries
might be added or removed from image_cache. If multiple clients try
hitting the cache at the same time with optimistic locking they will
always collide because num_requests is being incremented by every
client. If you've got an idea for improving on this I'd be more than
glad to try it!

Thanks,
Gili

Gentry, Michael (Contractor) wrote:

> Gili, why is it that you need pessimistic locking over optimistic
> locking?  In other words, what is your use-case?  I'd be hard-pressed to
> come up with a situation where I'd prefer pessimistic locking.
>
> Thanks,
>
> /dev/mrg
>
>
> -----Original Message-----
> From: Gili [mailto:[hidden email]]
> Sent: Saturday, August 27, 2005 3:41 PM
> To: [hidden email]
> Subject: Re: Lazily column retrieval
>
>
> Hi Andrus,
>
> You're one of the main developers of Cayenne if I remember
> correctly :)
> Could you possibly help me add pissimistic locking support to Cayenne?
> I've got a use-case that requires it.
>
> I'm hoping to add both pissimistic locking and streaming BLOBs
> to the
> next release but I am new to Cayenne so I don't know where exactly to
> start.
>
> Thanks,
> Gili
>
> Andrus Adamchik wrote:
>
>>I haven't heard of anyone posting a streaming [C|B]LOB ExtendedType.
>
> A
>
>>generic implementation would require some thought, but a simplified  
>>read-only custom version should be pretty easy to make.
>>
>>Reading blobs is much easier that writing them. Just implement  
>>"getClassName()" to return "java.sql.Blob" and "materializeObject
>
> (..)"
>
>>to call "getBlob()". Don't recall if some DB's (Oracle?)  required an
>>open connection that created a BLOB for the BLOB object  to be
>>accessible. If this is the case, I can think of a few  workarounds for
>
>
>>that.
>>
>>If you are going to try it and willing to share your code, please  
>>attach to CAY-316.
>>
>>Thanks
>>Andrus
>>
>>On Aug 27, 2005, at 1:05 AM, Gili wrote:
>>
>>
>>>Hi,
>>>
>>>    I'm looking for something similar to http://objectstyle.org/ 
>>>jira/secure/ViewIssue.jspa?key=CAY-316 which will allow me to  lazily
>
>
>>>retrieve certain blob columns. I was told someone on this  list might
>
>
>>>already have an ExtendedType which does this?
>>>
>>>Gili
>>>--
>>>http://www.desktopbeautifier.com/
>>
>>
>>
>

--
http://www.desktopbeautifier.com/
Reply | Threaded
Open this post in threaded view
|

RE: Lazily column retrieval

Gentry, Michael (Contractor)
In reply to this post by Gili
That's way too much for me to read through right now ...

For your image_cache_stats table, two things come to mind:

1) If your database supports it, you could write a stored procedure to
begin a transaction, update the values, and return them back to Cayenne.

2) Use SQLTemplate to send an UPDATE statement (something like: UPDATE
image_cache_stats set num_requests = num_requests + 1) and then do a
query to read the values back in again (if you need them at that point).

I'm not certain how your image_cache table functions, but a similar
approach might be sufficient for that, too.  This is far simpler than
adding pessimistic locking to Cayenne and should perform better, too.

/dev/mrg

PS. You could also use SQLTemplate to do a SELECT ... FOR UPDATE (to
lock the row), too.  Just be sure you release the locks.  Be especially
careful to use a try/catch/finally around everything you do and in the
finally clause, release the locks.  Otherwise if there is an exception,
deadlocks will get you.


-----Original Message-----
From: Gili [mailto:[hidden email]]
Sent: Monday, August 29, 2005 8:56 AM
To: [hidden email]
Subject: Re: Lazily column retrieval



        I've implemented the algorithm found here
http://www.usenix.org/events/usits99/full_papers/bahn/bahn_html/

        I've got two tables:

image_cache_stats, which contains [num_requests, num_hits]
image_cache, which contains cache entries [score, time, cachedImage]

        In a typical request, num_requests is incremented, and a few
entries
might be added or removed from image_cache. If multiple clients try
hitting the cache at the same time with optimistic locking they will
always collide because num_requests is being incremented by every
client. If you've got an idea for improving on this I'd be more than
glad to try it!

Thanks,
Gili

Gentry, Michael (Contractor) wrote:
> Gili, why is it that you need pessimistic locking over optimistic
> locking?  In other words, what is your use-case?  I'd be hard-pressed
to

> come up with a situation where I'd prefer pessimistic locking.
>
> Thanks,
>
> /dev/mrg
>
>
> -----Original Message-----
> From: Gili [mailto:[hidden email]]
> Sent: Saturday, August 27, 2005 3:41 PM
> To: [hidden email]
> Subject: Re: Lazily column retrieval
>
>
> Hi Andrus,
>
> You're one of the main developers of Cayenne if I remember
> correctly :)
> Could you possibly help me add pissimistic locking support to Cayenne?

> I've got a use-case that requires it.
>
> I'm hoping to add both pissimistic locking and streaming BLOBs
> to the
> next release but I am new to Cayenne so I don't know where exactly to
> start.
>
> Thanks,
> Gili
>
> Andrus Adamchik wrote:
>
>>I haven't heard of anyone posting a streaming [C|B]LOB ExtendedType.
>
> A
>
>>generic implementation would require some thought, but a simplified  
>>read-only custom version should be pretty easy to make.
>>
>>Reading blobs is much easier that writing them. Just implement  
>>"getClassName()" to return "java.sql.Blob" and "materializeObject
>
> (..)"
>
>>to call "getBlob()". Don't recall if some DB's (Oracle?)  required an
>>open connection that created a BLOB for the BLOB object  to be
>>accessible. If this is the case, I can think of a few  workarounds for
>
>
>>that.
>>
>>If you are going to try it and willing to share your code, please  
>>attach to CAY-316.
>>
>>Thanks
>>Andrus
>>
>>On Aug 27, 2005, at 1:05 AM, Gili wrote:
>>
>>
>>>Hi,
>>>
>>>    I'm looking for something similar to http://objectstyle.org/ 
>>>jira/secure/ViewIssue.jspa?key=CAY-316 which will allow me to  lazily
>
>
>>>retrieve certain blob columns. I was told someone on this  list might
>
>
>>>already have an ExtendedType which does this?
>>>
>>>Gili
>>>--
>>>http://www.desktopbeautifier.com/
>>
>>
>>
>

--
http://www.desktopbeautifier.com/
Reply | Threaded
Open this post in threaded view
|

Re: Lazily column retrieval

Gili

        I could do that but it would cause consistency issues. For one, I don't
want the image_cache_stats table to update if the image_cache failed to
update. I need a transaction to wrap the entire operation to ensure it
fully goes through or not.

        Secondly, the way the algorithm works, if I try adding a cache entry to
the pool and find out it is full I am supposed to discard the entries
with the lowest score. This requires me to update all rows in the table
(to update their individual scores) and then issue a query sorting them
in ascending score (to remove the lowest score entries first). When this
operation occurs, it pretty much requires me to lock the entire table
because all entries will be touched.

        I am pretty sure I could update their scores one a time without many
collisions but updating *all* their scores within a single transaction
is almost guaranteed to encounter collisions. I'm not sure it makes
sense to create a new transaction per row I update; this seems a bit
overkill.

Gili

Gentry, Michael (Contractor) wrote:

> That's way too much for me to read through right now ...
>
> For your image_cache_stats table, two things come to mind:
>
> 1) If your database supports it, you could write a stored procedure to
> begin a transaction, update the values, and return them back to Cayenne.
>
> 2) Use SQLTemplate to send an UPDATE statement (something like: UPDATE
> image_cache_stats set num_requests = num_requests + 1) and then do a
> query to read the values back in again (if you need them at that point).
>
> I'm not certain how your image_cache table functions, but a similar
> approach might be sufficient for that, too.  This is far simpler than
> adding pessimistic locking to Cayenne and should perform better, too.
>
> /dev/mrg
>
> PS. You could also use SQLTemplate to do a SELECT ... FOR UPDATE (to
> lock the row), too.  Just be sure you release the locks.  Be especially
> careful to use a try/catch/finally around everything you do and in the
> finally clause, release the locks.  Otherwise if there is an exception,
> deadlocks will get you.
>
>
> -----Original Message-----
> From: Gili [mailto:[hidden email]]
> Sent: Monday, August 29, 2005 8:56 AM
> To: [hidden email]
> Subject: Re: Lazily column retrieval
>
>
>
> I've implemented the algorithm found here
> http://www.usenix.org/events/usits99/full_papers/bahn/bahn_html/
>
> I've got two tables:
>
> image_cache_stats, which contains [num_requests, num_hits]
> image_cache, which contains cache entries [score, time, cachedImage]
>
> In a typical request, num_requests is incremented, and a few
> entries
> might be added or removed from image_cache. If multiple clients try
> hitting the cache at the same time with optimistic locking they will
> always collide because num_requests is being incremented by every
> client. If you've got an idea for improving on this I'd be more than
> glad to try it!
>
> Thanks,
> Gili
>
> Gentry, Michael (Contractor) wrote:
>
>>Gili, why is it that you need pessimistic locking over optimistic
>>locking?  In other words, what is your use-case?  I'd be hard-pressed
>
> to
>
>>come up with a situation where I'd prefer pessimistic locking.
>>
>>Thanks,
>>
>>/dev/mrg
>>
>>
>>-----Original Message-----
>>From: Gili [mailto:[hidden email]]
>>Sent: Saturday, August 27, 2005 3:41 PM
>>To: [hidden email]
>>Subject: Re: Lazily column retrieval
>>
>>
>>Hi Andrus,
>>
>> You're one of the main developers of Cayenne if I remember
>>correctly :)
>>Could you possibly help me add pissimistic locking support to Cayenne?
>
>
>>I've got a use-case that requires it.
>>
>> I'm hoping to add both pissimistic locking and streaming BLOBs
>>to the
>>next release but I am new to Cayenne so I don't know where exactly to
>>start.
>>
>>Thanks,
>>Gili
>>
>>Andrus Adamchik wrote:
>>
>>
>>>I haven't heard of anyone posting a streaming [C|B]LOB ExtendedType.
>>
>>A
>>
>>
>>>generic implementation would require some thought, but a simplified  
>>>read-only custom version should be pretty easy to make.
>>>
>>>Reading blobs is much easier that writing them. Just implement  
>>>"getClassName()" to return "java.sql.Blob" and "materializeObject
>>
>>(..)"
>>
>>
>>>to call "getBlob()". Don't recall if some DB's (Oracle?)  required an
>>>open connection that created a BLOB for the BLOB object  to be
>>>accessible. If this is the case, I can think of a few  workarounds for
>>
>>
>>>that.
>>>
>>>If you are going to try it and willing to share your code, please  
>>>attach to CAY-316.
>>>
>>>Thanks
>>>Andrus
>>>
>>>On Aug 27, 2005, at 1:05 AM, Gili wrote:
>>>
>>>
>>>
>>>>Hi,
>>>>
>>>>   I'm looking for something similar to http://objectstyle.org/ 
>>>>jira/secure/ViewIssue.jspa?key=CAY-316 which will allow me to  lazily
>>
>>
>>>>retrieve certain blob columns. I was told someone on this  list might
>>
>>
>>>>already have an ExtendedType which does this?
>>>>
>>>>Gili
>>>>--
>>>>http://www.desktopbeautifier.com/
>>>
>>>
>>>
>

--
http://www.desktopbeautifier.com/
Reply | Threaded
Open this post in threaded view
|

RE: Lazily column retrieval

Gentry, Michael (Contractor)
In reply to this post by Gili
Well, sending an "UPDATE image_cache_stats SET num_requests =
num_requests + 1" is highly unlikely to fail (your DB would have to
tank, in which case there is nothing Cayenne can do).

Are you trying to limit the image_cache_stats to a set size?  You could
do that, adding new entries and deleting others and then do a
dataContext.commitChanges(), but that seems like more trouble that it is
worth.

Could you not leave old entries in the image_cache_stats table do a
query against it with an ORDER BY and a LIMIT to control the data coming
back?  I'd let the database do the work since it is good at it.  If you
did have a need to prune the size of the table, you could always issue a
DELETE using SQLTemplate to get rid of rows that fall below a certain
threshold.

I guess what I'm saying is don't be afraid to use an SQLTemplate when
you have a valid need for it (that's what I do when it makes sense).
Let the database do work for you when appropriate.

As for the consistency issue, as soon as you refetch the table data,
you'll be up-to-date again in Cayenne.  It might make sense to put all
of this logic in a helper class (complete with it's own DataContext) to
manage it.  You should use a synchronize block around the update code to
let Java serialize that portion.

/dev/mrg


-----Original Message-----
From: Gili [mailto:[hidden email]]
Sent: Monday, August 29, 2005 10:24 AM
To: [hidden email]
Subject: Re: Lazily column retrieval



        I could do that but it would cause consistency issues. For one,
I don't
want the image_cache_stats table to update if the image_cache failed to
update. I need a transaction to wrap the entire operation to ensure it
fully goes through or not.

        Secondly, the way the algorithm works, if I try adding a cache
entry to
the pool and find out it is full I am supposed to discard the entries
with the lowest score. This requires me to update all rows in the table
(to update their individual scores) and then issue a query sorting them
in ascending score (to remove the lowest score entries first). When this

operation occurs, it pretty much requires me to lock the entire table
because all entries will be touched.

        I am pretty sure I could update their scores one a time without
many
collisions but updating *all* their scores within a single transaction
is almost guaranteed to encounter collisions. I'm not sure it makes
sense to create a new transaction per row I update; this seems a bit
overkill.

Gili

Gentry, Michael (Contractor) wrote:
> That's way too much for me to read through right now ...
>
> For your image_cache_stats table, two things come to mind:
>
> 1) If your database supports it, you could write a stored procedure to
> begin a transaction, update the values, and return them back to
Cayenne.
>
> 2) Use SQLTemplate to send an UPDATE statement (something like: UPDATE
> image_cache_stats set num_requests = num_requests + 1) and then do a
> query to read the values back in again (if you need them at that
point).
>
> I'm not certain how your image_cache table functions, but a similar
> approach might be sufficient for that, too.  This is far simpler than
> adding pessimistic locking to Cayenne and should perform better, too.
>
> /dev/mrg
>
> PS. You could also use SQLTemplate to do a SELECT ... FOR UPDATE (to
> lock the row), too.  Just be sure you release the locks.  Be
especially
> careful to use a try/catch/finally around everything you do and in the
> finally clause, release the locks.  Otherwise if there is an
exception,

> deadlocks will get you.
>
>
> -----Original Message-----
> From: Gili [mailto:[hidden email]]
> Sent: Monday, August 29, 2005 8:56 AM
> To: [hidden email]
> Subject: Re: Lazily column retrieval
>
>
>
> I've implemented the algorithm found here
> http://www.usenix.org/events/usits99/full_papers/bahn/bahn_html/
>
> I've got two tables:
>
> image_cache_stats, which contains [num_requests, num_hits]
> image_cache, which contains cache entries [score, time, cachedImage]
>
> In a typical request, num_requests is incremented, and a few
> entries
> might be added or removed from image_cache. If multiple clients try
> hitting the cache at the same time with optimistic locking they will
> always collide because num_requests is being incremented by every
> client. If you've got an idea for improving on this I'd be more than
> glad to try it!
>
> Thanks,
> Gili
>
> Gentry, Michael (Contractor) wrote:
>
>>Gili, why is it that you need pessimistic locking over optimistic
>>locking?  In other words, what is your use-case?  I'd be hard-pressed
>
> to
>
>>come up with a situation where I'd prefer pessimistic locking.
>>
>>Thanks,
>>
>>/dev/mrg
>>
>>
>>-----Original Message-----
>>From: Gili [mailto:[hidden email]]
>>Sent: Saturday, August 27, 2005 3:41 PM
>>To: [hidden email]
>>Subject: Re: Lazily column retrieval
>>
>>
>>Hi Andrus,
>>
>> You're one of the main developers of Cayenne if I remember
>>correctly :)
>>Could you possibly help me add pissimistic locking support to Cayenne?
>
>
>>I've got a use-case that requires it.
>>
>> I'm hoping to add both pissimistic locking and streaming BLOBs
>>to the
>>next release but I am new to Cayenne so I don't know where exactly to
>>start.
>>
>>Thanks,
>>Gili
>>
>>Andrus Adamchik wrote:
>>
>>
>>>I haven't heard of anyone posting a streaming [C|B]LOB ExtendedType.
>>
>>A
>>
>>
>>>generic implementation would require some thought, but a simplified  
>>>read-only custom version should be pretty easy to make.
>>>
>>>Reading blobs is much easier that writing them. Just implement  
>>>"getClassName()" to return "java.sql.Blob" and "materializeObject
>>
>>(..)"
>>
>>
>>>to call "getBlob()". Don't recall if some DB's (Oracle?)  required an

>>>open connection that created a BLOB for the BLOB object  to be
>>>accessible. If this is the case, I can think of a few  workarounds
for

>>
>>
>>>that.
>>>
>>>If you are going to try it and willing to share your code, please  
>>>attach to CAY-316.
>>>
>>>Thanks
>>>Andrus
>>>
>>>On Aug 27, 2005, at 1:05 AM, Gili wrote:
>>>
>>>
>>>
>>>>Hi,
>>>>
>>>>   I'm looking for something similar to http://objectstyle.org/ 
>>>>jira/secure/ViewIssue.jspa?key=CAY-316 which will allow me to
lazily
>>
>>
>>>>retrieve certain blob columns. I was told someone on this  list
might

>>
>>
>>>>already have an ExtendedType which does this?
>>>>
>>>>Gili
>>>>--
>>>>http://www.desktopbeautifier.com/
>>>
>>>
>>>
>

--
http://www.desktopbeautifier.com/
Reply | Threaded
Open this post in threaded view
|

Re: Lazily column retrieval

Gili
 > Well, sending an "UPDATE image_cache_stats SET num_requests =
 > num_requests + 1" is highly unlikely to fail (your DB would have to
 > tank, in which case there is nothing Cayenne can do).

        Regarding "UPDATE image_cache_stats SET num_requests = num_requests +
1" I am curious whether Cayenne is smart enough to issue this SQL
statement itself or is it impossible for it to know whether I am
incrementing a field versus setting it to an explicit value?

> Are you trying to limit the image_cache_stats to a set size?  You could
> do that, adding new entries and deleting others and then do a
> dataContext.commitChanges(), but that seems like more trouble that it is
> worth.
>
> Could you not leave old entries in the image_cache_stats table do a
> query against it with an ORDER BY and a LIMIT to control the data coming
> back?  I'd let the database do the work since it is good at it.  If you
> did have a need to prune the size of the table, you could always issue a
> DELETE using SQLTemplate to get rid of rows that fall below a certain
> threshold.

        That would work too but it won't guarantee the cache size is less than
a certain value. I *almost* do what you say though. I do ORDER BY and
remove rows I get back until the total cache size is under its expected
limit.

> I guess what I'm saying is don't be afraid to use an SQLTemplate when
> you have a valid need for it (that's what I do when it makes sense).
> Let the database do work for you when appropriate.
>
> As for the consistency issue, as soon as you refetch the table data,
> you'll be up-to-date again in Cayenne.  It might make sense to put all
> of this logic in a helper class (complete with it's own DataContext) to
> manage it.  You should use a synchronize block around the update code to
> let Java serialize that portion.

        But this still does not guarantee consistency. If I use two separate
transactions: one for incrementing num_requests, and another for
actually inserting/removing cache entries, there is no guarantee both
will succeed or fail together.

        Secondly, in the Hibernate documentation they explicitly warn not to
sychronize on database objects in memory because this does not guarantee
they won't actually get modified. The only safe place to lock on objects
is in the database. In Hibernate, for example, each thread sees a
difference object instance which points to the same database row.
Synchronizing in one piece of code in one application will not prevent
another piece of code in another application will attempting modification.

Gili
Reply | Threaded
Open this post in threaded view
|

Re: Lazily column retrieval

Gili

        What about if I do this?

        begin transaction
        increment num_references using SQLTemplate
        add/remove cache entries using object queries
        end transaction

        This should ensure everything occurs from within a single transaction,
while avoidding collisions for num_references. I assume that even if
optimistic-locking is enabled for the table containing num_references,
it'll be ignored if I use SQLTemplate?

        Basically I want to selectively increment num_references without any
optimistic locking. The same goes for deleting cache entries. If I
delete a cache entry and it has been modified since I last viewed it
(i.e. its score changed) I expect the delete to go through anyway.

        Is there a way to control optimistic locking on a per-query basis like
this? or am is it controlled on a per-table basis?

Thanks,
Gili

Gili wrote:

>  > Well, sending an "UPDATE image_cache_stats SET num_requests =
>  > num_requests + 1" is highly unlikely to fail (your DB would have to
>  > tank, in which case there is nothing Cayenne can do).
>
>     Regarding "UPDATE image_cache_stats SET num_requests = num_requests
> + 1" I am curious whether Cayenne is smart enough to issue this SQL
> statement itself or is it impossible for it to know whether I am
> incrementing a field versus setting it to an explicit value?
>
>> Are you trying to limit the image_cache_stats to a set size?  You could
>> do that, adding new entries and deleting others and then do a
>> dataContext.commitChanges(), but that seems like more trouble that it is
>> worth.
>>
>> Could you not leave old entries in the image_cache_stats table do a
>> query against it with an ORDER BY and a LIMIT to control the data coming
>> back?  I'd let the database do the work since it is good at it.  If you
>> did have a need to prune the size of the table, you could always issue a
>> DELETE using SQLTemplate to get rid of rows that fall below a certain
>> threshold.
>
>
>     That would work too but it won't guarantee the cache size is less
> than a certain value. I *almost* do what you say though. I do ORDER BY
> and remove rows I get back until the total cache size is under its
> expected limit.
>
>> I guess what I'm saying is don't be afraid to use an SQLTemplate when
>> you have a valid need for it (that's what I do when it makes sense).
>> Let the database do work for you when appropriate.
>>
>> As for the consistency issue, as soon as you refetch the table data,
>> you'll be up-to-date again in Cayenne.  It might make sense to put all
>> of this logic in a helper class (complete with it's own DataContext) to
>> manage it.  You should use a synchronize block around the update code to
>> let Java serialize that portion.
>
>
>     But this still does not guarantee consistency. If I use two separate
> transactions: one for incrementing num_requests, and another for
> actually inserting/removing cache entries, there is no guarantee both
> will succeed or fail together.
>
>     Secondly, in the Hibernate documentation they explicitly warn not to
> sychronize on database objects in memory because this does not guarantee
> they won't actually get modified. The only safe place to lock on objects
> is in the database. In Hibernate, for example, each thread sees a
> difference object instance which points to the same database row.
> Synchronizing in one piece of code in one application will not prevent
> another piece of code in another application will attempting modification.
>
> Gili
>

--
http://www.desktopbeautifier.com/
Reply | Threaded
Open this post in threaded view
|

RE: Lazily column retrieval

Gentry, Michael (Contractor)
In reply to this post by Gili
No, Cayenne isn't "smart" enough to automatically generate an
incrementing SQL statement, but you are.  :-)  I don't know of any ORM
that is smart enough to check for that.  And, personally, I think it
would be a performance drag to add it.

I wasn't suggesting locking on a database object.  Make a CacheHelper
class which you use exclusively to manipulate the cache.  Use Java
synchronization (on a method or variable, whichever makes more sense in
your helper class) to ensure only one thread is monkeying with the cache
at any point in time.  If you do this, you should be able to use plain
old Cayenne DataObjects and fire a single commitChanges() to do it all
in one transaction.  You might not even need SQLTemplate with this
approach.  The more I think about it, this might be the simplest
solution and will ensure you don't have cache manipulation code
scattered about.

/dev/mrg


-----Original Message-----
From: Gili [mailto:[hidden email]]
Sent: Monday, August 29, 2005 11:37 AM
To: [hidden email]
Subject: Re: Lazily column retrieval


 > Well, sending an "UPDATE image_cache_stats SET num_requests =
 > num_requests + 1" is highly unlikely to fail (your DB would have to
 > tank, in which case there is nothing Cayenne can do).

        Regarding "UPDATE image_cache_stats SET num_requests =
num_requests +
1" I am curious whether Cayenne is smart enough to issue this SQL
statement itself or is it impossible for it to know whether I am
incrementing a field versus setting it to an explicit value?

> Are you trying to limit the image_cache_stats to a set size?  You
could
> do that, adding new entries and deleting others and then do a
> dataContext.commitChanges(), but that seems like more trouble that it
is
> worth.
>
> Could you not leave old entries in the image_cache_stats table do a
> query against it with an ORDER BY and a LIMIT to control the data
coming
> back?  I'd let the database do the work since it is good at it.  If
you
> did have a need to prune the size of the table, you could always issue
a
> DELETE using SQLTemplate to get rid of rows that fall below a certain
> threshold.

        That would work too but it won't guarantee the cache size is
less than
a certain value. I *almost* do what you say though. I do ORDER BY and
remove rows I get back until the total cache size is under its expected
limit.

> I guess what I'm saying is don't be afraid to use an SQLTemplate when
> you have a valid need for it (that's what I do when it makes sense).
> Let the database do work for you when appropriate.
>
> As for the consistency issue, as soon as you refetch the table data,
> you'll be up-to-date again in Cayenne.  It might make sense to put all
> of this logic in a helper class (complete with it's own DataContext)
to
> manage it.  You should use a synchronize block around the update code
to
> let Java serialize that portion.

        But this still does not guarantee consistency. If I use two
separate
transactions: one for incrementing num_requests, and another for
actually inserting/removing cache entries, there is no guarantee both
will succeed or fail together.

        Secondly, in the Hibernate documentation they explicitly warn
not to
sychronize on database objects in memory because this does not guarantee

they won't actually get modified. The only safe place to lock on objects

is in the database. In Hibernate, for example, each thread sees a
difference object instance which points to the same database row.
Synchronizing in one piece of code in one application will not prevent
another piece of code in another application will attempting
modification.

Gili
Reply | Threaded
Open this post in threaded view
|

RE: Lazily column retrieval

Gentry, Michael (Contractor)
In reply to this post by Gili
I'm not certain that would work: your DB would have to support nested
transactions as would Cayenne (since commitChanges() will start a
transaction).

A safer way might be:

add/remove cache entries using object queries (might fail, catch
exceptions)
increment num_references using SQLTemplate (highly unliked to fail)

However, I think my previous suggestions (CacheHelper) would make the
most sense.

I'm not certain if there is a way to control OL on a per-query basis
(haven't had a need to look into it).

/dev/mrg


-----Original Message-----
From: Gili [mailto:[hidden email]]
Sent: Monday, August 29, 2005 1:24 PM
To: [hidden email]
Subject: Re: Lazily column retrieval



        What about if I do this?

        begin transaction
        increment num_references using SQLTemplate
        add/remove cache entries using object queries
        end transaction

        This should ensure everything occurs from within a single
transaction,
while avoidding collisions for num_references. I assume that even if
optimistic-locking is enabled for the table containing num_references,
it'll be ignored if I use SQLTemplate?

        Basically I want to selectively increment num_references without
any
optimistic locking. The same goes for deleting cache entries. If I
delete a cache entry and it has been modified since I last viewed it
(i.e. its score changed) I expect the delete to go through anyway.

        Is there a way to control optimistic locking on a per-query
basis like
this? or am is it controlled on a per-table basis?

Thanks,
Gili

Gili wrote:
>  > Well, sending an "UPDATE image_cache_stats SET num_requests =
>  > num_requests + 1" is highly unlikely to fail (your DB would have to
>  > tank, in which case there is nothing Cayenne can do).
>
>     Regarding "UPDATE image_cache_stats SET num_requests =
num_requests
> + 1" I am curious whether Cayenne is smart enough to issue this SQL
> statement itself or is it impossible for it to know whether I am
> incrementing a field versus setting it to an explicit value?
>
>> Are you trying to limit the image_cache_stats to a set size?  You
could
>> do that, adding new entries and deleting others and then do a
>> dataContext.commitChanges(), but that seems like more trouble that it
is
>> worth.
>>
>> Could you not leave old entries in the image_cache_stats table do a
>> query against it with an ORDER BY and a LIMIT to control the data
coming
>> back?  I'd let the database do the work since it is good at it.  If
you
>> did have a need to prune the size of the table, you could always
issue a
>> DELETE using SQLTemplate to get rid of rows that fall below a certain
>> threshold.
>
>
>     That would work too but it won't guarantee the cache size is less
> than a certain value. I *almost* do what you say though. I do ORDER BY

> and remove rows I get back until the total cache size is under its
> expected limit.
>
>> I guess what I'm saying is don't be afraid to use an SQLTemplate when
>> you have a valid need for it (that's what I do when it makes sense).
>> Let the database do work for you when appropriate.
>>
>> As for the consistency issue, as soon as you refetch the table data,
>> you'll be up-to-date again in Cayenne.  It might make sense to put
all
>> of this logic in a helper class (complete with it's own DataContext)
to
>> manage it.  You should use a synchronize block around the update code
to
>> let Java serialize that portion.
>
>
>     But this still does not guarantee consistency. If I use two
separate
> transactions: one for incrementing num_requests, and another for
> actually inserting/removing cache entries, there is no guarantee both
> will succeed or fail together.
>
>     Secondly, in the Hibernate documentation they explicitly warn not
to
> sychronize on database objects in memory because this does not
guarantee
> they won't actually get modified. The only safe place to lock on
objects
> is in the database. In Hibernate, for example, each thread sees a
> difference object instance which points to the same database row.
> Synchronizing in one piece of code in one application will not prevent

> another piece of code in another application will attempting
modification.
>
> Gili
>

--
http://www.desktopbeautifier.com/
Reply | Threaded
Open this post in threaded view
|

RE: Lazily column retrieval

Gentry, Michael (Contractor)
In reply to this post by Gili
Oh yeah, SQLTemplate bypasses the in-memory data object graph.  So you
don't get optimistic locking with it.  That's why if you do DELETEs/etc
in it, you can get your data object graph out of sync if you aren't
careful.

/dev/mrg

-----Original Message-----
From: Gili [mailto:[hidden email]]
Sent: Monday, August 29, 2005 1:24 PM
To: [hidden email]
Subject: Re: Lazily column retrieval



        What about if I do this?

        begin transaction
        increment num_references using SQLTemplate
        add/remove cache entries using object queries
        end transaction

        This should ensure everything occurs from within a single
transaction,
while avoidding collisions for num_references. I assume that even if
optimistic-locking is enabled for the table containing num_references,
it'll be ignored if I use SQLTemplate?

        Basically I want to selectively increment num_references without
any
optimistic locking. The same goes for deleting cache entries. If I
delete a cache entry and it has been modified since I last viewed it
(i.e. its score changed) I expect the delete to go through anyway.

        Is there a way to control optimistic locking on a per-query
basis like
this? or am is it controlled on a per-table basis?

Thanks,
Gili

Gili wrote:
>  > Well, sending an "UPDATE image_cache_stats SET num_requests =
>  > num_requests + 1" is highly unlikely to fail (your DB would have to
>  > tank, in which case there is nothing Cayenne can do).
>
>     Regarding "UPDATE image_cache_stats SET num_requests =
num_requests
> + 1" I am curious whether Cayenne is smart enough to issue this SQL
> statement itself or is it impossible for it to know whether I am
> incrementing a field versus setting it to an explicit value?
>
>> Are you trying to limit the image_cache_stats to a set size?  You
could
>> do that, adding new entries and deleting others and then do a
>> dataContext.commitChanges(), but that seems like more trouble that it
is
>> worth.
>>
>> Could you not leave old entries in the image_cache_stats table do a
>> query against it with an ORDER BY and a LIMIT to control the data
coming
>> back?  I'd let the database do the work since it is good at it.  If
you
>> did have a need to prune the size of the table, you could always
issue a
>> DELETE using SQLTemplate to get rid of rows that fall below a certain
>> threshold.
>
>
>     That would work too but it won't guarantee the cache size is less
> than a certain value. I *almost* do what you say though. I do ORDER BY

> and remove rows I get back until the total cache size is under its
> expected limit.
>
>> I guess what I'm saying is don't be afraid to use an SQLTemplate when
>> you have a valid need for it (that's what I do when it makes sense).
>> Let the database do work for you when appropriate.
>>
>> As for the consistency issue, as soon as you refetch the table data,
>> you'll be up-to-date again in Cayenne.  It might make sense to put
all
>> of this logic in a helper class (complete with it's own DataContext)
to
>> manage it.  You should use a synchronize block around the update code
to
>> let Java serialize that portion.
>
>
>     But this still does not guarantee consistency. If I use two
separate
> transactions: one for incrementing num_requests, and another for
> actually inserting/removing cache entries, there is no guarantee both
> will succeed or fail together.
>
>     Secondly, in the Hibernate documentation they explicitly warn not
to
> sychronize on database objects in memory because this does not
guarantee
> they won't actually get modified. The only safe place to lock on
objects
> is in the database. In Hibernate, for example, each thread sees a
> difference object instance which points to the same database row.
> Synchronizing in one piece of code in one application will not prevent

> another piece of code in another application will attempting
modification.
>
> Gili
>

--
http://www.desktopbeautifier.com/
Reply | Threaded
Open this post in threaded view
|

Re: Lazily column retrieval

Gili
In reply to this post by Gentry, Michael (Contractor)

        But... like I keep on saying... this doesn't help if the cache is
instantiated across different JVMs. In my case I have my webapp using
one JVM and my admin console using another and both hit the cache. In
the future I forsee clustering many other JVMs hitting the cache too. So
I don't see how locking in memory is going to help.

Gili

Gentry, Michael (Contractor) wrote:

> No, Cayenne isn't "smart" enough to automatically generate an
> incrementing SQL statement, but you are.  :-)  I don't know of any ORM
> that is smart enough to check for that.  And, personally, I think it
> would be a performance drag to add it.
>
> I wasn't suggesting locking on a database object.  Make a CacheHelper
> class which you use exclusively to manipulate the cache.  Use Java
> synchronization (on a method or variable, whichever makes more sense in
> your helper class) to ensure only one thread is monkeying with the cache
> at any point in time.  If you do this, you should be able to use plain
> old Cayenne DataObjects and fire a single commitChanges() to do it all
> in one transaction.  You might not even need SQLTemplate with this
> approach.  The more I think about it, this might be the simplest
> solution and will ensure you don't have cache manipulation code
> scattered about.
>
> /dev/mrg
>
>
> -----Original Message-----
> From: Gili [mailto:[hidden email]]
> Sent: Monday, August 29, 2005 11:37 AM
> To: [hidden email]
> Subject: Re: Lazily column retrieval
>
>
>  > Well, sending an "UPDATE image_cache_stats SET num_requests =
>  > num_requests + 1" is highly unlikely to fail (your DB would have to
>  > tank, in which case there is nothing Cayenne can do).
>
> Regarding "UPDATE image_cache_stats SET num_requests =
> num_requests +
> 1" I am curious whether Cayenne is smart enough to issue this SQL
> statement itself or is it impossible for it to know whether I am
> incrementing a field versus setting it to an explicit value?
>
>
>>Are you trying to limit the image_cache_stats to a set size?  You
>
> could
>
>>do that, adding new entries and deleting others and then do a
>>dataContext.commitChanges(), but that seems like more trouble that it
>
> is
>
>>worth.
>>
>>Could you not leave old entries in the image_cache_stats table do a
>>query against it with an ORDER BY and a LIMIT to control the data
>
> coming
>
>>back?  I'd let the database do the work since it is good at it.  If
>
> you
>
>>did have a need to prune the size of the table, you could always issue
>
> a
>
>>DELETE using SQLTemplate to get rid of rows that fall below a certain
>>threshold.
>
>
> That would work too but it won't guarantee the cache size is
> less than
> a certain value. I *almost* do what you say though. I do ORDER BY and
> remove rows I get back until the total cache size is under its expected
> limit.
>
>
>>I guess what I'm saying is don't be afraid to use an SQLTemplate when
>>you have a valid need for it (that's what I do when it makes sense).
>>Let the database do work for you when appropriate.
>>
>>As for the consistency issue, as soon as you refetch the table data,
>>you'll be up-to-date again in Cayenne.  It might make sense to put all
>>of this logic in a helper class (complete with it's own DataContext)
>
> to
>
>>manage it.  You should use a synchronize block around the update code
>
> to
>
>>let Java serialize that portion.
>
>
> But this still does not guarantee consistency. If I use two
> separate
> transactions: one for incrementing num_requests, and another for
> actually inserting/removing cache entries, there is no guarantee both
> will succeed or fail together.
>
> Secondly, in the Hibernate documentation they explicitly warn
> not to
> sychronize on database objects in memory because this does not guarantee
>
> they won't actually get modified. The only safe place to lock on objects
>
> is in the database. In Hibernate, for example, each thread sees a
> difference object instance which points to the same database row.
> Synchronizing in one piece of code in one application will not prevent
> another piece of code in another application will attempting
> modification.
>
> Gili
>

--
http://www.desktopbeautifier.com/
Reply | Threaded
Open this post in threaded view
|

Re: Lazily column retrieval

Mike Kienenberger
In reply to this post by Gili
Optimistic locking is currently on a per-object basis, but since the
status of whether any particular query does optimistic locking is
maintained as part of the query, it would be fairly simple to override
that behavior.  It could either be another flag
(setShouldUseOptimisticLocking) or you could just subclass the

On 8/29/05, Gili <[hidden email]> wrote:

>
>         What about if I do this?
>
>         begin transaction
>         increment num_references using SQLTemplate
>         add/remove cache entries using object queries
>         end transaction
>
>         This should ensure everything occurs from within a single transaction,
> while avoidding collisions for num_references. I assume that even if
> optimistic-locking is enabled for the table containing num_references,
> it'll be ignored if I use SQLTemplate?
>
>         Basically I want to selectively increment num_references without any
> optimistic locking. The same goes for deleting cache entries. If I
> delete a cache entry and it has been modified since I last viewed it
> (i.e. its score changed) I expect the delete to go through anyway.
>
>         Is there a way to control optimistic locking on a per-query basis like
> this? or am is it controlled on a per-table basis?
>
> Thanks,
> Gili
>
> Gili wrote:
> >  > Well, sending an "UPDATE image_cache_stats SET num_requests =
> >  > num_requests + 1" is highly unlikely to fail (your DB would have to
> >  > tank, in which case there is nothing Cayenne can do).
> >
> >     Regarding "UPDATE image_cache_stats SET num_requests = num_requests
> > + 1" I am curious whether Cayenne is smart enough to issue this SQL
> > statement itself or is it impossible for it to know whether I am
> > incrementing a field versus setting it to an explicit value?
> >
> >> Are you trying to limit the image_cache_stats to a set size?  You could
> >> do that, adding new entries and deleting others and then do a
> >> dataContext.commitChanges(), but that seems like more trouble that it is
> >> worth.
> >>
> >> Could you not leave old entries in the image_cache_stats table do a
> >> query against it with an ORDER BY and a LIMIT to control the data coming
> >> back?  I'd let the database do the work since it is good at it.  If you
> >> did have a need to prune the size of the table, you could always issue a
> >> DELETE using SQLTemplate to get rid of rows that fall below a certain
> >> threshold.
> >
> >
> >     That would work too but it won't guarantee the cache size is less
> > than a certain value. I *almost* do what you say though. I do ORDER BY
> > and remove rows I get back until the total cache size is under its
> > expected limit.
> >
> >> I guess what I'm saying is don't be afraid to use an SQLTemplate when
> >> you have a valid need for it (that's what I do when it makes sense).
> >> Let the database do work for you when appropriate.
> >>
> >> As for the consistency issue, as soon as you refetch the table data,
> >> you'll be up-to-date again in Cayenne.  It might make sense to put all
> >> of this logic in a helper class (complete with it's own DataContext) to
> >> manage it.  You should use a synchronize block around the update code to
> >> let Java serialize that portion.
> >
> >
> >     But this still does not guarantee consistency. If I use two separate
> > transactions: one for incrementing num_requests, and another for
> > actually inserting/removing cache entries, there is no guarantee both
> > will succeed or fail together.
> >
> >     Secondly, in the Hibernate documentation they explicitly warn not to
> > sychronize on database objects in memory because this does not guarantee
> > they won't actually get modified. The only safe place to lock on objects
> > is in the database. In Hibernate, for example, each thread sees a
> > difference object instance which points to the same database row.
> > Synchronizing in one piece of code in one application will not prevent
> > another piece of code in another application will attempting modification.
> >
> > Gili
> >
>
> --
> http://www.desktopbeautifier.com/
>
Reply | Threaded
Open this post in threaded view
|

Re: Lazily column retrieval

Mike Kienenberger
ack.  subclass Query and override UsingOptimisticLocking() -- can't
guarantee it will work, but I think it will.

On 8/29/05, Mike Kienenberger <[hidden email]> wrote:

> Optimistic locking is currently on a per-object basis, but since the
> status of whether any particular query does optimistic locking is
> maintained as part of the query, it would be fairly simple to override
> that behavior.  It could either be another flag
> (setShouldUseOptimisticLocking) or you could just subclass the
>
> On 8/29/05, Gili <[hidden email]> wrote:
> >
> >         What about if I do this?
> >
> >         begin transaction
> >         increment num_references using SQLTemplate
> >         add/remove cache entries using object queries
> >         end transaction
> >
> >         This should ensure everything occurs from within a single transaction,
> > while avoidding collisions for num_references. I assume that even if
> > optimistic-locking is enabled for the table containing num_references,
> > it'll be ignored if I use SQLTemplate?
> >
> >         Basically I want to selectively increment num_references without any
> > optimistic locking. The same goes for deleting cache entries. If I
> > delete a cache entry and it has been modified since I last viewed it
> > (i.e. its score changed) I expect the delete to go through anyway.
> >
> >         Is there a way to control optimistic locking on a per-query basis like
> > this? or am is it controlled on a per-table basis?
> >
> > Thanks,
> > Gili
> >
> > Gili wrote:
> > >  > Well, sending an "UPDATE image_cache_stats SET num_requests =
> > >  > num_requests + 1" is highly unlikely to fail (your DB would have to
> > >  > tank, in which case there is nothing Cayenne can do).
> > >
> > >     Regarding "UPDATE image_cache_stats SET num_requests = num_requests
> > > + 1" I am curious whether Cayenne is smart enough to issue this SQL
> > > statement itself or is it impossible for it to know whether I am
> > > incrementing a field versus setting it to an explicit value?
> > >
> > >> Are you trying to limit the image_cache_stats to a set size?  You could
> > >> do that, adding new entries and deleting others and then do a
> > >> dataContext.commitChanges(), but that seems like more trouble that it is
> > >> worth.
> > >>
> > >> Could you not leave old entries in the image_cache_stats table do a
> > >> query against it with an ORDER BY and a LIMIT to control the data coming
> > >> back?  I'd let the database do the work since it is good at it.  If you
> > >> did have a need to prune the size of the table, you could always issue a
> > >> DELETE using SQLTemplate to get rid of rows that fall below a certain
> > >> threshold.
> > >
> > >
> > >     That would work too but it won't guarantee the cache size is less
> > > than a certain value. I *almost* do what you say though. I do ORDER BY
> > > and remove rows I get back until the total cache size is under its
> > > expected limit.
> > >
> > >> I guess what I'm saying is don't be afraid to use an SQLTemplate when
> > >> you have a valid need for it (that's what I do when it makes sense).
> > >> Let the database do work for you when appropriate.
> > >>
> > >> As for the consistency issue, as soon as you refetch the table data,
> > >> you'll be up-to-date again in Cayenne.  It might make sense to put all
> > >> of this logic in a helper class (complete with it's own DataContext) to
> > >> manage it.  You should use a synchronize block around the update code to
> > >> let Java serialize that portion.
> > >
> > >
> > >     But this still does not guarantee consistency. If I use two separate
> > > transactions: one for incrementing num_requests, and another for
> > > actually inserting/removing cache entries, there is no guarantee both
> > > will succeed or fail together.
> > >
> > >     Secondly, in the Hibernate documentation they explicitly warn not to
> > > sychronize on database objects in memory because this does not guarantee
> > > they won't actually get modified. The only safe place to lock on objects
> > > is in the database. In Hibernate, for example, each thread sees a
> > > difference object instance which points to the same database row.
> > > Synchronizing in one piece of code in one application will not prevent
> > > another piece of code in another application will attempting modification.
> > >
> > > Gili
> > >
> >
> > --
> > http://www.desktopbeautifier.com/
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Lazily column retrieval

Joshua Pyle
In reply to this post by Gili
Actually tou could think about doing a programmatic times lock.

Something like adding a datetime field to the table in question that
is used to denote when the lock will expire.  Then in your code you
query the table and check the lock expiration field.  Some thing like
the following...

MY_DATA_TABLE
... Some Fields...
... LOCK_EXPIRE DATE NULL

you just have to check the LOCK_EXPIRE field before you let a user
edit the row in question.  This will give you a row level expiring
lock.

Don't know if this helps alot, but its an idea.

Joshua T. Pyle


On 8/29/05, Gili <[hidden email]> wrote:

>
>        But... like I keep on saying... this doesn't help if the cache is
> instantiated across different JVMs. In my case I have my webapp using
> one JVM and my admin console using another and both hit the cache. In
> the future I forsee clustering many other JVMs hitting the cache too. So
> I don't see how locking in memory is going to help.
>
> Gili
>
> Gentry, Michael (Contractor) wrote:
> > No, Cayenne isn't "smart" enough to automatically generate an
> > incrementing SQL statement, but you are.  :-)  I don't know of any ORM
> > that is smart enough to check for that.  And, personally, I think it
> > would be a performance drag to add it.
> >
> > I wasn't suggesting locking on a database object.  Make a CacheHelper
> > class which you use exclusively to manipulate the cache.  Use Java
> > synchronization (on a method or variable, whichever makes more sense in
> > your helper class) to ensure only one thread is monkeying with the cache
> > at any point in time.  If you do this, you should be able to use plain
> > old Cayenne DataObjects and fire a single commitChanges() to do it all
> > in one transaction.  You might not even need SQLTemplate with this
> > approach.  The more I think about it, this might be the simplest
> > solution and will ensure you don't have cache manipulation code
> > scattered about.
> >
> > /dev/mrg
> >
> >
> > -----Original Message-----
> > From: Gili [mailto:[hidden email]]
> > Sent: Monday, August 29, 2005 11:37 AM
> > To: [hidden email]
> > Subject: Re: Lazily column retrieval
> >
> >
> >  > Well, sending an "UPDATE image_cache_stats SET num_requests =
> >  > num_requests + 1" is highly unlikely to fail (your DB would have to
> >  > tank, in which case there is nothing Cayenne can do).
> >
> >       Regarding "UPDATE image_cache_stats SET num_requests =
> > num_requests +
> > 1" I am curious whether Cayenne is smart enough to issue this SQL
> > statement itself or is it impossible for it to know whether I am
> > incrementing a field versus setting it to an explicit value?
> >
> >
> >>Are you trying to limit the image_cache_stats to a set size?  You
> >
> > could
> >
> >>do that, adding new entries and deleting others and then do a
> >>dataContext.commitChanges(), but that seems like more trouble that it
> >
> > is
> >
> >>worth.
> >>
> >>Could you not leave old entries in the image_cache_stats table do a
> >>query against it with an ORDER BY and a LIMIT to control the data
> >
> > coming
> >
> >>back?  I'd let the database do the work since it is good at it.  If
> >
> > you
> >
> >>did have a need to prune the size of the table, you could always issue
> >
> > a
> >
> >>DELETE using SQLTemplate to get rid of rows that fall below a certain
> >>threshold.
> >
> >
> >       That would work too but it won't guarantee the cache size is
> > less than
> > a certain value. I *almost* do what you say though. I do ORDER BY and
> > remove rows I get back until the total cache size is under its expected
> > limit.
> >
> >
> >>I guess what I'm saying is don't be afraid to use an SQLTemplate when
> >>you have a valid need for it (that's what I do when it makes sense).
> >>Let the database do work for you when appropriate.
> >>
> >>As for the consistency issue, as soon as you refetch the table data,
> >>you'll be up-to-date again in Cayenne.  It might make sense to put all
> >>of this logic in a helper class (complete with it's own DataContext)
> >
> > to
> >
> >>manage it.  You should use a synchronize block around the update code
> >
> > to
> >
> >>let Java serialize that portion.
> >
> >
> >       But this still does not guarantee consistency. If I use two
> > separate
> > transactions: one for incrementing num_requests, and another for
> > actually inserting/removing cache entries, there is no guarantee both
> > will succeed or fail together.
> >
> >       Secondly, in the Hibernate documentation they explicitly warn
> > not to
> > sychronize on database objects in memory because this does not guarantee
> >
> > they won't actually get modified. The only safe place to lock on objects
> >
> > is in the database. In Hibernate, for example, each thread sees a
> > difference object instance which points to the same database row.
> > Synchronizing in one piece of code in one application will not prevent
> > another piece of code in another application will attempting
> > modification.
> >
> > Gili
> >
>
> --
> http://www.desktopbeautifier.com/
>


--
Joshua T. Pyle
Go has always existed.
Reply | Threaded
Open this post in threaded view
|

RE: Lazily column retrieval

Gentry, Michael (Contractor)
In reply to this post by Gili
Well, to avoid integrity issues or update failures, you really need a
common control mechanism for a shared (and changing) resource, otherwise
you are on a slippery slope.

I suspect you are using MySQL, so a stored procedure is out of the
question?  (I've not looked at MySQL in a while, so I don't know if that
is possible now).

You could write a web service (or similar) which managed the cache and
direct all requests to it.  Probably too much work.

You could turn on Cayenne's Remote Change Notification (under the Data
Domain's Cache Configuration) and use a CacheManager as I mentioned
before.  I've not used that mechanism before, so perhaps someone else
could comment on it.

/dev/mrg


-----Original Message-----
From: Gili [mailto:[hidden email]]
Sent: Monday, August 29, 2005 2:23 PM
To: [hidden email]
Subject: Re: Lazily column retrieval



        But... like I keep on saying... this doesn't help if the cache
is
instantiated across different JVMs. In my case I have my webapp using
one JVM and my admin console using another and both hit the cache. In
the future I forsee clustering many other JVMs hitting the cache too. So

I don't see how locking in memory is going to help.

Gili

Gentry, Michael (Contractor) wrote:
> No, Cayenne isn't "smart" enough to automatically generate an
> incrementing SQL statement, but you are.  :-)  I don't know of any ORM
> that is smart enough to check for that.  And, personally, I think it
> would be a performance drag to add it.
>
> I wasn't suggesting locking on a database object.  Make a CacheHelper
> class which you use exclusively to manipulate the cache.  Use Java
> synchronization (on a method or variable, whichever makes more sense
in
> your helper class) to ensure only one thread is monkeying with the
cache

> at any point in time.  If you do this, you should be able to use plain
> old Cayenne DataObjects and fire a single commitChanges() to do it all
> in one transaction.  You might not even need SQLTemplate with this
> approach.  The more I think about it, this might be the simplest
> solution and will ensure you don't have cache manipulation code
> scattered about.
>
> /dev/mrg
>
>
> -----Original Message-----
> From: Gili [mailto:[hidden email]]
> Sent: Monday, August 29, 2005 11:37 AM
> To: [hidden email]
> Subject: Re: Lazily column retrieval
>
>
>  > Well, sending an "UPDATE image_cache_stats SET num_requests =
>  > num_requests + 1" is highly unlikely to fail (your DB would have to
>  > tank, in which case there is nothing Cayenne can do).
>
> Regarding "UPDATE image_cache_stats SET num_requests =
> num_requests +
> 1" I am curious whether Cayenne is smart enough to issue this SQL
> statement itself or is it impossible for it to know whether I am
> incrementing a field versus setting it to an explicit value?
>
>
>>Are you trying to limit the image_cache_stats to a set size?  You
>
> could
>
>>do that, adding new entries and deleting others and then do a
>>dataContext.commitChanges(), but that seems like more trouble that it
>
> is
>
>>worth.
>>
>>Could you not leave old entries in the image_cache_stats table do a
>>query against it with an ORDER BY and a LIMIT to control the data
>
> coming
>
>>back?  I'd let the database do the work since it is good at it.  If
>
> you
>
>>did have a need to prune the size of the table, you could always issue
>
> a
>
>>DELETE using SQLTemplate to get rid of rows that fall below a certain
>>threshold.
>
>
> That would work too but it won't guarantee the cache size is
> less than
> a certain value. I *almost* do what you say though. I do ORDER BY and
> remove rows I get back until the total cache size is under its
expected

> limit.
>
>
>>I guess what I'm saying is don't be afraid to use an SQLTemplate when
>>you have a valid need for it (that's what I do when it makes sense).
>>Let the database do work for you when appropriate.
>>
>>As for the consistency issue, as soon as you refetch the table data,
>>you'll be up-to-date again in Cayenne.  It might make sense to put all
>>of this logic in a helper class (complete with it's own DataContext)
>
> to
>
>>manage it.  You should use a synchronize block around the update code
>
> to
>
>>let Java serialize that portion.
>
>
> But this still does not guarantee consistency. If I use two
> separate
> transactions: one for incrementing num_requests, and another for
> actually inserting/removing cache entries, there is no guarantee both
> will succeed or fail together.
>
> Secondly, in the Hibernate documentation they explicitly warn
> not to
> sychronize on database objects in memory because this does not
guarantee
>
> they won't actually get modified. The only safe place to lock on
objects
>
> is in the database. In Hibernate, for example, each thread sees a
> difference object instance which points to the same database row.
> Synchronizing in one piece of code in one application will not prevent

> another piece of code in another application will attempting
> modification.
>
> Gili
>

--
http://www.desktopbeautifier.com/