reverse engineering refactoring

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

reverse engineering refactoring

Andrus Adamchik
You've probably seen tons of semi-random commits from me lately :)

What I was trying to do is to bring some sanity into reverse-engineering and its plethora of configurations, settings, strategies and flows. I am still not fully finished, but some important steps are already complete:

* Code related to reverse engineering backend was split from cayenne-server into its own "cayenne-dbsync" module.

* The most challenging part of was moving MergerTokenFactory, as factories are adapter-specific. I came up with a few DI tricks to preserve per-adapter implementations. Long story short - "merge" package is now in "cayenne-dbsync" as well.

* Name generators for various model objects in various contexts were unified, heavily refactored and placed in "cayenne-dbsync" as well. There are just 2 visible ones: NameBuilder and ObjectNameGenerator. Both complement each other and are fairly easy to understand.

* Removed most logic forks in our reverse engineering flow. Now regardless of whether you do rev-eng from scratch, or merging into an existing DataMap, the sequence is pretty much the same:
 
  1. DbLoader reverse engineers the DB based on its filters into a temporary DataMap.
  2. This DataMap contains only the DB layer. No ObjEntities.
  3. Then we compare this DataMap with our target DataMap, generate MergerTokens and merge them.

Now of the above is backwards compatible API-wise, but this is not the core API that we absolutely need to preserve between releases. So I think this is not a problem. The only visible incompatibility for an average user is this:

https://issues.apache.org/jira/browse/CAY-2118

Next I am going to pull some stuff from downstream cayenne-tools into cayenne-datasync to make reverse engineering mechanism fully agnostic of the caller. There's probably more cleanup to do around configurations, etc.

After that going to finally fix a few rev-eng bugs that triggered this whole endeavor.

Also we'll need to spend some time on the new rev-eng tab in the modeler. It kinda works, but there are UI quirks, and simply lots of unused potential in how we can present the new backend capabilities via the UI.

Hope all this make sense :)

Andrus
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: reverse engineering refactoring

Tony Giaccone
Two suggestions and I'd be willing to help on any of this.


1) have the model "remember" which datasource was used to
Rev-engineer.  

2 it would be nice if the graph layouts for the data models actually used some kind of layout algorithm to prevent the tables from being the mess that they are..

Tony Giaccone

> On Oct 3, 2016, at 2:57 PM, Andrus Adamchik <[hidden email]> wrote:
>
> You've probably seen tons of semi-random commits from me lately :)
>
> What I was trying to do is to bring some sanity into reverse-engineering and its plethora of configurations, settings, strategies and flows. I am still not fully finished, but some important steps are already complete:
>
> * Code related to reverse engineering backend was split from cayenne-server into its own "cayenne-dbsync" module.
>
> * The most challenging part of was moving MergerTokenFactory, as factories are adapter-specific. I came up with a few DI tricks to preserve per-adapter implementations. Long story short - "merge" package is now in "cayenne-dbsync" as well.
>
> * Name generators for various model objects in various contexts were unified, heavily refactored and placed in "cayenne-dbsync" as well. There are just 2 visible ones: NameBuilder and ObjectNameGenerator. Both complement each other and are fairly easy to understand.
>
> * Removed most logic forks in our reverse engineering flow. Now regardless of whether you do rev-eng from scratch, or merging into an existing DataMap, the sequence is pretty much the same:
>
>  1. DbLoader reverse engineers the DB based on its filters into a temporary DataMap.
>  2. This DataMap contains only the DB layer. No ObjEntities.
>  3. Then we compare this DataMap with our target DataMap, generate MergerTokens and merge them.
>
> Now of the above is backwards compatible API-wise, but this is not the core API that we absolutely need to preserve between releases. So I think this is not a problem. The only visible incompatibility for an average user is this:
>
> https://issues.apache.org/jira/browse/CAY-2118
>
> Next I am going to pull some stuff from downstream cayenne-tools into cayenne-datasync to make reverse engineering mechanism fully agnostic of the caller. There's probably more cleanup to do around configurations, etc.
>
> After that going to finally fix a few rev-eng bugs that triggered this whole endeavor.
>
> Also we'll need to spend some time on the new rev-eng tab in the modeler. It kinda works, but there are UI quirks, and simply lots of unused potential in how we can present the new backend capabilities via the UI.
>
> Hope all this make sense :)
>
> Andrus
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: reverse engineering refactoring

Andrus Adamchik
Hi Tony,

> 1) have the model "remember" which datasource was used to
> Rev-engineer.  

Good point. Not sure if you are looking at M3 or M4 (this problem exists on both IIRC). In M4 reverse engineering UI has been changed completely. And we will still be reworking that version to make it more "production quality".


> 2 it would be nice if the graph layouts for the data models actually used some kind of layout algorithm to prevent the tables from being the mess that they are..

That's like the most requested feature, yet there are no developers working on the graph layout at the moment. Would be nice if someone could take ownership of this component and address all the issues.

> Two suggestions and I'd be willing to help on any of this.

Much appreciated.

Andrus


> On Oct 5, 2016, at 3:51 AM, Gmail <[hidden email]> wrote:
>
> Two suggestions and I'd be willing to help on any of this.
>
>
> 1) have the model "remember" which datasource was used to
> Rev-engineer.  
>
> 2 it would be nice if the graph layouts for the data models actually used some kind of layout algorithm to prevent the tables from being the mess that they are..
>
> Tony Giaccone
>
>> On Oct 3, 2016, at 2:57 PM, Andrus Adamchik <[hidden email]> wrote:
>>
>> You've probably seen tons of semi-random commits from me lately :)
>>
>> What I was trying to do is to bring some sanity into reverse-engineering and its plethora of configurations, settings, strategies and flows. I am still not fully finished, but some important steps are already complete:
>>
>> * Code related to reverse engineering backend was split from cayenne-server into its own "cayenne-dbsync" module.
>>
>> * The most challenging part of was moving MergerTokenFactory, as factories are adapter-specific. I came up with a few DI tricks to preserve per-adapter implementations. Long story short - "merge" package is now in "cayenne-dbsync" as well.
>>
>> * Name generators for various model objects in various contexts were unified, heavily refactored and placed in "cayenne-dbsync" as well. There are just 2 visible ones: NameBuilder and ObjectNameGenerator. Both complement each other and are fairly easy to understand.
>>
>> * Removed most logic forks in our reverse engineering flow. Now regardless of whether you do rev-eng from scratch, or merging into an existing DataMap, the sequence is pretty much the same:
>>
>> 1. DbLoader reverse engineers the DB based on its filters into a temporary DataMap.
>> 2. This DataMap contains only the DB layer. No ObjEntities.
>> 3. Then we compare this DataMap with our target DataMap, generate MergerTokens and merge them.
>>
>> Now of the above is backwards compatible API-wise, but this is not the core API that we absolutely need to preserve between releases. So I think this is not a problem. The only visible incompatibility for an average user is this:
>>
>> https://issues.apache.org/jira/browse/CAY-2118
>>
>> Next I am going to pull some stuff from downstream cayenne-tools into cayenne-datasync to make reverse engineering mechanism fully agnostic of the caller. There's probably more cleanup to do around configurations, etc.
>>
>> After that going to finally fix a few rev-eng bugs that triggered this whole endeavor.
>>
>> Also we'll need to spend some time on the new rev-eng tab in the modeler. It kinda works, but there are UI quirks, and simply lots of unused potential in how we can present the new backend capabilities via the UI.
>>
>> Hope all this make sense :)
>>
>> Andrus

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: reverse engineering refactoring

Michael Gentry-2
In reply to this post by Andrus Adamchik
On Mon, Oct 3, 2016 at 2:57 PM, Andrus Adamchik <[hidden email]>
wrote:

> Also we'll need to spend some time on the new rev-eng tab in the modeler.
> It kinda works, but there are UI quirks, and simply lots of unused
> potential in how we can present the new backend capabilities via the UI.
>

I'm still making slow, but steady, progress on the CM rewrite/prototype I
started.  Would we want to try to do this in that or continue with the
current CM and try to reuse what we can in the future?

mrg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: reverse engineering refactoring

Andrus Adamchik

> On Oct 5, 2016, at 2:33 PM, Michael Gentry <[hidden email]> wrote:
>
> On Mon, Oct 3, 2016 at 2:57 PM, Andrus Adamchik <[hidden email]>
> wrote:
>
>> Also we'll need to spend some time on the new rev-eng tab in the modeler.
>> It kinda works, but there are UI quirks, and simply lots of unused
>> potential in how we can present the new backend capabilities via the UI.
>>
>
> I'm still making slow, but steady, progress on the CM rewrite/prototype I
> started.  Would we want to try to do this in that or continue with the
> current CM and try to reuse what we can in the future?
>
> mrg

Yeah, I wish we'd stop spending time on the Swing version. But we have no choice but to keep it going until the new JavaFX version is ready for an average user and supports all main functionality.

Not sure if we can design the new system so that it can reuse more stuff with the old one. I'd imagine we do reuse a bunch of stuff already (org.apache.cayenne.map, cayenne-project, cayenne-dbsync). Wonder if we can increase this reuse somehow, e.g. via a shared Bootique service stack (??)

Andrus
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: reverse engineering refactoring

Michael Gentry-2
I still haven't had a chance to experiment with Bootique to see how it
would fit in yet...

I'm currently pulling in cayenne-server, cayenne-di, and cayenne-project,
but don't see any issues adding more modules as-needed.


On Wed, Oct 5, 2016 at 7:56 AM, Andrus Adamchik <[hidden email]>
wrote:

>
> > On Oct 5, 2016, at 2:33 PM, Michael Gentry <[hidden email]>
> wrote:
> >
> > On Mon, Oct 3, 2016 at 2:57 PM, Andrus Adamchik <[hidden email]>
> > wrote:
> >
> >> Also we'll need to spend some time on the new rev-eng tab in the
> modeler.
> >> It kinda works, but there are UI quirks, and simply lots of unused
> >> potential in how we can present the new backend capabilities via the UI.
> >>
> >
> > I'm still making slow, but steady, progress on the CM rewrite/prototype I
> > started.  Would we want to try to do this in that or continue with the
> > current CM and try to reuse what we can in the future?
> >
> > mrg
>
> Yeah, I wish we'd stop spending time on the Swing version. But we have no
> choice but to keep it going until the new JavaFX version is ready for an
> average user and supports all main functionality.
>
> Not sure if we can design the new system so that it can reuse more stuff
> with the old one. I'd imagine we do reuse a bunch of stuff already
> (org.apache.cayenne.map, cayenne-project, cayenne-dbsync). Wonder if we can
> increase this reuse somehow, e.g. via a shared Bootique service stack (??)
>
> Andrus
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: reverse engineering refactoring

Andrus Adamchik

> On Oct 6, 2016, at 3:32 PM, Michael Gentry <[hidden email]> wrote:
>
>>>>
>>>> Also we'll need to spend some time on the new rev-eng tab in the
>> modeler.
>>>> It kinda works, but there are UI quirks, and simply lots of unused
>>>> potential in how we can present the new backend capabilities via the UI.

BTW, my thinking on this UI now is to restore the old M3 reverse engineering UI, but hook it up to M4 backend that can do clean schema merging. This way we have a chance of releasing M4 in less than a year :)

Andrus
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: reverse engineering refactoring

Andrus Adamchik
In reply to this post by Tony Giaccone

> On Oct 5, 2016, at 3:51 AM, Gmail <[hidden email]> wrote:
>
> 1) have the model "remember" which datasource was used to
> Rev-engineer.  

Already fixed on master.

Andrus
Loading...