3.1 jenkins builds

classic Classic list List threaded Threaded
33 messages Options
12
Reply | Threaded
Open this post in threaded view
|

3.1 jenkins builds

Andrus Adamchik
Savva recently did some tweaks to 3.1 test bootstrap code to ensure that Jenkins is picking up the right DB type (until now it was testing everything with default DB - HSQL , no matter what Jenkins UI showed). I just edited Jenkins configs to make sure these changes can take effect, added Java 8 dimension, and manually started a 3.1 build. Now that the tests are running against HSQL, H2 and Derby, I expected a fair amount of random failures. Though this is pretty bad:

https://builds.apache.org/view/A-D/view/Cayenne/job/cayenne-31/126/

You may remember that the cause of failures is DB cleanup randomness in the tests (not bugs in Cayenne). Those were eradicated in 4.0, stabilizing the builds there. Porting the fixes to 3.1 felt like a huge undertaking and was never pursued. This unfortunately means that now it is hard to separate real errors from noise on the stable branch :-/

Andrus
Reply | Threaded
Open this post in threaded view
|

4.0 jenkins build failures

Andrus Adamchik
Hmm, now 4.0 started failing with random errors. Makes no sense what so ever.

A.


> On Mar 24, 2016, at 10:01 PM, Andrus Adamchik <[hidden email]> wrote:
>
> Savva recently did some tweaks to 3.1 test bootstrap code to ensure that Jenkins is picking up the right DB type (until now it was testing everything with default DB - HSQL , no matter what Jenkins UI showed). I just edited Jenkins configs to make sure these changes can take effect, added Java 8 dimension, and manually started a 3.1 build. Now that the tests are running against HSQL, H2 and Derby, I expected a fair amount of random failures. Though this is pretty bad:
>
> https://builds.apache.org/view/A-D/view/Cayenne/job/cayenne-31/126/
>
> You may remember that the cause of failures is DB cleanup randomness in the tests (not bugs in Cayenne). Those were eradicated in 4.0, stabilizing the builds there. Porting the fixes to 3.1 felt like a huge undertaking and was never pursued. This unfortunately means that now it is hard to separate real errors from noise on the stable branch :-/
>
> Andrus

Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Aristedes Maniatis-2
The log I just looked at failed inside Hessian serialisation and ultimately:

Caused by: java.lang.NullPointerException: null
        at java.net.URL.readResolve(URL.java:1311)

Could this be a socket problem with multiple jobs running at once on the same machine and trying to grab the same unix socket or port?


Ari


On 30/03/2016 6:40pm, Andrus Adamchik wrote:

> Hmm, now 4.0 started failing with random errors. Makes no sense what so ever.
>
> A.
>
>
>> On Mar 24, 2016, at 10:01 PM, Andrus Adamchik <[hidden email]> wrote:
>>
>> Savva recently did some tweaks to 3.1 test bootstrap code to ensure that Jenkins is picking up the right DB type (until now it was testing everything with default DB - HSQL , no matter what Jenkins UI showed). I just edited Jenkins configs to make sure these changes can take effect, added Java 8 dimension, and manually started a 3.1 build. Now that the tests are running against HSQL, H2 and Derby, I expected a fair amount of random failures. Though this is pretty bad:
>>
>> https://builds.apache.org/view/A-D/view/Cayenne/job/cayenne-31/126/
>>
>> You may remember that the cause of failures is DB cleanup randomness in the tests (not bugs in Cayenne). Those were eradicated in 4.0, stabilizing the builds there. Porting the fixes to 3.1 felt like a huge undertaking and was never pursued. This unfortunately means that now it is hard to separate real errors from noise on the stable branch :-/
>>
>> Andrus
>

--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Andrus Adamchik
In reply to this post by Andrus Adamchik
I wonder if we should experiment with Travis CI, with builds done off of GitHub? In my experience it gives full build isolation, and would also allow us to run our own Docker containers, meaning we can finally test against MySQL and PostgreSQL.

Andrus


> On Mar 30, 2016, at 10:40 AM, Andrus Adamchik <[hidden email]> wrote:
>
> Hmm, now 4.0 started failing with random errors. Makes no sense what so ever.
>
> A.
>
>
>> On Mar 24, 2016, at 10:01 PM, Andrus Adamchik <[hidden email]> wrote:
>>
>> Savva recently did some tweaks to 3.1 test bootstrap code to ensure that Jenkins is picking up the right DB type (until now it was testing everything with default DB - HSQL , no matter what Jenkins UI showed). I just edited Jenkins configs to make sure these changes can take effect, added Java 8 dimension, and manually started a 3.1 build. Now that the tests are running against HSQL, H2 and Derby, I expected a fair amount of random failures. Though this is pretty bad:
>>
>> https://builds.apache.org/view/A-D/view/Cayenne/job/cayenne-31/126/
>>
>> You may remember that the cause of failures is DB cleanup randomness in the tests (not bugs in Cayenne). Those were eradicated in 4.0, stabilizing the builds there. Porting the fixes to 3.1 felt like a huge undertaking and was never pursued. This unfortunately means that now it is hard to separate real errors from noise on the stable branch :-/
>>
>> Andrus
>

Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Andrus Adamchik
In reply to this post by Aristedes Maniatis-2
I thought all our tests are using LocalConnection and do not open server sockets. Could be wrong. Something to check.

Andrus

> On Mar 30, 2016, at 10:46 AM, Aristedes Maniatis <[hidden email]> wrote:
>
> The log I just looked at failed inside Hessian serialisation and ultimately:
>
> Caused by: java.lang.NullPointerException: null
> at java.net.URL.readResolve(URL.java:1311)
>
> Could this be a socket problem with multiple jobs running at once on the same machine and trying to grab the same unix socket or port?
>
>
> Ari
>
>
> On 30/03/2016 6:40pm, Andrus Adamchik wrote:
>> Hmm, now 4.0 started failing with random errors. Makes no sense what so ever.
>>
>> A.
>>
>>
>>> On Mar 24, 2016, at 10:01 PM, Andrus Adamchik <[hidden email]> wrote:
>>>
>>> Savva recently did some tweaks to 3.1 test bootstrap code to ensure that Jenkins is picking up the right DB type (until now it was testing everything with default DB - HSQL , no matter what Jenkins UI showed). I just edited Jenkins configs to make sure these changes can take effect, added Java 8 dimension, and manually started a 3.1 build. Now that the tests are running against HSQL, H2 and Derby, I expected a fair amount of random failures. Though this is pretty bad:
>>>
>>> https://builds.apache.org/view/A-D/view/Cayenne/job/cayenne-31/126/
>>>
>>> You may remember that the cause of failures is DB cleanup randomness in the tests (not bugs in Cayenne). Those were eradicated in 4.0, stabilizing the builds there. Porting the fixes to 3.1 felt like a huge undertaking and was never pursued. This unfortunately means that now it is hard to separate real errors from noise on the stable branch :-/
>>>
>>> Andrus
>>
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Aristedes Maniatis-2
In reply to this post by Andrus Adamchik
I've just added the travis file. Andrus, can you add me as an admin to https://github.com/apache/cayenne?

Ari


On 30/03/2016 6:47pm, Andrus Adamchik wrote:

> I wonder if we should experiment with Travis CI, with builds done off of GitHub? In my experience it gives full build isolation, and would also allow us to run our own Docker containers, meaning we can finally test against MySQL and PostgreSQL.
>
> Andrus
>
>
>> On Mar 30, 2016, at 10:40 AM, Andrus Adamchik <[hidden email]> wrote:
>>
>> Hmm, now 4.0 started failing with random errors. Makes no sense what so ever.
>>
>> A.
>>
>>
>>> On Mar 24, 2016, at 10:01 PM, Andrus Adamchik <[hidden email]> wrote:
>>>
>>> Savva recently did some tweaks to 3.1 test bootstrap code to ensure that Jenkins is picking up the right DB type (until now it was testing everything with default DB - HSQL , no matter what Jenkins UI showed). I just edited Jenkins configs to make sure these changes can take effect, added Java 8 dimension, and manually started a 3.1 build. Now that the tests are running against HSQL, H2 and Derby, I expected a fair amount of random failures. Though this is pretty bad:
>>>
>>> https://builds.apache.org/view/A-D/view/Cayenne/job/cayenne-31/126/
>>>
>>> You may remember that the cause of failures is DB cleanup randomness in the tests (not bugs in Cayenne). Those were eradicated in 4.0, stabilizing the builds there. Porting the fixes to 3.1 felt like a huge undertaking and was never pursued. This unfortunately means that now it is hard to separate real errors from noise on the stable branch :-/
>>>
>>> Andrus
>>
>

--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Andrus Adamchik

> On Mar 30, 2016, at 10:59 AM, Aristedes Maniatis <[hidden email]> wrote:
>
> Andrus, can you add me as an admin to https://github.com/apache/cayenne?

This will have to go through infra. They only made me a part of the Github "apache" org like a month ago, and I don't have access to the repo "settings" myself either.

Andrus
Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Aristedes Maniatis-2
On 30/03/2016 7:15pm, Andrus Adamchik wrote:
> This will have to go through infra. They only made me a part of the Github "apache" org like a month ago, and I don't have access to the repo "settings" myself either.


For any other committers who want to be joined to the apache github group, just go here https://id.apache.org/ and add your github id to your account yourself.

As for enabling travis, infra need to do that:

https://issues.apache.org/jira/browse/INFRA-11569


Ari



--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Aristedes Maniatis-2
On 30/03/2016 8:39pm, Aristedes Maniatis wrote:
> enabling travis

https://travis-ci.org/apache/cayenne/builds/119484078

--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Andrus Adamchik
Nice! Especially the status: https://travis-ci.org/apache/cayenne/builds/119484078 :) Let's add something like this:

- script:
  - mvn clean verify

This is needed to run integration tests.

Andrus

> On Mar 30, 2016, at 12:57 PM, Aristedes Maniatis <[hidden email]> wrote:
>
> On 30/03/2016 8:39pm, Aristedes Maniatis wrote:
>> enabling travis
>
> https://travis-ci.org/apache/cayenne/builds/119484078
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Aristedes Maniatis-2
I'm just heading home from work now, but go ahead and add whatever changes you need. I don't think maven clean is needed though, and doesn't 'test' already run 'verify' in maven?

I've tried to blot out all the maven stuff I have in my head...

Also, was my environment variable setting enough to trigger the different database choices? I'm not sure that was passed on correctly to maven.

Ari


On 30/03/2016 9:12pm, Andrus Adamchik wrote:

> Nice! Especially the status: https://travis-ci.org/apache/cayenne/builds/119484078 :) Let's add something like this:
>
> - script:
>   - mvn clean verify
>
> This is needed to run integration tests.
>
> Andrus
>
>> On Mar 30, 2016, at 12:57 PM, Aristedes Maniatis <[hidden email]> wrote:
>>
>> On 30/03/2016 8:39pm, Aristedes Maniatis wrote:
>>> enabling travis
>>
>> https://travis-ci.org/apache/cayenne/builds/119484078
>>
>> --
>> -------------------------->
>> Aristedes Maniatis
>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>

--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Andrus Adamchik
In reply to this post by Andrus Adamchik
I also just added a few status badges to our GitHub README.md. 



On Mar 30, 2016, at 1:12 PM, Andrus Adamchik <[hidden email]> wrote:

Nice! Especially the status: https://travis-ci.org/apache/cayenne/builds/119484078 :) Let's add something like this:

- script:
 - mvn clean verify

This is needed to run integration tests.

Andrus

On Mar 30, 2016, at 12:57 PM, Aristedes Maniatis <[hidden email]> wrote:

On 30/03/2016 8:39pm, Aristedes Maniatis wrote:
enabling travis

https://travis-ci.org/apache/cayenne/builds/119484078

-- 
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A


Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Andrus Adamchik
In reply to this post by Aristedes Maniatis-2

> On Mar 30, 2016, at 1:16 PM, Aristedes Maniatis <[hidden email]> wrote:
>
> I'm just heading home from work now, but go ahead and add whatever changes you need. I don't think maven clean is needed though, and doesn't 'test' already run 'verify' in maven?

Will do. No, verify is a separate stage that happens after package. So *Test and *IT are run at different stages of lifecycle.

A.

Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Andrus Adamchik
Builds with itests timeout and die because of too many logs generated: https://travis-ci.org/apache/cayenne/builds . I turned off all the "pmd" checks, but this wasn't enough. I guess we'll need to find other ways to trim the logs. Cayenne is too big for Travis :)

A.

> On Mar 30, 2016, at 1:18 PM, Andrus Adamchik <[hidden email]> wrote:
>
>
>> On Mar 30, 2016, at 1:16 PM, Aristedes Maniatis <[hidden email]> wrote:
>>
>> I'm just heading home from work now, but go ahead and add whatever changes you need. I don't think maven clean is needed though, and doesn't 'test' already run 'verify' in maven?
>
> Will do. No, verify is a separate stage that happens after package. So *Test and *IT are run at different stages of lifecycle.
>
> A.
>

Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Michael Gentry
It is probably all the SQL logging...


On Wed, Mar 30, 2016 at 7:18 AM, Andrus Adamchik <[hidden email]>
wrote:

> Builds with itests timeout and die because of too many logs generated:
> https://travis-ci.org/apache/cayenne/builds . I turned off all the "pmd"
> checks, but this wasn't enough. I guess we'll need to find other ways to
> trim the logs. Cayenne is too big for Travis :)
>
> A.
>
> > On Mar 30, 2016, at 1:18 PM, Andrus Adamchik <[hidden email]>
> wrote:
> >
> >
> >> On Mar 30, 2016, at 1:16 PM, Aristedes Maniatis <[hidden email]>
> wrote:
> >>
> >> I'm just heading home from work now, but go ahead and add whatever
> changes you need. I don't think maven clean is needed though, and doesn't
> 'test' already run 'verify' in maven?
> >
> > Will do. No, verify is a separate stage that happens after package. So
> *Test and *IT are run at different stages of lifecycle.
> >
> > A.
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Andrus Adamchik
Among other things.

@Ari: BTW, do you have control of where the failure (and success) emails go? Would be great to route them to commits@

Andrus

> On Mar 30, 2016, at 2:29 PM, Michael Gentry <[hidden email]> wrote:
>
> It is probably all the SQL logging...
>
>
> On Wed, Mar 30, 2016 at 7:18 AM, Andrus Adamchik <[hidden email]>
> wrote:
>
>> Builds with itests timeout and die because of too many logs generated:
>> https://travis-ci.org/apache/cayenne/builds . I turned off all the "pmd"
>> checks, but this wasn't enough. I guess we'll need to find other ways to
>> trim the logs. Cayenne is too big for Travis :)
>>
>> A.
>>
>>> On Mar 30, 2016, at 1:18 PM, Andrus Adamchik <[hidden email]>
>> wrote:
>>>
>>>
>>>> On Mar 30, 2016, at 1:16 PM, Aristedes Maniatis <[hidden email]>
>> wrote:
>>>>
>>>> I'm just heading home from work now, but go ahead and add whatever
>> changes you need. I don't think maven clean is needed though, and doesn't
>> 'test' already run 'verify' in maven?
>>>
>>> Will do. No, verify is a separate stage that happens after package. So
>> *Test and *IT are run at different stages of lifecycle.
>>>
>>> A.
>>>
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Aristedes Maniatis-2
On 30/03/2016 10:37pm, Andrus Adamchik wrote:
> @Ari: BTW, do you have control of where the failure (and success) emails go? Would be great to route them to commits@

As far as I know Travis doesn't send email. There are hooks into github which adds a comment to a pull request, but that's all.


Should we change most of CommonsJdbcEventLogger to DEBUG? For some reason the whole thing is bound to INFO?

        @Override
        public boolean isLoggable() {
                return logger.isInfoEnabled();
        }


logQueryError() should be ERROR or at least WARN. Pretty much everything else should be DEBUG.



Ari

--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Andrus Adamchik





On Mar 31, 2016, at 1:50 AM, Aristedes Maniatis <[hidden email]> wrote:

On 30/03/2016 10:37pm, Andrus Adamchik wrote:
@Ari: BTW, do you have control of where the failure (and success) emails go? Would be great to route them to commits@

As far as I know Travis doesn't send email. There are hooks into github which adds a comment to a pull request, but that's all.

Actually it does for me on other projects. E.g.:


Reply | Threaded
Open this post in threaded view
|

Re: 4.0 jenkins build failures

Andrus Adamchik
In reply to this post by Aristedes Maniatis-2

> On Mar 31, 2016, at 1:50 AM, Aristedes Maniatis <[hidden email]> wrote:
>
> Should we change most of CommonsJdbcEventLogger to DEBUG? For some reason the whole thing is bound to INFO?
>
> @Override
> public boolean isLoggable() {
> return logger.isInfoEnabled();
> }
>
>
> logQueryError() should be ERROR or at least WARN. Pretty much everything else should be DEBUG.

Cayenne has always logged SQL at INFO level. And I'd like to have it on by default. Turning it off for Jenkins (or your own apps) is not a problem - logger levels are configurable. I am not doing it for other reasons:

1. We often want SQL to be logged during tests for tracing purposes.
2. I am not yet convinced Cayenne logs are the culprit here. Consider that these builds are done with an empty local Maven repo, so Maven downloads a bunch of dependencies. Take a look at raw logs on Travis. It is not yet clear SQL logs are the majority.

So I'd like to play with it a bit, and see what we can trim without sacrificing clarity. Long logger names and timestamps are the first candidates.

Andrus
Reply | Threaded
Open this post in threaded view
|

Re: 3.1 jenkins builds

Andrus Adamchik
In reply to this post by Andrus Adamchik
We are not fully out of the woods, but we are in a much better shape now with Jenkins 3.1 builds:

https://builds.apache.org/view/A-D/view/Cayenne/job/cayenne-31/131/

All I fixed was (some) DB cleanup code in the unit tests and Java 8 javadoc profile. So no reasons to believe that anything in the code is broken.

Andrus


> On Mar 24, 2016, at 10:01 PM, Andrus Adamchik <[hidden email]> wrote:
>
> Savva recently did some tweaks to 3.1 test bootstrap code to ensure that Jenkins is picking up the right DB type (until now it was testing everything with default DB - HSQL , no matter what Jenkins UI showed). I just edited Jenkins configs to make sure these changes can take effect, added Java 8 dimension, and manually started a 3.1 build. Now that the tests are running against HSQL, H2 and Derby, I expected a fair amount of random failures. Though this is pretty bad:
>
> https://builds.apache.org/view/A-D/view/Cayenne/job/cayenne-31/126/
>
> You may remember that the cause of failures is DB cleanup randomness in the tests (not bugs in Cayenne). Those were eradicated in 4.0, stabilizing the builds there. Porting the fixes to 3.1 felt like a huge undertaking and was never pursued. This unfortunately means that now it is hard to separate real errors from noise on the stable branch :-/
>
> Andrus

12