cerc-io/laconicd-deprecated
19
0
Fork 1
Code Issues 21 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Tests in CI are flakey #142

New Issue
Open
opened 2024-01-23 17:45:45 +00:00 by telackey · 12 comments
telackey commented 2024-01-23 17:45:45 +00:00
Member
Copy Link

Here is an example failure for test-rpc:

=== RUN   TestEth_GetFilterChanges_NoTopics
    rpc_test.go:295: 
        	Error Trace:	/workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:295
        	            				/workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:367
        	Error:      	Expected value not to be nil.
        	Test:       	TestEth_GetFilterChanges_NoTopics
        	Messages:   	transaction failed
--- FAIL: TestEth_GetFilterChanges_NoTopics (16.62s)
=== RUN   TestEth_GetFilterChanges_Topics_AB
    rpc_test.go:454: 
        	Error Trace:	/workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:454
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	TestEth_GetFilterChanges_Topics_AB
--- FAIL: TestEth_GetFilterChanges_Topics_AB (19.72s)

There is nothing new that is broken here, and re-running the test passed without issue.

The test seems to be dependent on some sort of performance of other environmental factor that means sometimes it works and sometimes it does not, depending on what task runner it gets assigned to, what other actions are running on the machine, etc.

Here is an example failure for `test-rpc`: ``` === RUN TestEth_GetFilterChanges_NoTopics rpc_test.go:295: Error Trace: /workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:295 /workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:367 Error: Expected value not to be nil. Test: TestEth_GetFilterChanges_NoTopics Messages: transaction failed --- FAIL: TestEth_GetFilterChanges_NoTopics (16.62s) === RUN TestEth_GetFilterChanges_Topics_AB rpc_test.go:454: Error Trace: /workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:454 Error: Not equal: expected: 1 actual : 2 Test: TestEth_GetFilterChanges_Topics_AB --- FAIL: TestEth_GetFilterChanges_Topics_AB (19.72s) ``` There is nothing new that is broken here, and re-running the test passed without issue. The test seems to be dependent on some sort of performance of other environmental factor that means sometimes it works and sometimes it does not, depending on what task runner it gets assigned to, what other actions are running on the machine, etc.
telackey commented 2024-01-23 18:14:28 +00:00
Author
Member
Copy Link

Another example:

=== RUN   TestEth_GetFilterChanges_Topics_AB
    rpc_test.go:420: 
        	Error Trace:	/workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:420
        	            				/workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:445
        	Error:      	Expected value not to be nil.
        	Test:       	TestEth_GetFilterChanges_Topics_AB
        	Messages:   	transaction failed
--- FAIL: TestEth_GetFilterChanges_Topics_AB (16.58s)
=== RUN   TestEth_GetFilterChanges_Topics_XB
    rpc_test.go:420: 
        	Error Trace:	/workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:420
        	            				/workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:475
        	Error:      	Expected value not to be nil.
        	Test:       	TestEth_GetFilterChanges_Topics_XB
        	Messages:   	transaction failed
--- FAIL: TestEth_GetFilterChanges_Topics_XB (16.35s)
Another example: ``` === RUN TestEth_GetFilterChanges_Topics_AB rpc_test.go:420: Error Trace: /workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:420 /workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:445 Error: Expected value not to be nil. Test: TestEth_GetFilterChanges_Topics_AB Messages: transaction failed --- FAIL: TestEth_GetFilterChanges_Topics_AB (16.58s) === RUN TestEth_GetFilterChanges_Topics_XB rpc_test.go:420: Error Trace: /workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:420 /workspace/cerc-io/laconicd/tests/rpc/rpc_test.go:475 Error: Expected value not to be nil. Test: TestEth_GetFilterChanges_Topics_XB Messages: transaction failed --- FAIL: TestEth_GetFilterChanges_Topics_XB (16.35s) ```
dboreham commented 2024-01-23 18:15:29 +00:00
Owner
Copy Link

Can we get the test time for the passed case?

Can we get the test time for the passed case?
telackey commented 2024-01-23 18:15:56 +00:00
Author
Member
Copy Link

SDK tests:

    github.com/tendermint/tendermint/mempool/v0.(*CListMempool).CheckTx
    	github.com/tendermint/tendermint@v0.34.24/mempool/v0/clist_mempool.go:254
    account sequence mismatch, expected 8, got 7: incorrect account sequence
SDK tests: ``` github.com/tendermint/tendermint/mempool/v0.(*CListMempool).CheckTx github.com/tendermint/tendermint@v0.34.24/mempool/v0/clist_mempool.go:254 account sequence mismatch, expected 8, got 7: incorrect account sequence ```
dboreham commented 2024-01-23 18:17:16 +00:00
Owner
Copy Link

Some parallelism thing?

Some parallelism thing?
telackey commented 2024-01-23 18:18:05 +00:00
Author
Member
Copy Link

Can we get the test time for the passed case?

=== RUN   TestEth_GetFilterChanges_NoTopics
--- PASS: TestEth_GetFilterChanges_NoTopics (6.66s)
=== RUN   TestEth_GetFilterChanges_Topics_AB
--- PASS: TestEth_GetFilterChanges_Topics_AB (9.68s)
=== RUN   TestEth_GetFilterChanges_Topics_XB
--- PASS: TestEth_GetFilterChanges_Topics_XB (9.68s)

With an overall runtime of 4m23s

> Can we get the test time for the passed case? ``` === RUN TestEth_GetFilterChanges_NoTopics --- PASS: TestEth_GetFilterChanges_NoTopics (6.66s) === RUN TestEth_GetFilterChanges_Topics_AB --- PASS: TestEth_GetFilterChanges_Topics_AB (9.68s) === RUN TestEth_GetFilterChanges_Topics_XB --- PASS: TestEth_GetFilterChanges_Topics_XB (9.68s) ``` With an overall runtime of 4m23s
dboreham commented 2024-01-23 18:19:27 +00:00
Owner
Copy Link

Definitely shorter times than the failure case. Is there some global timeout we can set?

Definitely shorter times than the failure case. Is there some global timeout we can set?
dboreham commented 2024-01-23 18:19:48 +00:00
Owner
Copy Link

Also : what's it doing for 20 seconds??

Also : what's it doing for 20 seconds??
dboreham commented 2024-01-23 18:20:13 +00:00
Owner
Copy Link

Another idea: could we just try a faster runner and see if that fixes the problem?

Another idea: could we just try a faster runner and see if that fixes the problem?
telackey commented 2024-01-23 18:22:33 +00:00
Author
Member
Copy Link

I'm all for trying the that, we just need to know where to put it.

I'm all for trying the that, we just need to know where to put it.
dboreham commented 2024-01-23 18:23:50 +00:00
Owner
Copy Link

What does it entail? Could we run it temporarily on one of the big servers?

What does it entail? Could we run it temporarily on one of the big servers?
telackey commented 2024-01-23 18:31:42 +00:00
Author
Member
Copy Link

It is fairly simple to spin up one, and we'd need to give it some sort of unique tag and alter the workflows accordingly.

That would also have the side-effect of queueing all the tasks on the same runner to run sequentially, which might improve performance of individual tests vs the possibility of multiple jobs running on the same host machine at the same time.

I'm not sure how that works long term, because it is pretty inefficient, but it might be useful for diagnostic purposes.

It is fairly simple to spin up one, and we'd need to give it some sort of unique tag and alter the workflows accordingly. That would also have the side-effect of queueing all the tasks on the same runner to run sequentially, which might improve performance of individual tests vs the possibility of multiple jobs running on the same host machine at the same time. I'm not sure how that works long term, because it is pretty inefficient, but it might be useful for diagnostic purposes.
dboreham commented 2024-01-23 18:32:53 +00:00
Owner
Copy Link

My thinking was to just try it initially and see if it makes the tests reliable. Then we can think about next steps.

My thinking was to just try it initially and see if it makes the tests reliable. Then we can think about next steps.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something isn't working

  
C:CLI

   
C:Crypto

   
C:Encoding

   
C:Proto

   
C:Types

   
dependencies

Pull requests that update a dependency file

  
docker

Pull requests that update Docker code

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
go

Pull requests that update Go code

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
high priority

   
in progress

currently working on

  
invalid

This doesn't seem right

  
javascript

Pull requests that update Javascript code

  
low priority

   
medium priority

   
question

Further information is requested

  
Status: Stale

   
Type: ADR

   
Type: Build

   
Type: CI

   
Type: Docs

   
Type: Tests

   
urgent

Top priority issue

  
wontfix

This will not be worked on

  
Copied from Github

An issue or PR manually copied from GitHub.

  
Kind/Breaking

Breaking change that won't be backward compatible

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Security

This is security issue

  
Kind/Testing

Issue or pull request related to testing

  
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Medium
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

No Label
bug
C:CLI
C:Crypto
C:Encoding
C:Proto
C:Types
dependencies
docker
documentation
duplicate
enhancement
go
good first issue
help wanted
high priority
in progress
invalid
javascript
low priority
medium priority
question
Status: Stale
Type: ADR
Type: Build
Type: CI
Type: Docs
Type: Tests
urgent
wontfix
Copied from Github
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/laconicd-deprecated#142
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?