endpoints.ts unit tests unstable

ashten commented

2025-01-19 07:47:12 +00:00

Owner

💡 Summary

the unit tests definited in backend/test/e2e/endpoints.ts are unstable and sometimes fail, only to succeed when the job is rerun

🥰 Expected Behavior

the unit tests should succeed, especially when the code written does not affect those unit tests. they should be stable meaning that the same unit tests can be run on the same code and have the exact same results

🤬 Actual Behavior

the unit tests are erratic and sometimes fail, sometimes succeed, even when run on the same branch without any code modifications

📝 Steps to Reproduce

No response

💻 Frontend Environment

* Model and OS of the device(s):
* Browser:
* Server URL:
* Misskey:

🛰 Backend Environment (for server admin)

* Installation Method or Hosting Service:
* Misskey:
* Node:
* PostgreSQL:
* Redis:
* OS and Architecture:

Do you want to address this bug yourself?

Yes, I will patch the bug myself and send a pull request

### 💡 Summary the unit tests definited in backend/test/e2e/endpoints.ts are unstable and sometimes fail, only to succeed when the job is rerun ### 🥰 Expected Behavior the unit tests should succeed, especially when the code written does not affect those unit tests. they should be stable meaning that the same unit tests can be run on the same code and have the exact same results ### 🤬 Actual Behavior the unit tests are erratic and sometimes fail, sometimes succeed, even when run on the same branch without any code modifications ### 📝 Steps to Reproduce _No response_ ### 💻 Frontend Environment ```markdown * Model and OS of the device(s): * Browser: * Server URL: * Misskey: ``` ### 🛰 Backend Environment (for server admin) ```markdown * Installation Method or Hosting Service: * Misskey: * Node: * PostgreSQL: * Redis: * OS and Architecture: ``` ### Do you want to address this bug yourself? - [ ] Yes, I will patch the bug myself and send a pull request

ashten commented

2025-01-19 07:49:39 +00:00

Author

Owner

left steps to reproduce empty in case we find a pattern (maybe caused by one of the action runners acting up?)

ashten added the

bug

label 2025-01-19 07:49:52 +00:00

ashten referenced this issue

2025-01-19 07:55:35 +00:00

Replace notes(userId) index with (userId, id) #28

ashten commented

2025-01-19 08:03:12 +00:00

Author

Owner

job failure logs

test-backend-e2e (22.x)-784.log

260 KiB

ashten added the

high priority

label 2025-01-19 08:50:59 +00:00

ashten commented

2025-01-19 08:59:33 +00:00

Author

Owner

job success logs

test-backend-e2e (22.x)-799.log

254 KiB

itssophie commented

2025-01-28 18:00:08 +00:00

assert.strictEqual(newBob.followingCount, 0);

assert.strictEqual(res.status, 400);

Are the lines failing. The last 2, it explicitly only checks if the last fails, but it succeeds. It could be that the request before fails, and therefore the one after succeeds. There would be the need to check that too.

The first one,
Would the later check on Alice also fail?
I rn can think of that it maybe fails because the API is serving still old data, by either using cached data or for some weird reason it first fetches the follower count and then the unfollow happens (because it is async?) - would be interesting to introduce a delay and see if then it fails less often (or never) (the fact of one executing before the other, could be a reason of failure also for the second and third failure)

@ashten do you mind sharing another failure log (even better if 2)? So I could double check if there are other tests failing, or only these 3

https://git.woem.men/woem.men/forkey/src/commit/f4bda8582f66ed7ea5ea14d9a9a82c9941256020/packages/backend/test/e2e/endpoints.ts#L363 https://git.woem.men/woem.men/forkey/src/commit/f4bda8582f66ed7ea5ea14d9a9a82c9941256020/packages/backend/test/e2e/endpoints.ts#L848 https://git.woem.men/woem.men/forkey/src/commit/f4bda8582f66ed7ea5ea14d9a9a82c9941256020/packages/backend/test/e2e/endpoints.ts#L875 Are the lines failing. The last 2, it explicitly only checks if the last fails, but it succeeds. It could be that the request before fails, and therefore the one after succeeds. There would be the need to check that too. The first one, Would the later check on Alice also fail? I rn can think of that it maybe fails because the API is serving still old data, by either using cached data or for some weird reason it first fetches the follower count and then the unfollow happens (because it is async?) - would be interesting to introduce a delay and see if then it fails less often (or never) (the fact of one executing before the other, could be a reason of failure also for the second and third failure) @ashten do you mind sharing another failure log (even better if 2)? So I could double check if there are other tests failing, or only these 3

itssophie commented

2025-01-28 18:06:49 +00:00

There is also need for more descriptive test failures. So in this example, if the API fails or passes for an unexpected reasons, also that the responses before are logged would be useful.

itssophie commented

2025-01-28 18:22:13 +00:00

This appears to also be an upstream issue. While not at this rate, it still happens https://github.com/MisskeyIO/misskey/actions/runs/12816504556/job/35737592822#step:10:1103

endpoints.ts unit tests unstable #44

💡 Summary

🥰 Expected Behavior

🤬 Actual Behavior

📝 Steps to Reproduce

💻 Frontend Environment

🛰 Backend Environment (for server admin)

Do you want to address this bug yourself?