3d95b243e9
feat(core): Make queue mode settings configurable ( #7526 )
...
This PR allows users to configure the settings to Bull, possibly
reducing the errors with `maxStalledCount` and other issues, that
usually happen either when a worker crashes or when the event loop is
super busy. Increasing the lease time and the `maxStalledCount` settings
might improve UX.
Github issue / Community forum post (link here to close automatically):
2023-10-26 17:30:16 +02:00
ae8c7a635e
refactor(core)!: Remove webhook deregistration at startup and shutdown ( #7515 )
...
https://linear.app/n8n/issue/PAY-932/deprecate-flag-to-skip-webhook-deregistration-on-shutdown
2023-10-26 14:37:54 +02:00
5477e3fb45
refactor(core): Move execution save settings into lifecycle function (no-changelog) ( #7370 )
...
Move the handling of execution save settings into a tested lifecycle
function as discussed with Omar
2023-10-26 14:35:38 +02:00
1055bd3762
fix(core): Handle missing resultData in runData ( #7523 )
2023-10-26 13:15:19 +02:00
742c8a8534
fix(core): Ensure execution deletion in worker lifecycle hook ( #7481 )
...
Reported by customer
[here](https://n8nio.slack.com/archives/C05PUALKZHD/p1697446945481249?thread_ts=1697196557.638169&cid=C05PUALKZHD ),
apparently a very old long-standing bug for queue mode. Please review
closely as I am not familiar with queue mode.
2023-10-25 19:13:06 +02:00
05586a900d
refactor(core): Make Logger a service (no-changelog) ( #7494 )
2023-10-25 16:35:22 +02:00
0643487403
ci: Fix oclif manifest generation
2023-10-25 14:51:10 +02:00
6f45298d3d
fix(core): Ensure nodes post-processors run in the correct order ( #7500 )
...
Fixes #7497
2023-10-25 13:59:38 +02:00
93cfabbeac
fix(core): Fix workflow activation with history and workflow history for EE (no-changelog) ( #7508 )
...
Github issue / Community forum post (link here to close automatically):
2023-10-25 11:07:11 +01:00
671c95760b
refactor(core): Make executions pruning more resilient ( #7480 )
...
This PR converts the hard-deletion interval to a timeout:
- to prevent the interval from not being restored when hard deletion
throws, and
- to prevent a long-running hard deletion from leading to duplicate
deletions.
2023-10-24 16:16:45 +02:00
78243edd18
refactor(core): Make pruning via lifecycle configuration in S3 mode mandatory ( #7482 )
...
Since we do not store which executions produced binary data, for pruning
on S3 we need to query for binary data items for each execution in order
to delete them. To minimize requests to S3, allow the user to skip
pruning requests when setting TTL at bucket level.
2023-10-24 10:37:02 +02:00
3ef771ea38
feat: Collect usage metrics on license renewal (no-changelog) ( #7486 )
2023-10-23 21:39:29 +02:00
774fe202bf
fix(core): Fix frontend.settings external hook execution ( #7496 )
2023-10-23 11:46:13 -04:00
41236b7e08
fix: Save new version of the workflow instead of the previous (no-changelog) ( #7428 )
...
Github issue / Community forum post (link here to close automatically):
2023-10-23 16:30:36 +02:00
b6de910cbe
refactor(core): Abstract away InstanceSettings and encryptionKey into injectable services (no-changelog) ( #7471 )
...
This change ensures that things like `encryptionKey` and `instanceId`
are always available directly where they are needed, instead of passing
them around throughout the code.
2023-10-23 13:39:35 +02:00
76c04815f7
fix(core): Reduce logging overhead for levels that do not output ( #7479 )
...
all current logging calls execute `callsites()` to figure out what code
tried to log. This happens even for logging methods that aren't supposed
to create any output. Under moderate load, this can take up quite a lot
of resources. This PR changes the logger to make all ignorable logging
methods a No-Op.
In a small benchmark with a simple webhook, with log-level set to
`warn`, and using `ab -c 50 -n 500
http://localhost:5678/webhook/testing `, these were the response times:
### Before
### After
2023-10-20 18:26:33 +02:00
0b42d1aa71
refactor(core): Limit soft-deletions to pruning only ( #7469 )
...
Based on customer feedback, we should limit soft deletions to pruning
only, to prevent executions from piling up in very high volume cases.
2023-10-20 15:02:47 +02:00
647372be27
feat(core): Add support for oauth based service accounts with UM SMTP ( #7311 )
...
This PR adds support for using OAuth based service accounts for the User
Management SMTP connection.
Tested using a Google Service Account.
2023-10-20 11:36:40 +01:00
b50376cf52
refactor(core): Make executions pruning settings configurable ( #7468 )
2023-10-19 16:57:12 +02:00
55c6a1b0d3
fix(core): Do not return inviteAcceptUrl in response if email was sent ( #7465 )
2023-10-19 13:58:06 +02:00
83762e051d
fix(core): Handle gzip and deflate compressed request payloads ( #7461 )
...
NODE-870
2023-10-18 21:52:15 +02:00
40707fa692
feat(core): Make executions pruning interval configurable ( #7439 )
2023-10-18 17:01:57 +02:00
3ddc176dfa
feat(n8n Form Trigger Node): New node ( #7130 )
...
Github issue / Community forum post (link here to close automatically):
based on https://github.com/joffcom/n8n-nodes-form-trigger
---------
Co-authored-by: Giulio Andreini <g.andreini@gmail.com >
2023-10-17 07:09:30 +03:00
e01b9e5ae1
fix(core): Prevent false stalled jobs in queue mode from displaying as errored ( #7435 )
...
This is related to an issue with how Bull handles stalled jobs, see
https://github.com/OptimalBits/bull/issues/1415 for reference.
CPU intensive workflows can in certain cases take a long while to finish
up, thereby blocking the thread and causing Bull queue to think the job
has stalled, even though it finished successfully. In these cases the
error handling could then overwrite the successful execution data with
the error message.
2023-10-13 16:07:08 +02:00
c6ee1e30c0
refactor(core): Add binary data S3 to telemetry ( #7412 )
...
https://n8nio.slack.com/archives/C04B1GZ4T0U/p1697033523039729
2023-10-13 13:16:43 +02:00
45f2ef373e
fix(core): Add check that queue is defined and remove cyclic dependency ( #7404 )
...
In a rare edge case an undefined queue could be returned - this should
not happen and now an error is thrown.
Also using the opportunity to remove a cyclic dependency from the Queue.
2023-10-13 11:53:59 +02:00
46977a2aff
fix: Prevent undefined issues when restoring binary data ( #7419 )
...
Github issue / Community forum post (link here to close automatically):
---------
Co-authored-by: Iván Ovejero <ivov.src@gmail.com >
2023-10-12 16:41:51 +02:00
54e900955a
refactor(core): Move frontend settings to service (no-changelog) ( #7396 )
...
Quick follow-up to https://github.com/n8n-io/n8n/pull/7283
2023-10-11 17:12:19 +02:00
2f4d91b2cd
fix(core): Fix ignoring crashed executions without event msgs ( #7368 )
...
when the event logs do not contain messages for running executions, the
recovery/crash detection on startup would skip these. this PR fixes
that.
2023-10-11 14:22:43 +02:00
6479eb180f
fix(Webhook Node): Backward compatible form-data parsing for non-array files ( #7385 )
...
Fixes
https://community.n8n.io/t/possible-bug-0-added-to-end-of-files-sent-via-webhook/31169
2023-10-10 15:19:05 +02:00
0847623f85
feat(core): Switch binary filesystem mode to nested path structure ( #7307 )
...
Depends on #7253 | Story:
[PAY-863](https://linear.app/n8n/issue/PAY-863/switch-binary-filesystem-mode-to-nested-path-structure )
This PR introduces `filesystem-v2` to store binary data in the
filesystem in the same format as `s3`.
2023-10-10 10:06:06 +02:00
c5ee06cc61
refactor(core): Refactor nodes loading (no-changelog) ( #7283 )
...
fixes PAY-605
2023-10-09 16:09:23 +02:00
789e1e7ed4
fix(core): Add an option to enable postgres ssl with default certs ( #6889 )
...
CP-809
2023-10-09 13:55:03 +02:00
b8608cee6d
feat(core): Add Job Summary to Worker response ( #7360 )
2023-10-06 17:52:27 +02:00
34bda535e6
refactor(core): Create controller for binary data (no-changelog) ( #7363 )
...
This PR adds a controller for binary data + integration tests.
2023-10-06 16:21:13 +02:00
afa683a06f
refactor(core): Have one orchestration service per instance type ( #7303 )
...
webhook instances will not listen to either worker or event log messages
on the Redis pub/sub channel
2023-10-06 13:58:11 +02:00
f4d8c9eed5
feat: Improve error msg when attempting to redeem the same activation code multiple times (no-changelog) ( #7355 )
2023-10-05 17:08:30 +02:00
1a661e6d00
feat(core): Integrate object store as binary data manager ( #7253 )
...
Depends on: #7225 | Story:
[PAY-848](https://linear.app/n8n/issue/PAY-848 )
This PR integrates the object store service as a new binary data manager
for Enterprise.
2023-10-05 15:25:17 +02:00
5cfd87692f
refactor: Add log messages to debug worker init issues ( #7238 )
...
Github issue / Community forum post (link here to close automatically):
2023-10-05 13:37:25 +02:00
169175080a
ci: Fix WorkflowHistoryManager tests (no-changelog) ( #7356 )
...
[DB Tests](https://github.com/n8n-io/n8n/actions/runs/6418058186 )
2023-10-05 13:14:57 +02:00
1b4848afcb
fix(core): Fix pruning of non-finished executions ( #7333 )
...
This fixes a bug in the pruning (soft-delete). The pruning was a bit too
aggressive, as it also pruned executions that weren't in an end state
yet. This only becomes an issue if there are long-running executions
(e.g. workflow with Wait node) or the prune parameters are set to keep
only a tiny number of executions.
2023-10-04 16:32:05 +03:00
0adc533719
feat: Workflow History pruning and prune time settings ( #7343 )
...
Github issue / Community forum post (link here to close automatically):
2023-10-04 13:57:21 +01:00
101255d186
ci: Make builds release-channel aware. Add support for scheduled beta builds ( #7323 )
...
ADO-1121
---------
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com >
2023-10-03 20:49:04 +02:00
00a4b8b0c6
feat(core): Add support for building LLM applications ( #7235 )
...
This extracts all core and editor changes from #7246 and #7137 , so that
we can get these changes merged first.
ADO-1120
[DB Tests](https://github.com/n8n-io/n8n/actions/runs/6379749011 )
[E2E Tests](https://github.com/n8n-io/n8n/actions/runs/6379751480 )
[Workflow Tests](https://github.com/n8n-io/n8n/actions/runs/6379752828 )
---------
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com >
Co-authored-by: Oleg Ivaniv <me@olegivaniv.com >
Co-authored-by: Alex Grozav <alex@grozav.com >
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in >
2023-10-02 17:33:43 +02:00
1691223789
fix(core): Account for itemless case on restoring binary data ID ( #7305 )
...
https://linear.app/n8n/issue/PAY-862
2023-10-02 09:13:55 +02:00
d1b6c7fd79
feat(editor): Workflow history [WIP]- create workflow history list component (no-changelog) ( #7186 )
...
Co-authored-by: Omar Ajoue <krynble@gmail.com >
2023-09-29 17:48:36 +02:00
ec0379378e
feat(core): Add plan name to telemetry (no-changelog) ( #7296 )
...
Github issue / Community forum post (link here to close automatically):
2023-09-29 15:41:32 +02:00
53a7502d20
feat(core): Add secrets provider reload and refactor ( #7277 )
...
This PR adds a message for queue mode which triggers an external secrets
provider reload inside the workers if the configuration has changed on
the main instance.
It also refactors some of the message handler code to remove cyclic
dependencies, as well as remove unnecessary duplicate redis clients
inside services (thanks to no more cyclic deps)
2023-09-28 12:57:35 +02:00
a80abad3af
fix(core): Change WorkflowHistory nodes/connections columns to be json ( #7282 )
...
Github issue / Community forum post (link here to close automatically):
2023-09-28 10:37:33 +02:00
0132514f8b
fix(core): Use consistent timezone-aware timestamps in postgres ( #6948 )
...
Fixes:
* ENG-51 / N8N-2490
* PAY-397
* #2178
* #2810
* #3855
Supersedes #2813
[DB
Tests](https://github.com/n8n-io/n8n/actions/runs/6000780146/job/16273596338 )
2023-09-27 18:44:47 +02:00
Omar Ajoue