System Performance is down
Incident Report for TrustSource
Update
We were able to remove the limitations of LegalCheck, so that it can be used. But there still may be conditions, under which the results will not be available in time. If you are facing such a situation, please just re-process the scan/re-analyse.
We are working on a short term fix to reduce the impact of this issue.
Posted Jul 08, 2024 - 15:42 CEST
Update
We are continuing to monitor for any further issues.
Posted Jul 08, 2024 - 12:28 CEST
Monitoring
During repair we ran into issues with updating our infrastructure due to limitations of our provider. The changes require to re-deploy some stacks containing NICs. The replacement of NICs took hours instead of seconds which lead to unforeseen race-conditions. The resulting complications are understood and will be addressed.
As a result the legalCheck service will be affected for a while and analysis performance will be limited.
Posted Jul 08, 2024 - 11:30 CEST
Update
We are continuing to investigate this issue.
Posted Jul 08, 2024 - 11:04 CEST
Update
We were able to identify the causes and rebuilt the lost indexes. Performance should have returned to normal since 0913 CET. We keep monitoring behaviour and will conduct a root cause analysis.
However, our legalcheck service still has issues due to problems in re-creating the index. Thus very complex requests may not receive answers fast enough. We recommend to reprocess scans uploaded between 240707-2325 and now later, when legalcheck service will be fully restored.
Posted Jul 08, 2024 - 09:54 CEST
Update
We are continuing to investigate this issue.
Posted Jul 08, 2024 - 01:58 CEST
Investigating
Our monitoring recognised a significant performance decrease in all DB-related services. We are investigating the issue together with our service provider
Posted Jul 08, 2024 - 01:58 CEST
This incident affects: TrustSource Services (Core Service, LegalCheck Service, API).