Railway Users Must Rotate Secrets After Potential Environment Variable Breach
Cloud platform Railway warns that attackers may have accessed decrypted credentials, urging all users to rotate API keys and database passwords immediately.
Key points
- Railway detected suspicious activity targeting systems storing customer environment variables.
- An unauthorised third party may have obtained decrypted contents of some variables.
- All users are advised to rotate secrets and check logs for unusual activity.
Railway has issued an urgent warning to its user base after detecting suspicious activity on the systems responsible for storing customer environment variables. The cloud hosting provider states that its ongoing investigation indicates an unauthorised third party may have accessed encrypted variables. In some specific cases, the attackers appear to have obtained the decrypted contents of these sensitive configuration settings.
What data was potentially exposed?
Environment variables often contain critical secrets, including API keys, database credentials, and private registry passwords. Railway clarifies that while the incident involved encrypted variables, the decrypted contents were potentially accessed for some of them. This exposure means that credentials used to connect applications to external services or databases may now be known to an unauthorised party.
Who is affected by this incident?
Although initial analysis suggests that only a subset of users may be directly affected, Railway is urging all users to take precautionary measures. The platform hosts more than 16 million services for around 4.8 million users. Specifically, AI inference providers, which run AI models for Railway-hosted applications, were notified directly and advised to revoke all their keys as a precautionary step.
How is the service operating now?
Railway has paused all deployments until the investigation is complete, meaning users cannot roll out new versions of their applications during this period. As of 08:44 UTC on 22 October, builds had been restored for Pro and Enterprise workspaces. However, deployments remained delayed with many queued, and builds had not yet resumed for Hobby, Free, and Trial workspaces.
What should users do next?
Railway will directly notify users whose accounts are confirmed to be compromised and provide further details. In the meantime, the company urges all users to immediately rotate environment variables and secrets stored on the platform. This includes private registry credentials. Users should also check associated service logs for any unusual activity that might indicate unauthorised access or exploitation of the exposed credentials.
What affected people can do
Teams should verify whether their applications rely on credentials stored in environment variables. If so, generate new keys and update configurations immediately. Monitor application logs for unexpected traffic or access patterns that suggest misuse. Be alert for phishing attempts claiming to be from Railway support, asking for login details to 'verify' accounts. Legitimate providers will never ask for passwords via email or chat.
What to do and how to stay safe: Railway
- Rotate all API keys, database passwords, and registry credentials stored in environment variables on the platform.
- Review application logs for unusual activity, such as unexpected data exports or connections from unknown IP addresses.
- Monitor for phishing emails or messages impersonating Railway support that request login credentials or personal information.
- Wait for direct notification from the vendor if your account is confirmed compromised, but do not delay credential rotation.
Step-by-step guide: Identity Theft Protection: Answers to Critical Security Questions
General security guidance from the Phish Herald newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
Did Railway confirm that all user data was stolen?
No, the company states that an unauthorised third party may have accessed encrypted variables and obtained decrypted contents for some of them, but initial analysis suggests only a subset of users may be affected.
Are deployments currently working on Railway?
Railway has paused all deployments until the investigation completes, although builds were restored for Pro and Enterprise workspaces as of 22 October.
Which specific services were targeted by the attackers?
The incident was part of a series of attacks targeting AI inference providers, which were notified directly and advised to revoke all their keys as a precaution.
