Security update

**This post was updated on August 12, 2015 at 12:30 PM EST.**

On August 4, 2015 our security team identified and stopped unauthorized access to one of our systems. As a result, we reset all reseller Control Panel and Reseller Web Interface (RWI) passwords. Since then, we have worked 24/7 to complete an extensive investigation process and conduct a series of full and detailed sweeps of every infrastructural element of our systems. We would like to take this opportunity to share more details with you.

What happened?

– The unauthorized access to our systems was gained through the use of elevated credentials. There was no evidence of brute force failed login attempts.
– There was no attempt made to access production systems and data, nor to access or modify the system code base.
– We discovered a secured file transfer outbound from one of our backup servers attempting to copy data to an external address that we were able to disrupt and terminate before it was successfully completed.

We are a security conscious company. We consider any unauthorized access to a system to be a full compromise of data, and have taken a cautious approach of resetting all credentials that could be affected.

Is my reseller account safe?

We have no evidence that any OpenSRS reseller accounts have been accessed, but even the possibility that this could have happened moved us to err on the side of extreme caution.

What should resellers do?

As a precautionary measure, we have asked OpenSRS resellers to do two things:

1) Reset their Control Panel and Reseller Web Interface (RWI) passwords:

On August 4, we sent an email letting OpenSRS resellers know that we had reset their passwords. All passwords are encrypted to ensure their protection. If you have not already done so, go to the Reseller Control Panel at manage.opensrs.com and reset your password using the “Forgot password?” option. If you have already done it after receiving an email from OpenSRS asking you to do so, no further action is required.

2) Reset their API keys:

As an additional precaution, if you access our system through the API, we have asked you to reset your API key. These keys are also encrypted.

Here are some instructions on how you can do it. IMPORTANT: Once the new API key is generated, the old key will stop working. You must be ready to make the change to avoid service interruption.

Who do I contact if I have any questions or concerns about this issue?

We encourage you do get in touch with support at help@opensrs.com

Tucows is deeply committed to the improvement of Internet security practices. We are thankful to the strong and cooperative industry relationships we have built over the last decade and we’ll continue to be a key player in the fight against cybercrime.

We deeply regret this incident and apologize to everyone who relies on OpenSRS as a trusted service provider.

***********************************************************************************************

**Update: August 12, 2015 12:30PM EST**

After reading all the comments below, it’s important for us to clarify a few points. We once again apologize for the inconvenience and hope you understand that these measures have been implemented with the security of your account in mind.

3rd party software and API users:

For resellers accessing our systems via 3rd party software, we can activate reseller account level authentication for their accounts. We are also happy to activate this for resellers who access our system via custom API integrations. However, they will need to confirm that they authenticate users themselves and not based on the username and passwords we provide. We ask these resellers to please contact OpenSRS support at help@opensrs.com.

–Users of the WHMCS plugin: the WHMCS development team has updated the OpenSRS module to use an alternative authentication method which removes the reliance on registrant domain passwords.This module is compatible with both WHMCS v5.3.x* and v6.0. Once you deploy this updated module, all OpenSRS functionality will be restored.

OpenSRS end-user interface password retrieval process:

There are some concerns about password reset links being sent to an invalid email address. Resellers can log in to the Reseller Control Panel and edit owner’s contact information (including email address). We recommend you watch the Domain Management tutorial. If you are currently using the Reseller Web Interface (RWI), we encourage you to switch to the Reseller Control Panel as it provides a lot more options and functionality.

Reseller Control Panel functionality:

In the Reseller Control Panel you can manage all aspects of a domain name independently of registrant username and passwords. In very specific cases, where policy prevents us from allowing changes, you will need to contact OpenSRS support at help@opensrs.com and we’ll be happy to assist you with changes. We recommend you watch the Domain Management tutorial.

Domain management via API

Some resellers have expressed concerns about being unable to manage domains on behalf of their customers after the registrant password reset. We’d like to reiterate that the OpenSRS end-user management interface is not intended for reseller use. OpenSRS does provide resellers with APIs and Control Panels to manage domains on behalf of their customers using their reseller account login credentials or API key. We understand that some resellers may still be using older versions of our API, and our support staff will be happy to assist and guide you on how you can update your integrations.

Registrant password length requirements

We have rolled back the change that forced resellers to set registrant passwords that are at least 10 characters long. However, this change will be rolled out again on September 9, 2015 for additional security (a reminder will be sent in early September). This may require you to adjust your systems if you have been setting passwords that are less than 10 characters in length. WHMCS and Parallels/Odin will be able to handle 10-character registrant passwords when interfacing with our system.

Multiple domains under a single profile

We are currently working on making better tools available for identifying all domains under a single profile and merging domains into existing profiles in the Reseller Control Panel.

45 thoughts on “Security update

  1. A breach is one thing but now you have changed access to passwords in the api (removed them) and initiated a new password protocol requiring and I quote from your interface
    “Invalid password length: Password should contain at least 10 and at most 20 characters.”

    Security is one thing but you have now effectively caused all billing operations to cease to have any functionality !!
    All users of WHMCS can no longer use their billing software to modify or renew domains
    THATS just plain bad business.
    We need this functionality fixed immediately and you must never make a change such as this without notifying any one.
    Is Open SRS going to reimburse all its reseller using WHMCS (their are thousands) for the time now wasted by client calling to do anything with their domain? They can no longer renew, modify or do any thing else as the billing interface can no longer interact with the Tucows API .

  2. 100% agree. I just discovered further repercussions of your further arbitrary actions that you have not even had the courtesy to mention or document anywhere. The password retrieval has been shut down with absolutely no notice!!! Have you considered the repercussions? What kind of crap is this? You suggested that “We have no evidence that any OpenSRS reseller accounts have been accessed”, yet you made an overreaching arbitrary decision to not only reset all resellers passwords, but now are forcing a reset on all API keys, with zero indication of a timeline to complete this. I don’t see any valid explanation that would justify measures you refer to as “a precautionary measure”, but would be better described as overzealous an unnecessary. Unless more went on with this hack than you are revealing here, these measures add up to Overkill. So how about a detailed explanation of ALL the changes you’ve arbitrarily made, as a COURTESY to your resellers, Instead of having find out the hard way? You have made by job as a reseller more difficulty by your actions, and I am none too happy. This was a very bad decision.

  3. Hi Greg,
    Thank you for your comment and we do apologize for the issues you are experiencing at this time. If you would kindly reach out to our support team (if you haven’t already – help@opensrs.com) with your reseller account and let them know that you are a WHMCS user, they will be able to get you up and running again right away.

  4. Support reached out to me already and resolved..thanks for thst but in the same breath it’s disarming that the fix they had was and is not public knowledge. The fix Absolutly should be public and I can not imagine any reason ii is not.

  5. Just to further elaborate and illustrate why this was a “very bad decision”, I’ll give you a real world example that occurred earlier.

    A registrant contacted us wondering why their domain expired yesterday. I quickly discovered the reason was that the domain’s Admin contact no longer worked for the organization, so the contact email was invalid. Previously, it would be a simple matter for me as a reseller, to login to the RWI, fetch the username/password, provide it to the new contact for the domain, so they could login to the renewal page on our site, renew the domain, and update the invalid contact info. However today, much to my chagrin, I discover the RWI no longer provides the password, just the username. What purpose does that serve? Useless.

    The password retrieval tool now sends a password reset link to the Registrant, instead of the account username/password. How brilliant is that, if it is sent to an invalid email address? So now I’m a reseller with no way to help the Registrant. How embarrassing.

    To continue along my journey of unnecessary embarrassment and humiliation, I identify that because the registrants’ invalid email address (luckily) used to be on one of our domains, that I can a) set up the invalid email account, b) resend the password reset email, c) have our SysAdmin intercept the password reset email, d) click the link to reset, which now takes the user to an off-site link, non-branded and confusing at manage.opensrs.net, which would be A HUGE SECURITY CONCERN TO MOST USERS in itself. e) reset the password, and I can finally help the registrant, only after WASTING part of my afternoon. BUT!!!! That was out of sheer LUCK that I was able to do that. If the invalid email address was not previously on our system, used another domain, or was a Gmail or Hotmail address, etc., then guess what? WE’RE SCREWED! That will represent the majority of cases, and I won’t have “luck” to reply on.

    How inefficient does it have to be before you realize the repercussions of this change? Why block a reseller’s access to the domain password in RWI, when a potential hacker would have access to potentially wreak havoc in the RWI once accessed anyway? The only thing this change has done, is take away a valuable tool that resellers absolutely REQUIRE in order to support registrants.

    Personally, I don’t have time to waste pissing around jumping through extra hoops. Justify your actions, or restore the tools we need to do our jobs. And in the future, think about the repercussions of making arbitrary changes without consulting or advising the resellers whose businesses, and reputations rely on access to these tools. And before you suggest it, I don’t want to “reach out to our support team” as you suggested to Greg. Instead, have your “support team” and more importantly, your management and development teams reach out to this page, read it, and understand the repercussions of your actions.

    This is hugely disappointing.

  6. WTF? I just went to manage one of our own domains with a password I know is vaild, and got an Authentication Error. Then I read Tom Browns comment about resetting passwords for all domains. This can’t be true, can it? Have you reset ALL Registrant domain passwords across the board, without even telling us? No, it can’t be true. There has to be some other explanation. And I hope it’s good. Because if it is true, you’ve just FVCKED US! Seriously, what the Hell is going on?

  7. Derek, any system that can show passwords is storing plain text passwords (or an equivalent). Since they have now decided this was a bad idea (agreed), they are obviously switching to encrypted passwords, which means they no longer know what the password was. Plaintext passwords have been considered a “bad idea” for a _long_ time now… because if the bad guys get in, they’ve got everyone’s userid and password. So while the opensrs change is good, the implimentation… well, words fail :(

  8. “We have no evidence that any OpenSRS reseller accounts have been accessed” but then you go and reset ALL passwords including authentication codes? REALLY??
    You are either paranoid and have a bad management or you have been hacked beyond what you admit. I dunno which is worse…

    Do you realize what are you doing to our business? I only have a few domains and I already have problems. I have to ask the clients to go and reset passwords in order to have access to Manage your domain, change emails or have accesses to auth codes.
    You have screwed up. Big time.
    I just can’t go after each client asking them to spend time because you messed up.
    If they wanted to manage their own domains they would go to godaddy directly.
    I can only imagine the mess you have created to resellers with thousands of clients.

    please, give access to the resellers so they can at least change emails of the owners.

  9. I have to agree with everyone here… this is absolutely ridiculous. Resetting all of the passwords because you found a flaw in your security model is in no way acceptable. If a password is encrypted, it can be decrypted and presented to the resellers who may need this info to do their jobs.;

    As it is now, we will have to change the admin contact to an email address we control, send the password to ourselves then login and fix it for every client we have. How exactly does your security model create better security when more freaking red tape is all that is needed to get around it?

    We manage many domains for some of our clients who have NO IDEA at all how to consolidate them or even how important it is to keep their contact information current. How do we now explain to them that the passwords they had are useless and we have no immediate way to rectify this?

    Plain text passwords might be a bad idea, but to tell us that you can’t reverse the encryption is outright amateur. Supplying a decryption routine is one extra step but would allow the resellers to be able to function without all of the extra hassle you have just created. I can’t image that this is the best you can come up with.

  10. Derek, thinking it doesnt seem to be their strongest point. Otherwise they would realize what are they doing to their resellers.
    If I was in Usa or Canada I would probably sue them :D

  11. Even though we have no evidence that individual accounts have been accessed, we consider any unauthorized access to a system to be a full compromise
    of data, and have taken a cautious approach of resetting all credentials
    that could be affected. This also includes the registrant passwords, which end users need to manage their domains individually. As a reseller, you have full management access to all the domains in your account through our Reseller Control Panel at manage.opensrs.com. You will not need registrant passwords to manage domains through the Reseller Control Panel. Please feel free to reach out to help@opensrs.com if you need assistance logging into your reseller account.

  12. ok I APOLOGIZE for this, I was in the old control Panel (one day you must close the old one and leave the new only working).

  13. Just received the latest Security update “Auth codes and registrant passwords”
    Explaining changes coming. I believe there is a HUGE issue with one item.
    The New password settings being a minimum of 10. THAT can not be done on many systems WHMCS included.

    8 is often the maximum allowed in a system. This point should be looked at seriously. Beyond 8 by every paper I have ever read on the subject claims that beyond 8 little or no additional security is added.
    requiring upper and lower case, number and character is all that should be required.
    Most end users will have a huge issue with this. I know for sure our system will not handle it !!!!!.

  14. I agree with Greg. Any blocking issue with WHMCS that has a standard, across-the-board fix needs to be public knowledge. Forcing every WHMCS user to contact the help desk for a standard fix is ridiculous.

  15. Actually we do not have “Full” access to manage all of the information through the new reseller control panel. We are unable to access Domain owner information. We can access the email address, (thank god) but we cannot change the domain owners credentials.

    For instance.. A person who asked us to register a domain dies. Now their next in line wants us to update that information, we cannot. That action MUST be completed through the manage.opensrs.net end user interface.

    Believe it or not, clients have little to no interest in managing their domains, or in remembering a whole new set of passwords. They don’t understand how important this is, nor do they care.

  16. I don’t think this is true… You can change the owner contact information for the domain in the new reseller control panel. I have done it before.

  17. You can change the owners email address as I had already stated, you CANNOT change the Owners name.

    Also one additional little gem about these new changes, when the client logs into one domain they no longer have access to the other domains within their account. I may be missing something but it appears that the client will need to log into each domain separately now where they used to have access to all of their domains at once.

    Please correct me if I am wrong but this is a HUGE issue. I have clients that don’t even know how many domains they have, so how do I explain that to them when they call and ask?

  18. “encrypted” passwords are not reversible, they are more accurately called hashed passwords. To check a login, you repeat the hash operation, if you get the same value that is stored, then check succeeds.

  19. so, to restore the functionality you destroyed, we’re to use a web interface to hijack accounts (10,000 of them, one at a time), send the reset links to ourselves, and store the new (or re-use the old) passwords? No, that isn’t going to happen. I’m pretty sure you wouldn’t want that happening.

    You guys just arbitrarily wiped out our integrated backend management, where customers had a single login and could manage any/all their domains (because we cached the management passwords). Thousands of dollars of integration work, loss of face, loss of business. And trivial to fix, just give us the API ability to set passwords that we should have anyway.

  20. Not to be a stickler for detail but you are not correct for all types of encryption. If the encrypted password cannot be decrypted it is worthless. In the case of MD5 hashes and hashed passwords then yes they cannot easily be reversed which is part of why they are used for some functions. Encryption must me able to be reversed or it becomes completely useless since one cannot access what was encrypted ever. ie: drive encryption.

    All I was saying is that they can provide us the ability to actually service our clients in some manner that doesn’t require us to do as you have stated above.

  21. Anything that is reversible on the same system that stores the “encrypted” passwords is a waste of time, the hackers would steal both the encrypted data and the decryption keys/algorithm/whatever. In any common usage “encrypted passwords” means hashed passwords. Which are designed to be NOT reversible. In short there is a reason why they made this change, and one of the logical consequences is that they can no longer show you the raw/plaintext password.

    Encrypted hard drives are a different usage case entirely… which is why the cops had to get Ross Ulbricht while his laptop was powered on. But the opensrs systems are _always_ powered on.

  22. Ahhhh. A positive note with a partially helpful solution.

    OpenSRS has preemptively created a boatload of client handholding work.

    Makes me wonder if there is not more going on than “We have no evidence that …” to justify such guillotine actions.

    … is OpenSRS under new management?
    … trying to get rid of resellers?
    … are resellers now being viewed as middlemen instead of agents?

  23. At 8 characters long then assuming you have a completely random password including upper, lower case, digits and special characters (roughly 900 Trillion possible combinations) then if the hashed value were stolen and using SHA512 for example as the hash (different hashes would take different times to crack) and assume a unique salt was stored with the hash then I estimate it would take a single desktop with a good graphics card on average 25 hours to crack the password using GPU based software such as hashcat (roughly 5 billion hashes per second for SHA512 with a 128 GPU card). These are rough estimates – if anyone has more accurate figures please feel free to correct.

  24. That’s definitely a WHMCS problem that you should report to them (see Nick’s post below). I’m more upset by the 20 character maximum which stops the use of strong and secure passphrases!

  25. I’m not defending OpenSRS but if you look in the API docs there is a command to change the credentials for a domain. It has been there for many years and it still works, I have tried it just now. We don’t use the RWI but do everything via the API. If you do that then there is no reason to store the credentials for each client (which is a bad idea anyway). All we had to do was update our API key (after resetting the RWI password) and everything was fine. I would far rather OpenSRS did treat this as a full breach. Hopefully they have got an external security company in to do some forensics.

  26. I hope you now have a way in the Reseller Interface to see all domains within 1 profile. Clients rely on us to advise them what domains they still have and now we have no way of seeing that in the reseller interface and the client can’t even provide the password because now it is a password reset. Perhaps an immediate improvement to the interface is required before making such changes without any notice. Remember domain transfers etc are time sensitive.

  27. The Reseller Interface gives you the ability to search for domains by registrant email. Enter an email address in the searchbox, and the result will give you all domains using that email address.

  28. Will you at least add the ability for us to create a temporary token (15 minutes?) in order to manage a domain in the MWI (since we can no longer retrieve the user’s domain login)? The way we can do so for hostedemail in order to view a user’s mailbox when asked for support. The RWI does not give us access to do everything that the MWI does; there are occasions we need to get in there to change something on behalf of our clients.

  29. That would be good, we were lazy and based our code on the opensrs supplied manage.cgi, but that is meant to be stand alone and require authentication, so it may have been the wrong starting point.

  30. I was able to change the owner first and last name in the contact information for one of my domains in the new reseller interface with absolutely no issue, so there must be some other issue with yours.

  31. What’s this command?
    Every command of this kind I checked needs a cookie, and to generate this cookie you need registrant’s username and password…

  32. We once again apologize for the inconvenience these changes have caused. We have just updated the post with some information that will help address some of the issues (scroll down to the section labeled **Update: August 12, 2015 12:30PM EST**. As always, we encourage you to reach out to support at help@opensrs.com.

  33. Ah we have cookie bypass enabled on our account. The command is the CHANGE (Ownership) one.
    Perhaps you can get Support to enable cookie bypass on your account?

  34. On Aug. 12, you added…

    “There are some concerns about password reset links being sent to an invalid email address. Resellers can log in to the Reseller Control Panel and edit owner’s contact information (including email address). We recommend you watch the Domain Management tutorial. If you are currently using the Reseller Web Interface (RWI), we encourage you to switch to the Reseller Control Panel as it provides a lot more options and functionality.”

    As far as encouraging resellers to use the “Reseller Control Panel”, instead of the older “RWI”. I find I have to use both, but I use the RWI more because I find that the “Reseller Control Panel” can still be buggy. For example, using the RCP, I search our own domain ‘vaxxine.com’, it comes up with incorrect details – status, created and expiry dates are wrong – see screen shot: http://goo.gl/qmRCqk

    Then if I click the link (vaxxine.com) to manage the domain, I cannot access the domain management functions. Instead it offers a renewal page that is completely out of context, with an error – see screen shot: http://goo.gl/kOtjN9

    The old RWI does not have this issue, so I prefer the interface that works, or at least used to before you disabled some of the important tools I relied on.

    Here’s another issue as a result of these changes. In the RWI, I had the “Email domain password to customer” template configured to BCC me for cases where the registrant email was invalid, so I could help the registrant get access to their domain to renew or manage. Now that you have arbitrarily replaced the “Email domain password to customer” function with a “Email domain password reset link to customer”, the template for the latter has the CC and BCC functions completely disabled. So there is no way for us get a copy of the reset email in case the registrant email address no longer exists.

    Restoring the ability to add a BCC to the latter template would be far more efficient than what you have implemented. And I think this is the least you could do under the circumstances, that would in no way compromise security.

    The whole ICANN verification process that was introduced last year added a whole new layer of support overhead, and these changes have stepped that up a few more notches. Eventually resellers will have no ability, or desire to support Registrants!

  35. My first test was a .ca no access to owner outside of the end user platform. The .com you are correct I can edit the owner. So it’s probably a .ca thing.

  36. I am extremely familiar with how encryption works. I also understand why this method is used, but there still needs to be a way where we can reset the passwords without the freaking email process. If they can send us an email link to reset the password, they can provide us with a link to do so within the interface.

    Even then this still puts us in a very precarious position where we now have to have an alternate method to manage the passwords and domains that each client has.

    EG: I was trying to transfer my lawyers domains, the previous opensrs reseller registered the domains in his own name and with his own email. That emails domain had since been changed to some newly branded company and is no longer active at all.

    If I cannot get cooperation from the losing opensrs account holder then the lawyer loses his domains since the only way to retrieve his now changed password is an account that no longer exists.

    I agree there needs to be security, but there also needs to be a way where we as the main providers can provide the service our clients have come to expect.

    I have clients with 10 or more domains. Those clients now need 10 or more passwords to manage or even view each domain.. plus they have to login 10 different times to do it. How do I explain that when they go to :netsol,godaddy,domainsatcost, they can see all of their domains at once?

  37. Of course if we don’t know what email address they used, or if they updated the profile since or if one of their staff members changed details…. then this is pretty much useless.

  38. Having us contact support if we are a WHMCS user for generic instructions seems a bit overkill, especially when it takes a full day to get a STOCK answer from opensrs. From OPENSRS: Thanks
    for contacting OpenSRS reseller support today! Please upgrade your WHMCS plugin
    to use our domains pro plugin at: https://opensrs.com/integration/tools/whmcs/.
    If you do not wish to upgrade, please contact WHMCS support and they can provide
    a bug fix for their plugin for you.

    FYI, OpenSRS released a hotfix on Monday for this. one file to drop into the built-in opensrs module and poof, fixed. http://blog.whmcs.com/?t=104706

  39. The forget password option at manage.opensrs.net which has been added on Aug. 11th, is a very bad idea. It makes stealing a domain name registered in OpenSRS down to hacking an email account. A hacker now only needs to hack the owner/admin of a domain, then reset the password and he can transfer it in no time. We all know that in the control panel there is the Auth Code, as well as the option to unlock the domain.

    OpenSRS will probably reason that they have put this in place to solve the recent problems. But to solve a problem they have just created a bigger problem.

    Some resellers might be ok with this option, but I prefer to have this off and be able to turn it on per domain – after the end user has contacted me and I have verified that they are who they claim to be and have indeed access to the email address in the whois. Of course it makes a lot of burden for us, but I would rather deal with this than a domain theft.

  40. So OpenSRS team, no replies to the issues I raised in my last comments above??? Specifically regarding issues with the RCP that make it unusable, and senselessly disabling BCC ability from template ‘Email domain password reset link to customer’. Returning that functionality would in no way compromise security, but would be a huge asset!!!

    Do reply when you can, I’m only trying to run a business here, so hey, no rush. In the mean time, we’re already experiencing the negative affects of your action. Do keep in mind that your answers are kind of important to me and our Registrants, so it would be nice to get timely answers, and a better impression that things aren’t completely out of control at OpenSRS.

    No one is perfect, but WOW, this whole thing could have been handled A LOT BETTER!

  41. I agree that these password reset are absolutely stupid and cause a lot of havoc to
    resellers. I am very upset because it interrupt my system that store password locally to ease registrant’s request. I think OpenSRS people who commit these changes might be some newbies who know no difference between security and difficulty.

  42. We’ll have a senior support staff member contact you this morning, Derek. They’ll work with you to fix all the problems you have been experiencing.

Leave a Reply