There are different reasons why it pays to get a credit card that comes with an annual fee. In some cases, the money you spend on that fee could more than make up for itself in the form of perks and rewards.

Imagine that by getting an annual fee card, you’re entitled to an extra $200 in cash back on your yearly restaurant purchases. If the fee you pay for that card comes to $95, you’re already ahead financially. Plus, some travel reward credit cards that charge annual fees also come with money-saving benefits like free checked bags on flights, making that fee worth paying.

But at some point, you may decide you no longer want to keep a card with an annual fee. Maybe you feel you’re not getting good use out of that card, or the fee just isn’t something you want to deal with anymore. Before you rush to cancel an annual fee credit card, consider one specific drawback.

Will closing your account damage your credit score?

Annual fees can be costly, but so can having a not-so-great credit score.

Having a lower credit score can affect your financial life in many ways. For example, if your score isn’t the best, you may be charged more the next time you need to borrow money, whether in the form of a personal loan or mortgage. As such, you’ll need to be careful when closing a credit card, even if you’re eager to unload the annual fee that comes with it.

There are two ways closing a credit card could hurt you. First, if you’ve had that account open for a long time, closing it could shorten your credit history. That could, in turn, cause your score to take a hit.

Secondly, a big factor that goes into calculating your credit score is your credit utilization ratio, which measures the percentage of your total credit limit that you’re using at once. If that ratio surpasses the 30% mark, credit score damage can ensue. If your annual fee card comes with a generous spending limit, closing it could send that ratio of yours into unfavorable territory.

Imagine you owe $3,000 and have a total credit limit of $10,000 over several cards. In that case, you’re okay, because you’re not beyond that 30% mark. But if you close an annual fee card with a $3,000 credit limit, suddenly, your $3,000 balance puts you at about 43% utilization, which could drag your score down — a lot.

Do this before closing an annual fee card

If you’re really not getting any use out of an annual fee credit card, you should try to find a way to close it without hurting your credit score. One option is to apply for another card with a comparable credit limit so your utilization ratio doesn’t suffer.

That said, there may be another avenue to explore. Before you rush to close that card, contact its issuer and see if it’s possible to have your fee waived. You never know what steps a credit card company will go to in order to keep you on board, so it’s worth having that conversation before rushing to cancel your card.