The inflation voting and the federated voting in the SCP consensus algorithm are happening at totally different levels of abstraction. Inflation voting is a property of your wallet, stored in the ledger. Whatever you set it to stays in effect until you change it. The inflation algorithm, when it is run, just goes off the inflation votes in the ledger.
Now a validator in SCP could try to prevent you from changing your inflation destination, which would be the same as trying to prevent you from sending a payment. If that validator (or some validator it is colluding with) happens to be the highest priority node in the nomination protocol for one ledger, then it could indeed potentially censor that request for that ledger. But as soon as an honest validator leads the nomination protocol, the previously censored transaction will make it through nomination into the balloting process.
Once the inflation adjusting transaction makes it through nomination, bad validators could refuse to participate in SCP balloting, which is akin to crashing. Whether or not that crash brings the network to a halt depends on whether there are sufficient remaining good nodes to form a quorum.