The email from Yaesu's tech support was probably talking about non-rechargeable 1.5 V lithium (not lithium-ion) AA batteries, rather than rechargeable lithium-ion batteries with a nominal voltage of 3.6–3.8 V. (Apparently at least one company makes rechargeable lithium-ion AA batteries with an output voltage of 1.5 V that charge from a USB cable plugged into the side. I would guess that these batteries are really 3.6 V lithium-ion batteries that include a voltage regulator inside the AA-sized package to reduce the output voltage to 1.5 V. I don't think that tech support was talking about this type of battery.)
All batteries have an internal resistance, which reduces the terminal voltage (and therefore also the delivered power) when a high current flows through the battery. The internal resistance limits the current that can be delivered by a battery pack. The internal resistance of different battery types is a specification that designers of battery-powered devices need to plan around, but that ordinary consumers can ignore.
The VX-8DR could have been designed to use power from the internal battery pack in the same way as external power, or the power from the battery could go through a separate circuit with a different voltage regulator. Ideally the voltage and current expectations of the internal battery pack would be clearly laid out in the specifications, but to save space in the manual the specifications probably only state the battery types that were known to work at the time the radio was designed, so the specifications probably are no help. The only way to find out for sure, besides testing the radio the hard way, would be to look at the radio's schematic and the specification sheets of various ICs that are used. The radio's schematic can probably be found in the radio's service manual.
EDIT: I downloaded the service manual, and the block diagram shows that the current from the internal battery pack doesn't pass through a voltage regulator like the current from the DC jack. There is no exact schematic with every part in the radio, just the block diagram, which does show that the final amplifier is a 2SK2476 MOSFET made by Toshiba. The 2SK2476's data sheet says that it can be driven with a drain-source voltage as high as 20 V. I would think that the final amplifier is driven by battery voltage directly, but I've never experimented with MOSFET amplifier circuits, so I'm not sure about that.
There's also an NJU7231F30 voltage regulator that converts the battery voltage to 3 V. That part is rated for a maximum of 18 V input.
I would think that applying 10.8 V to the battery terminals instead of 4.5 V wouldn't hurt anything in receive mode. However I'd think that you'd run the risk that a higher battery voltage would drive the 2SK2476 final to produce too much power, overheat, and cook itself. Unfortunately I have no way of judging how likely that outcome might be. If you had test equipment, you could connect a variable-voltage power supply to the battery terminals, transmit momentarily, and measure the output power. Then you could gradually raise the input voltage and see what happens to the output power; if the output power went higher than the specified maximum, then I'd say you'd run a high risk of damaging the device. If the output power stays about the same, then you your plan of substituting the 14500 series lithium-ion batteries for the AAs that the radio expects might work.
