1

TL;DR: Xeon E5-2630 does not reach even the TDP, turbo boosts the clock only when idle, reduces when under load and stays cool as cucumber, despite removing all limits except for the built-in thermal ones.

I am trying to squeeze maximum power from an older Xeon processor (Intel Xeon CPU E5-2630 v3 @ 2.40GHz, Haswell architecture). The processor has 8 cores, is hyperthreaded and has a TDP of 85W, and should be able to boost to 3.2Ghz: https://ark.intel.com/content/www/us/en/ark/products/83356/intel-xeon-processor-e52630-v3-20m-cache-2-40-ghz.html

I have Ubuntu 22.04 installed, and governor has been set to "performance". Before that (with default governor "ondemand"), the processor kinda behaved as expected: 1.2 GHz when idle, and then ramped to 2.6 GHz when under load. After setting the governor to "performance" the following curious thing started to happen (I am using a plethora of tools to observe these parameters, but all give the same result - I mostly rely on s-tui and turbostat)

  1. When processor is (almost) idle, all cores run at 3.2 Ghz. turbostat reports PkgWatt of 20-25W (I run kvm with nearly idle windows VM at the same time,when observing that), and the temperature of all cores is 35-40 degrees C.
  2. When I stress the processor using s-tui (or some other tool), the processor frequency drops to 2.6 GHz, and the temperatures of all cores raise to 48-52 degrees C. All tools report full load on all cores. Turbostat PkgWatt increases to 48-52 Watts.

Now, understanding the TDP of the processor is 85W TDP, this does not make any sense! Reading the RAPL docs https://01.org/blogs/2014/running-average-power-limit-%E2%80%93-rapl, the processor should boost at least to TDP, if not over.

Of course, the thermal throttling can limit that, but all cores are way cooler than the limit, so that is probably not the issue?

Examining further, I tried to adjust the RAPL limits using powercap-set intel-rapl and I raised the power_limit_uw to 250000000 (250W) for both long and short term constraints, and also set very long times. The limit before that was 75W, so not even that was reached under load! The change was accepted, and now rapl-info reports the following:

  name: package-0
  enabled: 1
  max_energy_range_uj: 262143328850
  Constraint 0
    name: long_term
    power_limit_uw: 250000000
    time_window_us: 2046820352
    max_power_uw: 85000000
  Constraint 1
    name: short_term
    power_limit_uw: 250000000
    time_window_us: 2046820352
    max_power_uw: 170000000
  Subzone 0
    name: dram
    enabled: 0
    max_energy_range_uj: 65712999613
    Constraint 0
      name: long_term
      power_limit_uw: 0
      time_window_us: 976
      max_power_uw: 19500000

However, there is no change in processor behavior. Any ideas why or what to do to force processor to boost to 3.2Ghz and stay there under load? I have a lot of headroom with cooling so this really surprised me in a negative way.

EDIT. The possibilites that 1) perhaps only single core can boost or 2) that my thermal budget is exhausted are IMHO pretty remote, as 1) when idle, many cores are boosted to 3.2Ghz, my thermal budget stays way below even the stated TDP and the temperature of cores are low. I am attaching images of s-tui under load and idle.

Under load Idle

xmp125a
  • 203
  • 2
  • 6
  • The turbo boost speed is dependent on power budget as well as temperature and the chip itself decides how that is met. Max turbo frequency may only be available when a few cores are used rather than the whole set of cores. More cores in use means lower peak clock speed. – Mokubai May 16 '23 at 17:12
  • If you absolutely must have that higher clock frequency then you'll have to go down the route of proper overclocking. Turbo boost is an intelligent "we can grab a bit of extra performance for particular tasks" and is not the same. – Mokubai May 16 '23 at 17:14
  • At the very bottom of https://en.wikichip.org/wiki/intel/xeon_e5/e5-2630_v4 there is a list of the various number-cores-in-use vs boost clock speed: https://i.stack.imgur.com/HZtWh.png which states for 6 or more cores in use the boost will only be 2.6GHz. This is how Turbo Boost operates. At idle you are only briefly seeing one core in use at a time. – Mokubai May 16 '23 at 17:29
  • I edited the post and added two screenshots that show that the cpu is cool (plenty of thermal budget) and that indeed practically all cores boost when idle (so not an issue with single core only available for boosting) – xmp125a May 16 '23 at 17:30
  • @Mokubai thanks, the table you provided is the answer. If you post it, I will accept it. – xmp125a May 16 '23 at 17:31
  • @Mokubai what I still wonder, but it is really irelevant for my problem, is, why then the processor reports almost all cores boosted to 3.2 Ghz when idle :) ... It must be a bug somewhere, perhaps in clock reporting tools. It would save me bunch of time if the processor would not report "all cores at 3.2Ghz" immediately after switching the governor. – xmp125a May 16 '23 at 17:44
  • It shows as high clock speed at idle because each core is being used for an infinitesimally small time. There is almost nothing to do so under use the core quickly ramps up to maximum boost. The thing is that a lot of power management in processors is done by [power gating](https://en.wikipedia.org/wiki/Power_gating) where the main power drawing parts of the core essentially go to sleep. The front end may still be awake at a high clock speed but the power use has shifted to the next core that is actually being used. – Mokubai May 16 '23 at 17:56

0 Answers0