31

We have a system that uses an SSD (4TB Samsung 860 Pro) that we power on for 10 minutes to write data to and then off every hour for 24/7 for about six months via a Linux system. We manually turn on the power to the drive and wait for the O/S to to see the drive mounted. This usually takes between 12 seconds to 22 seconds to do. We consider a failure to mount if the drive wouldn't show up after 30 seconds of waiting to mount. The first time we did this, everything was working fine. We did a second round with the same drive, but the drives stopped mounting under 30 seconds after about 1 month to 3 months between the 5 systems we ran.

Basically, in the first round, the drive would have been on and off at least 4,320 times. With the drive failing to mount consistently during the second test round, it seems to be between 5,000 and 7,000 total power cycles. All the drives are still working if you wait more than 30 seconds, but they are considered not reliably mounting anymore with our system.

I can't seem to find any SSD drive specifications regarding power cycling and whether there's a limit to doing this. The 4TB 860 Pro drive was very expensive when we bought it (>$1k) and supposedly very reliable with very high Program/Erase (P/E) cycles. However, there are no specs on power cycling.

Is frequent power cycling a bad thing for the SSD drive? I know that most people probably don't do this and the drive probably doesn't get power cycled more than once a day. we basically ran 12 years' worth of everyday power cycling in 6 months.

Edit 1 (additional info from comments): We are running on batteries so power usage is very limited.

Edit 2 (additional info from comments): The SSD drive is connected to a RPi 2B v.1.2 using a modified USB 3 to SATA cable. We have an external power control to turn the power on and off to the cable. Basically, the Pi turns on the power to the SSD and then monitor that the SSD is connected to a specific USB port and then attempt to mount the drive. This is done via a bash script and it runs a mounting loop with 1 second delay until the SSD could be accessed. We give it up to 30 loop counts (1 sec delay each after a fail to mount).

Edit 3 (additional info from comments): The unmounting procedure is to do umount of the drive and then turn off the power. We verified that the data is completely written before unmounting and powering off. The data size is a compressed file typically around 1.2GB to 1.6GB. It's normally just a single file in one hour and it takes about 10 minutes or so to compress the file from the raw data on an SD card and transfer it to the SSD. So the SSD is on for 10-12 minutes before turning off.

Edit 4 After checking more drives, I have found one that already has over 13,000 power cycles and it's still mounting the way we want. I'm waiting to get the failed drives back to see what the counts are on them. We know that we have used them in at least 2 prior runs so I'm expecting to see over 10k power cycles for each of them.

Edit 5 File type on the SSD is Ext4.

Patratacus
  • 429
  • 4
  • 7
  • 19
    Power cycling any electronic device hourly for months may have a negative impact. Why not just leave the device powered on? – John Aug 29 '23 at 19:28
  • 1
    Are these drives in some kind of external enclosure? Could be the power supply in the enclosure dying... – Mokubai Aug 29 '23 at 19:28
  • 1
    So you just 'pull the plug', so to speak? – Joep van Steen Aug 29 '23 at 19:33
  • 11
    As @JoepvanSteen asks, are the drives being dismounted *gracefully*, allowing writes and journaling in file system to complete, or just powered down? If power is cut off abruptly, it would be surprising that a drive would last more than a few iterations of such abuse. – DrMoishe Pippik Aug 29 '23 at 20:26
  • 1
    We are using the drives as a data storage that only comes on every hour to receive compressed data converted from the SD Drives which have raw byte data. The idea is to have a massive storage (4TB) available to store the data but we don't want to leave it on more than 10 minutes to save the power. We are running on batteries so power usage is very limited. – Patratacus Aug 29 '23 at 23:15
  • 2
    I would be more concerned about every mechanical component in the system rather than the solid state component. I personally wouldn’t power cycle any device the way you describe. – Ramhound Aug 30 '23 at 00:51
  • 14
    Have you measured actual power use for the 10 mins every hour vs. just keeping it on? Usually turning on a device draws more current than just leaving it on the whole time (especially if left otherwise idle), so turning it off and on too quickly can cost more power than just leaving it on. – Miral Aug 30 '23 at 05:44
  • 3
    You don't tell what filesystem you are using and whether the drive was unmounted cleanly. For example if an automatic "journal replay" or "filesystem repair" takes place when mounting, the times can be quite variable. Also setting up your timeout value: Please try some statistics like "standard deviation (sigma)" and the decide on the likelihood of exceeding the average mounting time: 10%? 5%? 1% 0.1%, etc. The smaller the failure, the wieder the interval must be. Also: How is the SSD connected to the system? USB? PCI Express? etc. – U. Windl Aug 30 '23 at 06:28
  • 5
    I support @U.Windl hypothesis: if the drive is not unmounted properly before being powered off, then mounting it at the next power on can take longer, and depending on the filesystem all the more than it contains more data. – PierU Aug 30 '23 at 06:42
  • 1
    SATA is hot-plug, and has facilities for on-line (un)registering of drives. Are you unregistering the drive from SATA before you cut its power? – marcelm Aug 30 '23 at 09:40
  • 5
    Also, I agree with other comments that you should re-evaluate whether to cut power at all. According to [a review I found](https://tweakers.net/reviews/5995/3/samsung-860-evo-en-860-pro-rappe-maar-dure-satadrives-prestaties.html), a 1TB 860 Pro uses 35mW in idle. That's likely negligible on your total system consumption, and not worth the damage you're causing. – marcelm Aug 30 '23 at 09:43
  • @DrMoishePippik: you mean the disk hardware or the consistency of what is written on it? – WoJ Aug 30 '23 at 13:00
  • 1
    @Miral for a non-motorised device I'd expect the extra startup current to be equivalent to a few seconds of use maximum, nowhere near minutes – Chris H Aug 30 '23 at 13:11
  • 4
    @marcelm we don't know the battery setup here, so we don't know how significant the draw is (assuming we can ensure it's idling anyway). Saving 20hours/day of idling is 700mWh/day, or 13 Wh over the 6-month project (significant if we can't recharge in the field). In the context of an unattended sensing system, that could be huge. – Chris H Aug 30 '23 at 13:16
  • 8
    We lack quite a bit of information: 1. What is the filesystem? 2. What is your full "power off" procedure? Do you properly unmount the drive and eject it before disconnecting it? 3. How is the drive connected to the system (SATA directly, via USB, something else...)? 4. How do you actually "power on" the drive? Connect the SATA power? Some SATA hot-plug dock? Connect USB cable? Plug in external power transformer plug? Something else? Mounting a clean SSD filesystem should take seconds, not dozens of seconds or more. – jcaron Aug 30 '23 at 14:22
  • Something seems off. A 4 TB SSD should not draw enough power that leaving it on is a problem. You should be able to get USB 3 bus powered SSDs that use very little power. – Todd Wilcox Aug 30 '23 at 17:40
  • Could you put this on something like a raspberry Pi or other SBC, and then the power consumption would be so low you could just leave it on. Maybe use suspend to RAM instead of turning it off. – cybernard Aug 30 '23 at 17:52
  • I'd recommend to take a look at (1) the system logs (perhaps the drivers have a runtime or compile time debug facility?) and (2) the low-level data on the bus connecting the drive. The latter was extremely instructive when we debugged usb stick problems in a infotainment system, but back then it was only USB 2.0 -- both much easier and cheaper than modern buses. You may need to task a dedicated lab with the proper hardware with an analysis. – Peter - Reinstate Monica Aug 30 '23 at 20:23
  • It might be just fragmentation, especially if you are creating and deleting many small files. You could test this by letting a drive run for 6 months without turning it off, and see if it still exhibits an increase in power-up time. If it does, it's most likely fragmentation, and not a hardware issue at all. In this case you could either defragment the drive every month or so, or accept the degradation in power-up time. – TonyK Aug 31 '23 at 00:05
  • @TonyK what does fragmentation has to do with mounting time? – PierU Aug 31 '23 at 06:22
  • "We consider a failure to mount if the drive wouldn't show up after 30 seconds of waiting to mount." - Where? Show up where? – Joep van Steen Aug 31 '23 at 08:34
  • @PierU: well the drive is obviously doing _something_, which may well involve reading a file allocation table into memory. This would be affected by fragmentation. What do _you_ think it's spending its time on? – TonyK Aug 31 '23 at 09:23
  • @TonyK running file system repair after unclean shutdown, most likely – ojs Aug 31 '23 at 11:20
  • 2
    @TonyK Reading a file allocation table, even fragmented, should not be that long. And especially not on a SSD. – PierU Aug 31 '23 at 11:34
  • 1
    @Patratacus Could you answer this simple and very important question asked to you before: "What is your full "power off" procedure? Do you properly unmount the drive and eject it before disconnecting it? " – PierU Aug 31 '23 at 12:08
  • The SSD drive is connected to a RPi 2B v.1.2 using a modified USB 3 to SATA cable. We have an external power control to turn the power on and off to the cable. Basically, the Pi turns on the power to the SSD and then monitor that the SSD is connected to a specific USB port and then attempt to mount the drive. This is done via a bash script and it runs a mounting loop with 1 second delay until the SSD could be accessed. We give it up to 30 loop counts (1 sec delay each after a fail to mount). – Patratacus Aug 31 '23 at 12:08
  • The unmounting procedure is to do umount of the drive and then turn off the power. We verified that the data is completely written before unmounting and powering off. The data size is a compressed file typically around 1.2GB to 1.6GB. It's normally just a single file in one hour and it takes about 10 minutes or so to compress the file from the raw data on an SD card and transfer it to the SSD. So the SSD is on for 10-12 minutes before turning off. – Patratacus Aug 31 '23 at 12:14
  • Have you checked how long the "faulty" drives take to mount on other machines than the Pi? It is consistently long, or does it depend on the machines? – PierU Aug 31 '23 at 12:24
  • 1
    Other possible tests on the Pi itself: 1) power-on the drive, check how long it takes to mount, unmount it (without doing anything in between), then remount it: how long does it take the second time? Same, or shorter? 2) same test but with powering off the drive after unmounting and before remounting. – PierU Aug 31 '23 at 12:36
  • Another possible test: instead of using the custom cable, physically unplug and plug in the drive using an unmodified cable – ojs Aug 31 '23 at 13:23
  • 3
    BTW I'm not sure that simply unmounting the drive before manually powering it off is enough. It will prevent filesystem corruption, but the drive will still suffer from an "unexpected power loss". An external drive should be "ejected", such that it can properly power off by itself. – PierU Aug 31 '23 at 13:23
  • All this additional information should added to the question using the Edit option. – Joep van Steen Aug 31 '23 at 13:49
  • 1
    Hi. Please let us know what file system you are using. You can do this with the `df` command while one of the SSDs is mounted. – kmort Aug 31 '23 at 20:32
  • @PierU: Yeah, always being powered off just after some writes may not leave the drive's firmware enough time to finish migrating data from flash it's using as SLC (fast write buffer) to TLC or whatever (compact storage, multiple bits per cell), and do whatever background wear-leveling it wants. As the drive gets fuller, this might take more time. Obviously good SSDs will journal whatever they do so they can recover, but yeah perhaps it's resulting in the internal data structures being messier than they might have been. – Peter Cordes Sep 01 '23 at 05:30
  • 1
    Compression at RPi CPU speeds limits write bandwidth to 1GiB in 10 mins so that's really low by the standards of an 860 Pro (leaving it plenty of time to do stuff between writes for most of the 10 mins, unless it tries to wait longer for a more-idle time). But I wonder if the power-on time before mount can succeed is due to it processing its data structures (like loading and tidying the flash remapping data from its power-off emergency dump, or just loading it into its onboard DRAM to be ready to work, or sanity-checking it) and/or finishing background stuff that got interrupted. – Peter Cordes Sep 01 '23 at 05:32
  • Some of the things that might increase time from power-on to ready might happen even without frequent power cycling (like the SSD getting fuller, less unused / unTRIMmed space), others might be made worse by power cycling. I don't know SSD firmware internals so I can only make wild guesses. – Peter Cordes Sep 01 '23 at 05:35
  • Thank you for your updates. A few more questions: 1. What is the filesystem type on the SSD? ext3, ext4, FAT, NTFS...? 2. What Linux distribution and version are you using? 3. Have you verified that TRIM is active? 4. Have you measured the power used by the SSD while idle? It's probably insignificant compared to the RPi. There are probably also ways to send the SSD to a lower sleep state. 5. Do you shutdown the RPi between cycles? 6. Did you check dmesg or other system logs while the drive is mounted? 7. Is the power coming from the RPI or a separate supply? Is it sufficient? – jcaron Sep 01 '23 at 12:58
  • Also you may want to check out this option: https://core-electronics.com.au/guides/disable-features-raspberry-pi/ — it may or may not be suitable for you depending on what other USB device you may need to keep active. – jcaron Sep 01 '23 at 13:17
  • More info here https://github.com/mvp/uhubctl#raspberry-pi-b2b3b – jcaron Sep 01 '23 at 13:32
  • 1
    Bravo for parrying all the "you shouldn't do that" responses. Sounds like you have thought about your restrictions and alternatives and still need this problem solved. Great example of the [YX](https://news.ycombinator.com/item?id=10023951) [Problem](https://csf.github.io/blog/2018/05/01/the-yx-problem/). The [RPi Stack](https://raspberrypi.stackexchange.com/) may be useful to you as well. – mlhDev Sep 01 '23 at 19:02
  • TRIM should not be an issue as long as you're only adding data to the SSD. This topic turned into a rabbit hole and perhaps largely opinion 'driven' and losing focus. – Joep van Steen Sep 01 '23 at 23:02

9 Answers9

25

Rather than answer your question, I suggest you reevaluate how to control power to the drive(s). Have you factored in the added hardware cost and parasitic power consumption for the capability for directly controlling the power?

SoCs save on power by disabling the clock to a device, rather than disabling power to the device. Instead of denying it power, the device is put to sleep, and it responds by consuming (demanding) less power. So rather than turning off power to the drive, see if you can put the drive to sleep. See Device Sleep (DevSleep) Using the drive's low-power mode(s) eliminates any external power-switching hardware, and transfers the responsibility of conserving power to the drive itself. Presumably such a drive can sustain repeated sleep-wake cycles.

The need to consume less power and provide extended battery life is a critical part of today’s mobile devices. To meet the ever more aggressive power/battery life requirements in this new environment, the SATA interface is evolving. DevSleep is a new addition to the SATA specification, which enables SATA-based storage solutions to reach a new level of low power operation.

The DevSleep specification does not state what power levels a device will reach while in the DevSleep state, but SSDs are targeting 5mW or less.

sawdust
  • 17,383
  • 2
  • 36
  • 47
  • 8
    Yes, it's not an answer to the question, but modern SSDs have multiple levels of power-saving, where the lowest power level may be close to "power off". – U. Windl Aug 30 '23 at 06:33
  • @harrymc Considering this question specifically mentions Linux, not Windows, I don't see how that is relevant. – Mast Aug 30 '23 at 17:34
  • We are using a USB 3.0 to SATA to connect to the SSD. Do we have the ability to issue the DevSleep? The article you posted seems to be specific to having a direct control over the SATA commands which sometimes are not working using the USB to SATA adapter. – Patratacus Aug 31 '23 at 11:57
  • @Patratacus: If the drive isn't going into a sleep state, perhaps you can get your hands on a mobo + adapter that supports UASP = USB Attached SCSI (https://en.wikipedia.org/wiki/USB_Attached_SCSI). Apparently mobo vendors have to pay a license fee to enable this, so not all USB3 ports support it. (I don't know if RPi boards support it.). As well as better performance than standard USB storage protocols, it should allow passing through a more complete set of SATA commands. Some regular USB<->SATA adapters support commands like SMART and some don't, so even without UAS, try other adapters. – Peter Cordes Aug 31 '23 at 20:03
15

Yes, power cycles are a wear factor for SSDs, and are tracked as "Power Cycle Count" in the internal s.m.a.r.t. monitoring. Only the manufacturer can say how much is too much, but enterprise-level drives are designed to be powered on 24/7, at a consistent temperature, and with a clean power supply. The farther outside those bounds you go, the less reliable your drives may be.

That said, longer mount times are not really a common symptom of SSD wear, unless matched with read/write errors. If the SSD is working normally once it's mounted, then it's much more likely that something at the OS level is causing the mount operation to take longer - though the causes can vary based on OS, firmware, filesystem, etc.

Cpt.Whale
  • 4,501
  • 2
  • 13
  • 25
  • I have a 7 years SSD that is reported (SMART) with 8597 power cycle counts, and still the corresponding normalised attribute is at 100 (as if it was brand new). So, assuming there is a limit, it is much higher than that. – PierU Aug 30 '23 at 06:12
  • 3
    @PierU One sample isn't exactly what I'd use to support such a statement. FWIW 100 normalized value may be simply hard coded. – Joep van Steen Aug 30 '23 at 09:53
  • 3
    @JoepvanSteen Sure, but the point is not that my drive is still OK, but that the vendor apparently doesn't consider that 8597 power cycle count is a problem. Otherwise the attribute would not be at 100. – PierU Aug 30 '23 at 10:07
  • 1
    Like I said, some hard code certain normalized values to be 100. – Joep van Steen Aug 30 '23 at 13:15
  • 2
    @JoepvanSteen I think it's reasonnable to assume that in the case 100 is hard coded, it's because the vendor is considering that power cycles are essentially harmless. – PierU Aug 30 '23 at 14:47
  • Could be, but I doubt it. Same is true for for example ID 174, Unexpected Power Loss Count: hard coded normalized @ 100. We know this can be 'dangerous' but it's hard to pinpoint any critical threshold value. At the same time you'd like number of such events as small as possible. I can not find hard info on it (yet) but it seems to me you want to keep power-cycles limited as much as possible with any electronic device. I mean how many cycles can a capacitor endure to name one. A PMIC? – Joep van Steen Aug 30 '23 at 18:06
  • 3
    @JoepvanSteen I also have a 12 years old HDD, and the power cycle count is 25334. The normalised attribute is at 75. – PierU Aug 30 '23 at 20:47
  • 1
    Yes, and? Let me spell it out: SOME SSD manufacturers, for some SSD's, hard code the normalized values for certain attributes. All I meant was a normalized value 100 does not mean by definition the attribute is 'okay'. – Joep van Steen Aug 30 '23 at 20:53
  • @JoepvanSteen and: the vendor of this disk takes into account the number of power cycles to derive a normalised attribute (and not hardcoding it), and considers that 25000 cycles are not such a big deal (attribute at 75/100, no threshold). So 5000 or so for the disk the OP should not be an issue. That said, for the OP it could be rather "unexpected power loss count" in the case the disk is switched off without being unmounted first, which is another story... – PierU Aug 31 '23 at 06:21
  • I don't care about that, I believe that. Point is, you can not (unfortunately) generally state normalized value of 100 means all is well, nor can you make any claims based on one single sample. Which was basically what your first comment did, and it's the comment I was responding too. I'm not here for a pissing contest. – Joep van Steen Aug 31 '23 at 08:26
  • @JoepvanSteen my previous comment was indeed about a case where the normalised value is not hardcoded, and where the vendor considers that 25000 cycles are not a big deal: this is completely relevant for the initial question, whether you do care or not. – PierU Aug 31 '23 at 09:19
  • @JoepvanSteen Going to ad personam attacks is your choice... – PierU Aug 31 '23 at 11:40
  • 2
    Where is the evidence that hourly power cycling is a significant wear factor? As others have noted, low power modes are typically implemented as local power cycling. I suspect that they are included in SMART monitoring for diagnostic purposes, not because they are a wear factor. For example, SMART monitoring includes seek time performance, which is not a wear factor (might be a *result* of wear, just as power cycling could *result* from wear). Also the linked article doesn't provide a target value for power cycle count. – personal_cloud Aug 31 '23 at 16:31
14

No, there's no good reason for your SSD to be wearing out from just 7,000 power cycles.

But, if it takes 12-22 seconds to mount when empty, it could easily take twice as long to mount when full (it's hard to say what the drive needs to do to report itself as ready, but that activity could easily scale with the file count, for example). You haven't mentioned how you're filling your drive up over time, but you could try saving the mount time vs boot count for each drive. I am guessing you will then see a gradual increase of mount time with each boot, and further details should give clues to help explain this better.

bobuhito
  • 623
  • 1
  • 4
  • 15
  • 6
    If you can measure the mount time on the device, you can keep a rolling average, and set a soft upper limit of no more than say 120% of the last 10 mounts – Chris H Aug 30 '23 at 13:18
  • 6
    "No, there's no good reason for your SSD to be wearing out from just 7,000 power cycles." - Maybe, probably, but isn't partially at least the point of the question to get some citations on that claim? – Joep van Steen Aug 31 '23 at 08:31
  • 1
    After checking several SSD, we have some that had over 13,500 power on count and the SSD seems to still be mounting. I don't have the failed drives back yet to see how many counts are on them. These drives have been used at least in 3 runs of 6 months with hourly power cycling. – Patratacus Aug 31 '23 at 12:02
  • @Patratacus By the way, I have my biggest doubts about your "external power control to turn the power on and off to the cable". Whether there's a relay involved or not, such home-made controls often get flakey. – bobuhito Aug 31 '23 at 16:51
  • Yeah, the external power control is definitely a weak point. We have to do it because of RPI 2B deficiency since it can't provide enough power to SSD from the port due to being USB 2 only. This problem should be resolved when we go to RPi 4 with USB 3.0 ports. – Patratacus Aug 31 '23 at 19:04
10

Turning on an electrical device amounts to creating a power surge as power goes from zero to 100 percent. Power-on is the most dangerous operation for electronic equipment, which is why hardware problems are often detected when turning on the computer.

So yes, there is a negative impact, but for a good-quality SSD it would take a very large number of power cycles to see an effect.

SSDs are protected from power outages by either hardware or firmware PLP (Power Loss Protection). PLPs within SSDs have improved over the years, so the newer the drive, the more likely it is to be protected by the latest PLP technology. The Samsung 860 Pro seems to have come out in 2018, so is not the latest technology.

I don't believe that any SSD company will have ratings for the maximum number of power recycles, although all manufacturers test their SSD to assure a certain resiliency.

For example, I found that ATP SSDs undergo a testing schema that is described in the article Using Four-Corner, Temperature Cycling, and Power Cycling Tests to Verify SSD Resistance to Extreme Operating Conditions, wherein a disk passes if it can withstand 4000 such cycles. Divided by 365 days, this would mean a lifetime of more than 10 years for a typical consumer computer that is turned on once a day.

Your disk undergoes many more power cycles than the 4000 that ATP sees as the desired upper performance limit, so you're basically in uncharted grounds.

harrymc
  • 455,459
  • 31
  • 526
  • 924
  • I don't have the failed drives back yet but I was able to use Samsung Magician software to read the S.M.A.R.T parameters on a similar drive I have on hand. The drive I had listed 3,320 Power Cycles and this drive is still "good". It would be interesting to see the count on the failed drives. – Patratacus Aug 29 '23 at 23:18
  • Strictly speaking this is the manual power-off operations (with a mechanical switch) which is usually the dangereous operation, more than the power-on, and I would expect a total failure rather than longer mounting times. And from what I understand from the article it is much more about data integrity than about hardware reliability : the power is cut (or varying) during writing phases to see if the PLP can ensure the data is properly written anyway. – PierU Aug 30 '23 at 06:34
  • 3
    @Patratacus: Just remember that not all disks were created equal, even of the same model. – harrymc Aug 30 '23 at 09:34
  • 5
    This is the best answer, however there is one thing no one has spoken about. When you apply power, the chips heat up and expand. Switch off and they cool down and contract.This will induce weaknesses in the tracks which could fail. This was explained to me when I worked at the DEC chip manufacturing plant in Scotland. – Bib Aug 30 '23 at 18:57
  • @Bib, excellent comment! not uncommon: SSD failure due to crack in some solder joint. – Joep van Steen Aug 30 '23 at 21:10
  • There's no need to effectively power-on/power-off a drive to have temperature variations: alternaning between heavy workloads and idle state have the same effect. And this happens much more frequently... – PierU Aug 30 '23 at 22:05
  • @Patratacus have you mounted this particular drive to a machine that has the Samsung Magician software to see what it says about this drive you've been abusing? – FreeMan Aug 31 '23 at 14:39
  • I have mounted this into a PC via an internal SATA cable and used Samsung Magician to read the drive. According to all the stats, it's still considered "OK," even for the one with 13,000+ power on cycles. We did have one drive that actually self-erased and failed to mount even after a long time after the final run, but I don't have a chance to scan that drive yet since it's not back in the lab. – Patratacus Sep 03 '23 at 00:37
8

First it's important to recognise the 3 different layers that "damage" can be happening at here:

  1. Hardware: some physical component getting damaged. This makes sense for a spinning disk, and it's why power cycle count is a S.M.A.R.T. metric, but this isn't a spinning disk. We can't say anything for certain about whether or not power cycles are bad for the hardware of the SSD, but based on my experience working with electronics, I would call it extremely unlikely. An SSD is made of solid state components and they (mostly) don't care how many times you power cycle them. Resistors don't care. The impact on transistors and capacitors is negligible. Inductors can create voltage spikes when you cut power to them suddenly, but any good design will account for this.
  2. Firmware-level device state: stuff like bad sector relocations. SSDs have become complex. The firmware is performing all kinds of tricks behind your back, and SSD firmware is notoriously buggy. It is possible, for example, that your SSD is somehow marking sectors as bad if it happens to be in the middle of a write when you cut power. Lots of SSDs also have tiered storage, where writes are persisted to a small buffer that is invisible to the OS. This allows the SSD to re-order writes and to report writes as "durably stored" faster. Maybe something in that system is getting confused by all the power cycles. If that is what's happening, you might be able to fix it with an "ATA Secure Erase" or "NVMe Secure Erase" (this does delete everything on the drive). That said, I think this is unlikely to be the problem.
  3. Software-level device state: AKA, the filesystem. Mounting a filesystem on an SSD should take ~1 second, not 12-22 seconds. This suggests that the filesystem might not be getting cleanly unmounted. Many filesystems have to take some sort of recovery action when mounting a device which was not cleanly unmounted. This often involves "walking" the filesystem to make sure everything is valid. This gets slower as there is more data on the filesystem. Other filesystems keep a "journal" of what they were doing so they only have to check the parts of the filesystem they were working on if something gets cleanly unmounted. Other filesystems have (basically) no safeguards in place and mount very quickly, even when corrupted.

I think your issue exists at #3. There are several ways to test this:

  • If you re-format the drive, is it fast again? If so, you have a #3 problem.
  • If you byte-for-byte copy the partition from a "bad" drive to a new one (without mounting it and allowing the OS to clean anything up) is it still slow? If so, you have a #3 problem.
  • If you monitor disk I/O while you are mounting the drive, do you see a lot of activity? If so, you probably have a #3 problem.
Toby Speight
  • 4,866
  • 1
  • 26
  • 36
9072997
  • 199
  • 5
  • You make good points. I'd love to see citation for: "Resistors don't care. The impact on transistors and capacitors is negligible.". – Joep van Steen Aug 31 '23 at 08:38
  • 1
    Unfortunately, the premise here is pretty flawed at the most basic hardware level. An SSD is made from NAND flash, not resistors, transistors and capacitors. Flash _can_ be physically damaged, and this is in fact normal (that's why SSD's have a finite number of write cycles). Furthermore, a flash write *requires* 250 milliseconds of power. And all modern SSD's do wear leveling, which cause such writes even if the OS is silent (_especially_ when the OS is silent). – MSalters Aug 31 '23 at 09:55
  • 1
    And yet PCB's of SSD's *are* populated with resistors and capacitors which do contribute to SSD device failure. Point that's being made is also that pulling power from the device can cause corruption at several levels, FTL *and* file system, and SSD doing background maintenance with OS 'silent' would only add to the risk of such corruption. – Joep van Steen Aug 31 '23 at 10:47
  • 2
    I actually expect the OP's issue being your #2 instead. SSD firmware takes a while to recover its metadata from persistent storage and fill any blanks if they have appeared after an unexpected power loss or disconnection, especially in case of getting shut down while writing a sector. OP needs to issue a proper unmount anyway, the remainder depends on the driver used to operate the SSD, whether it had detected that the SSD requires a specific (SATA?) command to stop kicking its storage because it's about to get powered off, or just issued one just in case. – Vesper Aug 31 '23 at 13:47
  • We secure erase the drive between the run. The mounting behavior is the same whether the drive has data or not. I suspect the long mounting time is attributed to the USB<>SATA initialization. I'm getting data on different SSD model (4TB 870 Evo) to compare the mounting time. – Patratacus Sep 01 '23 at 20:32
  • @Patratacus just to double check, you are secure erasing via the "ATA Secure Erase" or "NVMe Secure Erase" commands, **not** a utility that secure-erases the drive by writing a bunch of random data to it (i.e. dban, badblocks, shred), correct? – 9072997 Sep 01 '23 at 20:51
  • We use Samsung Magician's secure erase and it's doing an ATA Secure Erase. In the past we used to do the random zero writing thing but that was extremely slow for a 4TB drive and we don't do that for at least a year now. Since all the drives are Samsung we just use the secure erase utility from it. – Patratacus Sep 03 '23 at 00:39
3

With regards to number of acceptable power-cycles: I can not find data on this.

But I doubt it matters. I'm inclined to believe that any sudden power-interruption is potentially harming the device at some level.

SSDs are hardly ever doing nothing

You being done writing does not mean the SSD is done writing as, as others already suggested, SSDs tend to perform all kinds of background tasks (garbage collection, wear leveling, scrubbing) in "idle time". Pulling the plug may therefore leave the FTL in an inconsistent state.

Pulling the plug does do harm at some level

So far it seems you haven't answered the question how you disable power to the SSD or how you 'shut it down'. If you sort of "pull the plug" or "flip the switch" you could indeed be damaging the SSD at some level. These are claims that can be backed up by research.

This paper examines one aspect of that data integrity by measuring the types of errors that occur when power fails during a flash memory operation. Our findings demonstrate that power failure can lead to several non-intuitive behaviors.

Apart from damage on the FTL level, file systems are not invulnerable to power interruptions either. I suppose every PC user knows this from personal experience.

The drive not mounting in x seconds does not mean it has failed

Just as the OS attempting to recover from an unclean shut-down or at least checking the 'dirty' file system we may assume a SSD's firmware will do a similar thing. These checks take time. Some manufacturers for example suggest to give the SSD 5 minutes or so to perform these.

Whether the drive is visible or not, let it sit in this state for a minimum of five minutes to allow the SSD to rebuild its mapping tables, then reboot the system and see if the drive is restored.

In the data recovery industry it's a known fact that a 'bricked' SSD may self recover by letting it sit for a while with power connected, data lines disconnected. I know of extreme cases where a SSD came back to life after being connected to power for 24 hours. But there's also cases where the firmware has failed to the degree where the controller can not even access the NAND. At some point the controller has to read the firmware from the NAND itself, and if that's too corrupt it typically comes to life but with reduced capacity.

No information about actual failure mode

Your device not mounting in x minutes does not mean by definition the SSD has terminally failed. Your device not 'mounting' in x minutes also tells us very little about the failure mode: Is a file system issue, a firmware issue, a hardware issue?

Back to SD Cards?

It is kind of 'funny' that the SD cards you have been using previously are better handling sudden power loss than more sophisticated (in many ways) SSDs. If you require a system where you can just flip the switch your choice may be switching back to the SD cards or switch to more expensive SSDs with physical power loss protection in the form of an array of 'super capacitors'.

Silent data corruption is what you perhaps should be worried about

In the end, every sudden power loss situation is bad and potentially crippling the SSD without any actual hardware component failing, but even without the unit failing it may corrupt your data, which if this goes unnoticed may be a far more serious issue.

Bit corruption hit 3 devices; 3 had shorn writes; 8 had serializability errors; one device lost 1/3 of its data; and 1 SSD bricked. The low-end hard drive had some unserializable writes, while the high-end drive had no power fault failures (tested: 15 drives)

EDIT because of edits to question.

"We are running on batteries so power usage is very limited."

I think it's worth investigating if perhaps this is the source of the problem. So test same setup but now with wall-power. EDIT: This was investigated and not the issue

"The unmounting procedure is to do umount of the drive and then turn off the power. We verified that the data is completely written before unmounting and powering off."

I am not convinced this is the proper way as unmount does not tell the SSD to stop its background processing so it may still be writing and a such sudden power loss may corrupt the FTL. But I'm no Pi nor Linux person. For inspiration see this answer.

"I have found one that already has over 13,000 power cycles and it's still mounting the way we want"

It isn't useful info, one may fail after n power-cycles, the other after m power-cycles, next one after first time. Next one may fail for an entirely different reasons. And then we have brands, models, firmware revisions and whatnot to account for.

EDIT in reaction to comment: "Sounds like this could be the answer to the unsafe power off: echo 1 | sudo dd of=/sys/block/sdX/device/delete"

Based on my experience with SSDs in different contexts, I am inclined to believe that this is what you should be exploring: graceful power-down of the SSD.

Other than sending direct ATA commands some tool may exist that can do this for you. This was the purpose of my 'inspirational link'. Graceful unmount isn't enough, it needs to be a command that tells the drive to power down, to stop it's internal house-keeping activities.

An extra hurdle can be the USB > SATA conversion: Sending the proper commands does not per se mean the USB bridge will pass the command to the SATA drive. Again from experience it seems to me the best chance the USB > SATA adapter passes on the command is if it's powered by a Asmedia controller (ASM1153, ASM1051).

Joep van Steen
  • 4,730
  • 1
  • 17
  • 34
  • I have done the test with the wall-power, and we saw the same mounting problem. We have had over 30 successful runs already using SSD but that's when all the drives were new and not too many re-use cycles. Going back to SD card is not an option since the largest SD card you can get is currently 1.5 TB. We have about 6 TB of compressed data. Uncompressed would be close to 12TB and we only have 2 SD slots. Maybe we can go to 12 SD slots but that's a major change. Sounds like this could be the answer to the unsafe power off: echo 1 | sudo dd of=/sys/block/sdX/device/delete – Patratacus Aug 31 '23 at 18:45
  • Maybe in the end, we'll just put in a new SSD drive every time we do a new run. We know that we could hit 13,000 power cycles and (~3 runs) the drive was still functioning the way we want. So maybe we could even run it twice. 4TB SSD drives are so much more affordable now than before. – Patratacus Aug 31 '23 at 18:45
  • @Patratacus I have answered latest comment in tail of my answer. Good luck! – Joep van Steen Aug 31 '23 at 22:06
  • 1
    I'm going to accept this as the answer since this has an actionable answer. I was able to use command to "eject" the drive after the dismount successfully. Now I'll implement it and see whether the Flash Recovery Count would be reduced. That might help with the grand scheme of things. – Patratacus Sep 03 '23 at 00:41
  • 1
    I'll add to the question edits when I get the "bad" drives back to analyze. It would be interesting to see how many power cycles are on them and whether there's any correlation at all to the failures. – Patratacus Sep 03 '23 at 00:43
1

This is entirely expected. Modern SSD's do wear levelling, which means they move logical blocks around physically. This is usually done as a low-prio background task in firmware, when the OS isn't writing. Because of this wear levelling, SSD's need to store a logical-physical block mapping. That is stored in flash as well.

Flash also requires 250 ms of stable power when writing a cell. This is hidden by the firmware, and in a sequence of writes this means you only need to have power on for 250 ms after the last physical write - but this does include the block mapping.

Since you turn off the device without warning, you risk corrupting the block mapping. Depending of the firmware, the SSD may be able to recover part or all of that mapping. But each time you turn off the SSD while it's doing wear levelling, you risk a total disk failure.

A factory reset may allow the firmware to discard the entire block mapping and generate a new one. If this is the case, all you lose is a bit of capacity from the flash blocks that were destroyed by the power-offs.

MSalters
  • 8,185
  • 1
  • 23
  • 29
  • 3
    And this is why SSDs (and actually HDDs as well) have Power Loss Protection: one of the objectives is to ensure that the mapping table is kept consistent even in the case of power failure: https://www.kingston.com/en/blog/servers-and-data-centers/ssd-power-loss-protection . – PierU Aug 31 '23 at 12:03
  • @PierU In cheaper SSDs (non enterprise) this is most likely firmware level PLP, there's no guarantees though, corruption due to sudden power loss is still an option. PLP, specially firmware PLP does not invalidate the answer. I'm not so certain about the 250ms number, seems and awfully long time. I have seen research working with μs units. – Joep van Steen Aug 31 '23 at 16:20
0

TL;DR

Read about Thermal Cycling Failure in Electronics and check out this cool image of thermal fatigue in solder.

enter image description here

We have a system that uses an SSD (4TB Samsung 860 Pro) that we power on for 10 minutes to write data to and then off every hour for 24/7 for about six months via a Linux system.

You're only power cycling the SSD, right? Not the entire system?

We manually turn on the power to the drive and wait for the O/S to to see the drive mounted. This usually takes between 12 seconds to 22 seconds to do. We consider a failure to mount if the drive wouldn't show up after 30 seconds of waiting to mount.

Manually?? Do you value your own worth in negative values?

MonkeyZeus
  • 9,026
  • 7
  • 26
  • 46
  • I didn't design the system, but we do need a way for the drive to be powered off. The system was built using RPi and unfortunately there's no power control for individual USB port. It's all or nothing. Additionally, USB 2 ports on RPi 2B cannot supply enough power to run the SSD drive from USB3 to SATA cable anyway so we need to supply the power from externally. We are migrating over to RPi 4 so the USB 3 ports can offer enough power to the SSD. However, they can still not be controlled individually. We use 2 SSD drives so we can have up to 8TB and also write redundancy. – Patratacus Aug 31 '23 at 18:15
0

Some experience relating the question:

  • The particular pattern of power cycling may or may not be bad, depending on the design of the power bus. It is usually not bad.

  • If something fails because of the power cycling itself, it does not fail gradually or gracefully. It fails, period.

  • SSDs have a lot of housekeeping work that they do when left powered on and idle. This includes, but is not limited to, erasing the blocks where the data is not valid anymore (i.e. overwritten or trimmed) and moving the recently written data from buffering SLC to permanent-storage MLC blocks. There may be other background tasks as well. Failing that, SSD do show reduced performance.

  • (may be related to your mount times) We have observed SSDs from different reputable brands reducing their performance at 3-5 orders of magnitude for both reads and writes, following prolonged use. We were not able to determine the particular usage pattern that leads to this loss of performance, but it is not big sequential writes for sure. In regard to reading, the disk develops "slow spots" at particular LBA ranges and it gets painful to salvage the data from it. On the other hand, no data lost so far. On the third hand, the disk at least temporarily recovers performance after being "security erased enhanced" and then left alone (powered on) for the time advertised for the "security erase enhanced" command.

fraxinus
  • 1,141
  • 5
  • 6