DPM 2016 – The storage involving the current operation could not be read from or written to. (ID 40003)

I had an issue with backing up some workloads with DPM 2016 with an error The storage involving the current operation could not be read from or written to. (ID 40003).

To fix the problem you need to remove the last recovery point of affected data source.

The first cmdlet will get the Protection Group specified in $PG variable. The second cmdlet will get the data source specified in $DS variable. The third cmdlet will get the last recovery point of the selected data source stored in $PObject, and the last cmdlet will remove the recovery point. You can run third cmdlet (Get-DPMRecoveryPoint) multiple times to see if the recovery point was removed from DPM Server.
After recovery point is removed, run the consistency check again.

Update

The issue may reappear after some time. Another solution is to move your backups to a new disk storage.

 

13 thoughts to “DPM 2016 – The storage involving the current operation could not be read from or written to. (ID 40003)”

  1. I highly recommend this workaround, but since i have tape backups also, i came up with the following script, which can be run on dpm server itself:

    $pg = Get-DPMProtectionGroup | ogv -PassThru
    $po = Get-DPMDatasource -ProtectionGroup $pg | ogv -PassThru
    $rp = Get-DPMRecoveryPoint -Datasource $po | ogv -PassThru
    Remove-DPMRecoveryPoint -RecoveryPoint $rp

  2. Hi,

    After investigations, because moving 60 datasource + new sources every weeks impacted was not possible, and this is a blocking feature, impossible to start new jobs for days, i tracked dpm with process monitor. I found FSCTL errors while accessing these file, suggestion there would be a fragmentation problem. Moving volume would reduce fragmentation probably.
    So i tried to move current vhdx job file from volume to a free volume and then put it back,outside dpm, with psexec from command line.
    Result : It worked, datasources are ok for weeks now. It’s a faster solution, not blocking dpm jobs.

    1. can you share how you did this? We had a power outage that resulted in the server shutting down. Ever since then the consistency checks fail, I cannot create local or online recovery point, and the above error message is common. I followed the above code for one VM and afterwards it passes the consistency test, but still fails recovery point creation. The only way I got the test VM to work was by removing it from the protection group along with its local files (which removed all local backups) and adding it back in. Microsoft has not returned my calls yet. I am not familiar with psexec and want to test one machine to make sure this process works. I also, cannot tell which vhd job file belongs to which VM.

  3. Hi,

    Great script, thanks for sharing. Just a note for a change I made should anyone else encounter the same, I had to change the following to use the .computer property for it to select the datasource

    $PObject = Get-DPMDatasource -ProtectionGroup $PGroup | where {$_.computer -EQ $DS}; $PObject

  4. Great script, thanks for sharing. I’ve found it works great with VM systems. But I still have SQL DB issues. I’ve even erased as far back as last cluster of backup jobs. Thought I’d post the script to look at single Database on a sql server backup as it took me a while to figure out the changes. Again, thanks for posting this. As for SQL, I find if I turn my backup system off for say 10 min. That clears the same issue with SQL DB’s and I am able to back those up again. So either Dedup or SQL is keeping files open or interfering.

    Single DB Instance on SQL Server.
    $PG = “Data – SQL – Online” #Protection Group name of affected data source.
    $DS = “SERVERNAME” #Data source name, for example Virtual Machine name or SQL Database name.
    $SD = “DATABASE” #Data source name
    $DPM = “Backup04.extend.pro” #FQDN of your DPM Server.

    $PGroup = Get-DPMProtectionGroup -DPMServerName $DPM | where {$_.Name -EQ $PG}; $PGroup

    $PObject = Get-DPMDatasource -ProtectionGroup $PGroup | where {$_.Computer -EQ $DS} | where {$_.Name -EQ $SD}; $PObject

    $RPoint = Get-DPMRecoveryPoint -Datasource $PObject | select -Last 1; $RPoint
    Remove-DPMRecoveryPoint -RecoveryPoint $RPoint

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.