Incremental file copy and xcopy with PowerShell

 

The piece of script discussed in this article helps you to incrementally copy large folders with the assistance of the venerable xcopy tool and PowerShell. Files can be copied with a couple of mouse clicks in File Explorer; however, if you have to copy hundreds or thousands of files, things get a bit more complicated.

I was recently assigned a very simple task: copy the sales department folders from an old file server to a new one. Drag and drop, yeah? With a few gigabytes of data and several thousands of files, drag and drop doesn’t look like a great idea. The biggest problem is not even the speed. What if the copy operation partly fails? There is no way of telling which files were copied and which were not.

It is also worth mentioning that one of the requirements was to build a tool that can be used by end users to repeat process. For instance, if somebody changed already-copied files while the rest were still copying, the tool needs to be able to run an incremental sync at some point.

First, I thought about utilizing robocopy. It is a very nice utility with a lot of options. However, it didn’t work for me because it spends too much time trying to retry copying failed files. Even with the minimal number of retries set, it wasn’t acceptable. Another problem was that there is no obvious way to log information about failed copies only. There is verbose logging, which is nice, but the last thing I needed were hundreds of thousands of lines to analyze.

I then decided to try the Copy-Item PowerShell cmdlet. This is a great tool, except it cannot always copy opened files and is not good with incremental copying.

Finally, I remembered the old, faithful xcopy utility that has always worked for me, even way before Microsoft rolled up PowerShell. However, it does not have the logging capabilities I needed. Thus, I decided to use xcopy in combination with PowerShell.

Below is the script I used for my copy task. I will explain how it works below.

<#
.SYNOPSIS
    Copies folder content from source location to destination including ACLs.   
.DESCRIPTION
    Copies folder content from source location to destination including ACLs.
    If the files and folders in the destination location already exist, the script performs an incremental copy.
    Every item that was changed before the copy process started will be copied to the destination.
    The error log file is created and populated with errors that occurred during the copy process, if any.
.PARAMETER Source
    Path to the folder of the network share that will be copied.
.PARAMETER Destination
    Path to the folder of the network share where the source content will be copied. 
.PARAMETER Log
    Path to the log file to save copy errors if any. If omitted, it will save logs to the c:\temp\copylog.txt file.
   
.EXAMPLE
    PS C:\Scripts\> .\Copy_Folders_and_subfilders_with_Error_Logging_Incrementaly.ps1 -Source \\windows10\share1 -Destination \\windows10\share2 -Log c:\temp\copylog.log
    This example copies files and folders from the \\windows10\share1 location to the \\windows10\share2 one and stores errors in the c:\temp\copylog.log file. 
#>
[CmdletBinding()]
Param(
   [Parameter(Mandatory=$True)]
   [string]$Source,
   [Parameter(Mandatory=$True)]
   [String]$Destination,
   [Parameter(Mandatory=$False)]
   [String]$Log
)
if (!($Log)) {
    $Log = "c:\temp\copylog.txt"
}
$Error.Clear()
$Dt = Get-Date
New-Item -ItemType file -Path $Log -Force
"Starting incremental sync process for: $Source folder at $Dt" | Out-File $Log -Append
xcopy $Source $Destination /c /s /e /r /h /d /y /x
$Dt = Get-Date
"Incremental sync process for: $Source folder has completed at $Dt The following errors occurred:" | Out-File $Log -Append
" " | Out-File $Log -Append
"$Error" | Out-File $Log -Append

I assigned parameter values in the script and had to give the end user some idea of how to use it. Thus, I wrote a help section with a description of what the script does. The Get-Help cmdlet is quite useful in such cases because it allows the end user to retrieve all the information required to successfully work with the script.

Users will get the output you see in the screenshot below if they run the Get-Help cmdlet against the script.

Using Get-Help to explain the usage of PowerShell script

Now, let’s take a look at the parameters section.

[CmdletBinding()]
Param(
   [Parameter(Mandatory=$True)]
   [string]$Source,
   [Parameter(Mandatory=$True)]
   [String]$Destination,
   [Parameter(Mandatory=$False)]
   [String]$Log
)

The first line is the CmdletBinding() keyword, which makes the script operate like a compiled cmdlet. This is followed by two mandatory parameters, $Source (source folder) and $Destination (destination folder), which the script expects. The $Log parameter is not mandatory, because a default value is assigned in the script:

if (!($Log)) {
    $Log = "c:\temp\copylog.txt"
}
$Error.Clear()

The line above is just a precaution. I use the $Error PowerShell object to log process errors, and I have to be sure that there are no old errors in the log file that are not related to the current operation.

$Dt = Get-Date

Here, we store the current date in the $Dt variable.

New-Item -ItemType file -Path $Log –Force

This command creates the log file. Each time the script runs, it recreates the log file.

"Starting incremental sync process for: $Source folder at $Dt" | Out-File $Log -Append

This line writes the source folder and the date and time into the log file.

xcopy $Source $Destination /c /s /e /r /h /d /y /x

As you can see, I use quite a few xcopy parameters. However, I won’t explain them here; extensive documentation is available.

$Dt = Get-Date

In the next line, I refresh the date and time information.

"Incremental sync process for: $Source folder has completed at $Dt The following errors occurred:" | Out-File $Log -Append
" " | Out-File $Log -Append

This command updates the log file with the time of completion.

"$Error" | Out-File $Log -Append

Here, I write the contents of the $Error object into the log file. If any errors occurred during the file copy process, all of them will be now be in the log file.